This is Jessica. A while back on the blog I shared some opinions about studies of human-decision making, such as to understand how visualizations or displays of model predictions and explanations impact people’s behavior. My view is essentially that a lot of the experiments being used to do things like rank interfaces or model explanation techniques are not producing very informative results because the decision task is defined too loosely. 

I decided to write up some thoughts rather than only blogging them. In Decision Theoretic Foundations for Human Decision Experiments (with Alex Kale and Jason Hartline), we write: 

Decision-making with information displays is a key focus of research in areas like explainable AI, human-AI teaming, and data visualization. However, what constitutes a decision problem, and what is required for an experiment to be capable of concluding that human decisions are flawed in some way, remain open to speculation. We present a widely applicable definition of a decision problem synthesized from statistical decision theory and information economics. We argue that to attribute loss in human performance to forms of bias, an experiment must provide participants with the information that a rational agent would need to identify the normative decision. We evaluate the extent to which recent evaluations of decision-making from the literature on AI-assisted decisions achieve this criteria. We find that only 6 (17%) of 35 studies that claim to identify biased behavior present participants with sufficient information to characterize their behavior as deviating from good decision-making. We motivate the value of studying well-defined decision problems by describing a characterization of performance losses they allow us to conceive. In contrast, the ambiguities of a poorly communicated decision problem preclude normative interpretation. 

We make a couple main points. First, if you want to evaluate human decision-making from some sort of information interface, you should be able to formulate the task you are studying as a decision problem as defined by statistical decision theory and information economics. Specifically, a decision problem consists of a payoff-relevant state, a data-generating model which produces signals that induce a distribution over the state, an action space from which the decision-maker chooses a response, and a scoring rule that defines the quality of the decision as a function of the action that was chosen and the realization of the payoff-relevant state. Using this definition of a decision problem gives you a statistically coherent way to define the normative decision, i.e., the action that a Bayesian agent would choose to maximize their utility under whatever scoring rule you’ve set up. In short, if you want to say anything based on your results that implies people’s decisions are flawed, you need to make clear what is optimal, and you’re not going to do better than statistical decision theory. 

The second requirement is that you communicate to the study participants sufficient information for a rational agent to know how to optimize: select the optimal action after forming posterior beliefs about the state of the world given whatever signals –visualizations, displays of model predictions, etc–you are showing them. 

When these criteria are met you gain the ability to conceive of different sources of performance loss implied by the process that the rational Bayesian decision-maker goes through when faced with the decision problem: 

• Prior loss, the loss in performance due to the difference between the agent’s prior beliefs and those used by the researchers to calculate the normative standard.
• Receiver loss, the loss due to the agent not properly extracting the information from the signal, for example, because the human visual system constrains what information is actually perceived or because participants can’t figure out how to read the signal.
• Updating loss, the loss due to the agent not updating their prior beliefs according to Bayes rule with the information they obtained from the signal (in cases where the signal does not provide sufficient information about the posterior probability on its own).
• Optimization loss, the loss in performance due to not identifying the optimal action under the scoring rule. 

Complicating things is loss due to the possibility that the agent misunderstands the decision task, e.g., because they didn’t really internalize the scoring rule. So any hypothesis you might try to test about one of the sources of loss above is actually testing the joint hypothesis consisting of your hypothesis plus the hypothesis that participants understood the task. We don’t get into how to estimate these losses, but some of our other work does, and there’s lots more to explore there. 

If you communicate to your study participants part of a decision problem, but leave out some important component, you should expect their lack of clarity about the problem to induce heterogeneity in the behaviors they exhibit. And then you can’t distinguish such “heterogeneity by design” from the real differences between decision-quality based on the differences between the conditions that you are trying to study. You don’t know if participants are making flawed decisions because of real challenges with forming accurate beliefs or selecting the right action under different types of signals or because they are operating under a different version of the decision problem than you have in mind.

Here’s a picture that comes to mind:

I.e., each participant might have a unique way of filling in the details about the problem that you’ve failed to communicate, which differs from how you analyze it. Often I think experimenters are overly optimistic about how easy it is to move from the left side–the artificial world of the experiment–to draw conclusions about the right. I think sometimes people believe that if they leave out some information (e.g.,  they don’t communicate to participants the prior probability of recidivating in a study on recidivism prediction, or they set up a fictional voting scenario but don’t give participants a clear scoring rule when studying effects of different election forecast displays), they are “being more realistic”, because in the real world people rely on their own intuitions and past experience so there are lots of possible influences on how a person makes their decision. But, as we write in the paper, this is a mistake, because people will generally have different goals and beliefs in an experiment than they do in the real world. Even if everyone is influenced in the experiment by a different factor that does operate in the real world, the idea that the composition of all these interpretations gives us a good approximation of real world behavior is not supported, as we say in the paper it “arises from a failure to recognize our fundamental uncertainty about how the experimental context relates to the real world.“ We can’t know for sure how good a simulacrum our experimental context is for the real world task, so we should at least be very clear about what the experimental context is so we can draw internally valid conclusions. 

Criteria 1 is often met in visualization and human-centered AI, but Criteria 2 is not

I don’t think these two criteria are met in most of the interface decision experiments I come across. In fact, as the abstract mentions, Alex and I looked at a sample of 46 papers on AI assisted decision-making that a survey previously labeled as evaluating human decisions; of these 11 were interested in studying tasks for which you can’t define ground truth, like emotional responses people had to recommendations, or had a descriptive purpose, like estimating how accurately a group of people can guess the post-release criminal status of a set of defendants in the COMPAS dataset. Of the remaining 35, only a handful gave participants enough information for them to at least in theory know how to best respond to the problem. And even when sufficient information to solve the decision problem in theory is given, often the authors use a different scoring rule to evaluate the results than they gave to participants. The problem here is that you are assigning a different meaning to the same responses when you evaluate versus when you instruct participants. There were also many instances of information asymmetries between conditions the researchers compared, like where some of the prediction displays contained less decision-relevant information or some of the conditions got feedback after each decision while others didn’t. Interpreting the results is easier if the authors account for the difference in expected performance based on giving people a slightly different problem. 

In part the idea of writing this up was that it could provide a kind of explainer of the philosophy behind work we’ve done recently that defines rational agent benchmarks for different types of decision studies. As I’ve said before, I would love to see people studying interfaces adopt statistical decision theory more explicitly. However, we’ve encountered resistance in some cases. One reason I suspect is because people don’t understand the assumptions made in decision theory, so this is an attempt to walk through things step by step to build confidence. Though there may be other reasons too, related to people distrusting anything that claims to be “rational.”