This is Jessica. Explainable AI has in some ways been highly successful as a human-oriented body of machine learning research. It’s motivated hundreds of research papers (many highly cited) and dozens of workshops over the last 7 or 8 years, without much evidence of slowing down. But it’s also been the source of considerable hand-wringing, with methods getting popular and then later being exposed as non-robust or otherwise misleading. One reason for this is that as a research area, it’s never done a great job of establishing what an explanation is supposed to do. And so there’s been a pattern where techniques get popular but are followed by critiques that point out blind spots. Earlier enthusiasm about saliency maps and feature importance methods were dampened by results showing how they could be foiled. More recently, mechanistic interpretability has caught on, but similar critiques are cropping up. Other embarrassments include evidence that the most successful techniques that make it to practice (like SHAP) are frequently misunderstood by those relying on them. And so explainable AI has the reputation of a research area that has largely missed the mark despite the obvious importance of finding ways to help people understand AI/ML systems.

By now, dozens of position papers have tried to explain what explainable AI is missing. These can sometimes be insightful, but often they lack sufficient concreteness in explaining what’s missing for authors to know when they have overcome it. Meanwhile, the state of the art in evaluating explanations has devolved into something like testing explanations against every task-agnostic criteria you can think of.

Against this backdrop, Ziyang Guo, Berk Ustun, and I write:

Modern methods for explainable machine learning are designed to describe how models map inputs to outputs without deep consideration of how these explanations will be used in practice. This paper argues that explanations should be designed and evaluated with a specific end in mind. We describe how to formalize this end in a framework based in statistical decision theory. We show how this functionally-grounded approach can be applied across diverse use cases, such as clinical decision support, providing recourse, or debugging. We demonstrate its use to characterize the range of possible performance that a task admits, preventing misuse due to ambiguity by forcing researchers to specify concrete use cases that can be analyzed in light of models of expected explanation use. We argue that evaluation should meld theoretical and empirical perspectives on the value of explanation, and contribute definitions that span these perspectives. 

At a high level, we argue that proposals of new explanation methods must be accompanied by 1) a specification of a task (or class of tasks) that they improve the performance of, and 2) a model of expected use of the explanation under which explanations can be shown to improve performance on that task. While it’s true that the world hardly seems to need more position papers on explainable AI, the message this one conveys has seemed so curiously absent in the literature that we felt it needed to be stated. We spend much of the paper dissecting what it means for an explanation to be useful and describing a concrete set of analyses that authors can do to establish that a new explanation method they are proposing has value. 

Regarding #1, the need to specify concrete tasks that an explanation method helps with, we suggest representing tasks as decision problems. A decision problem is characterized by a choice of action that must be made under uncertainty about the state of the world where the quality of the action conditional on a ground truth state is assigned by a scoring rule. At the time of making a choice, the decision-maker has access to some “signal” that is thought to inform them about the state. Performance is measured over some evaluation distribution, i.e., a joint distribution over signals and ground truth states. For example, tasks implied in the explainable AI literature include explaining a risk model’s prediction to a doctor making treatment decisions, explaining a prediction that led to a person being denied some good (e.g., a loan) so that they can decide whether to invest effort in changing some aspect of their application, or helping a model developer understand a model’s use of inputs so that they can further improve the model.  

It’s worth noting that many (most?) explainable AI papers don’t clearly distinguish target tasks or models of expected use. Many either don’t consider well-defined downstream tasks at all (focusing instead on internal validity like deriving guarantees on “faithfulness” of the explanation to the model’s underlying mechanisms, which may or may not help people use the explanation) or they conduct empirical studies focused on “upstream” outcomes like whether people can understand an explanation or say they find it intuitive. Studies that do examine how explanations improve task performance often find that they don’t help much, leaving us to wonder whether it was the explanations fault or it was just a task where the explanation didn’t have much potential to help. 

What it means for an explanation to help

The nice thing about decision theory is that once we have specified a task we expect an explanation to help with, we can get concrete about what it means for an explanation method to provide value. Namely, it should help the agent who uses it gain information about the state of the world (which is uncertain at decision time, but on which the quality of their choice of action depends, like whether the patient recovers). 

There are two ways an explanation of this form can help an agent gain information about the state. First, it can directly provide information about the state, like presenting the posterior probability that a prediction is correct. (As an aside, I find it ironic that uncertainty quantification is often not considered part of explainable AI). 

However, many explanation methods are not designed to directly provide information about the state. The most common form of explanation in the literature is an explanation function E: X x Y → Z that takes a specific instance and model prediction yhat=f(x) and outputs an explanation z=E(x, yhat) in feature space, such as a set of feature importance scores. Such explanations can help indirectly by enabling the person to extract more information about the state than they otherwise would from the model prediction or the instance features. Imagine a person has their own internal model of the relationship between features and labels, f^H: X -> Y, distinct from the prediction model. By presenting information in feature space, allowing the person to compare the model’s use of information to their own model of how features relate to labels. Whenever the explanation implies the posterior distribution more directly than the features (which we argue is another implicit assumption behind presenting explanations), then the explanation may help. Whether it does and how much depends on the value of getting more information about the state to performance on the task, the alignment between the person’s internal model and the information they get from the explanation, and how exactly they deviate from rational use of the information.  

Three ways to define the value of explanation

Given these two ways in which explanations can improve task performance, how can we define the value of an explanation technique? The most obvious way, given that an explanation is a type of intervention, is to run a randomized controlled experiment to estimate the treatment effect of having access to the explanation over only having the features and model prediction. We call this the “behavioral value of explanation.”

But, how do we make sense of the effect estimate we get when we do this? Like I mentioned above, lots of studies that do this find little impact of the explanation. Should we conclude those explanations don’t work? How do we know whether people are getting most of the possible value the explanation could provide? It’s hard to know what we are learning about explanation techniques if we don’t understand their potential to improve performance on a task. 

In a decision theoretic framework, one way we can contextualize a treatment effect associated with access to some information signal is by comparing the observed effect to the improvement we expect if we give the best possible decision-maker access to that signal. The “value of information” of some signal for a decision problem is the difference in the expected score of a Bayesian rational agent who knows the study set-up (i.e., knows all elements of the decision problem, including the data-generating distribution) versus the same agent’s expected score when they only have access to the prior. This gives a sense of the largest improvement we can expect purely from the fact that the signal is providing some relevant information. 

But when we calculate the value of an explanation method of the form z=E(x, yhat) using this method, we find that the rational agent gets no value at all from the explanations! This is because such explanations are a “garbling” of the information about the state contained in features. This may explain why some authors argue that such explanations can’t be useful. 

But as I said above, explanations can help imperfect agents like humans to gain information indirectly about the state. And despite the assymetry in how an idealized rational agent uses an explanation, we can use the rational agent framework to help us understand how much improvement humans might get. For one, the benefit of the extra info about a decision instance that a human could gain as the result of the explanation “unlocking” it can’t be more than the best possible performance the maximally informed, idealized agent could get on that task. We can also look at how much value we expect the instance-level information to have to a rational agent (which for the rational agent is all contained in the features x) to get a sense of how much a human could improve because an explanation helped them better extract instance-level information.

Consider a case where the rational agent expects little improvement in score due to getting instance-level information over just making the best fixed decision under the prior. If this “theoretic value of explanation” is low, then we haven’t identified a good task to show the value of explanations, because in the best case the information about the individual instances that an explanation might help the human get is not that valuable to the task. And so, we argue that those proposing new explanation methods first check that the theoretic value of explanation is high, relative to the expected performance of the idealized agent who knows only the prior distribution. To learn anything about explanations that work, we need to make sure we’re looking at a task where an explanation could help at least in theory.

We can use a similar logic to ask (again prior to deploying the explanation method with people), to check whether an explanation method has the potential to improve the decisions of a particular population. Imagine that we haven’t deployed the prediction model and explanations yet with human users, but we have the ability to query the humans’ to get their predictions yhat^H for some set of instances. We can then ask, What’s the value (i.e., the expected improvement in task performance) of a rational agent who goes from making decisions using only the human predictions to also having access to the features? This “potential complementary value of explanation” gives us a measure of the best case value of the additional complementary information in the instance-specific information that the explanation might unlock over the information already contained in the human judgments. If this is small, there’s little reason to expect the explanation to improve people’s performance by helping them extract more information about the instances.

Combining these three different definitions, we suggest that developers of new explanation methods proceed by 1) checking that the theoretic value of explanation of explanation is high for the tasks they target, 2) checking that the potential complementary value of explanation is also high, and then, assuming both checks pass, 3) deploying the explanation to estimate the behavioral value of explanation, and 4) interpreting the magnitude of the behavioral value by comparing it to the theoretic value and potential complementary value, as well as looking at how close the behavioral performance with the explanation comes to the upper bound on performance defined by the rational agent who has access to the features. If humans are still falling pretty short of that target, then there is in theory more information about the instances to be gained than the explanation is “unlocking” for them, so we may want to reconsider how we design it or train people to use it.

There are challenges to this approach of course, like avoiding overfitting in the benchmarks, which we provide some methods for. But overall I like this paper as writing it clarified a number of things for me that remain below the surface in the explainability arguments I was reading.