This is Jessica. I went to NeurIPS last week, mostly to see what it was like. While waiting for my flight home at the airport I caught a talk that did a nice job of articulating some fundamental limitations with attempts to make deep machine learning models “interpretable” or “explainable.”
It was part of an XAI workshop. My intentions in checking out the XAI workshop were not entirely pure, as its an area I’ve been skeptical of for a while. Formalizing aspects of statistical communication is very much in line with my interests, but I tried and failed to get into XAI and related work on interpretability a few years ago when it was getting popular. The ML contributions have always struck me as more of an academic exercise than a real attempt at aligning human expectations with model capabilities. When human-computer interaction people started looking into it, there started to be a little more attention to how people actually use explanations, but the methods used to study human reliance on explanations there have not been well grounded (e.g., ‘appropriate reliance’ is often defined as agreeing with the AI when it’s right and not agreeing when it’s wrong, which can be shown to be incoherent in various ways).
The talk, by Ulrike Luxburg, which gave a sort of impossibility result for explainable AI, was refreshing. First, she distinguished two very different scenarios for explanation: the cooperative ones where you have a principal with a model furnishing the explanations and a user using them who both want the best quality/most accurate explanations, versus adversarial scenarios where you have a principal whose best interests are not aligned with the goal of accurate explanation. For example, some company who needs to explain why it denied someone a loan has little motivation to explain the actual reason behind that prediction, because it’s not in their best interest to give people fodder to then try to minimally change their features to push the prediction to a different label. Her first point was that there is little value in trying to guarantee good explanations in the adversarial case, because existing explanation techniques (e.g.,for feature attribution like SHAP or LIME) give very different explanations for the same prediction, and the same explanation technique is often highly sensitive to small differences in the function to be explained (e.g., slight changes to parameters in training). There are too many degrees of freedom in terms of selecting among inductive biases so the principal easily produce something faithful by some definition while hiding important information. Hence laws guaranteeing a right to explanation miss this point.
In the cooperative setting, maybe there is hope. But, turns out something like the anthropic principle of statistics operates here: we have techniques that we can show work well in the simple scenarios where we don’t really need explanations, but when we do really need them (e.g., deep neural nets over high dimensional feature spaces) anything we can guarantee is not going to be of much use.
There’s an analogy to clustering: back when unsupervised learning was very hot, everyone wanted guarantees for clustering algorithms but to make them required working in settings where the assumptions were very strong, such that the clusters would be obvious upon inspecting the data. In explainable AI, we have various feature attribution methods that describe which features led to the prediction on a particular instance. SHAP, which borrows Shapley values from game theory to allocate credit among features, is very popular. Typically SHAP provides the marginal contribution of each feature, but Shapley Interaction Values have been proposed to allow for local interaction effects between pairs of features. Luxburg presented a theoretical result from this paper which extends Shapley Interaction Values to n-Shapley Values, which explain individual predictions with interaction terms up to order n given some number of total features d. They are additive in that they always sum to the output of the function we’re trying to explain over all subsets of combinations of variables less than or equal to n. Starting from the original Shapley values (where n=1), n-Shapley Values successively add higher-order variable interactions to the explanations.
The theoretical result shows that n-Shapley Values recover generalized additive models (GAMs), which are GLMs where the outcome depends linearly on smoothed functions of the inputs: g(E[Y] = B_0 = f_1(x_1) + f_2(x_2) + … f_m(x_m). GAMs are considered inherently interpretable, but are also undetermined. For n-Shapley to recover a faithful representation of the function as a GAM, the order of the explanation just needs to be as large or larger than the maximum variable interaction in the model.
However, GAMs lose their interpretability as we add interactions. When we have large numbers of features, as is typically the case in deep learning, what is the value of the explanation? We need to look at interactions between all combinatorial subsets of the features. So when simple explanations like standard SHAP are applied to complex functions, you’re getting an average over billions of features, and there’s no reduction to be made that would give you something meaningful. The fact that in the simple setting of a GAM of order 1 we can prove SHAP does the right thing does not mean we’re anywhere close to having “solved” explainability.
The organizers of the workshop obviously invited this rather negative talk on XAI, so perhaps the community is undergoing self-reflection that will temper the overconfidence I associate with it. Although, the day before the workshop I also heard someone complaining that his paper on calibration got rejected from the same workshop, with an accompanying explanation that it wasn’t about LIME or SHAP. Something tells me XAI will live on.
I guess one could argue there’s still value in taking a pragmatic view, where if we find that explanations of model predictions, regardless of how meaningless, lead to better human decisions in scenarios where humans must make the final decision regardless of the model accuracy (e.g., medical diagnoses, loan decisions, child welfare cases), then there’s still some value in XAI. But who would want to dock their research program on such shaky footing? And of course we still need an adequate way of measuring reliance, but I will save my thoughts on that for another post.
Another thing that struck me about the talk was a kind of tension around just trusting one’s instincts that something is half-baked versus taking the time to get to the bottom of it. Luxburg started by talking about how her strong gut feeling as a theorist was that trying to guarantee AI explainability was not going to be possible. I believed her before she ever got into the demonstration, because it matched my intuition. But then she spent the next 30 minutes discussing an XAI paper. There’s a decision to be made sometimes, about whether to just trust your intuition and move on to something that you might still believe in versus to stop and articulate the critique. Others might benefit from the latter, but then you realize you just spent another year trying to point out issues with a line of work you stopped believing in a long time ago. Anyway, I can relate to that. (Not that I’m complaining about the paper she presented – I’m glad she took the time to figure it out as it provides a nice example).
I was also reminded of the kind of awkward moment that happens sometimes where someone says something rather final and damning, and everyone pauses for a moment to listen to it. Then the chatter starts right back up again like it was never said. Gotta love academics!
Jessica:
Regarding your last point, I remember a small conference (1 day, approx 12 speakers, approx 100 people in attendance) several years ago on some topic related to the replication crisis in science.
The conf was held at a biostatistics department. The first talk was by Nicole Lazar, a statistician who was involved in the American Statistical Association’s statement on the problems with p-values and significance testing. She gave a strong talk on science reform. Next up was me, and I said some similar things, with various examples and lots of passion. The audience was alert the entire time and seemed to really groove on what we were saying.
Then at a later session someone gave a talk on something, I don’t remember exactly, some combination of methods and applied work. He was giving p-value after p-value, the kind of disjointed analysis that looks like is real trouble (and I think Nicole would agree with me), where you can get pretty much anything you want from your data by pulling out comparisons that are or are not statistically significant. Aaaand . . . everybody in the audience seemed to love his talk too!
It was like, some days you want vanilla, some days you want chocolate, some days you want strawberry, it’s all cool. I find that whole attitude so frustrating, even if it’s unavoidable; see the pluralist’s dilemma.
Conferences are perhaps the best environment for studying human intellectual fickleness.
It’s always good to hear that something is the best, in any dimension! In our short lives we want to spend as much time as we can on the efficient frontier.
Nice piece! I assume you’ve read Rudin’s article, “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead” which articulates much of the frustrations noted above. If not, it’s a fun read.
My thesis had a significant focus on the interaction between acute care clinicians and clinical decision support algorithms. ML/AI proponents tend to promote XAI as a way to build trust between clinician and model and create a kind of symbiosis, but I have yet to meet a clinician who finds SHAP values useful in a time-constrained acute environment. Most of these things get introduced then people stop using them after a month or two because they don’t provide added value. I’d love to spend a little bit of time doing some research quantifying the added value of different aspects of a model – the output, the covariates, etc., maybe using a decision-theoretic approach.
Yes, I know Rudin’s work.
Interesting about clinicians giving up on SHAP. The symbiosis line of thinking does often seem optimistic, at least in the literature, like we can just figure out the right configuration of explanation and deferral strategy to force on the decision-makers and their patients and voila, complementarity.
I wrote a blogpost last year on XAI:
https://errorstatistics.com/2022/03/23/the-ai-ml-wars-explain-or-test-black-box-models/
What provoked my interest is hearing criticisms of XAI based on their lack of error control or performance guarantees:
“[XAI] methods do not even bother to quantify expected error rates. This makes it impossible to subject algorithmic explanations to severe tests, as is required of any scientific hypothesis”. (Watson 2020)
Some argue that, at least in cases of high stakes medical contexts, instead of trying to explain black boxes in AI, we should focus on well-designed clinical trials (RCTs) to test the safety and effectiveness of treatments. This would be to test, rather than explain, AI predictions.
I agree with Jessica’s skepticism about the trajectory of the XAI line of work, at least how such work is often framed and presented. The parameters of the large neural networks are non-identifiable, so there is no particular reason to suspect that a feature attribution method will be stable across re-training. Such methods may still be useful for semi-supervised exploration of the data and model, but they are dependent on the inductive biases/assumptions of the methods. The analogy to clustering is apt.
There is, however, another line of work (parallel to the XAI area) from the last few years that arose from statistical machine learning and computational linguistics that I would argue is a much more practical approach to analyzing and deploying large language models (LLMs). We can approximate an LLM at the resolution of the available labels to derive quite robust uncertainty estimates, which then leads to other useful side-effects (e.g., “example/exemplar based checks/interpretability”, useful for examining the predictions relative to the observed/labeled data).
This line of work is quite mature at this point. And it does indeed address the case that we really need (i.e., “deep neural nets over high dimensional feature spaces”). My startup Reexpress AI has released no-code macOS software that does just that for large language models.
There are a lot of interesting and new things here. We’ve thought a lot about statistical communication in building the first version. Reexpress one has a very sophisticated and robust uncertainty quantification approach. It is non-parametric (i.e., it makes minimal distribution assumptions); however, it is unlike conformal frequentist approaches you may have seen in the ML literature. Rather than estimating marginal or approximately conditional coverage, it produces frequentist probabilities conditioned on reference classes formed by the Similarity to Training (depth-matches to training), Distance to Training, and a CDF-based per-class threshold on the output magnitude. Unique to Reexpress one, we also provide users with a second-order estimate of the reliability of the calibration process itself.
When the calibrated probability and/or calibration reliability are low, the user can then use the auto-visualization interface to examine the training and calibration sets. This can also be used to winnow the space of possible documents to label to improve the reliability for future predictions over such documents, and/or to reveal label quality errors in the current data.
Jessica, Andrew, I’d be happy to give a demo at your group meetings at Northwestern or Columbia if it would be of interest. We also have tutorial videos on our website.
Here is a link to our (Reexpress AI) website, which has tutorial videos, including how to add reliable uncertainty estimates to essentially any generative AI model when used for classification: https://re.express/
(The Tutorial 2 on the website demonstrates adding these capabilities to the Mistral 7b model, for example.)
If you ask a person why they ordered pizza today, sometimes it may be explainable. Like it is a habit/custom: “It’s pizza Thursday, I always order pizza on Thursday”.
Other times they just “felt like” pizza. Maybe they’ll tell you they were hungry, saw a pizza commercial and hadn’t had any for awhile. But then you can pay attention to all the times that happens and they *didn’t* order pizza. In those cases it is not “explainable”. It is more like an urge, gut feeling, or intuition. Whatever innumerable inputs lead to the decision to order pizza.
Same thing afaict.
One idea is that if you want an equation, then have the algorithm output one. When I was playing with the Julia symbolic regression package, I watched a few videos from the guy who wrote it. That is pretty much what he is trying to do. Eg, “Converting Neural Networks to Symbolic Models”: https://www.youtube.com/watch?v=wmQIcTOzH0k
Really, to understand an equation you need to derive it from some assumptions though.
I am genuinely interested in any methods that would actually improve explainability of AI decisions. I do however, support giving up on the technique for quantifying model variable importance they have named Shapley values (it really has little to do with the game theory concept).
In practice, Shapley values don’t actually answer any questions we are interested in. We did a presentation a few years ago at a practitioner-focused conference where we presented simulations that happened to demonstrate the limitations of Shapley values.
Back to the main topic though… I’m even interested in quantifying the objective we’re trying to solve. What is the goal? To provide examples of the evidence that was used to make an individual decision? To provide general guidelines on which evidence or combinations of evidence impact the decision in aggregate?
Was there any progress in defining some acceptance criteria for explainable AI? Even that would be useful.
I guess there are different definitions of “explainable” when referring to AI? Without knowing much about “AI”, this sort of thing seems like a practical demonstration of at least some level of explainability, no?
https://www.nature.com/articles/s41586-023-06887-8
https://news.mit.edu/2023/using-ai-mit-researchers-identify-antibiotic-candidates-1220
“‘appropriate reliance’ is often defined as agreeing with the AI when it’s right and not agreeing when it’s wrong, which can be shown to be incoherent in various ways”
Can you expand on this? I am actually inclined to agree with this, but I have a hard time articulating to myself why it seems intuitively true.
Imagine I’m a human who decides to accept an AI’s suggestion for some decision because I correctly ascertain that it has a higher probability of being correct than I do. But the AI prediction ends up being wrong. Most people studying AI reliance would call this overreliance. The simple definition confounds problems people might have judging the probability the AI is correct with problems using the utility optimal decision rule.
We have a short paper we are currently writing up proposing a better framework, I’ll post it when it’s done.
Jessica:
In your example, I’d distinguish between a prediction that is wrong only in retrospect and a prediction that is prospectively wrong.
For example, suppose I take a bet offering me even odds on a fair die landing on 1, 2, 3, or 4. That’s a prospectively good bet. If the die lands on 5 and I lose my money, that was a retrospectively bad bet, but it was not a mistake, prospectively, to make that bet.
Yes, I’m assuming we want a definition of reliance that does not penalize rational behavior at the time of the decision. And we want a method that allows us to distinguish different types of errors people could make so we have some clue about how to improve human performance.
You make a great point but Jessica’s statement:
“Most people studying AI reliance would call this overreliance”
still applies. If you lose the bet, most people – including many scientists and even many statisticians – will say you made a bad bet and your statistical model (or something else) is the problem. Unfortunate but true. Probably some of those people will even be the same people who said you *should* make the bet in the first place, but now that you lost and the social tide is against you, they see an opportunity to put themselves above you so they claim they knew it was a bad bet all along.
“most people” are often wrong.
Jessica
I don’t see how specifying some utility function will enable you to say whether or not an AI was “over-relied” upon or not. Certainly, once a utility function has been specified, it is possible to determine whether a particular decision was dominated by a choice not taken – that would permit comparing a human decision with a rational agent decision. But I don’t see how the actual outcome can be used to compare those decisions. Any model that yields probabilities is, by definition, possibly “wrong.” I think probabilistic outputs should not be thought of as “wrong” or “right” unless you have some sort of repeated experiment that permits that sort of estimation. For example, looking at many elections (assuming they can be compared) might permit one model to be said to be more accurate than another. There will still be the issue of whether any particular election can be compared with the others (e.g., the 2016 election where models had put the probability of a Clinton victory at 80% or more – if those same models are applied to many elections, then perhaps the accuracy can be considered, although there will still be the issue of whether the 2016 election was somehow different).
So, are you talking of comparing models or comparing decision rules? Certainly decision rules can be thought of as “wrong” even if the underlying model that produces probabilities might not be wrong. But the post is about “explainable AI” and I feel like we are now talking about “explainable decisions.” Are these the same things? I don’t think so. There is certainly a need for explainable decisions – both legally and ethically. It certainly helps explaining decisions when the model that produces probabilities is explainable. But even if the model is not explainable, there is still a need to explain the decisions. Perhaps we should separate the two – models and decisions – and clarify which we are referring to. When I hear the phrase “explainable AI” I am not thinking about the decisions – I usually think about the models.
This is an important distinction to make to the XAI audience. The deep learning crowd grew up around problems where neural networks perform well – typically computer vision and natural language processing. These tend to be epistemic uncertainty problems. If you have a model to predict if there’s a cat in an image, and the model gets it wrong, this is sensibly considered an error. Occasionally, that crowd stumbles onto an aleatoric uncertainty problem, and doesn’t realize the rules have changed. Getting this wrong can result in nonsense like the ProPublica recidivism article.
Kevin
I don’t understand your reference to neural nets. Don’t they produce probabilities as well? They do for me – so the predicted classification requires a decision rule to go from probabilities to predictions. The decision rule (such as if probability > 0.5, then A, otherwise B) may be an error, but that doesn’t mean the probabilities were in error.
Dale,
Sorry, that was in response to Andrew’s post, I didn’t realize it would show up down here.
A misclassification is a mistake in a way that a low probability event occuring by chance isn’t. People in AI will often incorrectly treat the two situations in the same way – in particular, by using metrics that condition on the outcome, like false positive or false negative rates.
Somebody (I think a famous poker player) said that you should evaluate the quality of the decision, not the quality of the outcome. One can object to this on mathematical grounds (in expectation, the outcomes should align), but I still like the statement a lot.
Eric:
As they say in poker, don’t evaluate the play, evaluate the strategy. See last line of this post. I’m not claiming this quote or this idea is original with me!
In practice for high-dimensional inputs, I would argue that three things are needed, and just presenting a probability alone has too high of a lossy compression. I recommend providing decision-makers with the following:
1. A calibrated probability that controls for similarity-distance-magnitude, and optionally, any additional attributes of focus
2. An estimate/signal/check on the reliability of the process that determined (1). For example, if the similarity-distance-magnitude partition of the test point reflects a relatively loose connection to the training support, and/or the size of the partition in the calibration set is small, that should be directly conveyed to the user.
3. Finally, some type of mechanism/interface/etc. is needed to evaluate and audit (1) and (2). This can take the form, for example, of a visual interface to explore the predictions relative to the data with known labels.
Aside: Curiously, my guess is that the lasting takeaway of deep learning won’t be higher predictive accuracy, per se, but that it enables (1)-(3) above over high-dimensional inputs for which constructing other types of models (e.g., physical models or rule-based models) is challenging.
Isn’t the terminology that the AI’s prediction “ends up being wrong” unnecessarily confusing the issues? The AI prediction gave a probability for some decision. That probability can only turn out to be wrong if we have enough data (such as repeated circumstances to render that probability “wrong” – at least beyond some reasonable doubt) to determine it was wrong. If it is a unique decision, you may regret the decision, but there is no way to determine the AI was wrong, nor whether it had a higher probability of being correct than you did.
I think your hypothetical circumstance should be reworded: you believe the AI’s probability of an outcome from a decision to be superior to your personal judgement. In some cases, that change in the probability might cause you to take a different decision. Once the decision has been made, the actual outcome cannot be sufficient to establish whether you or the AI had a better judgement of the probability of an outcome. You may come to regret relying on the AI or you may feel justified by using its output rather than your own. But I don’t see how regret about your decision (or lack thereof) can be used to judge the probability that the AI is correct. This whole language of “correct” or “wrong” seems at odds with the actual output of the model. If you specify a decision model that predicts outcome A rather than not A if the probability of A is greater than 50%, then that decision model can indeed be “correct” or “wrong” based on the outcome. But isn’t that binary thinking the biggest problem with NHST and is just being repeated if using AI models in this way?
Dale said:
“the actual outcome cannot be sufficient to establish whether you or the AI had a better judgement of the probability of an outcome. ”
Excellent. There is no probability or error once the outcome has occurred, so for a probabalistic forecast, there is no “wrong” in some sense, because there is always some probability it will be “wrong”.
Imagine we have a human predictor with some accuracy on a decision problem (e.g., predict which defendants will recidivate) and some AI with its own accuracy on the same problem. We know the data generating process from which the decision tasks are drawn. Using decision theory, we can define the utility optimal behavior of a rational (Bayesian) decision maker who has access to these two sources of predictions (basically we take a similar approach to what we do for visualization experiments here: https://arxiv.org/abs/2304.03432)
I look forward to your comments when I have the paper draft to share!
I think my comment came out in the wrong place – it belongs below after Jessica’s comment with the link in it.
Now it is definitely in the wrong place! Just replace “below” with “above.”
Hi Dale,
“Certainly, once a utility function has been specified, it is possible to determine whether a particular decision was dominated by a choice not taken – that would permit comparing a human decision with a rational agent decision.”
It seems we agree. Adede asked for more info on an aside in my post, where I mentioned having an issue with how researchers define reliance in controlled studies trying to understand if people over or under-rely on AI predictions. I was critiquing the status quo approach, which is to say that appropriate reliance is when the people in your study agree with the AI when its right and disagree when its wrong. Try reading my comments above with the understanding that we are talking about how researchers should set up a situation where we can provide a more coherent definition of optimal reliance.
I know that you just mentioned it as an aside, but I just want to make one more dig against the method of variable importance that has been named Shapley Values (not really related to the game theory concept). Since it’s easy to run, people keep including it in papers, etc. But it isn’t really useful in this case. If we care about explainability of the model (for example, a neural network model), then we should do sensitivity analysis using that model. Shapley values use an ensemble of very stupid models (e.g. just linear model with one variable, then two, then three) and average across them. Many of those models averaged across may not be likely given the data, and worse for this explainability case… none of them are the model you wish to explain. It’s trying to measure variable importance separate from any modeling. I’d be much more interested in methods that tried to explain the decision of the actual model we are interested in. For example, if these sets of cases from the training set were removed, we would have made an opposite decision, so these cases are critical to the model’s decision. Or building a sensitivity curve using 1 or more variables and showing at what thresholds the model would have made a different decision. These types of approaches seem plausible and actually move toward answering the question of why the model made the decision it did.
“Mechanistic interpretability” is the term i’ve heard applied to attempts to reverse engineering what a neural net learned. I.e., work that is more interested in trying to infer mechanisms over post-hoc linear approximations.
My impression is that the frontier of this kind of research has been for a while the work of Chris Olah, Neel Nanda & co. at Anthropic and DeepMind, and the academic XAI field is currently a ball rolling by inertia.
Anecdotally, last year I saw a seminar by an XAI researcher, who gleefully explained their new state of the art technique based of using another neural network to explain a neural network, and showed how this opened promising lines of research. But his final slide was: “However, now that GPT is out, I think maybe all this does not matter anymore at all. I don’t know what’s going to happen.”
Although I never saw a precise critique like the one you witnessed, 3 years ago, while applying to grad programs, I skimmed over the topic and had the hunch it was doomed. Luckily, I was not accepted in the place where they did mostly XAI, so I did not have to make that call.
“… and the same explanation technique is often highly sensitive to small differences in the function to be explained (e.g., slight changes to parameters in training).”
It seems the me that the situation is even worse. Before one can start to give any guarantees (or not) about explanations one has to fix the level of feature coarse-graining. And of course explanations (of the same function) are extremely sensitive to it. This is rarely discussed in XAI publications nor in theory papers – presumably because one cannot prove any theorems about it. But how should one know which level of coarse-graining is the correct one?