Skip to content
 

Zombie semantics spread in the hope of keeping most on the same low road you are comfortable with now: Delaying the hardship of learning better methodology.

Now, everything is connected, but this is not primarily about persistent research misconceptions such as statistical significance.

Instead it is about (inherently) interpretable ML versus (misleading with some nonzero frequency) explanatory ML that I previously blogged on just over a year ago.

That was when I first become aware of work by Cynthia Rudin (Duke) that argues upgraded versions of easy to interpret machine learning (ML) technologies (e.g. Cart constrained optimisation to get sparse rule lists, trees, linear integer models, etc.) can offer similar predictive performance of new(er) ML (e.g. deep neural nets) with the added benefit of inherent interpretability. In that initial post, I overlooked the need to define (inherently) interpretability ML as ML where the connection between the inputs given and prediction made is direct. That is, it is simply clear how the ML predicts but not necessarily why such predictions would make sense – understanding how the model works but not an explanation of how the world works.

What’s new? Not much and that’s troubling.

For instance, policy makers are still widely accepting black box models without significant attempts at getting interpretable (rather than explainable) models that would be even better. Apparently, the current lack of  interpretable models with comparable performance to black box models in some high profile applications is being taken as the usual situation without question. To dismiss consideration of interpretable models? Or maybe it is just wishful thinking?

Now there have been both improvements in interpretable methods and their exposition.

For instance, an interpretable ML achieved comparable accuracy to black box ML and received  the FICO Recognition Award. That acknowledging the interpretable submission for going above and beyond expectations with a fully transparent global model that did not need explanation. Additionally there was a user-friendly dashboard to allow users to explore the global model and its interpretations. So a nice very visible success.

Additionally, theoretical work has proceeded to discern if accurate interpretable models could possibly exist in many if most applications.  It avoids Occham’s-Razor-style arguments about the world being truly simple by using a technical argument about function classes, and in particular, Rashomon Sets.

As for their exposition, there is now a succinct 10 minute youtube  Please Stop Doing “Explainable” ML that hits many of the key points along with a highly readable technical exposition that further fleshes out these points: Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead .

However as pointed out in the paper the problem persists that “Black box machine learning models are currently being used for high stakes decision-making throughout society, causing problems throughout healthcare, criminal justice, and in other domains. People have hoped that creating methods for explaining these black box models will alleviate some of these problems, but trying to explain black box models, rather than creating models that are interpretable in the first place, is likely to perpetuate bad practices and can potentially cause catastrophic harm to society”.

This might be best brought home by googling “interpretable ML” – most of what you get is all about creating methods for explaining black box models that might mitigate some of the problems often using the synonym for explaining – interpreting. A semantic ploy to keep most on the same low road of trying to salvage black methods in applications were intepretable models have already been developed or likely can be?

Recall I started this post with “everything is connected” – the connection I see here is with persistent research misconceptions.

“Why do such important misconceptions about research persist? To a large extent these misconceptions represent substitutes for more thoughtful and difficult tasks. … These misconceptions involve taking the low road [to achieving and maintaining academic prestige in methodology] , but when that road is crowded with others taking the same path, there may be little reason to question the route.” Six Persistent Research Misconceptions. Kenneth J. Rothman

Same old same old of these misconceptions [better to explain black boxes than switch] representing substitutes for more thoughtful and difficult tasks [learning methods to obtain interpretable ML and getting them]. The more widely shared the misconception (more crowded the low road) the less the need to be thoughtful. The persistence will likely persist for a long time :-(

Now trying to stay with the inertia, especially if it is substantial, has its advantages. Might even be optimal in the short run. It’s the accountant/economist’s dilemma of disregarding suck sunk costs.

A forthright quote about this is available from a well established AI researcher Been Kim: “There is a branch of interpretability research that focuses on building inherently interpretable models … Right now you have AI models everywhere that are already built, and are already being used for important purposes, without having considered interpretability from the beginning. … You could say, “Interpretability is so useful, let me build you another model to replace the one you already have.” Well, good luck with that [convincing someone to not just fix something that not broken, but actually throw it out and replace it?!!!]”.

For me, the bottom line of this post is that in settings where there are yet a lot of black box models being used – avoid those sunk costs if at all possible – they really su*k!

 

 

17 Comments

  1. Rohin Shah says:

    It’s important to note that models that are inherently interpretable are only likely to get good accuracy on tasks where there are already nice features. The paper itself acknowledges this:

    > When considering problems that have structured data with meaningful features, there is often no significant difference in performance between more complex classifiers (deep neural networks, boosted decision trees, random forests) and much simpler classifiers (logistic regression, decision lists) after preprocessing.

    In contrast, many of the techniques that aim to explain black-box models, are looking at tasks where you do not have structured data with meaningful features, and you instead have to learn such features. (Many would argue that deep learning is primarily for learning good representations / features.) In this setting, there’s a lot more reason to expect a tradeoff between interpretability and accuracy — to get interpretability, you essentially have to enforce the constraint that the features are human-understandable, and in practice you have to enforce a stronger constraint of “the features must look like so-and-so”, which unsurprisingly tends to handicap the model.

    So re:

    > A semantic ploy to keep most on the same low road of trying to salvage black methods in applications were intepretable models have already been developed or likely can be?

    I disagree that “intepretable models have already been developed or likely can be”, and I think many others would as well.

    (The paper you link to cites one paper that develops interpretable models for vision, but notably they don’t apply their model to ImageNet.)

    You might respond with “well what about the Rashomon sets argument?” I don’t find that argument compelling — it’s basically saying “if there are a lot of good models, probably one of them is interpretable”. In the Rashomon set paper, none of the theorems ever talk about interpretability — just about smaller and larger function classes. The theorems are also based on old generalization bounds, and we already know that such bounds are extremely loose in practice for deep learning.

    It’s as though you said “there are thousands of computer programmers in the world, and more than one in a thousand humans is < 1 year old, so probably there is at least one computer programmer who is < 1 year old".

    • Keith O’Rourke says:

      Rohin:

      I should have been clearer that there are applications, where at least for now, interpretable models are not there yet and maybe never will be. This should be taken for granted.

      However, my concern is in regard to applications with structured data with meaningful features, in high stakes areas that are being addressed with black box methods (or in the cue). I am aware of many proposals for just that. In government and industry applications with structured data with meaningful features may be the vast bulk of applications. Does anyone actually know?

      > I disagree that “intepretable models have already been developed or likely can be”, and I think many others would as well.
      There certainly won’t be many if most analysts don’t ever try. Sort of self-fulfilling.

      > Rashomon sets argument?” I don’t find that argument compelling
      I wrote that the work has proceeded.

      > It’s as though you said “there are thousands …
      I didn’t I say that and never would have meant to. You present a deductive style claim, where I simply see an abduction – given there are thousand of models that predict with the same accuracy, maybe one or more are inherently interpretable (it’s a good bet).

      • Rohin Shah says:

        > I didn’t I say that and never would have meant to.

        Sorry, I didn’t mean to imply that you said that — “you” was referring to the Rashomon sets argument.

        > However, my concern is in regard to applications with structured data with meaningful features, in high stakes areas that are being addressed with black box methods (or in the cue). I am aware of many proposals for just that. In government and industry applications with structured data with meaningful features may be the vast bulk of applications. Does anyone actually know?

        My sense was that many of these sorts of applications are solved by linear or logistic regression. I’m sure there are also some that are solved with black box models.

        Regardless though, my main point is that there are settings where interpretable models are not yet up to snuff, and there are plausible arguments that they never will be, and as a result it isn’t justified to tar the entire field of explainable ML, which to my knowledge tends to focus on these settings.

  2. John Richters says:

    Keith:

    Rudin’s paper devotes considerable attention to ProPublica’s accusations that the proprietary COMPAS recidivism model (the most widely used in the U.S.) is racially biased. Here’s a link to the November 2018 Royal Statistical Society’s analysis of the statistical flaws underlying ProPublica’s accusations– which does an excellent job of stepping through the relevant issues:https://www.rss.org.uk/Images/PDF/influencing-change/2018/RSS_submission_Algorithms_in_the_justice_system_Nov_2018.pdf

    • Keith O’Rourke says:

      Thanks -I’ll have a look.

    • Annoyed at the utter lack of graphs in that paper, but I agree with the basic assertion that the percentage of people with a given score who recidivate should be unchanged across races, and this is the criterion to use to determine bias.

      It’s not the job of the algorithm to change the population, it’s just the job of the algorithm to accurately assign risk to each person conditional on all the factors.

  3. Ethan Steinberg says:

    I’m still really confused about the exact goal of “interpretable ML”. Very concretely, what are we hoping to get out of the model that we would otherwise not get? What exactly is our metric for whether a model is “interpretable” and how do we measure it?

    On a very practical level, how would you run a randomized trial to show that the interpretable model provides more value?

    It seems like a lot of interpretability work is premised on a very vague sense of “value”, sorta like the old joke of “A, then B, then ???, then Profit” without a lot of thought on why exactly we should care about interpretability and how that interpretability is useful for actual decision making.

    For example, I have a logistic regression model that performs quite well, but I am faced with the choice of deciding how many features to use. The more features, the more accurate in this particular setting (I have a lot of data). The models with more features however are not interpretable. What exactly is the decision tradeoff here? What exactly am I giving up by intentionally choosing a worse model?

    https://hci.iwr.uni-heidelberg.de/system/files/private/downloads/860270201/felix_feldmann_eml2018_report.pdf is the main paper I know that really tries to evaluate this in an empirical fashion. They define one potential value of interpretability to be the ability for humans to detect when the model is wrong. They run a randomized experiment to measure whether interpretability actually has a causal effect on whether humans can detect that the model is wrong. They got a null result. Of course, this is a very limited experiment (and has it’s own set of issues), but it’s the main one I know of that really tries to tackle and determine the actual effect of using a more “interpretable” model for practical decision making.

    • I think the goal is to be able to answer questions like “why does this model treat men differently than women?” or “why does this model work really well for white patients but not black patients?” or “why does this model fall apart for people in rural counties?” or things like that. We’re using models to figure out how to treat patients, give loans, deploy police officers, give people parole, etc. There are existing biases in the world that makes training models on our existing biased practices perpetuates the biases… so enabling us to understand how features that are known to be related to biases in society are used in the model allows us to diagnose problems with models trained from our existing biased practices.

      I’m sure there are other uses too.

  4. Rahul says:

    But how do we explain the amazing utility black box models have given us?

    Most of the popular triumphs of AI eg Alexa, Google cars etc. Are all heavy on the black box model side aren’t they?

    • A chainsaw is great for cutting down trees, but not so good for trimming a hedge or build furniture.

      By the way, last year when I pointing this interpretable approach out to a clinical colleague who currently works with Geoff Hinton – they accused me of trying to undermine their work.

      • Rahul says:

        Keith:

        My point is, people would gladly use interpretable methods in a lot of contexts………..if only it was possible to deliver results as good as a blackbox method.

        Of course, I do admit there’s a lot of hype-fueled proliferation of neural networks etc. based academic papers for various applications. GIGO.

        • When it comes to social or medical applications where bias in existing practices that form the training data is rampant, we should demand more. In many of these cases you can build some Neural network that “solves” the problem… according to the existing biased training data! We should do better.

          When the application is something like landing an aircraft in challenging crosswinds or driving a vehicle, the training data isn’t necessarily particularly biased, we can potentially afford to use a black box method.

          • Wonks Anonymous says:

            Biases in training data seems like a separate issue from whether an ML is interpretable. A complete black box will still show lower accuracy when given non-training data.

            • Biases in training data is a *reason why we should demand interpretable*. If we know there will be biases, we need to know what our algorithms are doing so we can hopefully undo the biases inherent in the system and get accurate results that don’t involve basically learning bigotry from the teacher.

              Where there isn’t a significant culture-wide bias, like say learning unstable aircraft flight dynamics from sensor data… the importance of interpretability becomes lessened.

        • > lot of contexts………..if only it was possible to deliver results as good as a blackbox method.
          That is exactly what the video and paper link challenge.

          In paricular, in the FICO challenge the blackbox method’s were not better.

          The larger point is not to accept black box models without significant attempts at getting interpretable models. I think that is a good bet, certainly in areas I have worked.

Leave a Reply