The opposite of “black box” is not “white box,” it’s . . .

In disagreement with X, I think the opposite of black box should be clear box, not white box.
A black box is called a black box because you can’t see inside of it; really it would be better to call it an opaque box. But in any case the opposite is clear box.

22 thoughts on “The opposite of “black box” is not “white box,” it’s . . .

  1. Opposite of “black box” could be “no box”.

    Which I guess would render “black” somewhat redundant in the original phrasing. But as I think Walt Whitman once said…Yeah, I’m inconsistent. So sue me.

  2. I find the claim that black box models are uninterpretable unhelpful. A combination of sampling predictions for a wide range of inputs (and graphing their relationship) and causal testing of the relationship between input parameters and predictions mean that any model that can produce predictions can be made interpretable.

    • Suppose your bank uses a black box model with 40 predictors. you think you’ve been wrongly denied loans, and you subpoena their model. they won’t let you see the model but you can query it all you want…

      so you decide to try putting 5 different values for each of the 40 predictors… 5^40 = 9×10^27 so there are more inputs to consider than gas molecules in the room you are reading this in

      • I don’t understand how interpretability is relevant to the question of being wrongly denied the loan. Let’s say that they showed me the linear regression and it shows that if I had more previous debt that I’d be more likely to qualify for the loan. I would then say that I don’t beleive that to be true. They say that it’s true in our training dataset. It seems like model soundness, accuracy and generalisability are all that’s relevant for that assessment. Both of those can be assessed by doing validation testing on independent data from the training set (i.e., just by querying the model).

        Even if it was somehow relevant, then I would explore the problem by seeing what variations to the initial input were necessary to change the predicted outcome (e.g., by doing a random walk around the initial input values). That requires far less searching than sampling the full parameter space.

        • Almost the entire purpose of interpretability is to understand mechanisms by which biases can induce bad outcomes which perpetuate or even worsen social problems… it’s all about applying machine learning to problems directly affecting people’s lives particularly when training data may itself represent unreasonable bias.

          Suppose the bank makes a policy of mostly giving out loans to rich white trust fund kids and occasionally to very low income black families when the loans are subsidized by government grants to improve poor neighborhoods.

          Now, the bank trains on this dataset, and then decides to expand its business into giving out small business loans to recent college graduates. A whole bunch of black and hispanic entrepreneurs show up looking for loans for their restaurants or bike repair shops or whatever, and they are denied, whereas white people with the same financial and educational stats are not denied… But… get this! The machine learning algorithm *doesn’t know* what race you are.

          Why were they denied? Why were white people with the same income, debt load, rent, car loans etc not denied?

          Now suppose I tell you that in effect, the large number of other predictors, like whether these people have ever had a monthly bus pass, or some location tracking information about where their phone has been, when put together, allow the ML algorithm to effectively deduce the race of the people applying, and in the training dataset, people with that race were almost entirely high risk people who only got loans because of the govt subsidies previously… But of course we don’t know that, because it’s a 15000 connection neural network.

          Do you think this is OK? Do you think this is easy to figure out by just sampling around? How are you going to sample around the possible values from the phone location database or the bus pass database or the 37 other weird and proprietary databases being used?

        • Daniel’s example sounds like it’s clear, but farfetched. It’s not farfetched at all.

          Segmentation schemes such as Acxiom’s Personicx and Experian’s Mosaic code households into “lifestyle segments”, and are often used because they attempt to code every household in the US into one of the segments (71 in Mosaic’s case). These segmentations explicitly do not include race in their descriptions and the oral claim I’ve heard from one of the companies is that race is not used to derive the segments. That’s probably true due to the legal liability involved.

          The segments might be used to market lifestyle products — e.g. are you in a lifestage where you are likely to be interested in buying furniture? Life insurance? Retirement annuities? and so on. Note Experian is a credit bureau, and many of its clients are in financial services, and no financial services firm in the US wants to say they explicitly use race in their modeling.

          So, if you just changed the race of a household on a data record and nothing else, you wouldn’t see a change in the outcome.

          But if you looked at, say, 2 households with the same income who have different races, you’d find a difference in prediction because the location of the household (urban core, say, versus rural) and other factors are likely to differ in ways that may or may not be valid for the marketing decision (because even in a large model that’s validated using a holdout sample, the individual predictors would usually not be validated).

        • Thanks zbicyclist! I don’t actually have any personal experience with this kind of data, but I did work in the finance industry for a few years, and I can easily see how this kind of thing might happen.

          Forcing important social decisions to be made by explicit modeling which can be modified to take into account known problems in the training data is a kind application of “prior knowledge”. In many cases, we don’t want pure predictive power in the hold-out dataset, because both the training and hold-out datasets have the same biased perspective on the world.

          It is extremely rare for these kinds of models to be fit to a random sample of the current population, the *sampling* is biased and so the predictions/outcomes are biased. If in the past mostly white college educated people opened coffee shops or bike stores, and suddenly through improvements to a local economy some minority groups are getting college educations and wanting to open small businesses… we shouldn’t judge them on the basis that in the past, all the minorities in this region were struggling to survive and often didn’t pay back their loans. That’s not necessarily relevant to today.

          If we have no ability to introspect the models, then the models will perpetuate the historical biases they are trained on.

  3. I think a “clear box” isn’t ideal either. A clear box indicates you can see all the complex machinery, but the contents might still be too tangled and dense to understand from the outside.

    • I find any protestations to the contrary insubstantial.

      “Despite the insubstantial protestations of some who would deny the connotations and impact of such language [43], the use of the terms “black” and “white” in the context of predatory publishing must be considered racist. ”

      This is the kind of outstanding, well argued, research that I expect from the Irish.

Leave a Reply to Adede Cancel reply

Your email address will not be published. Required fields are marked *