In his final book, Perplexing Plots, the late David Bordwell wrote:
It’s not accidental that mystery stories are drawn to tricky shufflings of viewpoint or chronology. A plotline built on a detective’s present-time inquiry into past events helps us understand when the order of events is rearranged or character perspective changes.
This reminds me of two things related to statistics:
1. From my article with Jessica, Is Your Chart a Detective Story? Or a Police Report?:
Every data visualization is a story, a plot to be unraveled—but some are more approachable than others. Modern statistical displays of data—grids of scatterplots for inspecting correlations, for example—succeed by being transparent and allowing trends in the data to stand out. In contrast, classic data visualizations often succeed, paradoxically, by being a bit opaque: a puzzle that a reader figures out. . . .
In science we are delighted by unexpected brilliance, which we immediately try to systematize. The same goes for visualization: When we see a new and revelatory graph, we want to take it apart and see how it works. . . .
We can liken this experience to narrative, a lens through which many great (and lesser) works of art have been interpreted. Narrative involves some interplay between plot and perspective, events and interpretation, storyline and characters. Similarly, the practice of science can be viewed as the interplay between data and models. Data are the facts. Models are the characters whose perspectives and assumptions shape what we take away from the story. At the simplest level, the choice of how to visualize data structures the viewer’s experience of those data by promoting certain comparisons over others. It’s a character choice, a choice of model. . . .
Much has been written about how different forms of narrative involve the reader in different ways, from the relatively passive engagement of viewers of a film, to the more active involvement of those following a serial television drama, to the experience of people reading novels who must in a sense create entire movies in their heads. Data visualizations can fall in different places along this continuum. The stories told by some are so strong and clear that they require little from the viewer. Others are far more demanding. One could draw an analogy to works of art that are more or less accessible to the audience—but with the difference that hard-to-follow art is often intentionally ambiguous, whereas challenging visualizations are meant to be understood. In that sense, visualizations are more like video games than art or music. They invoke a trial-and-error experience reminiscent of the “active learning” approaches studied by educational psychologists.
As with video games, it is often the more unconventional visualizations that are the most appealing ones, even to broad audiences. That which is not familiar is more challenging; and aesthetic choices, like the use of pleasing shapes and symmetry, can help entice the viewer to try and solve the puzzle. . . .
What is exciting and unconventional is also a function of our expectations. Music is said to be compelling to the extent that it balances expectation and surprise: A note is interesting when it catches us off-guard, but then it should also make sense within the larger pattern of the piece as it develops. The same is true for storytelling: The thrill of the unexpected can only come with reference to (and in confounding) some preexisting norm.
In addition to addressing the issue of the pleasures of difficulty in narrative, this seems closely related to another of Bordwell’s points, which is that genre fiction can be highly experimental in form and makes this accessible by placing those innovations in a stylized context that is comfortable to readers or viewers.
2. The forward logic of the data generation process and the reverse logic of inference. In a statistical “generative model” or “directed acyclic graph,” there is a logical order: decisions and outcomes happen in time and can influence what comes in the future. In statistical learning, we start with the data and go backward to make inference about parameters that have already been generated and forward to make inference about predictive quantities. When we fit a model and apply it to the future, we’re going back and forth in time.
I think I’ve written something on the logic of data generation and the logic of inference in statistics, but no amount of searching turns anything up. The closest is my article with Guido, Why ask why? Forward causal inference and reverse causal questions, which also appears in Regression and Other Stories as section 21.5, “Causes of effects and effects of causes.”
Re: reverse causal. You (Andrew) have written numerous times that reverse causation has no mathematical basis in statistics, while forward causation does (sorry if I misstated that but hopefully the concept I am referring to is clear.) I asked where that concept was derived and you graciously gave me a reference. I followed the reference (I have since lost it) and found the same direct statement that you made about no formal definition for reverse causation. Is there some paper somewhere in which it is actually shown within some logical framework that forward causation is statistically tractable but reverse causation is not? Sorry to be a pest!
Matt:
In our article, Guido and I say: “the search for causes can be understood within traditional statistical frameworks as a part of model checking and hypothesis generation. We argue that it can make sense to ask questions about the causes of effects, but the answers to these questions will be in terms of effects of causes.”
I wouldn’t say that reverse causation has no mathematical basis in statistics; rather, i’d say that the mathematical basis of reverse causal reasoning in statistics is connected to model checking. Except in some very restrictive settings, we can’t estimate “the cause of Y” or “what percentage of Y is caused by X,” but we can interpret such Why questions as pointing to aspects of data that are not explained by some existing model.
I don’t really have more on this beyond what’s in that paper with Guido.
The problem is that you can not answer CoE questions in general! That’s not a discussion about lack of statistical (or any other) framework. How could you ever know whether X is a cause for Y? Perhaps in some counterfactual sense, you could say: Imagine value of Y in absence of X (the very next problem is how to get the “right” counterfactual world…). There is whole philosophical approach to this problem.
On the other hand, answers about EoC are relatively easy to get: Difference in Y0 and Y1 (manipulate X, measure Y).
Roman Folw wrote:
“How could you ever know whether X is a cause for Y? Perhaps in some counterfactual sense, you could say…”
This is the epistomological/philosophical concept that seems to be widely held here at the blog that I cannot understand.
If my car won’t start, the goal of the scientific project is perfectly circumscribed. I want to get it to consistently start again. There need be no connection whatsoever to any interpretation of the wider world. First I make my model, a list of all the parts involved in starting the car, and then I individually test each one. If I find one part that fails and all others pass, and I then replace that part and the car starts again, I have literally – and really in every epistemological and philosophical interpretation as well – determined that x caused y. In this case, there is no “ceteris paribus” assumption, or any other assumption necessary because I found an extremely plausible causal relationship and my car starts again every time after the fix. Nor is there any squishy philosophical question about causation, because I already drove home!
This is the point where everyone wants to quibble. What if there is an intermittent? That is not an epistemological question, it just means I require more tests. But what if there is no intermittent? Because if there isn’t, and replacing the bad part does achieve my fully circumscribed goal, then I really did successfully determine that x causes y.
I presume that this is Andrew’s “very restrictive setting,” and indeed my approach is merely model checking. But I think the epistemology is reversed here. Having a full, testable list of possible causes does not require setting aside the real world and creating a thought experiment. Cars get fixed by reverse causation every day that way. I would argue that social science about human behavior, where ceteris paribus is unavoidable and there is no fully conclusive final test of the causal inference, is actually the “restricted” setting that forces us to think epistemologically about the limits of our ability to model complex phenomena.
Matt:
Your car example fits perfectly into the framework of my paper with Guido. You start with a Why question (Why does my care not start?) which represents an anomaly (The car is supposed to start. It usually starts. My implicit model was that when I turned the key, the engine would turn). This motivates a series of hypotheses (Maybe I’m out of gas? Maybe the starter motor is broken? Maybe the battery is disconnected?, etc.), which in turn motivates data gathering (insert a dipstick into the gas tank, check the battery cables, etc.) and then experimentation (add gas into the tank, detach and reattach the battery cables, etc.) with the goal of fixing the problem, which is a set of forward causal questions of the form, What will happen when I do X?
Matt Skaggs:
“If my car won’t start (…). I want to get it to start…”
This example does not, at least in my limited understanding of the problem, belong into the what-is-the-cause realm. You want to set value of Y to “my car starts”.
Then you proposed to “list of all the parts involved in starting the car, and then I individually test each one”.
Take a look, you just perfectly moved into the what-is-the-effect-of-X-on-Y (or in Andrew’s words into the forward questions). What you are actually doing in the example scenario, is that you change X_1…X_k holding other Xs constant and then you measure the difference of Y_0 – Y_1.
What I feel like solving the car example would look like if framed in what-is-the-cause realm: My car doesn’t start (eg. Y = 1), then what chain of events lead to the state. Now, I have to start a hypothesis generator: Y = 1 is because of X_1 = 1, X_1 = 1 is because of X_2 = 1 … etc. into the big bang. You usually want to stop somewhere, let’s say X_3 is the proposed cause (meaning if X_3 = 0, then Y =/= 1). The problem is how could you ever know that X_3 is THE cause? I think you ever can’t, the best you can do is to assume counterfactual world in which X_3 = 0 and Y = 1.
I re-read the paper with Guido. It is specifically the epistemological claims that I am uncomfortable with. The paper contends that it is the forward causation “tests” that define the approach, but I don’t agree.
You and Guido wrote: ” Following the work by Rubin (1977) causal questions are typically framed in terms of manipulations: if x were changed by one unit, how much would y be expected to change?”
This is a claim that when we try to understand the past, the inevitable evidence-gathering choices necessarily change the logic from reverse causal to forward causal. I can indeed reverse the logic this way, but my method would look entirely different. Now I am trying to figure out why my car won’t start by taking a statistically significant sample of starters and subjecting them to full life cycle testing. Then from the results, I am estimating the likelihood that my car failed at x number of miles from a worn out starter. This is how social science (and just about everything else) moves forward yet bears no resemblance to what we are trying to accomplish.
Andrew and Guido wrote:
“We label this historical reasoning as forward causal inference as well, as it is based on the estimation of effects of defined treatments.”
Again, this misses what we are doing. There is no treatment (driving the car?), so there can be no estimation of effects. When we do a component test, we are not even indirectly trying to estimate the effect of a treatment. We are exclusively trying to figure out what happened in the past. We don’t even care WHY the starter broke. The reason we don’t care is because the logic driving our endeavor is purely reverse causal.
I think the paper by Gelman and Imbens makes valid points about reverse causation as practiced in social sciences being an indirect form of forward causation. But any suggestions that this applies to all causal inference is a bridge too far.
Matt:
You write, “There is no treatment (driving the car?), so there can be no estimation of effects.”
In my discussion of your example in the above comment, I refer to experimentation (“add gas into the tank, detach and reattach the battery cables, etc.”). These are the possible treatments that I’m considering.
“Driving the car” cannot be the treatment, because the car does not run.
I appreciate the follow-up. Now I finally understand what is happening here.
First, I concede that I cannot get around the forward causation step that leads to the car starting again. (I took my best shot at eliminating forward causation and it didn’t work, so it would be crass to not concede.)
But I am still not happy. The problem is that I don’t need that forward causation step to claim that x caused y. I already executed my reverse logic approach and came up with the “x caused y” answer. Driven purely by reverse causation logic, I verified that there was gas in the tank, and I independently tested each component. This resulted in passing tests for all the components except the starter, which failed. Epistemologically, I have completed my reverse causation task. Now I move on to validation, and that is where I must use forward causation and swap out the bad starter with a new one. I then complete my validation by starting the car. From this perspective, it is clear that the forward causation step is the model-checking, a quality control step that tells me that I executed reverse causation properly.
Matt, I tend to agree with you. If the hypothesis is “Maybe I’m out of gas?” and the data gathering process is to “insert a dipstick into the gas tank” then the “add gas into the tank” is no longer about finding why the car doesn’t start.
However, I see how Andrew may claim that it’s an intervention intended to answer “What will happen when I do X?” because there could be other reasons why the car won’t start even if the tank is full.
I guess it all depends on what do we want the different parts in the analogy to stand for.
The analogies with crime/detective fiction seem applicable also to some of the psychological/organizational difficulties in model building, e.g. the tendency to lock into the first plausible narrative/model and resist pushing too much against it. There might be a parallel there with the fantasy of the Holmesian detective (although Holmes himself was much more careful with his analysis) going backwards from data to precise and certain inferences just through observation (“big data”) and reflection (“statistical software”).
It’s not like there is no fiction, or even no crime fiction, comfortable with the possibility of multiple competing explanations, intrinsic uncertainties, and soon, but I wonder if, overgeneralizing, that sort of fiction doesn’t have much overlap with mainstream fiction among the main consumers of statistical analysis. I still remember 18 years ago or so a client asking me if data analysis worked like in the “Numb3rs” show, and being disappointed when I said it very much didn’t.