Felix Pretis writes:
I came across your 2013 paper with Guido Imbens, “Why ask why? Forward causal inference and reverse causal questionson reverse causal questions,” and found it to be extremely useful for a closely-related project my co-author Moritz Schwarz and I have been working on.
We introduce a formal approach to answer “reverse causal questions” by expanding on the idea mentioned in your paper that reverse causal questions involve “searching for new variables.” We place the concept of reverse causal questions into the domain of variable and model selection. Specifically, we focus on detecting and estimating treatment effects when both treatment assignment and timing is unknown. The setting of unknown treatment reflects the problem often faced by policy makers: rather than trying to understand whether a particular intervention caused an outcome to change, they might be concerned with the broader question of what affected the outcome in general, but they might be unsure what treatment interventions took place. For example, rather than asking whether carbon pricing reduced CO2 emissions, a policy maker might be interested in what reduces CO2 emissions in general?
We show that such unknown treatment can be detected as structural breaks in panels by using machine learning methods to remove all but relevant treatment interaction terms that capture heterogeneous treatment effects. We demonstrate the feasibility of this approach by detecting the impact of ETA terrorism on Spanish regional GDP per capita without prior knowledge of its occurrence.
Predis and Schwartz describe their general idea in a paper, “Discovering what mattered: Answering reverse causal questions by detecting unknown treatment assignment and timing as breaks in panel models,” and they published an application of the approach in a paper, “Attributing agnostically detected large reductions in road CO2 emissions to policy mixes.”
It’s so cool to see this sort of work being done, transferring general concepts about causal inference to methods that can be used in real applications.
Maybe someone can explain this at a conceptual level. I like the idea of reversing the logic: we have observed data (e.g., CO2 levels) and search for breaks in the series. When we detect the breaks, we search a large number of variables (perhaps more than the number of observations) for potential candidates for a causal relationship with those detected breaks. It seems to me that this can be an alternative way to look at those maligned regression discontinuity models. Rather than imposing the break point, we first search for break points and then see if they are associated with potential causes.
While I like the reverse logic in this, it seems to me a sort of “residual explanation” approach. In other words, we find a break in some series, and then attribute it to the most likely variables that we have data on. I have a hard time thinking of this as a causal analysis, however. It seems subject to what variables we have data on and choose to explore as potential causes. Absent an experimental setup where we can test particular causal factors, I do prefer this reverse approach to the more traditional attempt to detect whether the causal variables have a significant relationship with a pre-specified break (e.g. point in time or geographical break as we’ve seen in those regression discontinuity examples). But aren’t we relying on machine learning models to identify correlations and then declaring these to be “causal” relationships?
Here’s an interesting paper on finding discontinuities in data, which can be used in regression discontinuity analyses: https://www.jmlr.org/papers/v24/21-0670.html
This sounds right to me. What we have, potentially, is a much-amplified lamppost effect. ML scours for data, but what it finds is what someone chose to measure or enumerate. If this is how it is, there’s another level to consider: the data themselves are a selective record of the phenomena they account. Complex, heterogeneous items are standardized to become data points, which is very powerful, of course, but inevitably leaves some stuff out. In addition to paying attention to what we have variables for, it also helps to pay close attention to measurement and categorization. In the work I’ve done in the past, that was often huge.
This is a topic that’s interested me recently. I’ve had discussions with a hydrology colleague about differences in the ability to perform reverse caution in that field and my work in transportation behavior. I decided to ask Susan Athey, as an economist noted for her work on ML, not expecting an answer. Her response:
“I think the problem in social science is that very few characteristics of people are independent of one another, and similar with markets, everything affects equilibrium outcomes. So we have to work very hard to either design experiments (in some cases impossible) or look for clever natural experiments to find even a few independence assumptions. Sometimes independence assumptions arise because of a temporal sequence, but in social science you would generally know that sequence so no need to discover it. In contrast, it is more plausible that there is a physical system with a lot of temporal structure (A triggers a reaction B, in a software system or a biological system) but the system is so complex that we don’t already know those relationships. So I think causal discovery can be more useful there.”
Having done a bit of work on climate, I see it as partway along the spectrum, a bit of a social problem in that people contribute in various ways and policy responds to heterogeneous human behavior but also a complex physical system.
Jfhawkin,
I’m skeptical about the search for “clever national experiments.” From what I’ve seen, these seem to do more harm than good.
I’m sorry, do you mean national experiments or natural experiments? The above quote was mentioning the latter…
Oh, yes, I meant “natural”; good catch.