Kevin Lewis points to a research paper and writes, “I’m not sure how robust this is with just some generic survey controls. I’d like to see more of an exogenous assignment.”
I replied: Nothing wrong with sharing such observational patterns. They’re interesting. I don’t believe any of the causal claims, but that’s ok, description is fine.
I won’t get into the details of this particular paper because that’s not the point of the current post.
What I want to talk about is an exchange I had with Alex Tabarrok, who was cc-ed on the discussion about that observational study. In response to my skepticism, Alex wrote:
Andrew, you are skeptical of pretty much all causal claims. But wait, causality rules the world around us, right? Plenty have to be true.
I replied: There are lots of causal claims that I believe! For this one, there are two things going on. First, do I think the claim is true? Maybe, maybe not, I have no idea. I certainly wouldn’t stake my reputation on a statement that the claim is false. Second, how relevant do I think this sort of data and analysis are to this claim? My answer: a bit relevant but not very. When I think about the causal claims that I believe, my belief is usually not coming from some observational study.
Regarding, “Plenty have to be true.” Yup, and that includes plenty of statements that are the opposite of what’s claimed to be true. For example, a few years ago a researcher preregistered a claim that exposure to poor people would cause middle-class people to have more positive views regarding economic redistribution policies. The researcher then did a study and found the opposite result (not statistically significant, but whatever). She then published the results and claimed that exposure to poor people would reduce middle-class people’s support for redistribution. So what do I believe? I believe that for most people, an encounter (staged or otherwise) with a person on the street would have essentially no effects on their policy views. For some people in some settings, though, the encounter could have an effect. Sometimes it could be positive, sometimes negative. In a large enough study it would be possible to find an average effect. The point is that plenty of things have to be true, but estimating average causal effects won’t necessarily find any of these things. And this does not even get into the difficulty with the recent study which is that the data are observational.
Or, for another example, sure, I believe that early childhood intervention can be effective in some cases. That doesn’t give me any obligation to believe the strong claims that have been made on its behalf using flawed data analysis.
To put it another way: the authors of all these studies should feel free to publish their claims. I just think lots of these studies are pretty random. Randomness can be helpful. Supposedly Philip K. Dick used randomization (the I Ching) to write some of this books. In this case, the randomization was a way to jog his imagination. Similarly, it could be that random social science studies are useful in that they give people an excuse to think about real problems, even if the studies themselves are not telling us what the researchers claim.
Finally, I think there’s a problem in social science that researchers are pressured to make strong causal claims that are not supported by their data. It’s a selection bias. Researchers who just make descriptive claims are less likely to get published in top journals, get newspaper op-eds, etc. This is just some causal speculation of my own: if the authors of this study had been more clear that their conclusions are descriptive, not causal, none of us would’ve heard about the study in the first place.
Again, this is a general phenomenon we’ve talked about many times. I’m not mentioning the particular study that motivated this particular discussion because I don’t want to get sucked up into the details.
“But wait, causality rules the world around us, right? Plenty have to be true.”
Perhaps this is taken out of context (we don’t see anything else about the exchange), but if not I don’t understand the statement at all. Of course, causality “rules” the world in that any action has consequences. Further, plenty of causal statements must be true just as many must be false (more importantly, as you point out, most causal statements are partially true and partially false). But the implication is that somehow this justifies making causal claims from observational data. There seems to be some failure in logic here. Even if more causal claims were true than false (which I doubt), would that justify basing decisions on a randomly chosen causal claim?
You’re right about the failure in logic, i.e. a study can be poor evidence for a causal claim even if the causal claim is true. I took Alex to be saying something like “If you think there’s some underlying causal connection, you can’t be skeptical of every side of the issue”
Everything is collectively caused by every event that happened before (with its past lightcone).
However, it is not possible to study the causality of something like “supporting redistribution” by looking at the average. It is an individual person who supports or doesn’t, no average person actually exists to support or not.
If you want to talk about causality at a population level, then you can but blurring the lines leads to confusion.
Like say at the individual level s9me dependent variable, y, is a logistic function of t. Then you collect data for 100 individuals. If you look at the average you are going to be mislead into thinking it is something else, like a linear relationship:
logist = function(t, a, r, m){ a/(1 + exp(-r*(t-m))) }
t = 0:20
n = 100
res = matrix (nrow = length(t), ncol = n)
for(i in 1:n){
res[,i] = logist(t, runif(1,0,1), runif(1, 0, 2), runif(1, 0, 20))
}
plot(t, rowMeans(res), type = “l”, lwd = 10, xlim = range(t), ylim = c(0, 1), ylab = “y”)
for(i in 1:n){
lines(t, res[, i])
}
And indeed, this is what happened for ~100 years in studying all sorts of learning curves. The learning being studied happens at the individual level, *not* group.
“…a researcher preregistered a claim that exposure to poor people would cause middle-class people to have more positive views regarding economic redistribution policies. ”
This study is typical of many that are discussed here: it had no hope of finding anything because the premise is false. An adult’s views regarding redistribution and many other things are surely shaped by many recurring experiences and adding one more typical experience has almost zero chance of causing a change. People might use n=1 for their views on, say, CEOs of successful rocket launching companies after having a conversation with such a CEO. But they don’t use n=1 for their views on aspects of life that they’ve encountered daily throughout their lives.
Sadly the researcher probably didn’t even get that there *was* some premise underlying the experiment, so they couldn’t know if it was true or false!
But to the subject at hand: if an experiment is based on false premise(s) or requires assumptions that are false to draw its conclusions, then it has no relevance to causal relationships or anything else. Thousands of experiments like this can be performed and will never find the cause of anything. So I agree with Andrew – the fact that an experiment was performed has nothing to do with whether it uncovers cause and effect. We can’t say “well, there 378,998 experiments have been done, some of them have to be right”. No, they don’t. If they’re all use inaccurate premises and/or assumptions, they could all be wrong, and in fact they could all be *worse* than wrong: they could all be meaningless.
However, in one sense I disagree with Andrew: I don’t see why data generated on false premises is interesting. To me the observations are not useful nor is any analysis based on them useful. The “experiment” is not science.
I find your initial reaction interesting – you read the quoted statement as indicating a study where one exposure to poor people is introduced. I read it as a study where some people are routinely exposed to poor people (for example, living in mixed socioeconomic neighborhoods) compared with people in more homogeneous settings. Your reading gave you an opportunity to express some of your favorite views (I believe this falls under the rubric of “mood affiliation”). Of course, I don’t know which of our readings is closer to the truth, but it is interesting how much we bring our world views into potentially unrelated issues.
Dale:
That is an assumption – possibly an accurate one – on my part. Good point.
But it derives from the fact that it’s typical of practices used in papers that are discussed here – 30 minute videos that are supposed to change people’s life-time income etc etc. Thus even though it aligns with my bias I rate it as reasonable assumption that doesn’t emerge from subconscious self-interest seeking. Expressing that a bit differently, the assumption aligns with an *expectation* – not a bias – for the way research is commonly performed in the social sciences. I expect this to be the case because it’s so frequently true, not because I have some secret ego-confirming self-interest in the outcome (although I might have that too!;).
I poked around a little trying to find the paper without success. Just the same, while I may be wrong in the assumption of a single interaction, I expect I’m in the ballpark in that the paper depends on very few interactions. I voice that expectation because, in addition to research practices in common use, just the constraints on the way experiments are commonly performed almost necessitates a short period to do the research. Again, I could be wrong – this could be an exhaustive study of a long term effort to realign peoples’ thinking about poverty. Probably not, but possibly.
I’ll postulate that our difference in views may come from my lesser depth in social science literature. I mostly read about what becomes a political hot potato, I don’t read the SS literature for its own sake and thus probably miss some higher quality less controversial work that you may be familiar with. But I could still be right! :)
Chipmunk –
> Expressing that a bit differently, the assumption aligns with an *expectation* – not a bias –…
How do you know this? How do you control for your biases in assessing whether it’s a product of your bias (or a mere un-biased expectation)? Your logic seems to me to be that because it’s an expectation it’s not a bias, as if someone else might not have a different expectation (because of their bias).
Chipmunk -.
I find your level of confidence in your views to be quite interesting.
> But they don’t use n=1 for their views on aspects of life that they’ve encountered daily throughout their lives.
How do you so adeptly get into people’s heads to know what they do and don’t think and why?
Not to say that you aren’t touching in an important issue here (overconfidence that research can have regarding causality). I think you are. But I also think it’s unfortunate that your argument is burried beneath so much cultural cognition.
Anyway, I don’t know how well this research holds up:
https://www.science.org/doi/10.1126/science.aad9713.
But I think there’s always bound to be evidence that if you express views so categorically, you’re going to be wrong.
(inrember that particular study was interesting because it came on the heels of a fraudulent study that found a similar effect. I would imagine that Andrew blogged about it.
Related:
https://fivethirtyeight.com/features/how-two-grad-students-uncovered-michael-lacour-fraud-and-a-way-to-change-opinions-on-transgender-rights/
>For example, a few years ago a researcher preregistered a claim that exposure to poor people would cause middle-class people to have more positive views regarding economic redistribution policies….
This sentence also highlights the confusing (at least to me) lack of detail in ‘causal claims’. Is “exposure to poor people” really a definable cause? I doubt it. Thinking of a simpler example, say you have a drug that lowers blood pressure via vasodilation. There is a causal chain where the drug is absorbed, then flows to certain receptors, has a certain binding, etc etc (I’m not a biologist) in a lengthy chain of ‘causal’ events that eventually lead to vasodilation and thus reduced blood pressure, in some definable deterministic math function (if you had enough knowledge). The causal function is the same among people (not really, but mostly close), but the variation occurs in all the variation of the components of that function (blood volume, number receptors, etc) along the causal chain. You can get an ‘average effect’ with a large enough random sample. But “exposure to poor people” is like doing the blood pressure experiment with many kinds of drugs on participants that consist of many different kinds of animals, all mixed together. What part of “exposure to poor people” is the cause? Does it matter if they look poor? Something they said? Every ‘poor’ person is so different, this doesn’t even seem like a ‘treatment’ that can even be sufficiently defined. Then you get into all the causal paths where all the various manifestations of ‘poor people’ are ‘exposed’ in all the various manner of ‘exposures’ on all of the various ‘participants’ and their past exposures. Yeah, I guess you could get a big enough sample of ‘poor’ people, defined as making less than $X, and get a big enough sample of middle-class people making between $X and $Y, and get an “average effect”, but it wouldn’t seem clear that this number would have any meaning at all in terms of causality… But I do have a suspicion that the researcher has an idea of what they mean! For example, maybe once they have their ‘finding’ then they write about how exposure to poor people may induce feelings of empathy that cause changed views on redistribution of wealth. But even a study of colossal sample size of poor and middle-class people with a treatment that was ‘exposure’ would never measure that.
No one is obligated to believe any particular causal claim made in a research paper, unless the evidence presented is comprehensive and conclusive beyond reasonable doubt. But this is true of any claim, not just causal ones. What makes causal claims special is that (on the potential outcomes view) they pertain to unobserved counterfactuals. This means that it is not enough to trust that the paper reports what was observed accurately. The reader must also credit the background assumptions supporting the inferences made about these unobserved counterfactuals.
This too is not especially different from other kinds of statistical claims, in which inferences are made from the specific (observed) to the general (unobserved) on the basis of background assumptions. Causal background assumptions are trickier, however, since they are frequently not evaluable using the observed data, whereas statistical assumptions supporting population inferences often are evaluable in this way.
A reader is then always free to dispute a causal claim in a paper on the basis that the paper does not introduce any new data to change their mind about their background assumptions. When Andrew critiques papers like this, however, it isn’t always so clear which background assumptions he is disputing, if any. It seems like his objections are more often statistical in nature.
For example, he often observes that a treatment effect may be so heterogeneous that even a credible and precise estimate of an average treatment effect is meaningless (in the same way that any average is a poor summary of a distribution when the variance of that distribution is sufficiently large). Or he might complain about how researchers often think “clean identification” plus statistical significance is sufficient for good research, when these studies may have very low power and so produce unreliable claims.
These points are well-taken, but they don’t really clarify his differential attitude towards observational studies, where as far as I can tell the most important distinguishing feature is the absence of randomization. It’s one thing to say these issues arise in any study making any kind of inferential claim, but it’s not clear what is special about observational studies other than that they lack randomization and so rely more on trust in causal background assumptions to get off the ground.
The biggest problem with causal inference from observational studies is usually that there is virtually nothing in the way of detailed mechanistic/generative models of the process. If you want to convince me that “exposure to poor people” affects “attitudes towards redistribution” then just start telling me how you think that happens and what measurable outcomes you’d predict and what unobserved parameters you’d need to predict the outcomes.
For example perhaps that “exposure” is exclusively seeing them begging for change at freeway off ramps… That’d probably do something different than say seeing them line up outside day-laborer work camps, and again different than if they are janitors at office buildings, or drug addicts in tents under bridges or whatever.
The sad thing is people don’t even realize they’re being stupid when they conceptualize “exposure” as just one thing. The lack of imagination amongst researchers is really shocking. It’s one of the reasons I like to utilize lots of concrete examples… Try to jog people out of their limited world views.
I think the phenomenon you’re describing would be understood as treatment effect heterogeneity, meaning that the treatment of observing a poor person has effects that depend on other factors—in your example, the context of the observation. In the presence of such heterogeneity, one approach is to try to estimate an average treatment effect of some kind, though it is often unclear what kind of population distribution of contexts a data set represents. If you think the effect varies dramatically with context, then the average effect may not be very interesting, but you can still explore average effects within subpopulations. People sometimes use causal trees and forests for this kind of exercise, but there’s simpler ways to do this as well. Don’t think you need to write down a comprehensive structural model of the system of interest to make progress on these problems, though that is certainly one reasonable approach.
The question for me is how does randomization solve any of this? If you’re trying to write down a structural model, I suppose randomization deletes the arrows into the treatment assignment variable, but effects could still be highly heterogeneous. Even if the RCT administers identical treatment, the circumstances vary from unit to unit, so the effect can vary, possibly a lot. So I don’t know what’s special about observational studies here.
Again, I would say what’s special about them is that they require background assumptions for causal inference that cannot be evaluated with the data being analyzed, which can be a real shortcoming. But otherwise I’m not sure why they’re deserving of special scrutiny here. Also worth noting that RCTs have issues of their own that observational studies don’t, or at least not in the same way (e.g., external validity).
Think about it this way. When you a model a coinflip as a sample from a Bernoulli distribution, where does causality enter the picture?
You’re right that you can model things as heterogeneous, but this just pushes the problem back… Which factors affect the heterogeneity? And how many of each type of situation are there “out there”.
When you at least describe the factors that cause the heterogeneity you have a chance to estimate the individual subgroups. Like for instance perhaps wealthy people whose only exposure to poor people is panhandling and homeless encampments…. You’ve now specified a pretty identifiable group and you can imagine quantifying the size of that group and the group variation in effect.
Then perhaps you have lower middle class service industry workers working more than one job and exposed to poor people as family and friends… Brothers, cousins, aunts, whatever…
Again you have a chance of identifying the size of the group and the mechanisms involved…
Now how about upwardly mobile children of poor families who have succeeded in school and professional degree programs… Etc
Just saying “heterogeneity” without specifying the groups and their relative size and the important contextual mechanisms is going to get you zilch IMHO.