Awhile ago, Kevin Lewis pointed me to this article that was featured in the Wall Street Journal. Lewis’s reaction was, “I’m not sure how robust this is with just some generic survey controls. I’d like to see more of an exogenous assignment.” I replied, “Nothing wrong with sharing such observational patterns. They’re interesting. I don’t believe any of the causal claims, but that’s ok, description is fine,” to which Lewis responded, “Sure, but the authors are definitely selling the causal claim.” I replied, “Whoever wrote that looks like they had the ability to get good grades in college. That’s about it.”
At this point, Alex Tabarrok, who’d been cc-ed on all this, jumped in to say, quite reasonably, “Andrew, you are skeptical of pretty much all causal claims. But wait, causality rules the world around us, right? Plenty have to be true.”
I replied to Alex as follows:
There are lots of causal claims that I believe! For this one, there are two things going on. First, do I think the claim is true? Maybe, maybe not, I have no idea. I certainly wouldn’t stake my reputation on a statement that the claim is false. Second, how relevant do I think this sort of data and analysis are to this claim? My answer: a bit relevant but not very. When I think about the causal claims that I believe, my belief is usually not coming from some observational study.
Regarding, “Plenty have to be true.” Yup, and that includes plenty of statements that are the opposite of what’s claimed to be true. For example, a few years ago a researcher preregistered a claim that exposure to poor people would cause middle-class people to have more positive views regarding economic redistribution policies. The researcher then did a study, found the opposite result (not statistically significant, but whatever), and then published the results and claimed that exposure to poor people would reduce middle-class people’s support for redistribution. So what do I believe? I believe that for most people, an encounter (staged or otherwise) with a person on the street would have essentially no effects on their policy views. For some people in some settings, though, the encounter could have an effect. Sometimes it could be positive, sometimes negative. In a large enough study it would be possible to find an average effect. The point is that plenty of things have to be true, but estimating average causal effects won’t necessarily find any of these things. And this does not even get into the difficulty with the study linked by Kevin, where the data are observational.
Or, for another example, sure, I believe that early childhood intervention can be effective in some cases. That doesn’t give me any obligation to believe the strong claims that have been made on its behalf using flawed data analysis.
To put it another way: the authors of all these studies should feel free to publish their claims. I just think lots of these studies are pretty random. Randomness can be helpful. Supposedly Philip K. Dick used randomization (the I Ching) to write some of this books. In this case, the randomization was a way to jog his imagination. Similarly, it could be that random social science studies are useful in that they give people an excuse to think about real problems, even if the studies themselves are not telling us what the researchers claim.
Finally, I think there’s a problem in social science that researchers are pressured to make strong causal claims that are not supported by their data. It’s a selection bias. Researchers who just make descriptive claims are less likely to get published in top journals, get newspaper op-eds, etc. This is just some causal speculation of my own: if the authors of this recent study had been more clear (to themselves and to others) that their conclusions are descriptive, not causal, none of us would’ve heard about the study in the first place.
Summary
There’s a division of labor in metascience as well as in science. I lean toward skepticism, to the extent that there must be cases where I don’t get around to thinking seriously about new ideas or results that are actually important. Alex leans toward openness, to the extent that there must be cases where he goes through the effort of working out the implications of results that aren’t real. It’s probably a good thing that the science media includes both of us. We play different roles in the system of communication.
Is there something mistaken with my belief that having longitudinal data should be a pretty hard prerequisite for making claims about causality?
As for exactly what meets a bar for longitudinal data…I suppose that’s somewhat arguable.
Joshua:
In theory you don’t need longitudinal data, you just need a treatment/control comparison and a clean design. Or some strong theory. I can make causal statements about all sorts of future events based on my ability to generalize from past data and strong theory. For example, I have a good causal model of what will happen if I swing a hammer into my bathroom mirror. No longitudinal data required!
I’m not sure that feels analogous to me.
Seems to me that a snapshot (cross sectional data) of a broken mirror with a hammer next to it, and also a wrench and a rock and a barbell is what would be analogous to the divorce —>> lower social integration causality.
I would need to know that (1) the mirror wasn’t broken prior to the placement of the other items and that (2) the hammer had been moved (while the other items hadn’t been moved or were moved later, etc.). IOW, I to really have insight into causality I would need something longitudinal more than cross-sectional.
Going back to the causality in the paper – longitudinal data helps a great deal for understanding potential confounds.
I think the whole “this paper proves causality” thing is a shame. Many papers make such claims. But I cannot think of convincing reasons why any statistical claim should prove causality. It may provide some evidence that pushes the credibility of a theoretical model towards 1. But it can only converge towards 1, never reach it. For this reason, I consider statements like “this paper proves causality” to be misleading. Either they are hype, or the author’s philosophy is so poorly developed that they actually believe that a study can provide conclusive evidence.
I don’t think I’ve ever read a paper that claims to have “proved” causality; typically a journalist or editor or PR team might use that kind of language to get some clicks. I do see paper often use terms like “demonstrates” which I find less troublesome.
From the paper “Banning the purchase of sex increases cases of rape: evidence from Sweden”, which was discussed yesterday on this blog:
“An analysis of the results of this paper as a whole offers important conclusions about
the effects of the Nordic model. The introduction of a ban on the purchase of sex has
the unwanted effect of changing the behavior of prostitutes’ customers. Specifically,
it shifts their demand outside and towards other crimes, such as rape — in particular,
completed rapes. This mechanism is also aligned with considering prostitution as paid
rape, if this is the case, it is no surprise that restricting the purchase of sex raises
rapes and specifically, completed rapes. To this extent, it might be that customers rape
prostitutes now that prostitution is more expensive. A potential channel in this case
might be even not paying for sex.”
To say “…and thus it is!” is causal language. I agree, usually authors do not write “We have proved a causal relationship”. But causal language like the quotation above is not uncommon. *That* is the thing that I find problematic – especially when the evidence base is weak.
“I don’t think I’ve ever read a paper that claims to have ‘proved’ causality”
Because so many scientists were involved in the project, there is no single paper to point to, but the Oklo Natural Nuclear Reactor is an example where causation was proven. (From an epistemological standpoint, the lack of a single paper is a technicality.) Initially there was a conference and then an article in Scientific American in 1976 (see: “A Natural Fission Reactor,” by George A. Cowan, July 1976)
Here is a brief summary from a 2005 Scientific American article:
“Physicists confirmed the basic idea that natural fission reactions were responsible for the depletion in uranium
235 at Oklo quite soon after the anomalous uranium was discovered. Indisputable proof came from an examination of the new, lighter elements created when a heavy nucleus is broken in two. The abundance of these fission products
proved so high that no other conclusion could be drawn. A nuclear chain reaction very much like the one that Enrico
Fermi and his colleagues famously demonstrated in 1942 had certainly taken place, all on its own and some two billion years before.”
I find this example to be quite enlightening because it shows fundamental differences in how causation can or cannot be proven in different fields. In this case, it would be particularly pointless to argue that they didn’t REALLY prove it, which is why I like it as an example.
My reply to Alex would be that excluding the trivial case of a system with one input and one output, the number of possible causal claims that are false will always exceed those that are true and the false/true proportion will increase exponentially with increased system complexity.
Yes, exactly this!
Yeah, this seems to have an easy reply: “yes, I believe there are plenty of causalities in the world – in fact, I believe there are *so* many that any specific causal claim is usually wrong, and the simpler the claim, the more likely to be wrong (particularly of the form “X causes Y after controlling for a few covariates”).”
The linked article at the start of this post is part of a large series of articles which seem to be underwritten by the Templeton Foundation. At least one of the authors has been very open about his conservative views on religion and marriage. Perhaps unsurprisingly, given the source of funding, these studies regularly find that society would be better off, in most ways, if there were more marriages between men and women that resulted in more children and if religion played a foundational role in people’s lives. The author is clearly brilliant, but this initiative is a horrifying example of the insidious encroachment of the religious right into higher education in the U.S. What better way for right wing organizations to shape American society than to hire the most brilliant minds to generate (superficially) scientific findings to support their agenda?
Es:
Good catch! Let me just add to this that there is similar advocacy from the left pushing shaky science, for example all that stuff with the Implicit Association Test. And then there’s all the unreplicable Nudge stuff, which has a center-left vibe (the government will nudge you to do what’s best for you) and a center-right vibe (nudge as a form or corporate marketing).
Yes, there’s no shortage of ideologically-driven research these days. If a group has deep enough pockets, it will be able to find someone to do a “study” that supports the ideology. This trend is incredibly pernicious because the glaring intellectual conflicts of interest won’t be recognized by the average reader, who also won’t usually follow the money trail. But the “results” will surely be used, in political contexts (e.g., public hearings), to lend a veneer of objectivity/credibility to ideologically-driven policy proposals.
The fact that even the most high profile, ostensibly methodologically rigorous researchers will succumb to the lure of a vast sum of money, no matter its source, speaks very poorly of the state of higher education in the US. These types of articles do an incredible disservice to researchers trying to identify practical uses for more modern observational research methods.
RCTs of therapeutic interventions in medicine are certainly not foolproof either. But at least gross intellectual/ideologic biases are largely neutralized by the sheer number of people involved in multinational clinical trials, involvement of ethics/ monitoring boards, and guaranteed harsh subsequent scrutiny of results by regulatory agencies, practitioners, and funders (versus politicians unschooled in detection of slimy research practices). And let’s not fool ourselves. A discriminatory public policy underpinned by compromised observational social science research can do just as much harm as approval of an unsafe drug.
Let’s think about it. What could the possible motivation be for research into the effects of religion and marriage ? Is any particular result going to encourage people to get or stay married or start believing in God? Obviously, the answer is “no.” So that leaves politics as the only plausible underlying motivation. I’m not sure if these types of trends are strictly a US phenomenon (given the disproportionate religiosity of the US population), but the corrupt influence of money in US political discourse is mortifying.
Anon:
I don’t agree with the last paragraph of your comment, where you write that politics is “the only plausible underlying motivation” for research into the effects of religion and marriage. Religion and marriage are an important part of many people’s lives; of course it makes sense for social scientists to study their effects.
My response to Alex: if you have 25 variables affecting the outcome of a given event, how do you effectively isolate and test each variable? The answer is: you can’t.
The simple social science problems remain:
1) dozens more variables
2) very low precision in quantifying variables
3) lack of controls that allow variables to be isolated and tested.
Pretty straightforward.
But more importantly: Alex, no, there does not have to be an indivdual cause of every social event. Physical reasoning doesn’t necsessarily apply to the social world.
Comments above have fully debunked the topic title, but just to pile on: at the basic, quantum level, there is a whole lot of randomness involved in events. Afterall (I say), random evolution created us, and our causality addiction.
This is an area of active debate in the social sciences. See Besbris and Khan 2017 ‘More description. Less theory.’ for one side of the polemic.
Theory in the social sciences inevitably makes causal claims, so it’s rather hard to ‘do’ social science while both engaging with theory and respecting the limits of your data, measuring process, and design.
Thanks for flagging this article. That was my comment above (misfiled under “Anonymous”), questioning the motivation of researchers who try to identify, using observational data, any *causal* “effects” of marriage or religion.
I wasn’t denying that marriage and religion are important to a lot of people, nor that *descriptive* studies of these topics might be valuable in the right context. Rather, I was trying to point out that application of observational *causal inference* methods to such complex topics feels utterly inappropriate. Anyone who understands the limitations of these methods would recognize the inherent unreliability of any “results” derived from their application to such complex questions. In turn, since results from such studies are unlikely to be believed by anyone with deep methodologic understanding, the only possible *purpose* of such articles, arguably, is to influence people who are powerful enough to set social policy, but who lack a deep understanding of research method limitations (i.e., politicians).