George Dickinson points to this article and writes:
“.. analyses .. show that .. more intelligent individuals were less satisfied with their lives during the COVID-19 global pandemic *because* they were more intelligent” (emphasis in original)
Seems to me that a causal assertion such as the above needs a stronger tool than the multiple ordinal regression analysis they use in the paper. What do you think?
My reply: Oh yeah, this is terrible. The first author of this paper is notorious for publishing articles with bad statistics; see here. I’ll say this for the guy, though: he has an absolute talent for getting this bad research published in real journals.
As often seems to be the case, the problem appears to a mixture of: (1) genuine confusion about the distinction between descriptive and causal inference, (2) overconfidence about what can be learned from statistical data, (3) journals like to publish big claims, and (4) big claims get noticed.
P.S. Zad sends along the above picture of someone who is probably making inappropriate causal inference all the time—that’s what the executive function of the brain does for a living!—but that’s ok because adorable. Also I guess it all worked out because evolution.
This might be an interesting application of NLP.
Causal Inflation Index = Causal Claims Score / Causal Methods Score, where:
Causal Claims Score: 1 no use of causal language — 10 among the highest uses of causal language
Causal Methods Score: 1 if only discuss regressions — 10 among the highest uses of causal methods language (potential outcomes, experiment, etc).
One could use it to score articles and journals….
I would not recommend “to calculate” inputs such as certain analytical methods. Causation (causal effects) is about meeting the assumptions.
Agreed, but the methods are there to make those assumptions more credible like e.g. random assignment, no?
This is from Judea Pearl’s paper on “External validity: From Do-Calculus to transportability across populations”
“On the theoretical front, the standard literature on this topic, falling under rubrics such as “external validity” (Campbell and Stanley, 1963, Manski, 2007), “heterogeneity” (Höfler, Gloster and Hoyer, 2010), “quasi-experiments” (Adelman 1991) consists primarily of “threats,” namely, explanations of what may go wrong when we try to transport results from one study to another while ignoring their differences. Rarely do we find an analysis of “licensing assumptions,” namely, formal conditions under which the transport of results across differing environments or populations is licensed from first principles.
The reasons for this asymmetry are several. First, threats are safer to cite than assumptions. He who cites “threats” appears prudent, cautious and thoughtful, whereas he who seeks licensing assumptions risks suspicions of attempting to endorse those assumptions. Second, assumptions are self-destructive in their honesty. The more explicit the assumption, the more criticism it invites, for it tends to trigger a richer space of alternative scenarios in which the assumption may fail. Researchers prefer therefore to declare threats in public and make assumptions in private.”
Perhaps what is needed is a ratio between number of potential threats to validity (unobserved confounders, missing data, selection bias, interference, etc) and number of licensing assumptions where the causal query is identifiable
I googled Journal of Personality and it says it has an impact factor of “5.117” (I love all those decimal places) and is ranked 7 out of 66 in the category, “Psychology, Social.” So I guess they’re doing something right!
From the Kanazawa paper:
“The savanna theory of happiness (Kanazawa & Li, 2018) applies the evolutionary logic to the realm of happiness. Building on observations behind the Savanna Principle, the theory proposes that it is not only the current consequences of any situation that influences current levels of happiness but also its ancestral consequences—what the situation would have meant for our ancestors and their happiness in the ancestral environment.”
This is what I call “two bridges too far.” The Savanna Principle is some really dodgy “just so” science, a bridge too far. Then just take that one as an assumption in the next one, and presto, you are two bridges too far.
Sort of The Next Bridge of the River Why?
Matt:
I agree with you . . . but this was all good enough for Freakonomics, NPR, and, of course, our favorite such venue, the Edge Foundation.
Don’t forget to include the Santa Fe Institute on the dodgy list of just-so science.
Thomas:
From what I know of the Santa Fe Institute, it’s my impression that they do speculative science; I don’t think this is the same as junk science of the sort being discussed here.
Let me put it this way: If you just take the “Savannah Principle” and do some mathematical modeling, maybe fit some of those models to data, then, sure, that’s in Santa Fe Institute territory. But the bit about making strong and ridiculous claims from dead-on-arrival data analyses, that’s full-out NPR/Freakonomics/Ted territory. The Edge Foundation is all of the above plus some sexism, which I guess made the sponsor happy.
Andrew-
The SFI is just as guilty of making ‘strong and ridiculous claims from dead-on-arrival data analyses’ insofar as they explicitly posit S-shaped, logistic growth curves as a universal mathematical form underlying everything. See Geoffrey West’s 2015 book for confirmation, Scale: The Universal Laws of Growth, Innovation, Sustainability, and the Pace of Life in Organisms, Cities, Economies, and Companies. https://www.google.com/books/edition/Scale/Pf0jDwAAQBAJ?hl=en&gbpv=0
This is patently bogus pseudo-science for which the SFI needs to be aggressively called out.
I haven’t read it.
But if a success makes future success more likely, and a failure makes future failure less likely, then you get the “S-curve”:
https://www.tandfonline.com/doi/abs/10.1080/00221309.1930.9918225
Further, averaging individual S-curves with different learning rates can yield artifactual curves of other shapes:
https://www.pnas.org/doi/full/10.1073/pnas.0404965101
So it wouldn’t surprise me if this “law” is more general than currently realized.
Anoneuoid-
Does this example constitute a universal law generalizable to ‘Growth, Innovation, Sustainability, the Pace of Life in Organisms, Cities, Economies, and Companies’?
I don’t think so.
I don’t know, but the fact a sigmoidal curve drops out of such a basic model of reward/punishment makes me think it is possibly a very general rule.
Also that averaging is so popular, and this can hide the sigmoidal curve, makes me think it could be commonly missed when it is there.
But, I am just repeating myself and I didn’t read the book. Just sharing some interesting info.
The data they use, as described in section 4.1, in their words is “population data, not sample data” because the original survey from the 1950’s include “all” births in Great Britain for one specific week. However, there’s very clear attrition for all subsequent surveys after the second. Even the first survey, going by the FAQ section on the NCDS website (https://ncds.info/faqs/#why-am-i-unique) was opt-in. Which means unless every single mother consented, which I would find hard to believe, it’s not really census data.
This is a lot more nitpicky than the complaint about reckless causal language, but it’s telling that it literally only took me a handful of seconds glancing at the paper to find another exaggerated claim. It really bugs me too, because it’s not like there’s anything wrong with the NCDS data that it warrants being misrepresented like that. The data is fine, you don’t need to lie about it! I just don’t get it.
Djad:
You say you “just don’t get it.” I think the answer is in four parts:
1. The reward is not to get things right, to get publication, publicity, tenure, sweet Edge Foundation gigs, etc.
2. To follow up on point 1, why is “to get things right” not a goal? Dude’s a scientist, right? Ultimately he wants to learn about reality. The issue, I fear, is that he thinks he already knows the answer, so that the point of his research is not to learn about reality but rather to prove something he already knows.
3. Given 1 and 2 above, an easy answer to your question is “incentives.” Given that this author and others like him can publish papers without solid data analysis, they have no incentive to do the work.
4. But it’s more than that. I think that it’s often the case that doing bad statistics can be an absolute benefit to getting your work published and publicized. Bad statistics gives you more degrees of freedom (“forking paths”) and more opportunity for exaggerated claims. Edge Foundation, here we come!
There aughta be a law: No casual language allowed unless you have longitudinal data.
It’s kind of simple. Cross-sectional data won’t cut it.
Ha. Now looking at the article I see they say they analyzed longitudinal data.
But did they really, to test the causality they are speaking to? I say no – because they didn’t have data on changes over time in intelligence or “comprehension of severity of problems.”
What’s the response to Angus Deaton and his points about observational vs RCT data?
“AD: I think one of the big issues is that these RCTs are typically small and localized. And a lot of the observational studies use nationally representative data sets.
TO: That’s what I’m trying to home in on: the difference between a critique of a method and a critique of a sample.
AD: Maybe I’ll come at that indirectly. If you go back 50 or 60 years when economists started playing with regression analysis, they thought they had a magic tool that would reveal just about everything. They would run multi-variable regressions on all sorts of things and interpret that within, in a way completely unjustified by today’s standards, a causal framework. Then over the years economists and other people learned that there were all sorts of problems with that. If you go to an econometrics course now, they’re not teaching the magic regression machine. It’s more like the regression diseases and what’s wrong with regression. I think economists, especially development economists, are sort of like economists in the 50’s with regressions. They have a magic tool but they don’t yet have much of an idea of the problems with that magic tool. And there are a lot of them. I think it’s just like any other method of estimation, it has its advantages and disadvantages. I think RCTs rarely meet the hype. People turned to RCTs because they got tired of all the arguments over observational studies about exogeneity and instruments and sample selectivity and all the rest of it. But all of those problems come back in somewhat different forms in RCTs. So I don’t see a difference in terms of quality of evidence or usefulness. There are bad studies of all sorts.”
Full interview here…https://medium.com/@timothyogden/experimental-conversations-angus-deaton-b2f768dffd57
Thomas:
I pretty much agree with Deaton on this one. See my discussion here from a few years ago. Also see this discussion.
Well said.
“So I don’t see a difference in terms of quality of evidence or usefulness. There are bad studies of all sorts.”
Applause! :)
The question is not about whether you have natural x-section or long section data. It’s about your understanding of the strengths and weakness of the data and the processes involved.
RE: “I think one of the big issues is that these RCTs are typically small and localized. And a lot of the observational studies use nationally representative data sets.”
A lot of big observational studies use datasets of national scope which are assumed to be representative. Or which are assumed to act as though they are representative once they’ve passed through some sort weighting and adjustment procedure.
Does the cat need causal inference? I don’t think so. Prediction quality alone will work just fine for it. “That’s what the executive function of the brain does for a living!” Does it? My intuition is that this imposes a human concept inappropriately on the cat, but then I’m not an expert.
Christian:
I disagree. The cat needs to make decisions all the time. When to crouch, when to pounce, etc.
But you don’t need causality to make a decision, or do you?
All organisms that survive do so by transforming sensations (signals) from the environment (and memory) into representations of the environment to decide on how to act. Can’t be otherwise as (Christian often reminds us) there is no direct access to reality for any organism.
If those representations are not causal enough (too wrong) the organism won’t survive. But they need not be conscious in any way.
I plan to do a blog post on this in the future.
Christian:
You wrote, “Prediction quality alone will work just fine for it.” Causal inference is just prediction conditional on potential decision options. If you’re making predictions about what will happen under various possible decisions, that’s causal inference.
My view on this isn’t really elaborated. I’d have thought the term “causal inference” implies a certain indirectness; a representation of expected relations that goes beyond directly doing something in order to satisfy a need. But I see how it may make sense to not differentiate too much between these. A blog post on this would be very welcome.
To me the key thing is a desire/need to anticipate expected results from taking possible actions.
Just to say that I love the fact that someone has already posted this blog page on PubPeer against the Kanazawa article.