Mark Goldstein points us to this post by Alex Dimakis, who writes:
A paper was recently published in Science on highest level of human performance across athletics, science, math and music. I think the paper makes some classical statistics mistakes that still fool many smart people. The paper “Recent discoveries on the acquisition of the highest levels of human performance” by Gullich et al. claims: “In summary, when comparing performers across the highest levels of achievement, the evidence suggests that eventual peak performance is negatively associated with early performance.”
The paper makes two mistakes. Base-rate fallacy and . . . Berkson’s paradox . . .
The study says simply that the very top at young age are not identical with the very top adults. (As one would expect, since there are *many many more non-elite young candidates*). Still, elite young performers are 40 times more likely to be in the top adults compare to general population. This is acknowledged in the paper but in page 6-7, a bit buried in the technical analysis and not sufficiently discussed in abstract or conclusions. . . .
The paper claims “Across the highest adult performance levels, peak performance is negatively correlated with early performance.” This is a classic example of Berkson’s paradox. Here is a simplified example to understand this: Assume that to be a successful actor you have to be either extremely good looking or extremely talented. Assume also that talent and looks are independent in the population. However, among sucessful actors you will observe a negative correlation between looks and talent. This doesn’t meant anything beyond the selection process and should not be extrapolated. My favorite example-joke of this is that basketball points scored is negatively associated with height among NBA players. (because to be an NBA player you have to be very tall OR be very good at scoring). From this, I extrapolated that since I’m 5’7, I will be scoring 80+ points per NBA game. . . .
Here’s paper in question, “Recent discoveries on the acquisition of the highest levels of human performance.”
Yeah, this sort of thing comes up all the time! For example, some celebrity academics a couple years ago wrote a book that included the false statement, “while correlation does not imply causation, causation does imply correlation.” Even more amusingly, they prefaced this by “We must, however, remember that”. I guess we must remember a lot of false things! Economist Rachael Meager gave a quick example showing why they were wrong; See details here.
This new example also looks a lot like the well-known regression-to-the-mean fallacy (for more on that, I recommend Section 6.5 of our book, Regression and Other Stories, which includes some simulation code to demonstrate the problem). Of course, just because lots of people know about a fallacy, that doesn’t stop people from making the error in new settings. That’s why it’s a fallacy!
P.S. An anonymous commenter points out that Dimakis (and, by extension, Goldstein and me) are being unfair to this paper. The descriptive results are what they are. I remain skeptical of the paper’s claim that “similar developmental pattern across different domains suggests widespread, and possibly universal, principles underlying the acquisition of the highest levels of achievement,” as I do suspect that much of what they have seen arises from the usual statistical selection artifacts. So maybe it’s ok to caution about the interpretation of these numbers. But now I’m thinking it wasn’t fair of us to slam the paper for presenting some interesting data findings.
I’m not saying that the criticism of the statement that “causation implies correlation” is wrong, but I’ve always hated that specific example. Opening the throttle doesn’t directly cause speed. It induces the engine to produce more power. To maintain a constant speed as the car goes up the hill, work needs to be done, so the engine has to produce more power during that time. So depressing the “accelerator” and opening the throttle is causing exactly what it is supposed to cause and maintaining the constant speed is evidence of this. This seems to me to be evidence that a failure to observe a correlation where you expected one is actually just evidence that you haven’t properly identified what the action is causing.
“Cause” is one of those words that everyone seems to know but nobody can define in a satisfying way. Your comment makes me think two things:
1. In the example, the accelerator absolutely does cause speed! Because in the counterfactual world where the accelerator wasn’t pressed as much, speed would have been lower.
2. Nothing directly causes anything. Pressing the accelerator doesn’t cause the engine to produce more power, it causes more air to enter the intake manifold. Or actually, it causes the throttle body to open wider, which causes more air to enter the intake manifold. Or actually, it causes a cable to be pulled that causes the throttle body to be open wider which causes… We have to allow “cause” to mean the entire downstream cascade, or it can’t mean anything larger than a planck.
I don’t think we can allow cause to encompass everything downstream of an action. At least while keeping it meaningful. In this case it “causes” an increase in speed only subject to specific physical situations. In this case it seems like it’s more a case of gravity “causing” the speed to not increase. Speed doesn’t seem to me to inherently causally well related to the act of depressing the pedal. It’s a misreading of how causality could even work in this situation.
On the other points, I think it’s equally meaningless to say nothing directly causes anything. Anyone who believes a strong form of that statement must live a fascinating and terrifying life. But, yes, I do agree that it’s really hard to define causality, and most of my point is that I hold that in this case, making the causal connection between the pedal and the speed of the car in the presence of the hill is incorrect. What does that make the hill, a collider or something, i dont recall. But assuming that causality can flow from the pedal to the speed in this case is a modeling problem.
Causality is not something we can observe, its an assumption made when deriving a model. We can only observe correlations.
I still insist that the proper way to think about cause is counterfactuals. It’s like saying a medicine didn’t have an effect on your health if you still got sick. If the degree of sickness was less than it would have been if you hadn’t taken it, that’s still effective medicine! Id est, don’t compare to the previous state, compare to the other possible world where you didn’t intervene in this way. DAGwise: pedal -> combustion -> speed speed path means that the pedal really does cause speed.
It’s fair to say that “entire downstream” is excessive in a colloquial sense. If a friend stopped me to say hi in the hallway and then later that day I crash my car because I got distracted by, I dunno, some cool animal I saw in the woods, it’s true that nobody would say my friend caused me to crash. But there is still a causal path feeding from my friend’s conversation to the exact timing of my driving to the crash if that animal wouldn’t have been there 30 seconds earlier. It’s not a particularly interesting causal path, but it’s there.
As far as direct cause, maybe I only believe in a weak form of the statement, but the point is that every path in a DAG can be decomposed and decomposed again until you’re talking about quantum something or other, and even then we’re only stopping because we’ve reached the limits of our understanding. The division between direct cause and indirect cause is purely convenience. Worthwhile convenience, but convenience nonetheless.
You are of course correct about the throttle example.
After thinking about this a good bit the last time it came up, I am now convinced that causation cannot exist without correlation unless the phenomenon is modelled incorrectly in pretty much the same way as the throttle example, which is to take an independent variable and insist that it is the dependent variable. Then just put your fingers in your ears and hum really loud when someone claims that there is a meaningful dependent variable that does correlate and throttle setting is not it.
Nkh:
The point about the throttle example is that opening the throttle does have an effect–as you say, it causes the car to maintain its speed, compared to the deceleration that would happen if the throttle were closed–but, in the data, you won’t necessarily see a positive correlation between throttle position and speed. The point is that it would be mistaken to see a lack of correlation and infer from this a lack of causation.
This is an obvious and well-known issue with observational data. The only reason for bringing up the example was that three public intellectuals on an off day wrote, “causation does imply correlation,” and they put it in a book which naive outsiders might have been likely to take seriously.
“…in the data, you won’t necessarily see a positive correlation between throttle position and speed. The point is that it would be mistaken to see a lack of correlation and infer from this a lack of causation.”
Once it is noticed that throttle position does not correlate to speed under varying load, the logical thing to do, initially at least, is to reject the initial assumption of direct causation. It is far more likely that the variable that you assumed was “being caused” was incorrectly identified as the dependent variable. From there, you should search for the true dependent variable, which in the throttle case is thrust. Now when you graph the phenomenon, you show throttle position vs. thrust over a varying load, which gives you a useful portrayal of the performance of the system, as opposed to a graph of throttle position vs. speed, which is just going to be a mess.
Observational data is indeed full of this stuff. But these are not pits that you might fall into, they are clues that you should use to improve your understanding of the phenomenon. Because if you keep looking, you are certain to find a better way to state the causal relationship, and you will know it’s better because it correlates across the operational regime.
“The paper claims “Across the highest adult performance levels, peak performance is negatively correlated with early performance.” This is a classic example of Berkson’s paradox.”
Can someone explain how this statement is fallacious or a classic example of the paradox? I don’t really see a problem with the authors making a descriptive claim about the selected population (high performing adults). In the snippet, they are not extrapolating the claim to the general population (people across all performance levels).
The original post goes on to describe an example using NBA players where they end with “From this, I extrapolated that since I’m 5’7, I will be scoring 80+ points per NBA game. . . .” I understand how this would be a “mistake” since they are presumably not an NBA player
Anon:
Good point. The result from the data is interesting. Indeed, it can be misinterpreted–that’s where Berkson’s paradox comes in–but that doesn’t knock out the descriptive finding.
Anoneuoid: “Causality is not something we can observe, its an assumption made when deriving a model. We can only observe correlations.”
Whitehead once argued that, at least in some circumstances, we can in fact directly perceive causation. In fact he thought the idea we couldn’t to be a philosophical folly, obviously wrong. “I saw the mess he made. I saw it clearly.” This is actually a pretty strong argument. But of course statistics would still go on even if we weren’t all dogged Humeans.
If you share a link to the argument I will read it.
Haven’t read this all the way through but it seems to get the main bits right.
https://philarchive.org/archive/GOMWOT
Thanks, that is written by someone else (Samuel Gomes in 2015) though. Nothing against them personally, but too many times I’ve seen second/third parties misinterpret or misrepresent the original persons argument (eg, Fisher/Neyman-Pearson and NHST, Einstein and constant speed of light + “spacetime curvature”). If the original Whitehead is available that would be much better.
Try this
https://brocku.ca/MeadProject/Whitehead/Whitehead_1927/1927_01.html
Can you quote the relevant lines? Seems more like he just arguing that humans perceive causality.
I’m not an expert on this but have thought about it while studying the history and philosophy of science and statistics a bit: It’s an easy habit to fall into, to think that when there’s statistical variation, that’s because there’s a single cause that ends up with varied results due to chance or free will. There’s a real spot you dropped the steel ball and chance causes it to fall through the quincunx in a random pattern, and you can reconstruct where you dropped the ball from the pattern. But that’s not how complex systems like automobiles and living things work. Not only are there multiple causes, there are feedback loops and other mechanisms that are fully predictable but are interposed between the start and end of the process.
A kind of statistics-skeptical way of putting the problem, in part, is that regression analysis, etc., can give you Aristotelian “experience” but not Aristotelian “knowledge.” (Though most Aristotelians probably don’t consider knowledge of a mechanism to be real knowledge either.) You only get knowledge if you know the mechanism is one that produces a random distribution of a certain kind.
You see that a fair amount in the press though. A study showed X has no influence at all on Y because a study showed no correlation. Or we have to think of some demographic group as having an essence we can determine from its correlation with some trait.