Paul Alper writes:
You have written a few times to correct the oft-heard relationship between causation and correlation; but here is a Dana Milbank article in the Washington Post about congressmember Scott Perry’s unusual take:
There have been recent shortfalls in military recruitment, and research shows that economic and quality-of-life issues are to blame, as well as a declining percentage of young people who meet eligibility standards.
But Republicans argued that the real culprit is “woke” policies, though they offered no evidence of this.
“Just because you don’t have the data or we don’t have the data doesn’t mean there’s no correlation,” argued Rep. Scott Perry (R-Pa.).
At first Perry’s statement might sound ridiculous, but if you reflect upon it you’ll realize it’s true. He was making a claim about the correlation between two variables, X and Y. He did not have any data at hand on X or Y, but that should not be taken to imply that the correlation is zero.
Indeed, I can go further than Perry and say two things with confidence: (a) the correlation between X and Y is not zero, and (b) the correlation between X and Y changes over time, it is different in different places, and it varies by context. With continuous data, nothing is ever exactly zero. I guess it’s possible that some of these variables could be measured discretely, in which case I’ll modify my statements (a) and (b) to say that the correlation is almost certainly not zero, that it almost certainly changes over time, etc.
Setting aside all issues of correlation, the mistake that Perry made is what we’ve called the fallacy of the one-sided bet. Yes, he’s correct that, even though he has no data on X and Y, these two variables could be positively correlated. But they also could be negatively correlated! Perry is free to believe anything he wants, but he should just be aware that, in the absence of data, he’s just hypothesizing.
The all time best, imo.
“If we stop testing right now, we’d have very few cases, if any,” Trump asserted.
“With continuous data, nothing is ever exactly zero.” True, but usually unhelpful. In all practical circumstances (yes, I do mean all) there is a limit to the number of decimal places available and so there is no such thing as truly continuous data outside of theory. In many types of data the available number of decimal places is quite small.
I know how much some people enjoy using “nothing is exactly zero” as an argument against significance and hypothesis testing, but to do so ignores the practical in favour of the theoretical. In practice there is usually (maybe always) an effect size that might as well be zero, just as there is a strictly non-zero effect size that is numerically indistinguishable from zero. I argue that thinking about ‘too small to matter’ is a very useful process for every scientific analysis.
Michael:
I don’t know anything about paleontology so I can’t really comment there. When it comes to social sciences, I think your comment is wrong. The problem is that “statistically indistinguishable from zero” is not in general the same as “numerically indistinguishable from zero” or “too small to matter.”
For example, consider the famous case of subliminal priming and slow walking. Experiments when analyzed carefully show no evidence of any effect. This does not mean the effects are zero, or even that they are too small to matter. The effects can be moderate or even large (ok, I doubt they are large) and highly variable, depending a lot on context. I think this can be the case for a lot of things that are studied in social science: studies are designed to estimate an average effect, but the whole concept of an “average effect” depends so much on the person being studied and what else is happening at the time. Similarly for, say, education interventions: different teaching strategies work on different people at different times.
Or, to step back even further, many studies don’t even have the power to estimate reasonably-sized average differences. So, again, “statistically indistinguishable from zero” does not imply “too small to matter.”
I agree that sometimes an effect can be estimated so precisely that we can say with confidence it is too small to matter. This is just not what I’ve usually seen.