A couple of faithful correspondents pointed me to this recent article, “Fertility Fails to Predict Voter Preference for the 2020 Election: A Pre-Registered Replication of Navarrete et al. (2010).”
It’s similar to other studies of ovulation and voting that we’ve criticized in the past (see for example pages 638-640 of this paper.
A few years ago I ran across the following recommendation for replication:
One way to put a stop to all this uncertainty: preregistration of studies of all kinds. It won’t quell existing worries, but it will help to prevent new ones, and eventually the truth will out.
My reaction was that this was way too optimistic.The ovulation-and-voting study had large measurement error, high levels of variation, and any underlying effects were small. And all this is made even worse because they were studying within-person effects using a between-person design. So any statistically significant difference they find is likely to be in the wrong direction and is essentially certain to be a huge overestimate. That is, the design has a high Type S error rate and a high Type M error rate.
And, indeed, that’s what happened with the replication. It was a between-person comparison (that is, each person was surveyed at only one time point), there was no direct measurement of fertility, and this new study was powered to only be able to detect effects that were much larger than would be scientifically plausible.
The result: a pile of noise.
To the authors’ credit, their title leads right off with “Fertility Fails to Predict . . .” OK, not quite right, as they didn’t actually measure fertility, but at least they foregrounded their negative finding.
Bad Science
Is it fair for me to call this “bad science”? I think this description is fair. Let me emphasize that I’m not saying the authors of this study are bad people. Remember our principle that honesty and transparency are not enough. You can be of pure heart, but if you are studying a small and highly variable effect using a noisy design and crude measurement tools, you’re not going to learn anything useful. You might as well just be flipping coins or trying to find patterns in a table of random numbers. And that’s what’s going on here.
Indeed, this is one of the things that’s bothered me for years about preregistered replications. I love the idea of preregistration, and I love the idea of replication. These are useful tools for strengthening research that is potentially good research and for providing some perspective on questionable research that’s been done in the past. Even the mere prospect of preregistered replication can be a helpful conceptual tool when considering an existing literature or potential new studies.
But . . . if you take a hopelessly noisy design and preregister it, that doesn’t make it a good study. Put a pile of junk in a fancy suit and it’s still a pile of junk.
In some settings, I fear that “replication” is serving a shiny object to distract people from the central issues of measurement, and I think that’s what’s going on here. The authors of this study were working with some vague ideas of evolutionary psychology, and they seem to be working under the assumption that, if you’re interested in theory X, that the way to science is to gather some data that have some indirect connection to X and then compute some statistical analysis in order to make an up-or-down decision (“statistically significant / not significant” or “replicated / not replicated”).
Again, that’s not enuf! Science isn’t just about theory, data, analysis, and conclusions. It’s also about measurement. It’s quantitative. And some measurements and designs are just too noisy to be useful.
My criticism of the ovulation-and-voting study is ultimately quantitative. Their effect size is tiny and their measurement error is huge. My best analogy is that they are trying to use a bathroom scale to weigh a feather—and the feather is resting loosely in the pouch of a kangaroo that is vigorously jumping up and down.
At some point, a set of measurements is so noisy that biases in selection and interpretation overwhelm any signal and, indeed, nothing useful can be learned from them. I assume that the underlying effect size in this case is not zero—if we were to look carefully, we would find some differences in political attitude at different times of the month for women, also different days of the week for men and for women, and different hours of the day, and I expect all these differences would interact with everything—not just marital status but also age, education, political attitudes, number of children, size of tax bill, etc etc. There’s an endless number of small effects, positive and negative, bubbling around.
Bad Reporting
Bad science is compounded by bad reporting. Someone pointed me to a website called “The National Pulse,” which labels itself as “radically independent” but seems to be an organ of the Trump wing of the Republican party, and which featured this story, which they seem to have picked up from the notorious sensationalist site, The Daily Mail:
STUDY: Women More Likely to Vote Trump During Most Fertile Point of Menstrual Cycle.
A new scientific study indicates women are more likely to vote for former President Donald Trump during the most fertile period of their menstrual cycle. According to researchers from the New School for Social Research, led by psychologist Jessica L Engelbrecht, women, when at their most fertile, are drawn to the former President’s intelligence in comparison to his political opponents. The research occurred between July and August 2020, observing 549 women to identify changes in their political opinions over time. . . .
A significant correlation was noticed between women at their most fertile and expressing positive opinions towards former President Donald Trump. . . . the 2020 study indicated that women, while ovulating, were drawn to former President Trump because of his high degree of intelligence, not physical attractiveness. . . .
As I wrote above, I think that research study was bad, but, conditional on the bad design and measurement, its authors seem to have reported it honestly.
The news report adds new levels of distortion.
– The report states that the study observed women “to identify changes in their political opinions over time.” First, the study didn’t “observe” anyone; they conducted an online survey. Second, they didn’t identify any changes over time: the women in the study were surveyed only once!
– The report says something about “a significant correlation” and that “the study indicated that . . .” This surprised me, given that the paper itself was titled, “Fertility Fails to Predict Voter Preference for the 2020 Election.” How do you get from “fails to predict” to “a significant correlation”? I looked at the journal article and found the relevant bit:
Results of this analysis for all 14 matchups appear in Table 2. In contrast to the original study’s findings, only in the Trump-Obama matchup was there a significant relationship between conception risk and voting preference [r_pb (475) = −.106, p = .021] such that the probability of intending to vote for Donald J. Trump rose with conception risk.
Got it? They looked at 14 comparisons. Out of these, one of these was “statistically significant” at the 5% level. This is the kind of thing you’d expect to see from pure noise, or the mathematical equivalent, which is a study with noisy measurements of small and variable effects. The authors write, “however, it is possible that this is a Type I error, as it was the only significant result across the matchups we analyzed,” which I think is still too credulous a way to put it; a more accurate summary would be to say that the data are consistent with null effects, which is no surprise given the realistic possible sizes of any effects in this very underpowered study.
The authors of the journal article also write, “Several factors may account for the discrepancy between our [lack of replication of] the original results.” They go on for six paragraphs giving possible theories—but never once considering the possibility that the original studies and theirs were just too noisy to learn anything useful.
Look. I don’t mind a bit of storytelling: why not? Storytelling is fun, and it can be a good way to think about scientific hypotheses and their implications. The reason we do social science is because we’re interested in the social world; we’re not just number crunchers. So I don’t mind that the authors had several paragraphs with stories. The problem is not that they’re telling stories, it’s that they’re only telling stories. They don’t ever reflect that this entire literature is chasing patterns in noise.
And this lack of reflection about measurement and effect size is destroying them! They went to all this trouble to replicate this old study, without ever grappling with that study’s fundamental flaw (see kangaroo picture at the top of this post). Again, I’m not saying that they authors are bad people or that they intend to mislead; they’re just doing bad, 2010-2015-era psychological science. They don’t know better, and they haven’t been well served by the academic psychology establishment which has promoted and continues to promote this sort of junk science.
Don’t blame the authors of the bad study for the terrible distorted reporting
Finally, it’s not the authors’ fault that their study was misreported by the Daily Mail and that Steve Bannon associated website. “Fails to Predict” is right there in the title of the journal article. If clickbait websites and political propagandists want to pull out that p = 0.02 result from your 14 comparisons and spin a tale around it, you can’t really stop them.
The Center for Open Science!
Science reform buffs will enjoy these final bits from the published paper:
The SCORE project is a large replication project that directly replicates a stratified sample of published studies to empirically examine predictors of replicability. This study was part of the sample, so it had to be directly replicated.
This seems like an important detail that should have been mentioned. OTOH the abstract[1] discusses this study in isolation, as a test of the hypothesis mentioned. It does not describe that it is supposed to be just one datapoint in a larger project. Therefore it seems fair to say whatever about this paper as it stands alone, because that is how it is presented[2].
[1] I only have access to the abstract.
[2] SCORE is mentioned in the abstract but not explained. I don’t know what it is.
Daniel:
That makes sense. It’s too bad they are wasting effort on such noisy designs, but if you’re going to replicate a random sample of papers from this literature in the 2010-2015 period, I guess that is unavoidable.
That said, i.e., conditioning on the decision to replicate a study using a noisy design and indirect measurements to study an effect that will be small at best, I think the new article makes a major error by not discussing these statistical problems. If they’re going to do the study as part of some larger project, fine; then, when writing up the study it’s appropriate to say the study was essentially doomed from the start for reasons discussed clearly in the literature.
To put it another way, if for methodological or historical or sociological reasons it is considered a good idea to perform a study that is fatally flawed in the sense of being too noisy to have any chance of observing any realistic signal, then it would be a good idea to say so right at the start, and then say so again when the study is over. As it is, the paper has lots of discussion and explanation but without ever recognizing this problem, and that bothers me.
Also, what’s up with a project funded by the Center for Open Science that states that “no datasets were generated or analyzed during the current study”? That’s not my main problem with the paper; it’s just weird.
I do not know how funding through SCORE works. However,
(1) the research team decided to do the replication,
(2) someone from SCORE decided the replication study was worth funding.
These people share responsibility for the existence of this replication experiment.
In the researcher’s defense, just because they did not mention the measurement difficulties in their study, it does not mean that they were unaware of them.
On the other hand, maybe the funders wanted to pick a study that would certainly not replicate!
Raphael:
I would say it’s a scientific error to give multiple paragraphs of discussion of empirical results of a study without ever mentioning that the study was essentially a noise-production exercise and this should have been clear before any data had been collected, given the design, measurements, and plausible effect sizes.
This would be an error if the authors were unaware of this issue and it would also be an error if the authors were aware of the problem and didn’t mention it in the paper.
Regarding the choice of study: it sounds from the above comment by Daniel that they picked studies at random from a literature where it would be expected that many studies would not replicate. That’s fine—as meta-science, such an effort could be valuable (but, if so, they obviously should be sharing their data, not reporting that “no datasets were generated”). The key error in the published paper was not an error of meta-science; it was an error of science in implying that such a noisy study could tell us anything useful about fertility, voter preference, evolutionary psychology, etc.
Check out the first four paragraphs of the published article, which is all about the substantive topics. It would be natural after reading these paragraphs to suppose that this article is going to usefully address the substantive topics; but it does not. Whether or not this work is valuable as meta-science—I’ll make no judgement either way on this—, I don’t think it is valuable as science, in the same way that I generally don’t think it’s valuable as science to study a topic by gathering some very noisy data related to that topic and then performing some statistical analyses and using these as the basis of stories. I’d much rather for them to just jump straight from the theoretical speculations to the stories and not confuse the matter by chasing noise.
I can’t access the paper. What puzzles me is how this is a replication–conceptual or not–if no data were collected as it says in the data non-availability statement (really, that’s what it should actually be called more generally).
BTW, even if one had the original 2010 data (which one doesn’t), the probability that the original analysis would even reproduce is less than 30% anyway. So who even knows what the original data showed or didn’t show.
A good time to revisit this XKCD classic.
https://xkcd.com/882/ Significant
Just wanted to applaud the jumping-kangaroo/feather analogy.
From the above:
“… if you take a hopelessly noisy design and preregister it, that doesn’t make it a good study.”
“Bad science is compounded by bad reporting.”
You nailed it.
Instead of “But a pile of junk in a fancy suit” did you mean “Put a pile of junk in a fancy suit”?
P.S. I like the kangaroo/feather example.