Adam Zelizer writes:
I saw your post about the underpowered COVID survey experiment on the blog and wondered if you’ve seen this paper, “Counter-stereotypical Messaging and Partisan Cues: Moving the Needle on Vaccines in a Polarized U.S.” It is written by a strong team of economists and political scientists and finds large positive effects of Trump pro-vaccine messaging on vaccine uptake.
They find large positive effects of the messaging (administered through Youtube ads) on the number of vaccines administered at the county level—over 100 new vaccinations in treated counties—but only after changing their specification from the prespecified one in the PAP. The p-value from the main modified specification is only 0.097, from a one-tailed test, and the effect size from the modified specification is 10 times larger than what they get from the pre-specified model. The prespecified model finds that showing the Trump advertisement increased the number of vaccines administered in the average treated county by 10; the specification in the paper, and reported in the abstract, estimates 103 more vaccines. So moving from the specification in the PAP to the one in the paper doesn’t just improve precision, but it dramatically increases the estimated treatment effect. A good example of suppression effects.
They explain their logic for using the modified specification, but it smells like the garden of forking paths.
Here’s a snippet from the article:
I don’t have much to say about the forking paths except to give my usual advice to fit all reasonable specifications and use a hierarchical model, or at the very least do a multiverse analysis. No reason to think that the effect of this treatment should be zero, and if you really care about effect size you want to avoid obvious sources of bias such as model selection.
The above bit about one-tailed tests reflects a common misunderstanding in social science. As I’ll keep saying until my lips bleed, effects are never zero. They’re large in some settings, small in others, sometimes positive, sometimes negative. From the perspective of the researchers, the idea of the hypothesis test is to give convincing evidence that the treatment truly has a positive average effect. That’s fine, and it’s addressed directly through estimation: the uncertainty interval gives you a sense of what the data can tell you here.
When they say they’re doing a one-tailed test and they’re cool with a p-value of 0.1 (that would be 0.2 when following the standard approach) because they have “low signal-to-noise ratios” . . . that’s just wack. Low signal-to-noise ratio implies high uncertainty in your conclusions. High uncertainty is fine! You can still recommend this policy be done in the midst of this uncertainty. After all, policymakers have to do something. To me, this one-sided testing and p-value thresholding thing just seems to be missing the point, in that it’s trying to squeeze out an expression of near-certainty from data that don’t admit such an interpretation.
P.S. I do not write this sort of post out of any sort of animosity toward the authors or toward their topic of research. I write about these methods issues because I care. Policy is important. I don’t think it is good for policy for researchers to use statistical methods that lead to overconfidence and inappropriate impressions of certainty or near-certainty. The goal of a statistical analysis should not be to attain statistical significance or to otherwise reach some sort of success point. It should be to learn what we can from our data and model, and to also get a sense of what we don’t know..
You nailed it – contortions to achieve statistical significance when they could just report what happened. I briefly looked at the paper and noted a couple of things. The map of the control and treatment counties is a little strange – all of the counties in Alaska included in the study were control counties, as was a large swath of land in northwest Nevada. Those patterns don’t look random to me – both very sparsely populated areas and all in the control group. More careful analysis is needed, but it reminded me of Andrew’s discussion of the Wikipedia advertising results, where a supposed random assignment turned out not to be so random.
Striking (to me, at least) was the first table in the paper showing Republican voters trust (there’s that word again) in various sources of information about vaccines – Trump was ranked above all other sources, including science – by a factor of 2 in the later study. It constantly amazes me when I see these kinds of results – who are these people? I have no problem understanding skepticism of Fauci, science, Biden, etc., but trusting Trump on vaccine advice? Yes, this is tribal blind trust, with the emphasis on “blind.” I try not to succumb to tribal polarization, but it is getting harder and harder to avoid it.
Dale –
Maybe just to walk you back in off the ledge a bit, who really knows what “trust” means here? I suspect it’s little more than identity-aggressive signaling. Mostly, I suspect, “I trust Trump” = “I’m angry at libz and Trump makes fun of them.”
I believe many Trump supporters believe he lies constantly. So what does “trust” mean in that context? I suspect when many Trump supporters see evidence that he was lying, their support grows even stronger. But that wouldn’t mean that they “trust” him, exactly. They just like how he lies.
You say:
who are these people?
I think the point is that “these people” are pretty much just like everyone else, but they are just a group who have a strong identity-based affiliation with Trump. The basic underlying processes are those that affect pretty much everyone: confirmation bias, motivated reasoning, etc. Strong identity-orientation channels and concentrates those cognitive processes in a particular direction.
Yes, this is a perfect example of the problems with the word “trust.” If you ask me about whether Trump will do whatever he feels is in his best interest, regardless of how it affects anybody else, then I actually do “trust” him more than almost anybody. As for whether “these people” are “pretty much like everyone else,” I’m not so sure. I can’t fathom how anybody can blindly follow someone – particularly somebody that is such a bully. I have my own beliefs and hold them strongly, but I can’t think of any time in my entire life that I have willingly followed someone the way his supporters seem to follow him.
Andrew,
Why do you imply the two-sided test is “the standard approach”?
By no means would I defend statistical hypothesis testing… I’ve been hating on them long before it was cool. But it seems to me that the two-sided test has even more philosophical issues than a one-sided test. And the authors’ provide a reasonable second justification for applying their one-sided test.
If the authors are ‘moving the goal posts’ by first trying the two sided test and switching to the one sided test, then yeah, that’s obviously problematic.
Jyd:
I guess I’d say that the standard approach is to compute the 95% interval and then declare statistical significance if the interval excludes zero. This is mathematically equivalent to the two-sided test. I’m pretty sure that if their 95% interval had excluded 0, they would’ve reported the p-value of 0.03 or whatever and had been done with it. And, had they had this statistically-significant result, I still would’ve been doubtful about the result’s replicability for all the other reasons given in the above post. The point of talking about one-sided tests here is not because I call about the test itself; it’s to get some insight into the workflow that led to the published summaries.
There’s something crazy in scientists taking a set of analysis results, and when they look at in one way (2-tailed 0.05 threshold), they proclaim they’ve failed to find anything—even going so far as interpreting it as zero effect. But when they look at the exact same results through an alternative lense (1-tailed 0.1 threshold), they proclaim there’s definitely an effect.
I sometimes wonder how such a silly paradigm of binary thinking has maintained credibility for so long. Yet I know scientists who genuinely do believe in it all, and go off speculating on underlying mechanisms whenever they find an interaction with p<0.05 in the nosiest of environments lacking main effects.
Kj:
The common theme here is that they always think their theory is true. So it’s ok for them to find no effect if they can tell a story by which “no effect” is consistent with their theory. Conversely, if they feel that, for heir theory, there needs to be an effect, they can do what it takes to find statistical significance; if that fails, they can move the goalposts as in the above-discussed study; if that fails, they can say that they failed to find an effect because of low statistical power. It’s a heads-I-win, tails-you-lose kinda game.
Well, I think there are two issues that lead to incredible waste in academia. The first, like you said, are those who treat gathering statistical evidence as a hoop to jump through. They have a strong prior on their theory, and update as if data has no power.
The second is the opposite. The scientists put essentially flat priors on all hypotheses and believe statistical significance has incredible power. So whenever exploratory research turns up a p<0.05, it demands a causal explanation, a write-up in a paper, and perhaps follow-up experiments. It doesn't matter if the causal explanations seems like a stretch, or seems inconsistent with other outcomes in the data.
What's most frightening are that often is the same researchers who fall into both traps. They latch onto noise and then don't let go.
There's just so much wasted effort out there. Then again, I often wonder about just how many practical or "important" results can come out of causal models in social science, when there's so much heterogeneity across people, time, and minute changes to the situation. But that's an entirely different problem, I suppose.
The larger NHST issue with post hoc deciding to do a one-tailed hypothesis test is that it solves nothing. This needed to be determined a priori, not after the fact. Same with shifting your error rate. My question would be: if they had expected a negative effect and used a one-tailed test but the effect in the data was positive would the authors have then said “because of this, even if statistically significant, we must ignore the test”? I am sceptical.
> until my lips bleed
Ha! The writing on this blog is as good as the statistics!
Yup, my writing and my statistics both feature the deft usage of clichés.