Anthony Fowler writes:
As you may recall, Pablo Montagnes and I wrote a paper on college football and elections in 2015 where we looked at additional evidence and concluded that the original Healy, Malhota, Mo (2010) result was likely a false positive. You covered this here and here.
Interestingly, the story isn’t completely over. Graham, Huber, Malhotra, and Mo have a forthcoming JOP paper claiming that the evidence is mostly supportive of the original hypothesis. They added some new observations, pooled all the data together, and re-ran some specifications that are very similar to those of the original Healy et al. paper. The results got weaker, but they’re still mostly in the expected direction.
Pablo and I wrote a reply to this paper, available here, which is also forthcoming in the JOP. We ran some simulations showing that their results are in line with what we would expect if the original result was a chance false positive, and their results are much weaker than what we would expect if the original result was a genuine effect of the magnitude reported in the original paper.
They wrote a reply to our reply, which we only learned about recently when it appeared on the JOP site.
We have written a brief reply to their reply. We assume that the JOP won’t be interested in publishing yet another reply, but if you think this is interesting, we would greatly appreciate you covering this topic and sharing our reply.
There is a lot more to discuss. For example, Graham et al. say that they are conducting an independent replication using the principles of open science. But the data and design are very similar to the original paper, so this is neither independent nor a replication. They argue that they pre-registered their analyses, but they had already seen very similar specifications run on very similar data, so it’s not so clear that we should think of these as pre-registered analyses. They appear to have deviated from their pre-analysis plan by failing to report results using only the out-of-sample data (they just show results using the in-sample data and the pooled data, but not the out-of-sample data). They also exercise some degrees of freedom (and deviate from their pre-analysis plan) in deciding what should count as out of sample.
I have three quick comments:
First, much of the above-linked discussion concerns what counts as a preregistered replication. It’s important for people to consider these issues carefully but they don’t interest me so much, at least not in this setting where, ultimately, the amount of data is not large enough to learn much of anything without some strong theory.
Second, although I’m generally in sympathy with the arguments made by Fowler and Montagnes, I don’t like their framing of the problem in terms of “false positives.” I don’t think the effect of a football game on the outcome of an election is zero. What I do think (until persuaded by strong evidence to the contrary) is that these effects are likely to be small, are highly variable when they’re not small, and won’t show up as large effects in average analyses. In practice, that’s not a lot different than calling these effects “false positives,” but I don’t like going around saying that effects are zero. It’s enough to say that they are not large and predictable, which would be necessary for them to be detectable from the usual statistical analysis.
Third, when reading Fowler and Montagnes’s final points regarding political accountability, I’m reminded of our work on the piranha principle: Once you accept the purportedly large effects of football games, shark attacks, etc., where do you stop? To put it another way, it’s not impossible that college football games have large and consistent effects on election outcomes, but there are serious theoretical problems with such a model of the world, because then you have to either have a theory of what’s so special about college football or else you have a logjam of large effects from all sorts of inputs.
Dear Andrew,
Thank you for your post.
For the record, I agree with you that the term “false positive” is an imperfect shorthand here. Of course, college football probably has some effect on elections, although it’s probably a substantively negligible effect. It’s more correct to say that we think the previous studies, through some combination of bad luck, forking paths, publication bias, etc. significantly overestimated the effect.
In the papers, when we do use the more binary language, we talk about whether there is a substantively meaningful effect. We’re pretty comfortable saying that we think there is most likely not a substantively meaningful effect, and to the extent that HMM detected one, that was probably more noise than signal. The most efficient way we can think to communicate that is to say that it’s probably a chance false positive. But we certainly don’t want to give the impression that the world is binary and that the effect of college football on elections is exactly zero.
Best,
Anthony
Anthony:
It still seems possible to me that for some states in some years, the football game can change moods and swing a bunch of votes. But I think such special cases would be rare, and they could go in either direction, so I guess I pretty much agree with you on the substance.
It seems to me that “large effects from all sorts of inputs” is a perfectly reasonable possibility for reality, at least in some systems. I’m not sure that always has to imply a logjam, but in cases where reality is a logjam, it might be best to accept that reality early on to get a more realistic outlook on what might constitute useful insight. I mean, a clear picture of (some aspect of) the nature of things is surely the first goal of modeling, even if that nature is so confusing that the main lesson is a better understanding of the limits of understanding.
I keep thinking back to Clinton’s 2016 election loss and all the cottage industry of articles highlighting one or another “one thing” that cost her the election (Russian interference, James Comey, inattention to the Midwest, generational shifts in policy interest, …). Many of these articles’ theses were probably true in that she may have won if such-and-such element had gone the other way. If so, then it’s clear that no single-variable approach (nor viable multivariable approach, which I think is Andrew’s point) will ever have decent predictive power. But my point is that there’s more to life than prediction. Knowing which issues have potential effects of this size or that can still have value to the next campaign trying to navigate a path to victory (e.g., the football thing might actually be worth speaking to in some states – talk is free!).
“…[Clinton] may have won if such-and-such element had gone the other way. If so, then it’s clear that no single-variable approach (nor viable multivariable approach, which I think is Andrew’s point) will ever have decent predictive power.”
I think the default assumption should be that this situation would be best covered by a Pareto distribution. There would need to be a specific reason why the distribution would NOT be Pareto, and I am not sure what that would be.
Josh,
The thing about that election is that it was really close. It would only have taken a small effect to push it the other way. And yes, there are lots of things that really can have small effects.
But I don’t think there are a lot of things that can have large effects on voting preferences. If college football games had large effects and shark attacks had large effects and news about child abductions had large effects and P, Q, R, S, T, U and V also all had large effects, you would never be able to detect any of them because the one you were trying to study would always be masked by all of the other things happening at the same time.
The idea that many phenomena have small effects on voting preferences is not controversial, indeed it’s obviously true. It’s the idea that there are many phenomena that have large effects that doesn’t seem right. And, yes, when the election is really close a small effect on voting preferences can lead to a completely different election outcome, in our winner-takes-all system.