We’ve been here before.

Back in 2002, political scientists Chris Achen and Larry Bartels presented a paper “Blind Retrospection – Electoral Responses to Drought, Flu and Shark Attacks.” Here’s a 2012 version in which the authors trace “the electoral impact of a clearly random event—a dramatic series of shark attacks in New Jersey in 1916” and claim to “show that voters in the affected communities significantly punished the incumbent president, Woodrow Wilson, at the polls,” a finding that has been widely discussed in political science over the past several years and was featured in Achen and Bartels’s recent book, Democracy for Realists.

In 2016, Anthony Fowler and Andy Hall reanalyzed the data and concluded that “the evidence is, at best, inconclusive”:

First, we assemble data on every fatal shark attack in U.S. history and county-level returns from every presidential election between 1872 and 2012, and we find little systematic evidence that shark attacks hurt incumbent presidents or their party. Second, we show that Achen and Bartels’ finding of fatal shark attacks hurting Woodrow Wilson’s vote share in the beach counties of New Jersey in 1916 becomes substantively smaller and statistically weaker under alternative specifications. Third, we find that their town-level result for beach townships in Ocean County significantly shrinks when we correct errors associated with changes in town borders and does not hold for the other beach counties in New Jersey. Lastly, implementing placebo tests in state-elections where there were no shark attacks, we demonstrate that Achen and Bartels’ result was likely to arise even if shark attacks do not influence elections. Overall, there is little compelling evidence that shark attacks influence presidential elections, and any such effect—if one exists—appears to be substantively negligible.

Fowler and Hall published their article in the Journal of Politics, and Achen and Bartels replied with a comment of their own, including these points:

Attributing to us [Achen and Bartels] the notion that, in general, “shark attacks influence presidential elections,” much less that “irrelevant events generally influence presidential elections”, reflects a profound misreading of our argument. As we spelled out, the 1916 attacks were politically relevant because substantial economic losses ensued and the president was explicitly blamed. Neither of those things is true of the typical shark attack; thus, we would not expect it to matter at the polls. . . .

Fowler and Hall’s reanalysis of county-level voting patterns in New Jersey in 1916 produces “substantively smaller and statistically weaker” estimates of the impact of shark attacks on electoral support for Woodrow Wilson “under alternative specifications” . . . Even so, their point estimates mostly differ only modestly from ours. The estimates become statistically insignificant only because their ahistorical bad statistical fits inflate standard errors and thus make the t-statistics smaller. We show that a variety of regression models that get the politics right all fit better than Fowler and Hall’s. Those models all show a substantively and statistically significant shark effect. . . .

Fowler and Hall employ a series of “placebo tests” comparing election outcomes in coastal and noncoastal counties to suggest that “Achen and Bartels’s result for New Jersey in 1916 was somewhat likely to arise even if shark attacks have no effect on presidential elections” . . . They find that 27% of these comparisons produce “statistically significant” differences in the vote swing from one election to the next in counties bordering the ocean; hence, they argue that other factors besides shark attacks could have produced the marked electoral shift in Jersey Shore counties in 1916. But Fowler and Hall provide no indication of what those other factors might be. . . .

This last point seems to represent a statistical misunderstanding on the part of Achen and Bartels. Fowler and Hall’s point regarding the placebo tests—a point which I think is valid—is that correlations in the dataset make statistical significance easier to attain than would be expected under the simple default model of independent outcomes. Thus, the statistical significance of Achen and Bartels’s original analysis cannot be taken as strong evidence: the patterns they found could be explained by various systematic differences between counties unrelated to shark attacks.

Achen and Bartels are right that Fowler and Hall do not “demonstrate that the shark attacks made no difference.” That’s right. What Fowler and Hall find is there is no strong evidence for any effects of shark attacks.

Step back a minute. Before Achen and Bartels (2002, 2012), I assume that few scholars would’ve considered shark attacks to have important effects in 1916 or in any other presidential election. The Achen and Bartels papers made the surprising claim that, yes, shark attacks did make a difference, and they took this as evidence for their “blind retrospection” theory. As I wrote earlier when discussing all this, I think Achen and Bartels have some good points in their book; their larger arguments do not rely on the validity of that shark-attack study. Anyway, the point is that Fowler and Hall don’t need to demonstrate that shark attacks made no difference. The burden is on Achen and Bartels to support their counterintuitive statement that shark attacks mattered. Or, to put it another way, you can believe that shark attacks mattered in 1916, even if the data don’t really show it. All sorts of things could’ve mattered, and you can’t disprove any of them.

OK, fine. That all said, Achen and Bartels have two substantive points to make. The first is that irrelevant events can matter in presidential elections if “substantial economic losses ensued and the president was explicitly blamed.” The second is that Fowler and Hall’s estimates are similar to theirs, and that Fowler and Hall find non-statistical-significance only by fitting a crappy model and thus obtaining artificially high standard errors.

Now let’s turn to Fowler and Hall’s reply in that same journal.

I’ll pull out two parts of this reply.

First, regarding the effects of shark attacks:

[Achen and Bartels] agree with us that shark attacks do not, in general, lead voters to punish incumbents, stating that they “would not expect” the “typical shark attack” to affect a presidential election. This consensus is important since many readers have thought that their claims were stronger and more general than they are. . . . Writing in Pacific Standard, Seth Masket states that “voters punish their leaders for . . . shark attacks.” In his review of Achen and Bartels’s book in the Journal of Politics, Neil Malhotra writes that “voters frequently punish incumbents for things they cannot control such as shark attacks.” Achen and Bartels have now clarified that their claim is specific to only the shark attacks in New Jersey in 1916. . . . Future discussions of “blind retrospection” should take note of this new consensus. Rather than claiming that shark attacks indicate a general failure of electoral accountability, Achen and Bartels say that voters do not blame incumbents for shark attacks in about 99 percent of all recorded shark attacks in American history.

Well put.

Second, regarding the new analyses that give larger standard errors and thus diminish the claims of strong evidence regarding the 1916 election:

We show that the 1912 election, which Achen and Bartels use to control for the baseline political preferences of counties and towns, was anomalous. Figure 3 of our paper clearly shows that their county-level result is driven by the unusualness of 1912, not anything that happened in 1916. . . .

We show that Achen and Bartels’ standard errors are misleading. If we apply their inferential strategy to state-elections with no shark attacks, we detect an effect as large as theirs 32 percent of the time, and the estimate is statistically significant (p < .05) 27 percent of the time.

To see more on this, I recommend you go back and look at our earlier post, in particular this graph of adjusted data from Achen and Bartels:

and this graph of raw data from Fowler and Hall:

I agree with Fowler and Hall that the evidence isn’t nearly as strong as implied by Achen and Bartels’s regressions.

Let me be clear: I don’t think that what Achen and Bartels did is any sort of scandal. They have an interesting idea regarding blind retrospection, the shark attack example is a cool case study, and yes their data are consistent with no effect of shark attacks but their data are also consistent with a positive effect, perhaps for the reasons they stated regarding economic costs and the president being blamed. They did some analyses which confirmed their beliefs and they published. Fair enough. Later, some other researchers looked at their data more carefully and found the evidence, both for this particular case and for shark attacks more generally, to not be so strong. That’s how we move forward.

Fowler adds:

Andy Hall and I wrote about this a bit in the conclusion of our sharks paper, but it’s very difficult to show evidence either way regarding the competence of rationality of voters. For one, lots of seemingly irrational behaviors have rational explanations. For example, it’s not necessarily irrational for voters to change their beliefs about their elected officials as a result of shark attacks–maybe the voters learned that the government doesn’t have their back when a major crisis comes their way. And even then, once you’ve adopted irrationality or incompetence as your “theory,” there’s nothing constraining your empirical testing. Maybe sharks affect people in the beach towns, maybe the beach counties, maybe the coastal counties, or maybe the whole state. You can run 10,000 different regressions and each one is just as (poorly) grounded in theory as the next, so all bets are off. And whichever regression gives you the desired result, you just say “well, that must be how irrational voting works.”

One fun solution would be for people like Chris Achen and Larry Bartels to get together with people like me and Andy Hall to come up with some ex-ante tests of rationality/competence/etc. We could all agree on some compelling tests that would partly adjudicate some of these debates and then go out and run the experiments or collect the relevant data.

P.S. I have no financial conflicts of interest here, but in the interests of full disclosure I should inform you that I’ve been involved in disputes with Larry Bartels before. In one dispute, Bartels and I were on the same side (this was in dealing with the annoying Thomas Frank); in the other case, we disagreed with each other. So, sometimes we agree, sometimes we don’t.