Bernie electability update

The other day we discussed an article, “New research suggests Sanders would drive swing voters to Trump — and need a youth turnout miracle to compensate,” by David Broockman and Joshua Kalla.

A commenter pointed us to a post by Seth Ackerman, “Study Showing Bernie Needs Huge Youth Turnout Is Nonsense,” which states:

In the scenario [Broockman and Kalla’s paper] depicts, “youth turnout” appears to jump implausibly by 11 percentage points — from 43% in 2016 to 54% in 2020 — beating its all-time modern record of 48% (in 2008). . . .

The authors get this idea from the fact that a certain category of young people — mainly Democrats and independents in the 18-34 age bracket — currently boost Sanders’ relative standing in the polls by saying they would turn out to vote for Sanders but would not turn out for a moderate Democrat.

According to the authors, these Bernie-or-Bust respondents represent about 11 percent of all non-Republican 18-34-year-olds, which means that if they all turned out to vote for Sanders, the turnout rate for the “non-Republican 18-34-year-old” demographic would be 11 percentage points higher than it would be if Joe Biden were the nominee and they all stayed home.

At some point in the authors’ thought process they appear to have confused this counterfactual difference in turnout — the difference between a world where Sanders wins the nomination and a world where Biden does — with a sequential increase in turnout that occurs over some historical time period — say, from 2016 to 2020. . . .

A Sanders nomination in 2020 could result in a youth turnout rate 11 points higher than what Biden, counterfactually, would have gotten, without any need for Sanders’ turnout to be 11 points higher than the 2016 level. For example, hypothetical Sanders could get a 49% turnout rate, with hypothetical Biden getting 38%. Or he could get 47% with Biden getting 36%.

In no sense would Sanders have to attain any particular level of turnout — let alone the 54.4% rate drawn in the chart — for his turnout rate to be 11 points higher than what Biden would have gotten had he been the nominee. . . .

In addition, about one-fifth of the pro-Sanders respondents the authors counted in the numerator are spuriously included, because they did not actually indicate in the authors’ survey that they would stay home if a moderate Democrat were nominated. (They said they were undecided or would vote third party.) The authors themselves explain this problem in the caption of their Figure 5, but inexplicably use the 11% figure anyway. . . .

Even with zero increase in youth turnout, Sanders still beats Biden. . . . When they re-weight their matchup polls to simulate 2016 turnout patterns — thus ruling out any youth turnout differential between the candidates, by assumption — they still find Sanders outperforming Biden: Sanders’ margin over Trump is +0.8%; Biden’s is +0.2% (Fig. 6, p.12). Despite the authors’ rhetoric about Sanders being dependent on an implausible youth turnout surge, their own analysis finds that he isn’t. . . .

Broockman and Kalla wrote a response:

The critique misrepresents the analyses our paper conducted and seems to misunderstand how our paper draws its conclusions. . . . At the same time, given that three of the five candidates we examined in our paper have dropped out since we wrote it, we also agree it is reasonable for observers of the Democratic primary to pay renewed attention to a finding in our paper about Joe Biden specifically, although we think the right interpretation of our data is that it is statistically ambiguous with respect to Biden . . .

In our paper (p. 16), the following text leads up to and describes what Figure 8 shows: “It is beyond the scope of this short paper—and indeed likely impossible—to determine whether nominating Sanders would in fact stimulate an 11pp or larger increase in turnout (or decrease in third party voting) among Democrats and Independents under 35 than would otherwise occur if another Democrat were nominated. It may well be that nominating Sanders would increase turnout among this group some. However, other results help contextualize how large a turnout increase of a full 11pp among under 35 Democrats and Independents is.” . . . that 11pp estimate represents the counterfactual increase the survey toplines implicitly credit Sanders with relative to what would otherwise occur if a different Democrat were nominated. Second, that the purpose of Figure 8 is not to calculate that “overall youth turnout must be 54.4% for Sanders to win” but instead, quoting our paper, “[to] help contextualize how large a turnout increase of a full 11pp among under 35 Democrats and Independents is”. . . .

We can see why, without reading the text of the paper describing what the graph is intended to show and how we constructed it, one might interpret this graph as indicating something else. . . . This makes the same mistake of treating Figure 8 as if it is what we use to reach our estimate instead of what we use to visualize the estimate we already reached. . . . It is true that an 11pp turnout surge among non-Republican youth is a smaller absolute number of votes than an 11pp turnout surge among all youth; but nowhere do we make the mistake of equating the two. Again, an 11pp surge among non-Republican youth is what Sanders’ topline poll numbers appear to be banking on; and, as we note in the paper and above, the purpose of this graph is to help readers contextualize what an 11pp turnout increase (among a given demographic) looks like by placing it in the context of historical variation in overall youth turnout. . . .

The 9.2pp estimate [suggested by Ackerman] is not obviously “correct” as it involves another assumption that is also plain to see in Figure 5. Blank ballots (what a literal interpretation of “undecided” would mean given how these responses are counted in polls) are relatively rare in Presidential elections, so we think it is reasonable to discount the 4.7pp decrease in these responses among left-leaning young people with Sanders present, and think of them instead as reflecting a need for increased participation. . . . A central message in our paper is that Sanders’ electability numbers are dependent on whether “Bernie or bust” respondents actually behave this way on election day. . . .

Our paper did not discuss specific comparisons of Sanders to Biden; it compared Sanders to an average of the three moderate candidates in our data: Bloomberg, Buttigieg, and Biden (Klobuchar was not included). . . . There is a lot more statistical uncertainty when throwing out the data on Buttigieg and Bloomberg (who have since dropped out) — and so we think the main take-away should be that our data is somewhat ambiguous about Biden specifically. . . . We think it is reasonable to expect Biden to be fairly similar to the other moderates and for differences between them to be largely due to noise (sampling variability), a suspicion that appears to be confirmed in other public polls as we discuss below. On the other hand, our data also can’t reject that he is equal to Sanders. In other words, both Biden and Sanders supporters could look at our data and reasonably say “it doesn’t rule out my view of things” . . . we think the main takeaway of our paper for discussions of electability at this point in the primary should not be our somewhat noisy results for Sanders vs. Biden. Rather, it should be that analysts of other polls, which appear to find an edge for Biden on average, should check whether assumptions about changes in youth participation underpin their conclusions about Sanders’ performance. . . .

There’s a lot more detail, both in Ackerman’s post and in Broockman and Kalla’s response. As I wrote in my earlier post, the data are generally consistent with the baseline assumption that the more moderate candidate would do better in the general election. But, as Ackerman and Broockman/Kalla each point out, there’s enough variation from election to election that we can’t conclude too much from any general pattern. I expect that in 2016, Martin O’Malley or Amy Klobuchar or various other people would’ve won the election, had they been the Democratic nominee. I’m guessing that Sanders would’ve received fewer votes than Hillary Clinton but he still could’ve won.

Overall, I’m in agreement with Broockman and Kalla in the above debate. I don’t think Broockman and Kalla were confused about counterfactuals, and I think they were working carefully with the data. The bigger picture is that there’s only so much we can learn from head-to-head general election polls at this point. Even in this era of polarization, you can’t completely trust what voters say, this early in the election. Broockman and Kalla were not saying that the polls showed Sanders was less electable than Biden; what they were doing was pointing out weaknesses in the argument claiming that polls showed Sanders being more electable than Biden. There are reasons to think that Sanders is less electable than Biden, and there are reasons to think that Biden is less electable than Sanders. Broockman and Kalla are pointing out difficulties in naive interpretation of the head-to-head polls.

In Ackerman’s defense, I can understand his annoyance with headlines such as “Sanders . . . would need a youth turnout miracle” or “New Survey Suggests Bernie Can Win Only With Enormous Youth Turnout.” Broockman and Kalla’s study never suggested that Bernie could win only with enormous youth turnout. Ackerman’s mistake, perhaps, was to assume that their study made that claim, and then go backward looking to see what went wrong. Once we see the result as a mapping from data and assumptions to predictions, the whole thing makes more sense. That said, I have not looked at these claims with great care. As of now, what I’m saying is that Broockman and Kalla’s reasoning made sense to me. You can feel free to read all the above-linked documents and fill in the steps yourself.

12 thoughts on “Bernie electability update

  1. They lost me at “Decades of evidence from academic studies suggests that more moderate nominees tend to perform better in general elections than more ideologically extreme nominees.” That postulates that Sanders is extreme, which may fit some political narrative, but is quite funny to foreigners such as myself. Furthermore, would that imply that Trump is moderate?

    • If anything that postulates that Sanders is “more ideologically extreme” than Biden, not extreme. Saying that Sanders is more on the left than Biden doesn’t seem unfounded to the foreigner I am (although we could spend quite some time arguing over a definition of “left”)! As for the Trump case, the “tend to” is there for a reason.

      • Have been looking at exit polls lately. Am intrigued that the ideology question has the choices “very liberal” “somewhat liberal” “moderate” and “conservative”. Why isn’t it “very liberal” “somewhat liberal” “somewhat conservative” and “very conservative”?
        The way it is defined seems to say there are two degrees of liberal, one of moderate and one of conservative. And why is a somewhat liberal person not moderate while a somewhat conservative person is moderate?
        Has the wording evolved over time? Anyone knows?

        • Agreed: They need to define their terms clearly. To me, defining one’s terms is one of the first and most important thing to do in any scientific (or even somewhat scientific) paper. Without this, there are just words, not rational arguments.

        • It’s an unbalanced likert scale, presumeably used because there’s very few dem primary voters who identify as very conservative, whereas making a distinction on the leftwing end is more useful. A different scale would be used in the general.

  2. I haven’t followed this closely, but Sanders supporter Matt Bruenig makes what sounds like a fair point on Twitter (I know, I know):

    His tweets:
    It’s incredibly unbelievable to me that [B&K] consistently conflated counterfactual turnout advantage with a sequential turnout increase over and over and over again and then said “ah if you close read, we never said that.” Literally everyone read it that way!

    It is hard to believe that they did the conflation and then subsequently made no effort to correct that interpretation even as it popped up in every mainstream outlet. Only to finally after being pushed go “yeah we didn’t meant to imply that.”

    • Kyle:

      I dunno about that. I agree that Broockman and Kalla did not give their paper any pro-Senders spin, but I felt that their original article was pretty careful in its conclusion. Various reports of their study were not so scrupulous (for example, “New Survey Suggests Bernie Can Win Only With Enormous Youth Turnout”).

      I can see how Bruenig could be annoyed. I agree with Bruenig that Sanders very well could be a strong general election candidate conditional on receiving the nomination. But the Broockman and Kalla paper does not say otherwise. It just says that the existing head-to-head polls of various Democrats vs. Trump don’t clearly imply that Sanders is a stronger general election candidate than the other Democratic options. Their paper does not imply that Sanders could only win with a “youth turnout miracle.” The headlines, though, that’s another story.

      • One point in the BK paper (and in the authors’ Vox article) which might mislead readers is the box in their Figure 8 with the text “To believe Sanders is as electable as the moderate candidates, his nomination must increase turnout by 30% over 2016”. Another point is the abstract of their paper, which says “we find that Sanders performs similarly to more moderate Democrats against Trump in our survey only when assuming that (1) young people vote at much higher rates than they usually do”.

        When I read the Vox article and skimmed their paper, I certainly thought their claim was that Sanders needs the *level* of youth turnout to be 11pts higher than it was in 2016. (In particular, I missed their Table A6 in the appendix, which shows that self-reported intent to turn out among their survey respondents is much higher *for all age groups* than actual turnout, with or without Sanders on the ballot.) Judging by the reaction to the piece and to Ackerman’s post, I think most readers of the Vox piece and the original paper had the same impression. So I think BK are responsible for people interpreting it this way, although I agree if you read the fine print (as Ackerman and Bruenig did) you see that that’s not what they show. Of course it’s not uncommon (at least in my field, economics) for results to be presented in this slightly misleading way, and BK are not the worst offenders – but it still seems bad, especially when a study gets wider media coverage as this one did.

    • The distinction between counterfactual and sequential is important but maybe not as profound as implied. I mean, yes, a 11% change could be Biden -5, Bernie +6, but Obama -> Hillary was a +3, Kerry -> Obama was +3, Gore -> Kerry was +10.5 … An 11% counterfactual could well be Biden +10, Bernie +21, which would seem even more implausible than taking Sanders to 54%. Perhaps a lesson is that candidates don’t have as much of an impact on youth turnout as assumed.

Leave a Reply to Andrew Cancel reply

Your email address will not be published. Required fields are marked *