Four years ago Merlin Heidemanns and I worked with Elliott Morris at the Economist magazine to produce a state-by-state election forecast, combining national polls, state polls, economic and political “fundamentals,” and a hierarchical Bayesian model allowing for correlation among states, variation over time, and sampling and nonsamplng error of surveys. The model, which built off that of Lock and Gelman (2010), was described in this journal by Heidemanns et al. (2020), with further discussion of communication in Gelman et al. (2020). We fit the model in Stan (Carpenter et al., 2017), and our forecast updated daily as polls came in during the summer and fall and, with some hiccups, it performed reasonably well, albeit with some concerns regarding the quantification of uncertainty (Gelman, 2020), issues that arose with poll-based forecasts more generally (Gelman, 2021).
This year, Ben Goodrich, Geonhee Han, and I accepted the invitation of Dan Rosenheck of the Economist to help with their 2024 forecast. We started with the code from 2020, which we altered in several ways, including: (i) improving the fundamentals-based model to better account for the declining importance of the economy as a predictive factor in an increasingly polarized electorate; (ii) more carefully estimating the state-level correlations of polling errors and time trends in opinions; (iii) accounting for more nonsampling error in polling. As before, we checked our model by fitting it to data from the 2012, 2016, and 2020 campaigns along with existing polls from 2024, to check that it produced inferences that seemed reasonable given our current political understanding.
It might seem silly to check a model by comparing its inferences to reasonable expectations—if we knew what to expect, what is the purpose of the model at all?—but there are two reasons why this procedure seems reasonable to us. First, we are forecasting a multivariate outcome—50 state elections—and it require a lot of care to construct a full forecast with all its correlations. Second, we are constructing a sort of robot–a forecast that should be able to update itself over time as new polls and economic and political information arrive—so our checking is not just on the current forecast probabilities but also on how they develop over time. For example, if a new poll comes in from Ohio showing a stronger-than-expected support for the Democratic candidate, how much should this shift the forecast in Ohio and in other states, and how does that map to the probability of each candidate winning?
Our model currently gives the Republican candidate an expected 51% share of the national two-party vote and a 3/4 probability of winning the Electoral College (Economist, 2024). With the current state of public opinion and the expected relative distribution of votes among the states, it makes sense that the Republicans have the Electoral College edge, and a probability of 75% expresses an appropriate uncertainty given the closeness of the polls and the possibility of large polling errors and national swings between now and November.
Here are a few possible failures that we anticipated with our forecast going forward:
– What if one candidate or another takes a solid lead in the national polls? It would not take much for that party to get assigned a probability of 90% or more of winning–but then what if there is a big swing in the other direction, leading to that candidate’s win probability going below 10%? A probabilistic forecast should be a martingale—that is, if the forecast at time t of a certain future event has a probability of X(t), then E(X(t+s)), given all information available at time t, should be equal to X(t). So a swing in predicted probability from 90% to 10%, while possible, should be very unlikely, and a forecasting procedure that regularly shows such swings has problems (Taleb, 2018). We do not expect this to happen, but it could! Polling has been very steady during the past several election campaigns, but large swings were common in decades past (Gelman and King, 1993). The relevant parameter in our model is the standard deviation of the random walk of national vote preference over time. When implementing our model for the Economist, we set this scale to a value that seemed high enough to allow for plausible changes during the half year leading up to the election while still allowing informative inferences during those early months. But large enough variation over time could break this model and yield overconfident predictions.
– What about third parties? Following our practice in previous elections, we model preferences for the Democrat and the Republican, ignoring other candidates, which has seemed reasonable given that no third-party nominee has won any states since 1968. For awhile, though, RFK Jr. appeared to be a strong alternative to Biden and Trump, which could affect our forecast directly if Kennedy were to win any states and indirectly to the extent that changes in his support were to go unevenly to the major-party candidates. Presumably other minor parties won’t matter much, at least not compared to 2016, when the Libertarian and Green candidates did not win many votes despite widespread discontent with the options of Clinton and Trump.
– Actuarial concerns. Biden and Trump are both around 80 years old, with a nontrivial risk of death or disability between now and election day. What happens if one or the other candidate needs to be replaced? Even before the recent presidential debate, this was a vigorously-discussed topic, with pundits arguing that both parties were hobbled by weak candidates; see Gelman (2024). We did not have anything on this on our model, implicitly assuming that any replacement candidate would do about as well as the existing nominees. Ever since Rosenstone (1983), there has been a consensus in political science that candidates don’t matter so much for presidential voting, except that there is a slight advantage to political moderation. Given that most prominent alternatives within their parties are no more politically moderate than Biden or Trump, it seemed safe to not worry about specific candidate effects. That said, if, as it now seems likely, Biden is replaced on the Democratic ticket, we could well see some fluctuation in the polls beyond what might be expected from our default time-series model.
– Concerns specific to 2024. This is the first presidential election where either major-party candidate has been convicted of a felony, and the first since 1984 where there are serious concerns about either candidate’s mental deterioration. Pundits have also noted the unusual disconnect between relatively strong economic performance and the president’s low approval ratings. Another noteworthy feature, with effects already apparent in the 2022 midterm elections, has been a series of controversial Supreme Court decisions on issues ranging from abortion to presidential immunity. On the other hand, other recent campaigns have had historically unique features: the 2020 election was complicated by covid, early voting, two already elderly candidates, and justified concerns that one of these candidates would not accept the election outcome; and the three elections before that had the first African-American, Mormon, and female nominees, all of which might seem commonplace today, but at the time many people polled expressed resistance to voting for candidates with these attributes. This is not to say that it is a bad idea to adjust for what we can, just that we would hope our existing error terms to capture some of the unexpected. The Supreme Court issue is related to concerns about partisan balance, another tricky feature this year, with both houses of congress up for grabs.
– Polling errors were major concerns in 2016 and 2020. What about 2024? It’s hard to say. Our model allows for systematic errors at the national and state level. A study of state-level polling errors since 2000 found a positive correlation among successive elections—that is, if state polls are biased toward the Republicans or Democrats one year, they are likely to have a similar bias in the next election. Our model does not include this autocorrelation (because we assume that pollsters are trying to correct for such biases), so we may be leaving some information on the table. We hope that a reasonable range of possible polling bias is included in our predictive uncertainties.
Traditionally, the general election campaign is said to begin on Labor Day, after the two parties’ nominating conventions. This year, neither party’s candidates faced serious primary challenges, the two candidates appeared to be set in the spring, and observers were anticipating a long slog through November. Recently we have seen two shocks—Trump’s felony conviction and subsequent erratic performance in campaign events, and concerns about Biden’s age culminating in his calamitous debate performance—and here we are at the beginning of the summer with a new and potentially volatile race. In the modern era of extreme political polarization, we expect our state and national forecasts to still be reasonable, but ultimately they are conditional on model assumptions, hence the importance of transparency in methods and data.
References
Bob Carpenter, Andrew Gelman, Matt Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell (2017), Stan: A probabilistic programming language. Journal of Statistical Software 76 (1).
The Economist (2024). US Election 2024. Accessed 3 Jul. https://www.economist.com/interactive/us-2024-election/prediction-model/president
Andrew Gelman (2020). Concerns with our Economist election forecast. Statistical Modeling, Causal Inference, and Social Science, 28 Oct. https://statmodeling.stat.columbia.edu/2020/10/28/concerns-with-our-economist-election-forecast/
Andrew Gelman (2021). Failure and success in political polling and election forecasting. Statistics and Public Policy 8, 67-72.
Andrew Gelman (2024). How would the election turn out if Biden or Trump were replaced by a different candidate? Statistical Modeling, Causal Inference, and Social Science, 12 Jun. https://statmodeling.stat.columbia.edu/2024/06/12/how-would-the-election-turn-out-if-biden-or-trump-were-not-running/
Andrew Gelman, Jessica Hullman, Christopher Wlezien, and G. Elliott Morris (2020). Information, incentives, and goals in election forecasts. Judgment and Decision Making 15, 863-880.
Andrew Gelman and Gary King (1993). Why are American Presidential election campaign polls so variable when votes are so predictable? British Journal of Political Science 23, 409-451.
Merlin Heidemanns, Andrew Gelman, and G. Elliott Morris (2020). An updated dynamic Bayesian forecasting model for the 2020 election. Harvard Data Science Review 2 (4).
Kari Lock and Andrew Gelman (2010). Bayesian combination of state polls and election forecasts. Political Analysis 18, 337-348.
Steven J. Rosenstone (1983). Forecasting Presidential Elections. Yale University Press.
Nassim N. Taleb (2018). Election predictions as martingales: An arbitrage approach. Quantitative Finance 18, 1-5.
Or, you could just use Allan Lichtman’s Keys to the White House. He’s been correct about every election, and you don’t even need to know statistics….
Yup!
Could something like the Keys to the White House be useful? 10 or so true/false questions that could predict the election outcomes?
Otoh:
No, I do not think it is useful, for reasons explained at the above-linked post. The short answers are:
(a) Some elections are not close at all and any prediction method will get them right,
(b) Some elections are so close that for a prediction method to pick the winner is just chance, like picking a coin flip,
(c) There is information in the vote margin that is being thrown away if you just try to predict the winner; additional information is being thrown away by using true/false questions.
Andrew, you say “Ever since Rosenstone (1983), there has been a consensus in political science that candidates don’t matter so much for presidential voting, except that there is a slight advantage to political moderation”, but that outlook seems barely supportable now. It seems to me — or at, least, it seems plausible to me — that for decades each party’s candidates were pretty much the same in many ways, so those elections convey very little information about what might happen if candidates were wildly different from those norms. Among other things, most candidates spoke like politicians; kept most of their lies within a standard range of simplifications, exaggerations, and small lies; didn’t make promises that were _obviously_ false; paid at least lip service to veterans and in any case did not mock them.
If you’re choosing candidates from a pool of very similar candidates then yeah, it probably doesn’t matter who you pick. Indeed, I’ve been saying for months that I wish Biden would step aside and support any of the plausible replacements (which, in my view, includes Harris, Newsom, Whitmer, Beshear, Buttegeig, Booker, and several others); it’s not that they’re exactly interchangeable, but they’re all pretty much standard presidential candidates in most dimensions.
But if you’ve got one or more candidates who don’t fit the mold then the candidate matters a lot. I think Trump disproves the old view, no?
Phil:
Could be. On the other hand, Trump did about as well in 2016 and 2020 as might have been expected from the fundamentals, despite all his unusual features as a candidate, so that supports or at least does not contradict the idea that the candidate doesn’t matter much, at least when it comes to the party’s share of the national two-party vote.
Ah, yeah, I was interpreting “candidate doesn’t matter much” far too broadly, thinking of policies, probability of election violence, stuff like that.
Yankees fans aren’t going to root for the Red Sox, and vice versa, no matter who is on which team.
Oh, yes, candidates can matter a lot for policy and execution; I was just talking about votes in the upcoming general election.
> improving the fundamentals-based model to better account for the declining importance of the economy as a predictive factor in an increasingly polarized electorate
The non-specificity of the word “fundamentals” always bugs me.
The abstract from Merlin’s 2020 paper makes me much happier cuz I can see more clearly what’s going on:
> a forecast based on historically relevant economic and political factors such as personal income growth, presidential approval, and incumbency
I’m glad to see that the model is back.
It’s interesting that 538’s 2024 election model (“based on [Morris’s] past forecasting work”) gives a quite different prediction. It was 50/50 before the debate and it has moved only a bit to 53/47. (I’ve not checked in detail where the difference may be coming from.)
> The relevant parameter in our model is the standard deviation of the random walk of national vote preference over time.
A noticeable change from 2020 is that there is no longer a “election-day prediction” in the “two party popular vote” chart. For individual states one can find a vote margin (median) prediction – which may be wider or narrower than the current “forecast (?) vote intention” in the “popular vote” chart. However, there is nothing similar in the “National forecast” section as far as I can see – there is a “poll average” but no prediction to be found.
I never quite understood what was the meaning of the projection in the old chart anyway. The width of the forecast bands is (almost?) constant but one would expect uncertainty to increase with time. (Also, the “noise” in the projection seems meaningless but that’s a minor issue.)
https://www.researchgate.net/figure/Our-election-forecast-for-The-Economist-on-the-day-of-its-release-in-June-2020-Before_fig1_356841905
> (I’ve not checked in detail where the difference may be coming from.)
It seems that all the “competitive” states are now on Trump’s side (by a margin of at least 2%) for The Economist while FiveThirtyEight has Democrats winning Pennsylvania, Wisconsin and Michigan (0.5%, 1.1% and 1.7% forecasted margins respectively). The closest races for The Economist are Minnesota, New Hampshire and Nebraska 2nd (all for Biden with a 1% margin).
FiveThirtyEight popular vote forecast seems remarkably stable despite the movement in polls: https://imgur.com/a/KOheMJr
Andrew,
could you explain how you model the possibility of large polling errors? Specifically, I do not understand what you mean here: “Our model allows for systematic errors at the national and state level.”
You write:
– Polling errors were major concerns in 2016 and 2020. What about 2024? It’s hard to say. Our model allows for systematic errors at the national and state level. A study of state-level polling errors since 2000 found a positive correlation among successive elections—that is, if state polls are biased toward the Republicans or Democrats one year, they are likely to have a similar bias in the next election. Our model does not include this autocorrelation (because we assume that pollsters are trying to correct for such biases), so we may be leaving some information on the table. We hope that a reasonable range of possible polling bias is included in our predictive uncertainties.
Is what you mean by “Allowing for systematic errors at the national and state level.” the autocorrelation of biases? But then you say that your model does not include this autocorrelation, so I am kind of confused.
Huan:
Our model includes an error term representing polling bias. There is a national error term and state error terms—or maybe just state error terms, but with a high enough correlation that they induce a potentially large national error. The error terms have prior mean 0 and prior sd high enough that, even on election day, there is still something like 2.5 percentage points posterior uncertainty in the national average polling bias.
Our model includes autocorrelation of these biases between states, cross-sectionally. What our model does not include is autocorrelation of the biases within a state, from one election to the next.
Andrew, thank you.
Let me see if I understand you: You added noise, that represents systematic polling error, to the predictions in every state. This noise has mean 0 and some relatively high sd. If these random variables were not autocorrelated across states it would be very unlikely that the systematic errors affect the national polling average in a meaningful way, so you specify some autocorrelation across states. How did you choose the amount of autocorrelation?
There is one more thing I do not understand: You think temporal autocorrelation of biases will be 0, since pollsters will correct their biases. But you also think there might be correlation across states. What could make both statements true at the same time? I imagine your reasoning is something like this: The presidential debate was more terrible for Mr. Biden. Assume that polls find that more people support Trump now. However pollsters have to estimate the influence of the debate not for today but for the election in November. Since pollsters in different states use similar methods they all might be getting this estimation wrong, so there is correlation across states.
Huan:
No, that’s not what we did. We did not add noise to a prediction. Rather, we included a systematic error term into the model, and our predictions account for this variation.
Also, no, we don’t think temporal autocorrelation of biases is zero. We found that statewide polling biases were correlated from one election to the next; that is, if a state shows a polling bias in favor of the Republicans one election, it’s likely to show this bias the next election too. We did not include that in the model, however.
Here’s a description of our model from 2020. I think that some of our Stan code is somewhere on Github too.
Interesting to reread this post linked from your post about replacement candidates. That was presenting the arguments of Nate Silver & Paul Campos, and I’d already heard Silver give his more recent update on the odds, but it was only after reading that I decided to check out Campos’ most recent posts at LGM. It seems he didn’t make a large update right after the debate (“throwing out three and a half years of governing, not to mention a major party’s entire primary process, because of a bad night under the TV lights, is raising theater criticism to almost theological status”), but over time has come much closer to Silver’s position. One could say that he’s updating on the same kind of observational evidence about Biden that he previously used with respect to Trump, but as noted the debate didn’t initially change his opinion that much. Instead he’s looking at the opinions of others (“I spoke to four prominent neurologists directly, and got the opinion of maybe 20 of their colleagues indirectly, via the comments of those I spoke to regarding what those colleagues were thinking, and my views changed dramatically“). His personal conversations aren’t public information in the way polls are (he can report that he had them, but it’s still akin to private info), but he adds that “the level of sheer denial in the commentariat here has taken me aback”. That could be more a comment on the particular commentariat over there, but it raises the question of at what point it would be considered “denial” for a layman to say Biden wasn’t more vulnerable to a charge of dementia (among voters), the topic of my first link, or just generally considered too old for the presidency. I know he’s behind in polls now, but it’s my understanding that he was behind before the debate (even if he’s slipped further since then).
Wonks:
From the outside, it seems like both candidates are severely mentally incapacitated. The Democratic party seems slightly better than the Republican party at addressing this. Nate advised both parties to replace their nominees with younger, more capable alternatives, but it seems so unlikely that the Republicans would do this, that most of the focus seems to be on the Democrats.
A bit too much focus on age for me (full disclosure – I am getting old). Frankly, I see age as having little relevance to the presidency. You say “both candidates are severely mentally incapacitated” but I don’t think that is an accurate description. Biden certainly shows signs of age and perhaps mental decline, but I don’t see why that should impede his ability to be president given that he can surround himself with many competent people. Trump is more an issue of ethical incapacitation than mental, in my opinion – and that cannot be compensated for by who he surrounds himself with. The way that age matters in this election – in my view – is that there is a strong public perception that it matters and the unfortunate reality that our political system has not been able to produce younger candidates. The Democrats, in particular, should have been able to produce a candidate who can beat a weak Republican opponent.
When I say Trump is a weak candidate, I am not ignoring the strong, almost religious support he gets from his base. But he is a seriously flawed candidate and would have a hard time outside his base if the Democrats put up a candidate without the obvious limitations of Biden. And I think those “obvious limitations” are mostly due to public perceptions – misguided in my view. Many people seem to believe it is somehow very important that Biden not suffer any cognitive decline. Why is that so important? Cognitive decline has little to do with the nuclear codes or physical stamina or even golf handicaps. Ethical deficiencies have much to do with ability to be a good president.
At my age (I’m not as old as either candidate, but not that far behind), I suffer from some mental (shows up as memory lapses) and physical (so much for my golf handicap) decline. I don’t believe it hinders my ability to be an effective teacher or researcher. I think the focus on Biden’s (or Trump’s) cognitive ability is ageism and not particularly sensible.
Dale, suppose there’s a crisis, something along the lines of the cuban missile crisis, and suppose Biden has many good advisors, and they advise him, but it’s extremely stressful and he gets agitated and can’t follow the arguments and soforth. That’s a really bad thing. I’ve seen my wife’s mother decline to the point where some evenings she just isn’t there at all, doesn’t know what is going on around her. She’s about as old as Biden will be at the end of his term.
Ageism is an appropriate accusation when the person is mentally fit, when they express bouts of confusion and soforth, it’s no longer ageism its concern for competence. I don’t want my wife’s mother making investment decisions, she’s already been scammed once over the phone. My concern is not ageism, it’s legitimate concern that she could be easily scammed by phone as evidenced by the fact that… she HAS been already. My grandfather died at 92, by that point he had been scammed out of easily hundreds of thousands of dollars (by various people, including his employees at a business he was still in charge of). He was mentally fit at say 82, he wasn’t at 92. Somewhere between those two ages he went downhill enough that he was an easy target for scams, and he and his wife refused to accept help from family. But if we were helping him, there’d be almost nothing for him to do, he got to the point where he wasn’t sure what a corkscrew was for.
In my view there’s no “fixed age” we should set, but we should set a maximum age to serve in the government as the age where your life expectancy drops below 16 years (4 terms). That means you’ve got on average twice the needed life expectancy to not die in office while serving 2 terms as president.
Today that would be about 69 years old according to the total population life tables for 2021 https://www.cdc.gov/nchs/products/life_tables.htm
In the past that would have been a lower number. And maybe in the future we will extend people’s lives a bit, and it’ll rise… but for now I think that’s a fairly reasonable criterion.
Daniel
Your cherry-picked concern is real, but so limited. If we want to start imagining circumstances (such as the Cuban missile crisis) where cognitive decline could be an issue, then we should recognize that competence for decision making is multidimensional. It includes physical stamina, memory, judgement, wisdom, rationality, intelligence, and many other things. There are certainly medical conditions that warrant special attention, and cognitive impairment/dementia is one of these. If it is serious enough, then I would agree that it is grounds to not be eligible to be president. But it is a continuum, and I don’t think that “any” evidence of it is sufficient for elimination.
My parents were both around 90 years old and both were mildly crazy in their own ways. When they voted (absentee ballots), they would fill out the ballots together and check each other’s ballot to make sure they did what they intended. I wouldn’t say either were qualified to be president (though the bar is getting continually lower), but my point is that many people make adjustments to their physical and mental conditions. And age is but one dimension that matters.
I really don’t see Biden exhibiting behavior sufficient to say he should not be allowed to be president. Nor would I support any fixed age limit. Nor do I think that expected survival until the next election makes sense as a criterion – although it is certainly a dimension that voters should pay attention to. But to single it out as a bright line for eligibility is to ignore all those other dimensions that should also have bright eligibility lines.
I do think Biden should withdraw from the race. Not because I believe him incapable of being president – and I think he is far more capable than his opponent. But he carries unnecessary baggage that diverts attention from Trump’s many deficiencies, and because it speaks ill to the future of American politics. If the parties are unable to put up candidates better than these two, that is a sorry state of affairs. This week’s Economist has a number of stories about American elections that I think are quite insightful. And the fact that American leaders are considerably older than those in other countries despite the fact that our population is younger than many of those is striking and not admirable.
Yes, Nate says both parties would be better off with a replacement (the subject of the post you linked above). But he’s also saying that Biden is currently behind, which is why they’re more in need of a replacement. If they replace Biden with someone expected to win, then the Republicans will be in the position of needing to replace Trump in order to have the better odds. He won the overwhelming majority of delegates in the primaries, and the convention is next week, so his nomination would appear to be a done-deal right now and he’d have to be convinced to drop out (there’s no 25th Amendment for candidates not currently in office). Since the Democrats are having their convention later, they could wait for Trump to receive the nomination and then replace Biden with someone else. But here I am speaking of “the Democrats” when there’s not really any such organization with agency, but instead just a bunch of individuals with partisan commitments and no strong party hierarchy.
Do you think the observed polling errors from this cycle’s state-level primaries could be useful in estimating at least the direction (if not the magnitude) of the systematic error? There were some very large errors in the Trump vs. Haley polls – in the 20 to 30 points range – and most overestimated Trump’s margin of victory. Conversely, the Democratic primary polls consistently underestimated Biden’s margin of victory. These were pretty much ignored in the press, I suspect because everyone knew who would win.
We expect primary polling errors to be larger than general election polling errors, but is their direction still informative? Because their direction is consistent with your previous work on using enthusiasm to explain differential non-response (e.g. the “convention boost”). Trump voters this year are very enthusiastic – their guy was supposed to be dead and buried but now he’s back with a vengance! And anti-Trump voters are dejected, for the same reason. It makes sense that Haley voters and Biden voters would be relatively less inclined to talk to pollsters this time around, and Trump voters relatively more inclined.
This is just speculation on my part, but it fits with the observed state-level primary polling errors. And it would suggest pro-Trump bias this time around.
(I haven’t been able to find any polls that release detailed unweighted results, otherwise I’d be looking to see if raw counts of Biden 2020 voters have been getting smaller relative to Trump 2020 voters over the past couple years.)
I am British, perhaps that colours my opinion because we have had meaningful third parties in every general election since at least 1983.
Why do you ignore third parties? In Michigan 2016 the Libertarian Party won 3.6% (172k) and the Greens 1% (51k). Neither are anywhere near close enough to win the state but Trump had a winning margin of just 11k so it seems plausible that those voters decisions could affect the outcome in very close races (like Reform or UKIP have done here in Britain).
Is it the time to integrate to the model, lack of data, that these “vote sapping” parties don’t affect the model until they have a realistic chance of winning a state, or something else?