There’s some confusion regarding jumps in election forecasts. New information is coming in every day, so it makes sense that forecasts change too. But they don’t change very much. Each new piece of information tells you only a little bit.
Here’s something that I don’t think is so well understood. A 10% swing in win probability corresponds to about a 0.4% swing in predicted vote. For example, suppose Harris is predicted to win 51.6% of the national two-party vote and that, the way the states line up, this would correspond to a 50/50 chance of winning in the electoral college. Then a quick calculation shows that, approximately, a vote share prediction of 52% would yield a 60/40 chance of winning.
A shift of 0.4 percentage points in vote share is small—indeed, undetectable in any poll or even in any aggregation of polls, given what we know about nonsampling error. But it could be consequential. The point is that, sure, different forecasters or bettors or whoever could disagree by this tiny 0.4% of vote share (maybe I’ll make that more dramatic by saying, this tiny 0.004 proportion of the vote), and this will show up roughly as a 10% difference in win probability. That’s just the way it is. Here’s the math.
So what about the snippet above from today’s Economist? They write, “Of the eight pollsters who published polls of Pennsylvania, the average result was a one-point lead for her, compared with a one-point lead for Donald Trump in yesterday’s forecast”—that’s a 2-percentage-point difference in vote differential, i.e. a 1-percentage-point difference in vote share. We have roughly national partisan swing, so . . . a 1% swing, why didn’t that shift the win probability by 25%, which would’ve brought Harris to a roughly 75% probability? The reason is that the model doesn’t just take the latest polls; it also combines with all the other information already out there. And, from a Bayesian point of view, this new information is shifting the expected vote share by much less than 1%.
Prediction markets
I also recommend you read this post from Rajiv Sethi on the limits of arbitrage, where he explains what’s going on here:
So, when you look at the recent swing in prediction markets of approximately 10% in win probability, that roughly corresponds to a 0.4% swing in the candidates’ predicted vote share. Again, 0.4% is not a lot; it’s not unreasonable to see this sort of change in probabilities.
How polls are reported now, how they were reported in the past
Decades ago, polling results were reported as vote shares, or vote differentials, not as win probabilities. Then the uncertainty was more clear. If a poll gave a candidate 52% of the two-party vote, it was well understood that this was within the margin of error of a tie, also it was well understood that public opinion could shift, and that polls could be off.
I do think that in many ways we were better off thinking about public opinion and elections in terms of vote shares rather than win probabilities. That said, I do think probabilistic forecasts have value, even if sometimes the value is to express how uncertain we are.
More explanations
Here are my answers to some questions from the news media about election forecasting.
Have you seen this: https://www.theguardian.com/us-news/2024/nov/02/what-polls-mean-so-far-trump-harris-election-voters
Thoughts?
What are the probabilities that in 2020 the outcome could have swung based on just 1% of the vote in PA, or 0.2% in Georgia?
Adede:
It’s hard to say. A good poll will adjust for many factors, including partisanship of respondents. This should give stable results, as in our Xbox surveys discussed here.
From the Guardian article:
““Some of the tools pollsters are using in 2024 to address the polling problems of 2020, such as weighting by partisanship, past vote or other factors, may be flattening out the differences and reducing the variation in reported poll results,” they write.”
I mentioned variance loss in polling in a previous thread. Even if you start with good polls, adding in a bunch of adjusted polls waters down the signal. Adjustments that pull the actual results towards the expected results, when averaged over the entire sample, result in two things, a desired increase in “n,” and an undesired (or is it?) incremental variance loss.
The reason is that the model doesn’t just take the latest polls;
Why not?
If previous polls were done by the same pollster with the same methodology – why would p their revious polls be still included in an aggregation. Presumably, their previous polls are based obsolete data that measures a situation that no longer exists.
…it also combines with all the other information already out there.
I could see where maybe it would makes sense to include older polls by the same pollsters if it were significantly down-weighted. I mean maybe the most recent poll is for some unknown reason a real outlier and so you have to protect against that to some extent. I guess polls are given less and less weight in an aggregation as you go back in time. But stillzsswms to me if you have confidence in the polling methodology you should more or less reject the previous polls and only aggregate the most recent polls by all the pollsters (weighted relative to each other to some degree based on past performance – again with the relevance of past performance diminishing as you go back in time to less similar circumstances).
Joshua:
It’s not “an aggregation”; it’s a statistical model. Each poll provides some information. The model is described (but not in complete detail) here.
Perhaps it will also be helpful to read this post from 2012, A Non-random Walk Down Campaign Street.
I feel like random walk and means reversion are both derived models from more fundamental processes. One such process might be that voter preferences are reasonably stable and the various polls are measures designed to uncover what those preferences are (partially to measure nonresponse, partially to measure turnout, and partially the candidate preference). A very different view would be that voter preferences are shifting and the polls are fairly consistent attempts to keep up with these shifting preferences (with associated errors over time and over geography). With a mixture of these two, it seems to me that polls are imperfect measures of a non-stationary process – which seems more complex than either the random walk or means reversion theories.
So, in relation to Joshua’s post, it seems like his statement about the latest polls aggregating prior results would apply more strongly when voter preferences are fixed over time. Each poll would be revealing information about these preferences, but repeated polling by the same organizations should mean that the latest polls would have much higher weight than prior polls. To the extent that voter preferences are shifting, then each poll provides imperfect information about preferences at that point in time – if these imperfections can be modeled, then perhaps all the polls are useful. It still seems to me that the more recent polls should receive relatively greater weight.
I’d propose Anoneuoid’s common reaction to such issues: measuring past performance seems like it should be key to evaluating such models. Each election has a mixture of shifting preferences as well as imperfect measurements. We have lots of elections and lots of polls, so there is plenty of data to look at. But don’t we have to make strong assumptions about stability of the underlying processes in order to use that data? What if voter preferences are more stable in some elections than in others? Wouldn’t that undermine the use of such data? In particular, for presidential candidates, some elections have more intense personalities than others, and this could make voter preferences more stable in some elections than others. It starts to seem like a lot of hand-waving explanations could be used to account for why some polls have been more accurate than others.
I’m sure these are the kind of things you have thought about and written about, but it seems intractable to me unless voter preferences are reasonably stable. If they are, then why is so much money spent on campaigns? And, if preferences are not stable (at least at the margin), then most of the people being polled are largely irrelevant to what we want to understand (is that related to the asymmetry between the large swings in probability associated with much smaller swings in votes?).
I don’t think I’ve been particularly clear – combining polling results over time seems like an unusually thorny problem. My objections to polling stem from my belief that the polls themselves influence voter preferences (they certainly seem to influence campaign efforts, so that would influence voters even if the polls did not directly do so).
Andrew,
A short personal story: I was canvassing for Kamala Harris in Lebanon, Pa. yesterday. On the long bus ride back to the Upper West Side, I announced that The Economist election forecast had Harris at a 52% probability of winning the race. Cheers erupted from the entire bus. ( I did not mention the 538 or Silver election forecasts, which had lower probabilities of Harris winning.)
Let me start by saying that I am all for democracy. It is one of the principles I am most committed to, of all my values. But I am repulsed by the idea that if a handful of people in one place vote in a slightly different way, the political outcomes will be very different. I am not just talking about the US presidential election(s), but also referenda like the Brexit vote, the Scottish independence vote, and the ubiquitous referenda in Switzerland. I find it crazy: As a rule, there is no option for moderation or a middle way. The options are to leave the EU/UK or keep everything as it is (the Brexit vote and the Scottish independence referendum); to write environmental protection into the constitution or not to pass any legislation to protect the environment (the Swiss biodiversity referendum). The same goes for the US elections. Of course, some outcomes have to be binary: either Ms Harris or Mr Trump will be the next POTUS. But I have noticed that in several democratic countries there are big swings in policy, even though the electorate has changed very little. I love democracy, but this is something that irks me and makes me wonder if and how a democratic system could be designed to be more stable.
Raphael –
…makes me wonder if and how a democratic system could be designed to be more stable.
Ranked choice voting and proportional representation systems that encourage coalitions would seem (to me) to result in less volatility and less sharply divisive political struggles.
AI tells there’s a body of research that supports my conjecture, although of course that’s far from authoritative.
Some argue that our first to the pole system creates stability because there’s always so much gridlock that government doesn’t achieve much. Well, looks like there’s about to be a period where that really gets put to the test. Of course where’s always a huge difference between campaign rhetoric and what presidents actually achieve (or even try to achieve) once in office, but it seems to me that Trump/Vance/Musk/RFKjr are promising change on a scale that dwarfs anything I’ve seen before. Given that at the most extreme range of the uncertainty at least 45% of the voting public are opposed to (at least a sizeable portion of) those proposals – yeah, the system seems a bit nuts at this point.
Methinks instead of looking to the possibility of a 3rd party, it’s pretty clear a structural change would be a more feasible way to improve the political environment and net better outcomes.
>>Trump/Vance/Musk/RFKjr are promising change on a scale that dwarfs anything I’ve seen before.
While I don’t expect Trump to win, I think radical change on a level the US hasn’t seen since about the 1960s-early 70s is becoming increasingly likely, though it may well take a decade or so to arrive. (I imagine redistricting after 2030 Census will radically change the US political balance, unless population trends change radically in the next 6 years.)
The gridlock setup can more or less work while the “conservative” side is
conservative in the sense of wishing to preserve the status quo – as long as the side that wants dramatic change cannot gain a sufficient majority to succeed, the system is stable and roughly half the people are happy with it. And if the parties are not too far apart, the losing side isn’t utterly disenchanted to the point of “let’s smash the system”.
But the divide in the US has gotten so deep, and very few really want the status quo on either side, that this is no longer working.
I’ve always thought the US handles some things like that well by having different standards for ballot measures/laws and constitutional rights. Rights need strong majorities to institute, and once in, require strong majorities to remove.
Despite shifts on popular opinion, 1st and 2nd amendments aren’t remotely threatened.
Something like entering the EU would I believe require a constitutional amendment here, so 51/49 just wouldn’t cut it. That’s why I’d be receptive to some things like say marriage equality being explicitly added to the Constitution, so that any potential backsliding over another century or two won’t have the laws overturned by a small majority.
>>I’ve always thought the US handles some things like that well by having different standards for ballot measures/laws and constitutional rights.
I agree in theory, but the problem is a 5-4 Supreme Court decision can totally change the constitution in practice if not in theory… Which rather messes up the careful procedure for constitutional amendment. I think this was a problem introduced by basically copying over the 18th century British court system from a nation with an unwritten constitution, and worsened as generations passed – we started to see weird swings in constitutional interpretation post-Reconstruction (for example, at one point even state minimum wage laws had been found unconstitutional as infringing on the fundamental right to freedom of contract).
Nate is back at p hacking again: https://x.com/NateSilver538/status/1852915210845073445
Anon:
Good to see he’s still using Stata. Old school. Keep it real, Nate!
lol. I think I should reach such a point of zen. I still take him too seriously :(
Lol at the 9 significant figures
Mr Silver’s model, if I read it correctly, is a parsimonious model. It has one input and one constant. I would not call it p-hacking, although there is always a risk of forking paths, the omnipresent problem in data analysis. I gather from his comment that inflation (conditional on a few things) correlates with voting for Trump. He could also have produced a map showing inflation and predicted support for Trump, which would have been cleaner and shown the same message I think he wanted to convey.
I am not saying to leave it uncommented, this post can easily mislead the public as it may sound to them as if there is a causal relationship when there is only a correlation which may be spurious. (The possible spuriousness is acknowledged by stating the existence of confounding variables).
Anyway, I will take a statistician-turned-pundit any day over a zoologist-turned-homeschooling-lobbyist who abuses his offspring, just to put things into perspective.
> Lots of glib answers (in the zoologist’s case, supported by irrelevant Bible quotations; in the economist’s case, supported by irrelevant mathematical arguments), not much thought.
Ha! We need a greatest hits list of Andrew’s most memorable put downs.
I’m trying to reconcile the claim from this post:
“So, when you look at the recent swing in prediction markets of approximately 10% in win probability, that roughly corresponds to a 0.4% swing in the candidates’ predicted vote share. Again, 0.4% is not a lot; it’s not unreasonable to see this sort of change in probabilities.”
With this claim from the 2020 “Information, incentives, and goals in election forecasts” paper.
“If large opinion shifts are allowed with high probability, then there should be a correspondingly wide uncertainty in the vote share forecast a few months before the election, which in turn will lead to win probabilities closer to 50%.”
Is the difference here that we’re closer to the election? Or that 0.4 is a small shift in vote share, so 10% is not a large change in win probability? Still, it seems that if a series of polls close to election could swing things by 10 points, that should have been factored in previously. In other words, at least Kalshi and Polymarket should have been less bullish on Trump given the volatility of polling data.
William:
It’s possible that the prediction markets are overreacting to information, but maybe less so than before. I’m guessing that the popularity of probabilistic forecasts such as the Economist’s have grounded prediction markets and kept them from fluctuating too wildly.
Yes, and I was also indirectly trying to say that less volatility may be an advantage of models likes the Economist’s over prediction markets
Andrew,
> I do think that in many ways we were better off thinking about public opinion and elections in terms of vote shares rather than win probabilities.
Doesn’t the electoral college system mean that equivalent vote share posteriors can correspond to entirely different outcome probabilities? E.g., even if I were sure that a candidate’s vote share is going to be 50.5%, it could mean entirely different things politically depending on how those 50.5% are distributed across states.
Maybe a distribution over electoral college votes? It’d be more informative about uncertainties that a single win probability, but a have less ambiguous political interpretation. It’d probably also mean moving from a left-to-right scalar time series to say a top-to-bottom column of electoral college vote distributions, which is less familiar but would help show things like “we are more certain this week that it’s going to be close.”
Marcelo:
Swings are mostly national, so the national popular vote gets you most of the way there. But, sure, when we’re forecasting the election we do it at the state level, as we’ve discussed in many papers over the years:
[1993] Why are American Presidential election campaign polls so variable when votes are so predictable? {\em British Journal of Political Science} {\bf 23}, 409–451. (Andrew Gelman and Gary King)
[1993] Review of {\em Forecasting Elections}, by M. S. Lewis-Beck and T. W. Rice. {\em Public Opinion Quarterly} {\bf 57}, 119–121. (Andrew Gelman)
[1998] Estimating the probability of events that have never occurred: When is your vote decisive? {\em Journal of the American Statistical Association} {\bf 93}, 1–9. (Andrew Gelman, Gary King, and John Boscardin)
[2010] Bayesian combination of state polls and election forecasts. {\em Political Analysis} {\bf 18}, 337–348. (Kari Lock and Andrew Gelman)
[2010] What do we know at 7pm on election night? {\em Mathematics Magazine} {\bf 83}, 258–266. (Andrew Gelman and Nate Silver)
[2012] Estimating partisan bias of the electoral college under proposed changes in elector apportionment. {\em Statistics, Politics and Policy} {\bf 3}, 1–13. (Andrew C. Thomas, Andrew Gelman, Gary King, and Jonathan Katz)
[2020] An updated dynamic Bayesian forecasting model for the 2020 election. {\em Harvard Data Science Review} {\bf 2} (4). (Merlin Heidemanns, Andrew Gelman, and Elliott Morris)
[2020] Information, incentives, and goals in election forecasts. {\em Judgment and Decision Making} {\bf 15}, 863–880. (Andrew Gelman, Jessica Hullman, Christopher Wlezien, and George Elliott Morris)
[2024] Grappling with uncertainty in forecasting the 2024 U.S. presidential election. {\em Harvard Data Science Review} {\bf 6} (4). (Andrew Gelman, Ben Goodrich, and Geonhee Han)
The national popular vote calculation is enough to make the rough mapping of 10 percentage points in win probability to 0.4 percentage points of the vote.