Forecasting elections using prediction markets has a theoretical appeal, as people are betting their own money so are motivated to get things right. On the other hand there’s been concern about the thinness of the market, especially early on during the campaign. Thin markets can be easier to manipulate, also when there’s not much action on the bets, the betting odds can be noisy. There’s also a concern that bettors just follow the news, in which case the betting odds are just a kind of noisy news aggregator.
Ultimately the question of the accuracy of betting odds compared to fundamentals-based and polls-based forecasts is an empirical question. Or, to put it another way, the results of empirical analysis will inform our theoretical understanding.
A quick summary of my understanding of past empirical work on election prediction markets:
1. For major races, markets are not perfect but they generally give reasonable results.
2. Markets fail at edge cases, consistently giving unrealistically high probabilities of extremely unlikely events.
3. It’s difficult-to-impossible to compare different forecasting approaches because the uncertainties in the outcomes of different races in a national election are highly correlated; in this sense, a national election is giving you only one data point for evaluation.
The best thing I’ve read on the topic is this article by Rajiv Sethi et al., “Models, Markets, and Prediction Performance.”
What happened in 2022?
The recent election featured some strong pre-election hype on markets, along with the usual poll-based forecasts. This time the polls were very accurate on average, while the markets were a bit off, predicting a Republican wave that did not happen. I’d be inclined to attribute this to bettors following the conventional wisdom that there would be strong swing toward the out-party, which ultimately is a “fundamentals”-based argument that made a lot of sense a year ago or even six months ago but not so much in the current political environment with a highly partisan Supreme Court.
But I wanted to know what the experts thought, so I contacted two colleagues who study elections and prediction markets and sent them the following message:
Here are 4 stories for what happened in 2022:
1. Just bad luck. You can’t evaluate a prediction based on one random data point.
2. Overreaction to 2020. Polls overstated Democratic strength in the past election and bettors, like many journalists, did a mental adjustment and shifted the polls back.
3. Bettors in prediction markets have a distorted view of the world, on average, because they are more likely to consume conservative media sources such as Fox, 4chan, etc.
4. Prediction markets don’t really add information; bettors are regurgitating what they read in the news, and in 2022 the news media pundits were off.
What do you think?
The experts reply
David Rothschild:
– The comment section in PredictIt is dominated by the right, politically. Obviously comments are just a portion of traders, but this likely has some effect on some traders. This is especially important because PredictIt is constrained in trade size per person, so in some markets the price looks a little closer to the average trader than the marginal trader (i.e., very confident person cannot swoop in and correct biased market).
– Markets tend to converge towards polls late in the cycle, so while they provide information early in cycle and when information is breaking, final predictions in elections are heavily influenced by polls.
– Markets proved extremely good, faster than anything else, in incorporating information on Election Night.
Rajiv Sethi:
– Markets actually do very well early in the cycle, for example they had Dobbs beating Lake for AZ GOV in early August. But anecdotes like this aren’t evidence. I also feel that the reaction to the PA debate on social media and the markets was absurd – I must have seen hundreds of tweets saying that the Fetterman staff and family should never have allowed him to debate, etc. But he was perfectly capable of making the decision himself, and made a good call, that most people saw as courageous. But aside from PA I think the markets didn’t do badly, just the roll of the dice made them look bad this cycle.
From market fundamentalism to conspiracy theories
I was thinking about some of the above topics after reading this post by statistician Harry Crane, which ran a few days before the November 2022 elections:
When something doesn’t fit the official narrative: Regulate, legislate, or censor it out of existence. . . . It’s central to the Establishment’s strategy on a wide range of issues. Start looking and you’ll start to notice it just about everywhere. Here I focus on how it applies to elections. . . .
Who’s going to win the 2022 midterms? Specifically, which party will win control of the Senate?
According to the polls and pundits in the media, Democrats have the advantage. Voters are upset about Roe v. Wade. Democrats were 75% to win the Senate a couple weeks ago. Now it’s a toss up according to the forecasting website FiveThirtyEight.com.
But if you look at the prediction markets hosted at PredictIt — where savvy politicos risk real money on their opinions – you’ll see that the Republicans are 90% to win the House and 72% to win the Senate. . . .
So which is more accurate? As you’d probably expect, the markets are. . . .
Crane talked about how the prediction markets were favoring the Republican candidate from Jersey Pennsylvania:
Within a few moments after Fetterman opened his mouth for the first time [in their televised debate], Oz shot up to 65% and stayed near that price for the rest of the debate and the week following . . . FiveThirtyEight has Fetterman leading the entire time. We’ll know in a week or so which was more accurate at predicting the Pennsylvania Senatorial race outcome.
We’ll know in a week, indeed. Seriously, though, for the reasons discussed earlier in this post, we shouldn’t take one year’s failure as a reason to discard markets; we should just recognize that markets are human constructs which, for both theoretical and practical reasons, can have systematic errors.
And then Crane went all-in on the conspiracy theorizing, with an image of “Thought Police” and the following text:
When I gave this example [the Pennsylvania Senate race] in a recent interview about the upcoming election, the reporter was disturbed. The interview concluded shortly thereafter. The article was never written.
These markets pose an existential threat to legacy media . . . controlling the narrative before an election is integral to controlling what happens afterwards. Could this be why the media and current administration are putting extra effort to destroy all credible alternatives to biased polling?
When someone is so committed to an idea that he posits conspiracy theories . . . that’s not good.
I conjecture that some of this represents a theoretical misunderstanding on Crane’s part, a bit of what’s called market fundamentalism, a lack of appreciation for the complexity of information flow and behavior. It’s complicated, because if you read his post, Crane is not saying that he knows that markets are better. He says that markets do better empirically, but as discussed above we don’t really have so many data points to assess that claim. So calling him a “fundamentalist” is a bit too strong. I guess it would be more accurate to say that Crane overstates the evidence in favor of the performance of betting markets, he avoids looking at their problems, and then this puts him in a position of explaining the lack of dominance of markets in election forecasting by positing malevolent forces that suppress markets, rather than considering the alternative explanation that people are concerned about market manipulation (a topic as relevant nowadays as it’s ever been).
You might ask why discuss that post at all. Short answer is no, I wasn’t looking to be trolled, nor was I searching the web for the most extreme thing posted by a statistician that week. Given that tens of millions of Americans believe outlandish conspiracy theories, it’s no surprise that the some statistics professors are open to these ideas. I’d guess you could find quite a few believers in ghosts among the profession too, and even the occasional adherent of the hypothesis that smoking does not cause cancer.
Crane’s post interested me not so much for its conspiracy theorizing as much as for its ideological take on prediction markets. Crane loves prediction markets the way I love that Jamaican beef patty place and the way someone I know loves Knives Out. These are topics we just can’t stop talking about.
But let’s unpack this for a moment.
A prediction market, like any human institution, can be viewed as a practical instantiation of a theoretical ideal. For statisticians and economists, I think the starting point of the appeal of prediction markets comes from the theory. Betting is probability come to life, and betting on many related events induces a multivariate distribution. Real-life betting markets are too thin, too noisy, and have too many biases for this derive-the-distribution idea to really work, but it’s cool in theory. Indeed, even at the theoretical level you can’t be assured of extracting probabilities from markets, given possibilities such as insider trading and feedback. Anyway, seeing a post from someone who is such an extreme prediction-market fan gives us some sense of the appeal of these markets, at least for some segment of the technically-minded population.
Summary
My own views on prediction markets are mixed.
I like that there are election prediction markets and I get the points that Rothschild and Sethi make above about their value, especially when incorporating breaking news.
From the other direction, I would paradoxically say that I like markets to the extent that the bettors are doing little more than efficiently summarizing the news. I wouldn’t be so happy if market players are taking advantage of inside information; or using the markets to manipulate expectations; or, even worse, throwing elections in order to make money. I’m not saying that all these things are being done; I’m just wary of justifications of election markets that claim that bettors are adding information to the system. Efficient aggregation of public information would be enough.
I do like the idea of prediction markets for scientific replication because, why not? For me, it’s not so much about people “putting their money where their mouth is” but rather a way to get some quantification of replication uncertainty, in a world where Harvard professors are flooding the zone with B.S.
At the other extreme, no, I don’t favor the idea of a government-sponsored prediction market on terrorism run by an actual terrorist. In the abstract, I’m in favor of the rehabilitation of convicted criminals, but I have my limits.
Prediction markets are worth thinking about, and we should understand them in different contexts, not just as competition with the polls or as some idealized vision of free markets.
The problem I have with prediction markets is that they aren’t very useful until they have converged, and that only happens near in time to the event that is being predicted. By then, it’s usually too late for the results – good or -bad – too matter.
What is the appealing theoretical aspect of betting markets? It just seems like magical thinking to me.
The bettors don’t have any better information than anyone else, just money to throw around. Its easy to see how the opinions of the bettors appeal to right wingers, given the widespread belief on that end of the political spectrum that smart people and rich people are the same demographic. The only way I can see that one could conclude that the predictions are actually better is to make the same nutty assumption as Harry Crane, that pollsters and pundits are doing whatever they can to bolster the chances of Democrats and thereby produce information that is systematically skewed from reality.
See https://arxiv.org/abs/1201.6655
There are plenty of people in prediction markets who aren’t rich. If the rich people are systematically wrong and wind up distorting prices away from actual probabilities, it will be a golden opportunity for others to siphon money away from them. That’s the real justification for taking prediction markets seriously.
Andrew writes
“At the other extreme, no, I don’t favor the idea of a government-sponsored prediction market on terrorism run by an actual terrorist. In the abstract, I’m in favor of the rehabilitation of convicted criminals, but I have my limits.”
Chasing down the reference, I found that he is referring to what amounts to ancient history [1990], The Iran-Contra Affair and John Poindexter.
https://en.wikipedia.org/wiki/Iran%E2%80%93Contra_affair
As always, I am happy to see that an age mate of mine is still alive, even if he is a convicted criminal and later pardoned. The above Wikipedia reference ends with
Poindexter has endorsed the false[17][18][19][20] conspiracy theory that the 2020 presidential election was rigged to favor Joe Biden and claims that the United States “has taken a hard left turn toward Socialism and a Marxist form of tyrannical government.”
Some bright people [he graduated first in a class of 899 at the Naval Academy and has a PhD in nuclear physics from Cal Tech under Mossbauer] are very slow learners.
Paul:
I love that the people who put together that letter thought it would add credibility to include Poindexter as a signatory.
Hello, Andrew.
The “Open Letter from Senior Military Leaders” was signed by 317 retired generals and admirals. Paul Alper beat me to the chase: Poindexter graduated 1st in a class of 899, and his PhD thesis advisor at CalTech won the Nobel Prize in physics. Yes, Poindexter was indicted and convicted on April 1990 on 3 counts of obstruction and 2 counts of false statements to Congress for which he was sentenced to 6 months in prison, but his conviction was overturned on appeal. While his trial was in progress, back in 1989, he developed one of-or the very first-digital real time imaging systems; it was needed for physical security at nuclear power plants. Later, he was head of DARPA (when he introduced and ran that in/famous Policy Analysis Market). At the age of 70, he developed and promoted use of content addressable memory to help the IRS with fraud detection during the 1st Obama Administration.
Those are a few reasons why Poindexter’s name might have added credibility to the 316 other names on that letter.
You refer to Poindexter as a terrorist. I read your 2007 post about the Policy Analysis Market. Neither there nor here did you mention Poindexter’s funding of the Contras in Nicaragua, while National Security Advisor (which he did in cooperation with the US Secretary of Defense). Now THAT seems like state-sponsored terrorism. Instead, you mention weapons sales to Iran. Please be aware that Poindexter, North, and Weinberger did so in hopes of getting American citizen hostages in Lebanon released. They were held by genuine Hezbollah terrorists loyal to Iran’s Ayatollah Khomeini. Here’s a 1988 Politico article for Paul Alper’s entertainment https://www.politico.com/story/2012/03/this-day-in-politics-074074
I do have some thoughts about why election prediction markets might not be so great. I’ll make a separate comment about it. I’ll explain why Fetterman’s decision to debate was a good idea, as Rajiv suggested. I am Twitter friends with Harry Crane. He is a bit of a libertarian but not a nut. Fox leans right, but 4chan is a cesspool of the worst of right and left, maybe deserving a strike out.
Thank you for your patience with me.
more Poindexter trivia: as head of DARPA, he drew up the “Total Information Awareness” project (TIA) – logo says it all: https://en.wikipedia.org/wiki/File:IAO-logo.png, which, until program got shut down, funded alias-i, a 2-person company doing natural language processing, one half of which was Bob.
Ellie:
Yes, when talking about Poindexter as a terrorist, I was thinking of the entire Iran-Contra affair, including what they did in Nicaragua.
Mitzi!
That is a great find! Thanks so much for sharing. I started laughing as soon as I saw it because it is so hard-core surveillance. Now I’m laughing again because of what the first Wiki talk page said:
“This logo was dropped after awhile…fairly generally acknowledged as just too creepy. Masonic, Orwellian, you name it.” It sure is :)
Seems like betting market participants would be subject to the same rules that Phil Tetlock uncovered in his investigation of prediction capability. The market will function well to the extent that the bettors are “foxes” rather than “hedgehogs”. IMO in the recent elections there were, not just in betting markets but across the board, a lot of ideologists talking up the red wave, and that bled into the forecasts. I would say that, rather than following the news, bettors are part of the milieu from which the news emerges.
I’m suspicious of the claim that betting markets have some special essence early in an election. There are so many unpredictable events that occur in an election, it’s hard to see how those could be incorporated into a forecast before they occurred. In Tetlock’s projects, the most successful forecasters are constantly revising their positions, rather than sticking to their initial forecast.
To the prediction-market skeptics in the comments:
One way to think about a market is as a special case of a forecast aggregator. Start with the perspective that we have various sources of information that need to be aggregated. One way of doing this is to build a Bayesian model, which is what I did in 2022 with the Economist model. This can work, but it can take a lot of work and is not immediately scalable to other elections. Another approach is to build an algorithm, not a generative model of the data but a procedure that shifts and averages different estimates. That is what they do at fivethirtyeight.com, and it works pretty well, but again it can take a lot of work and is not immediately scalable to other elections. Betting markets are a form of information aggregation that don’t require any special statistical work. They’re not perfect, but they’re scalable. So that’s one reason they should always be in the conversation.
I’m not really satisfied by this explanation though. Many other things are forecast aggregators. Looking at the net sentiment of my twitter feed is an information aggregate. Asking in polls who people will think will win the election is an information aggregate. Reading the front pages of newspapers is an information aggregate. You have to actually validate the aggregation methodology.
In terms of your questions, I like to think of betting markets as essentially equivalent to the “who do you think will win” survey question, plus the confounding factor of a sample set biased towards whoever likes to place bets, and questions of the wealth of the bettors. Counterbalanced against that is the suggestion that a monetary interest might keep people honest – but that is a tenuous assumption at best looking at who participates in such markets. After all, that would suggest betting markets would be dominated by neutral experts with no partisan bias, but that is hardly the case. Just like how sports gambling is full of sports *fans*, not professionals coolly evaluating the odds.
As “who do you think will win” already does not always perform well (especially in this election), there’s little reason to think prediction markets are a reliable methodology.
“Many other things are forecast aggregators. ”
Sure. But your twitter feed and other news sources aren’t quantitative. They don’t return a probability. And surveys have many of the same problems you ascribe to betting markets – and they require someone to prepare and execute the survey and calculate the results.
Neither do betting markets. They *imply* a probability under a specific set of assumptions about the betters’ motivations, utility for money, rationality (in the dutch book sense), and frictionlessness of betting transactions. A billionaire could drop a fat sum on a betting market that Donald Trump will win California in 2024 just to raise eyebrows and the implied probability would jump to nearly 1; that doesn’t make it so.
Well, in terms of being quantitative I don’t think that makes a big difference. I can just declare “if 50% my twitter feed like Biden he has a 100% chance of winning” and boom, that’s a quantitative prediction. Having someone to prepare and execute the survey also isn’t a bad thing. It’s only really an argument if running surveys is really hard so we have to rely on what is a “citizen science” style dataset with a self-selecting sample. But if we already are doing surveys with hopefully decent methodology, what exactly are these betting markets adding? Not much.
> Just like how sports gambling is full of sports *fans*, not professionals coolly evaluating the odds.
The professionals don’t only coolly evaluate the odds (and don’t just aggregate). In the sports world they take steps to explicitly manipulate the odds – not the odds in the outcomes (over which presumably they can have no effect) but the betting odds.
Not sure how or if that might apply to betting odds for political outcomes.
“Betting markets are a form of information aggregation that don’t require any special statistical work. They’re not perfect, but they’re scalable. So that’s one reason they should always be in the conversation.”
Well, Jeane Dixon was a forecast aggregator, she had no problem with scaling, and her forecasts required no stats work. And you could even derive a distribution by including other seers. Yet I see no reason to include them in this conversation.
For me, this gets back to one of the themes of this blog, the need for some sort of theoretical framework when things are really noisy. I need to be able to lift the hood and see how the bettors make their decisions, like I can at fivethirtyeight or in your (Andrew’s) Bayesian approach. (I read the abstract of the paper Nuno Sempere linked above, but it starts by assuming betting markets work as intended.)
I began with a rather uninformed opinion that the source of panache afforded to betting markets is their inscrutable nature coupled with what are essentially conspiracy theories about pollsters and elites, and now I am more confident in that because none of the more knowledgeable people here have been able to come up with anything else.
Matt –
>… the need for some sort of theoretical framework when things are really noisy.
FWIW, that’s how I see it also.
https://statmodeling.stat.columbia.edu/2022/11/16/how-good-are-election-prediction-markets/#comment-2128225
Microprediction is using the market (specifically parimutuel) as a forecast aggregator, see here: https://www.microprediction.com/about
Thanks Andrew. Markets are opinion aggregators but with the property that if they come to be dominated by those with a particular slant, they will attract traders with the opposite slant. In this sense they are endogenously depolarizing, unlike most online platforms. Whether they forecast better or worse than models is an empirical question – our work (in the paper you linked to in your post) shows that in 2020 they did better than the Economist model several months out but worse as the election approached. A trader who believed the Economist model and bet accordingly would have made a 16% return in 2020 but would have made a 7% loss if the three closest states (GA, AZ, WI) had gone the other way. So it was close. But that’s just one data point. The mechanism that allows markets to predict well (to the extent that they do) remains somewhat mysterious, but those who thing they are easy to beat should try thir hand at beating them, it can be a sobering experience. A couple of links below that expand on these points:
https://ojs.aaai.org/index.php/AAAI/article/view/8808
https://rajivsethi.substack.com/p/prediction-markets-in-a-polarized
With my standard caveat of not being statistically literate…
Seems to me that finding patterns underlying polling error (and then explaining them by conspiracy theories). is exactly what we’d expect, even of “expert statisticians,” even when there’s actually not enough data to understand whether they’re really “patterns.”. And alignment of the patterns found (and conspiracy theories) with ideological predilections is a red flag. Theories of cognitive bias, particularly motivated reasoning, predicts exactly that.
Aren’t most of the polling errors actually within the margins of error? That seems to be what Nate Silver is usually arguing.
And when they aren’t, maybe even for two events in a row, couldn’t that be viewed as anomalous rather than explainable by some grand theory of everything? People scurried about to explain a pattern of error coinciding with Trump’s influence. But maybe Trump just mediated some other underlying causal mechanism, and perhaps that mediating effect is just basically a moment in time and not sustainable.
Seems to me that as long as polling is mostly within margins of error, it should be viewed as mostly useful but often wrong. Explanatory theories can be interesting to look at but should be viewed with a high degee of skepticism. If markets are more predictive of outcomes than poling over a limited period of time I will consider that to prolly be basically coincidence – unless someone can describe a theory of causal mechanism for the predictive ability of markets that seems more plausible than the causal mechanism of polling – where you ask people who they’re going to vote for, so a somewhat reasonable adjustment to make the sample representative, and then average the results with reasonably comprehensive crosstabs.
Is there really a difference between predictor markets and stock markets where the “predictors” are no more correct (and prolly less predictive) on balance, than index funds?
This comparison of the accuracy of 538 versus Predictit — https://mikesaintantoine.substack.com/p/scoring-midterm-election-forecasts?sd=pf — appears to accidentally demonstrate that Predictit was better calibrated than 538 for this election cycle. 538 did “better” when measured with a Brier score because 538 was overconfident about races in the tails.
Err.. specifically: outcomes that 538 said would happen ~25% of the time basically never happened, and outcomes 538 said would happen ~75% of the time almost always did. Whereas PredictIt seemed to get closer to the actual outcome fraction.
Peter:
Yes, I remember fivethirtyeight.com giving Biden a 6% chance of winning South Dakota in 2020, which seemed farfetched in the modern polarized environment but I guess you could’ve imagined Trump imploding somehow. They also gave Trump a decent chance of getting 58% of the vote in Florida which never seemed possible. And then there were the conditional probabilities . . .
But, as I wrote in my one of my other posts, every forecasting method I’ve ever seen, including models and prediction markets, give some implausible predictions. It’s just hard to make everything work out in this high-dimensional space. What distinguishes Nate is his unwillingness to admit that his method could have problems. He might not have heard of Georg Cantor.
I think the error you are making is that you are considering each of these predictions as independent. Taking the collection of 2022 midterm predictions, events with a <25% probability correspond more to "welp, the entire model basically failed and the outcome looks nothing like what we predicted." So in reality, the model isn't saying that 25% of these 25% probability outcomes will happen, it's closer to saying that with 25% probability ALL of these low probability outcomes happen, and 75% of the time none of them happen.
I had a similar finding, see https://www.microprediction.com/blog/longshot
One reason markets can work better is that anyone can spot a potential shortcoming with a model and take advantage of it, without having the approval or ability to improve the model itself. Here’s one thing not in the Economist model (Andrew can correct me if I”m wrong, but I’m pretty sure it isn’t in there) https://www.microprediction.com/blog/election
Peter:
Yeah, I don’t think we had any such effect in our model.
Andrew, I think someone should give you a huge betting account, and then we’ll have the best of both.
Or, we could have meta-prediction markets on forecast models.
By the way, there’s pre-midterm data at https://github.com/microprediction/midterms and if it is possible to add the Economist’ numbers it would improve the mini-study.
Peter:
I have another post coming on why I didn’t bet on the 2020 election.
Thanks I’ll look it up. I’m being somewhat whimsical of course, but I do think that there might be a germ of an idea for prediction markets that are shrunk towards expert opinion somehow. Perhaps that is subsumed in a meta-model though, which anyone with access to the individual contributions to a market could wield.
I’m honestly pretty disturbed by this Crane post–he’s either lost his damn mind or knowingly laundering his credentials to subvert American Democracy. From the post:
Clicking through his link to his own research article
“if you doubt these claims, you’re welcome to read the articles yourself and comment publicly at the link provided”–indeed!
I also love the implication that polls are easier to manipulate than prediction markets in a world of pareto distributed capital. If some voters have many times as much spending ability as other voters, and spending ability correlates with political interest, and spending ability is directly related to how much you get to influence “the narrative”–well that seems like asking for manipulation.
If “which is better” remains a matter of opinion, I’d think that needs to be worked out before bothering with any more forecasts.
What is the point if you can’t tell (in principle) which method produces the better results?
If one method is better at minimizing loss function A, but the other better for loss function B, then say that.
Anon:
“Which is better, prediction markets or poll-based forecasts?” is not a well-defined question, given that there are more than one of each of these. I would not say that any of these things minimize a loss function: they’re social endeavors that create some outputs. It’s true that you can’t learn much conclusive from N=1 or even N=5, but just cos we can’t learn anything conclusive, that shouldn’t stop us from trying to understand what we’ve got. Again, I recommend the above-linked paper by Rajiv Sethi which gets into much more detail.
Well yea, the first step to getting an answer is to figure out the right question(s) to ask.
You seem to be saying this is all exploratory research. I assume you think both are better than praying or rolling dice to generate forecasts though.
I think there is an aspect of this analysis that is missing: prediction markets might be systematically off if people are using them for hedging rather than making money.
For example, if you are a Democrat it makes a certain amount of sense for you to bet on Trump to “hedge” against losing the election. This way, at least if your preferred candidate wins, you get some money that can help you (emotionally or otherwise) overcome that loss.
For a good example, think of the $1,200 promise if the Democrats won in 2020. It would be rational for a person to bet $600 on Trump winning to hedge their odds so that they would make $600 either way if either Trump or Biden won.