tl;dr. In 2024, pre-election polls were off by about 2 percentage points. Election forecasters were aware of this and incorporated this into their uncertainties. But if you were not following the forecasts carefully, you might’ve not realized this. You might have mistakenly thought that, when predictions happened to be close to 50-50, the forecasters “gave up.” You might have mistakenly thought that “Whatever happened, it was supposed to be razor thin.” I guess news reports should be emphasizing forecast uncertainties even more! None of this means that polls are “useless” or that “the survey industry needs a tear-down.”
tl;dr of the tl;dr. Surveys are pretty good. They’re not perfect. The imperfections—nonsampling error—represent real difficulties. The lack of easy answers doesn’t mean that pollsters are chumps.
OK, this is just annoying. Ben Recht and Leif Weatherby write:
2024 was the year the election forecasters gave up. On the Monday before the election, the New York Times polling average showed Donald Trump and Kamala Harris within one point of each other in 6 critical swing states. They put the final election popular vote prediction at 49-48 in favor of Harris. Effectively, a tie. Poll aggregator Real Clear Politics split the difference even finer, predicting the result 48.5-48.5. Poll forecaster Nate Silver put the probability of either candidate winning at exactly 50-50.
Yes, the forecast was highly uncertain. No, that’s not “giving up.” Let me introduce you to some sports bookies in Vegas. They give lines on every game. Sometimes the two teams are, to the best of all information, evenly matched, and the betting line will be even. That doesn’t mean the bookies “gave up”; it means that, their best estimate is that the two teams are equally likely to win. (OK, not quite, it’s really some combination of their best forecast of the election and their best estimate of what it would take for equal amount of bets to come in each direction. The point is, they’re not “giving up”; they’re doing their best.)
So, yeah, to say that “2024 was the year the election forecasters gave up” is wrong, for the same reason that it’s wrong to label the National Weather Service as “giving up” on days where they announce a 50% chance of rain.
Recht and Weatherby continue:
Whatever happened, it was supposed to be razor thin.
Again, no. Here’s Nate Silver, the best-known forecaster, a couple weeks before the election:
I [Nate] have a guest essay up at the New York Times with a fun headline: “Here’s What My Gut Says About the Election. But Don’t Trust Anyone’s Gut, Even Mine.” . . . Most of the column is about how Kamala Harris could beat her polls — or Donald Trump could beat his again. One thing that might be counterintuitive is that even a normal-sized polling error — polls are typically off by around 3 points in one direction or the other — could lead to one candidate sweeping all 7 key battleground states. . . . the baseline assumption of the Silver Bulletin model is that while the polls could be wrong again — and in fact, they probably will be wrong to some degree — it’s extremely hard to predict the direction of the error.
I’m not saying Nate’s always right. We’ve had our disagreements. I’m just saying that, no, he was not saying the election “was supposed to be razor thin.” The forecast electoral vote outcome was a distribution with expected value 50-50 but with a substantial variance.
And here’s Elliott Morris, who runs Fivethirtyeight.com, the best-known forecasting site:
Trump and Harris are both a normal polling error away from a blowout
The race is uncertain, but that doesn’t mean the outcome will be close.
As of Oct. 30 at 11:30 a.m. Eastern, the margin between Vice President Kamala Harris and Trump in 538’s polling averages is smaller than 4 points in seven states: the familiar septet of Arizona, Georgia, Michigan, Nevada, North Carolina, Pennsylvania and Wisconsin. That means that, if the polling error from 2020 repeats itself, Trump would win all seven swing states and 312 Electoral College votes. . . . In a scenario where the polls overestimate Trump’s margin by 4 points in every state, Harris would win all seven swing states and 319 electoral votes. . . .
Both of these outcomes — and everything in between — are very much on the table next week. . . . the model is expecting a roughly 2020-sized polling error — although not necessarily in the same direction as 2020. (In 50 percent of the model’s simulations, Trump beats his polls, and 50 percent of the time, Harris does.)
This point is worth dwelling on. Because our average expectation is for there to be a decently large polling error at least half of the time, there is actually a very low probability that the polls are perfect and the election plays out exactly how the polls suggest. . . . Polls are inherently uncertain. This is why we model. . . . this is the big, fundamental problem with preelection polling: We don’t know what the demographic and political composition of the actual electorate will be, so pollsters are just making the best guesses they can. Those guesses have always, and will always, come with error attached to them.
Here’s some dude who worked on the Economist election forecast:
Why forecast an election that’s too close to call? . . . I think the main value of forecasts is not in the predictions themselves, but in how they portray uncertainty and the stability of the race over time. . . . In the end, elections will always be uncertain, because it is up to the individual to decide how to vote, and whether to vote at all.
And here’s Harry Crane, a forecaster who did better than the name brands in 2024 by incorporating additional information on party registration and early voting:
The forecast makes Trump about a 2-to-1 favorite . . . based on an analysis of fundamental data, polls, and early voting data. This is more or less in line with other opinions out there, such as the betting markets and other forecasters. But because this forecast likes Trump a bit more than markets and a bit more than the other forecasters (who are favorable to Harris), it is inevitable that I will be called an idiot, or worse, should Harris pull it out. The same people will still call me an idiot if Trump wins, so what’s the difference.
The point is that every serious forecaster in 2024 understood—and vocally communicated to the world—that their forecasts were uncertain. Nobody thought Harris or Trump had much of a chance of getting 400 electoral votes, but everybody was saying that 300+ was a possibility. Even the forecasters who were loudly disagreeing with each other on social media agreed on this point, that the forecasts had a lot of uncertainty in the electoral vote.
Recht and Weatherby continue:
The result was that, in a “close” race, he won every swing state. That stark truth seems like precisely the sort of thing the prognosticators should have been able to tell us, at least in the aggregate. Instead, the Republicans defeated the pollsters.
OK, 2 things. First, you can put “close” in scare quotes if you want, but the election really was close! The popular-vote margin was less than 2 percentage points, and a swing of 2 percentage points also would’ve swung the electoral college. That’s a close election.
Second, “that stark truth” that Trump (or, for that matter, Harris) could’ve won every swing state was explicitly stated by the forecasters.
Recht and Weatherby continue:
Polls attempt to divine big-picture answers about the sentiment of millions of people from the responses of a vastly smaller group, many of whom aren’t especially eager to tell the truth about their opinions. The internet, ubiquitous cell phones, and the widely varying use of technology among different age demographics have all contributed to the problem, thwarting techniques honed at a time when landlines were in every American household.
With response rates in the single digits, pollsters are now forced to apply “statistical corrections,” backed by a series of guesses about the pre-existing beliefs and tendencies of the populace.
This is not new. Polls have never been random samples. Old-time Gallup polls used quota sampling, which is just a statistical correction implemented in the design, and it requires all the same assumptions that any adjustment would. Modern polls have nonsampling errors—enough to roughly double the stated margin of error—but that’s always been the case. Final polls were way off in 1980 and 1948 as well.
They continue:
Low response rates and statistical corrections make polling into a special kind of obfuscated punditry, undermining its claim to neutral objectivity and rendering it useless.
Well, no. As I discussed in my article, Failure and success in political polling and election forecasting, yeah, we’d prefer if polls had no nonsampling error—but nonsampling error of 3 percentage points is not so bad. It just happens that when the forecast is very close, the potential nonsampling error is consequential.
There’s a big difference between imperfect and “useless.”
They conclude that “the survey industry needs a tear-down.” I can’t argue one way or another on that claim. It’s a matter of opinion.
In the comments section, Recht adds, “If you can’t randomly sample, you shouldn’t survey.”
Again, that’s a statement of opinion, so nothing to argue about. There are essentially no random samples of humans, whether you’re talking about political surveys, marketing surveys, public health surveys, or anything else.
I think that organizations will keep doing surveys, and I think that applied survey researchers will continue to recognize problems of measurement, nonresponse, coverage, and generalization, and they will use statistical models to estimate uncertainty.
I get it that many people are frustrated when news reports focus on point estimates. But in 2024 I think the news media were pretty good about recognizing uncertainty! Even on election day, when forecasts were at 50-50, there were lots of news reports accurately stating that anything could happen. It wasn’t like 1992, 1996, or 2008, when on election day there was a clear expectation of who would win.
So, yeah, it’s too bad that polling isn’t random sampling—it never was!—and a 3 percentage point error isn’t nothing. I just don’t think it’s appropriate to say “the election forecasters gave up” or that “Whatever happened, it was supposed to be razor thin,” given that the forecasters did not say that; indeed they took pains to emphasize their uncertainty.
Why go into all this detail?
Why bother writing the above post? Is it a notorious case of a blogger being upset because, in the immortal words of Randall Munroe, “someone is
- wrong
on the internet”? I guess so!
As a statistician who works on probabilistic inference sampling, I hate to see credentialed academic experts get things so wrong. To think that a 50-50 forecast is “giving up” or to think that the pollsters were “defeated” by being off by 2 percentage points is just so naive, especially given that this is no worse a polling error than was typical in the “time when landlines were in every American household.” Naive mistakes by non-experts are ok and, in some sense, necessary steps in building our understanding. I make naive mistakes all the time, and sometimes these even find their way into this blog. What bothers me more is that air of assurance which I associate with a certain style of writing on the internet. In the meantime, yeah, polls and forecasts are imperfect–indeed, this imperfection is built into forecasts, hence the wide uncertainties and the completely reasonable pre-election statements by Nate Silver and other forecasts.
The problem with a lot of these ml folks is they completely give up thinking about the data generating process
Florian:
It is a long-respected tradition to model the data collection process but not the data generation process: that is, to model the probability of inclusion in the sample or the probability of treatment assignment but not to model the outcome itself. This is sometimes called sampling or randomization inference. Sampling or randomization inference has its problems–it leaves on the table all that might be achieved by fitting a model to the outcome–but it can be useful as a baseline. For example, classical sampling inference from a survey gives a margin of error which is a lower bound on uncertainty (because it does not include nonsampling error; see here: https://sites.stat.columbia.edu/gelman/research/published/polling-errors.pdf) and it gives population-level estimates, but more needs to be done to get small-area estimation (for example, MRP; see here: https://sites.stat.columbia.edu/gelman/research/published/misterp.pdf). It still can be a useful baseline; the problem comes when people get all fundamentalist on you and say that you can’t do statistical inference if you don’t have a random sample, which, if true, limits the concept of “statistical inference” to exclude just about all surveys of humans as well as just about all experiments on humans (even if you have random treatment assignment, the people in the experiment won’t be a random sample of the people and scenarios of interest).
A real loss for political data and polling community that ABC decided to shut 538 down…
https://x.com/gelliottmorris/status/1897363074073682360
“As reported, everyone at 538 was laid off this morning.”
No big loss to the world at large, however.
One very small note is that
“That doesn’t mean the bookies “gave up”; it means that, their best estimate is that the two teams are equally likely to win. (OK, not quite, it’s really their best estimate of what it would take for equal amount of bets to come in each direction. The point is, they’re not “giving up”; they’re doing their best)”
Is actually a common misconception. This article, https://www.nytimes.com/athletic/2644177/2021/06/17/james-holzhauer-how-sportsbooks-really-make-their-lines-and-early-tips-on-where-to-beat-them/, from a professional sports bettor (more famous as a record-breaking jeopardy champion) specifically addresses it:
“One note before we begin: A popular misconception is that sportsbooks set their lines in order to get an equal amount of money on each side. Aside from rare exceptions like the Super Bowl or 2017’s Mayweather-McGregor fight, public money is generally not enough of a factor to move the odds. The book typically prefers to keep the line close to the “correct” number and gamble on the result, rather than move to an off-market number and attract a flood of action from advantage players. This means that the popular strategies to look for “sharp vs. square” or “reverse line movement” games will not show an automatic profit”
Michael:
Yes, good point. I changed the above post to fix.
Thanks for reminding me of my frustration when I saw that post :-)
>What bothers me more is that air of assurance which I associate with a certain style of writing on the internet
I assume this will persist as long as there are readers who like to collect quick stylized sound bytes and move on.
Jessica:
Probably also a problem for all of us who are teachers, which is that our students are likely to believe just about anything we tell them, especially if we say it with an air of great assurance.
I almost have the opposite problem now. Great assurance while being quite wrong makes it hard for me to remember to still try to learn from other things they write.
The linked blog also seems to dismiss literally every single use of statistics just because social scientists are bad at it.
There is a post where he argues that the Higgs boson discovery did not actually happen, and it is just nonsense, as the first comment on it points out in detail.
https://www.argmin.net/p/the-higgs-discovery-did-not-take
Fivethirtyeight, when Nate was there, spent a good while agonising over the fact that people just don’t intuitively grasp probabilities. They simplified their language a lot, and it barely helped. This is yet another area where AI will vastly outstrip primates.
There is a difference between politics and basketball though. A week before an election, the result largely exists in the world. An oracle could make a very good prediction.
Nate Silver just posted over on his Substack that Disney pulled the plug on 538 and it is no more. All the staff were let go.
https://www.natesilver.net/p/a-few-words-about-fivethirtyeight
Fraac wrote:
“[Intuitively grasping probabilities] is yet another area where AI will vastly outstrip primates.”
Since “intuition” is something humans employ to reach understanding without the need to reason, I would say that this claim is not even wrong.
To reach decisions, not reach understanding. You could always reason to a decision at least as good, given enough time. Problems arise when your intuition – formed from both personal experience and population-based evolution – is applied carelessly in a new domain. AI will be able to reason about this process, and avoid pitfalls, much better than a primate.
“So how should the public interpret poll numbers?
[Dr. Josh] Clinton suggested that readers should take the margin of error stated on the poll, double it, and see whether the difference between the two candidates is within the doubled margin of error. If it is, then it is hard to conclude which candidate is actually ahead in the race.”
https://news.vanderbilt.edu/2021/07/19/pre-election-polls-in-2020-had-the-largest-errors-in-40-years/
I remember you said something pretty similar once. Basically, assume that the systematic error is the same as the statistical error as a baseline rule of thumb.
This kind of reminds me of the analogous statement among historians: there are no unbiased sources, because everyone needs to make a choice about what to include and what to exclude in their narrative/analysis, and different people make different choices for different reasons.
yea yea whatever, we all know the real problem is lack of Martingale Rigor!
https://statmodeling.stat.columbia.edu/2020/10/12/more-on-martingale-property-of-probabilistic-forecasts-and-some-other-issues-with-our-election-model/
Ah the good old days, back before *gesticulates broadly*.
At least this time we could rigorously update from 50-50 to republic collapse only ^after^ the votes came in!
Well, a spirited defense I must say.
Just the same there are some points here that need to be contested.
“…the National Weather Service as “giving up” on days where they announce a 50% chance of rain.”
In an election there is no useful distinction between 51/49 and 99/1 to the general public. In a precip forecast, 51% chance of [some weather] vs 99% chance of [some weather] matters a lot to the general public. You choose “some weather” = “rain” to trivialize the claim. But in fact “some weather” is often life threatening and a difference in the forecast of 10-20% matters to the public. So, no, election forecasting is not remotely analogous to weather forecasting, except that both offer probabilities of some event.
“In the end, elections will always be uncertain, because it is up to the individual to decide how to vote”
Fine, but we already know that >80-90%? of the population has it’s mind made up a year ahead of time and it’s only a few people who swing the vote. So any sensible forecast is already restricted to the 60/40 – 40/60 range. In that sense, the margin of error that you quote is seriously understates the real error of the forecast. A 2% difference in the popular vote when the range of possible variation is only 10-20% is really equal to a 10-20% forecast error within the possible range.
You’re overlooking some other factors too. The most obivous of which is that we already have polls, which are a direct measurement of what the weather actually is. An election forecast uses [favorite statistical magic box] to aggregate, weight and average polls, throwing in a few general factors like the current economy along with another [favorite statistical magic box] to aggregate, weight and average those factors. While you’ve contended that all of this amounts to some form of science, I don’t think you’re there yet. Obviously, there is a rationale for the various gears and buzzers inside [favorite statistical magic box] . But it’s just not clear that [favorite statistical magic box] actually works any better than a just a quick look up at the sky (polls). Worse yet, it’s not even clear if there is a way to tell if it works – that puts alot of pressure on the claim that it’s “science”.
Moreover forecasts don’t even provide any value over the brutally obvious in the face of major events. Biden’s performance in the debate in the most recent election was analogous to the sudden appearance of a tornado down the street. Clooney didn’t need a forecast to recognize the certainty of disaster. But there was nothing in the “forecast” the day before that alerted anyone to the storm. IOW, the forecast only a day before the debate was completely useless.
So, overall, I agree with you that “forecasters haven’t thrown in the towel”. And yes, many forecasters qualified their forecasts with statements that their forecasts could not predict the outcome, so in that sense the forecasters were’t “wrong”. And yes, different forecasters insert different gears and buzzers and dogfoodifiers into [favorite statistical magic box], and in some cases the mix of equipment generates an outcome that matches the real outcome, but no one knows if that will be true for the next election.
So I don’t have a problem with people making election forecasts any more than I have a problem with the tarrot card reader in the little strip mall on highway 99. But I do think each of these professions provide about the same level of value in their forecasts of the future events they are concerned with, and I think each has a similar mistaken certainty as to the quality and value of the information they provide. The fact that they both excuse themselves with occassional claims that the cards aren’t quite clear doesn’t add value to their forecasts.
Surely there is work comparing the forecasting models that eg Gelman works on to a naive poll average as a sort of ‘null model’. I suspect we can actually answer the question how much value/information is provided.
Chris:
A null model will get you most of the way there. I think a good analogy is sports analytics, where there’s a lot of available information that’s easy to grab, but analysts will put in a lot of work to get more precise estimates.
Because of the the sampling and non-sampling error there is no easy way to evaluate whether the additional tea-leaf reading beyond what was already “most of the way there” — even after the fact. It’s also where the forecasters squirt in all their special assumptions, parameters, and computational condiments. “More precise” estimates are neither automatically “more accurate” or even “more useful”.
This is what I think Recht is saying. His weighting example of forecasters massaging the numbers to fit (for all we know) their preconceived notions or biases. The forecasting industry’s fetish with more and more precise simulation models is less than useless to drive any action more consequential than a water-cooler argument.
Harsha:
Of course a new estimate is not automatically better! We put a bit of work into our forecasting models. Our work on election forecasting is just like our work on modeling home radon levels, or baseball performance, or toxicology, or marketing, or the many other applications we work on. Vague insult-words such as “squirt,” “special assumptions,” “condiments,” “fetish,” “less than useless,” and “water-cooler argument” add nothing to the discussion. If people want to make decisions based on non-quantitative reasoning and if people want to ignore available data because the data are not perfect, that’s their choice. In the meantime the rest of us can proceed to do our best.
Anon:
If, for you, “in an election there is no useful distinction between 51/49 and 99/1,” then, yeah, I wouldn’t recommend you spend any time looking at election forecasts. You correctly write, “no one knows,” and that’s correct: if you demand certainty, then forecasts are not for you. But you’re not the only person out there. There are many consumers of the news who do find this distinction to be useful; that’s who the forecasts are for. Just cos a topic doesn’t seem important to you, that doesn’t mean that nobody should care about it.
I’ve long been on record as saying there are too many polls. The polls are already out there. Given that the poll results are available, any reporting of the polls will involve some aggregation. Poll aggregation is going to be done, and we’d like to do it as well as possible, given the resources that are assigned to the task. As I wrote in my article, In the absence of prediction models, political observers would be inclined to spin a story around each campaign event and every poll. Forecasting models don’t stop the storytelling, but I think they make the stories more sophisticated and more politically accurate.
Andrew:
My intention was not to insult anyone. If it came across that way, I apologize. I do take issue with models that produce probabilistic forecasts for singular phenomena and therefore cannot be evaluated post-hoc objectively. What value is the model that in places where it deviates from the obvious forecasts from a null model it also deviates from equally well-founded and minutely-argued models from other respected forecasters? It is opinion masquerading as objective data analysis.
To state it differently, to the extent that the deviations from the obvious forecasts are due to the particular choices of parameter values (e.g., how to weight historical poll accuracies) or assumptions (weighting based on which previous election results to disqualify as an outlier) it is no different from TV pundits placing some story upon the poll results. At least they are honestly, if implicitly, admitting that their interpretation is opinion, and not what the “data is telling us”, because some fancy statistical methods were used to process the assumptions.
Harsha:
It’s fine for you to “take issue” with models that I or others have fit in political science, sports, toxicology, etc etc. I think it’s fine that the statistical methods we use are “fancy.” My preference is to use the simplest possible method that will solve the problem; unfortunately it often turns out that I need to develop new methods to do what I need to do.
There’s nothing stopping you or anyone else from using simpler models in all these application areas, or even just giving your opinions. There’s room in the world for people who perform simple analyses, and there’s room for pundits. We typically use more complicated (in your terminology, “fancy”) methods in settings where simple methods fail, or give conflicting answers. One thing I like about our statistical models is that they make their assumptions and data clear. A statistical inference can be seen as a mapping from assumptions and data to conclusions.
I don’t get it. Let’s say you’re predicting whether it will rain or not. You understand that predictions have uncertainty and so you provide a confidence value. If you say 0.7 certainty of rain, you ought to also be willing to say 0.3 no rain.
Now we can measure your calibration. The catch is that if you assign 0.5 to everything, even a coin tosser will get perfect calibration. So we also need to add a surprisal metric, which rewards you for predicting further away from 0.5.
Reality is binary, either it rains or it doesnt. If your best guess is 0.5, I can get that level of calibratalion and surprisal with a nickel. I think it’s fair to refuse to answer some hard questions if you just don’t know, but you still are refusing to answer, you know? It says our methods have been defeated by the difficulty of the question.
Anon:
Perhaps a sports analogy will help. Suppose you do some statistical analysis to estimate the probability that a pro basketball player would make an undefended shot taken from distance x. Suppose this success probability is 75% for x=15 feet (free throws), 60% for x=20 feet, and 40% for x=30 feet, and suppose some modeling yields an estimate of a 50% probability for x=25 feet. It’s informative to say that Pr(y=1|x=25) = 0.5.
This also works in the weather context. In NYC, the probability of rain is approximately 1/3 in any given day at any time of the year. If you think the probability of rain on a particular day is 1/2, that’s informative.
Hi Andrew, late response because I’ve been mulling it over :)
I think what the 50% is telling you is that the model can’t predict on that point of the curve. It’s meta information.
I appreciate it when the weatherman has no idea and tells me as much. Ideally he’d give me a 100% prediction, but failing that it’s better to say 50% and be right by definition, than say 70% and miss half the time.
The 40-50-60-75 curve you provided is informative because it is not the the 50-50-50-50 of the coin oracle. The further away from the coin oracle it gets, the more information it contains. Ideally we can use more predictive input variables than just distance, getting it to output straight (and still correct!) 100% or 0% predictions.
> Now we can measure your calibration. The catch is that if you assign 0.5 to everything, even a coin tosser will get perfect calibration. So we also need to add a surprisal metric, which rewards you for predicting further away from 0.5.
This is only true if the observed frequency of the event you are predicting is 0.5. A coin toss weather forecaster will be justifiably ridiculed if they predict rain with probability 0.5 every day when they are stationed in Death Valley. Their predictions will be in no way calibrated.
If you predict 0.5 rain, you also predict 0.5 not rain. This is where the coin comes in. The coin toss weather forecaster will predict rain roughly half the time in Death Valley, with 50% certainty, and will be correct 50% of the time. Perfect calibration, just lacking in surprisal.
> Perfect calibration
You may want to revise your definition of calibration – at least if you want to communicate successfully with other people regarding this subject.
Carlos, could you be more specific and skip the guessing game? To the best of my knowledge, if you make predictions claiming 70% certainty, and those predictions indeed turn out to be correct 70% of the time, that is perfect calibration. Cursory googling seems to confirm this.
If every midnight you predict that the sun will rise with 50% certainty, and the sun rises 100% of the time, your prediction about the sun rising is not perfectly calibrated.
If every night at one you flip a coin to make a prediction about the sun rising (some days you predict it will rise with 100% certainty – some days you predict it won’t) I can predict every midnight that you will get it right with 50% probability. My prediction about your coin flip will be perfectly calibrated. That has nothing to do with the calibration of your prediction about the sun rising though.
If every night you are willing to predict that the sun will rise with 50% certainty, you have to be consistent and also be willing to predict that the sun will not rise with 50% certainty. 70% rain is the same thing as 30% not-rain. Which side of the bet you take has to be chosen randomly, otherwise your calibration measurement is really measuring some underlying trend in how the questions/predictions are phrased (you are always predictive positive and for some reason questions are phrased such that positive predictions are less likely to occur, or vice versa).
This “trick” only works for 50% predictions, because they contain zero information. At 70%, you could take a 70%-weighted coin and use it to pick, but, you would fail because you need information to correctly decide whether to assign the weighted side of the coin to “sun will rise” or to “sun will not rise”.
On further reflection, I think Im wrong. If you say 0.3 sunrise, 0.7 no sunrise, and flip a weighted coin, you will be right 0.3 of the time. Oops, my bad..
Here is an example of what I mean. You can ask this guy any binary question and if you record his answers, his accuracy will trend toward his predicted certainty (50%). This makes him perfectly calibrated.
import random
def ask_oracle(question):
response = random.choice([“yes”, “no”])
print(f”Q: {question}\nA: {response} (certainty: 50%)”)
if __name__ == “__main__”:
while True:
question = input(“Ask a binary question (or type ‘exit’ to quit): “)
if question.lower() == “exit”:
break
ask_oracle(question)
Anon:
Probabilities exist in a network, not in a vacuum. Consider my basketball example above. If a forecaster were to say Pr(success) = 0.5 always, that’s the same as saying Pr(success) = 0.5 when x=25, Pr(success) = 0.4 when x=30, etc.
Similarly with weather forecasts. They are always conditional. The more information they are conditional on, the better they are. To say Pr(precipitation on a given day in NYC in any given month) = 1/3 is already very informative: there are lots of cities where Pr(precipitation is much higher or lower than 1/3, and lots of cities where Pr(precipitation) varies during the year. But of course weather forecasters can do much better than that; they can much more informative forecasts the day before. Sometimes the forecast happens to be 50%. That doesn’t make it useless; that 50% has to be interpreted in context.
Okay, I understand your point here.
I think this is maybe more a statement on the inadequacy of evaluating a binary prediction purely on the predicted outcome probability in a single instance. The election forecasts that predicted ~50-50 are contingent — in the sense that the prediction could have been different — whereas the coin flip prediction will always be the same regardless of the information provided to it.
In any case, to reduce evaluation of an election forecast to the win probability discounts the numerous other predictions that go along with it. Sure, you can come up with the same win probability by flipping a coin, but that statement doesn’t apply to predicted popular vote share or electoral vote counts or state-specific predictions.
Anon 2, Yes I’m surprised Andrew hasn’t mentioned it yet here but the real target of their election forecasts is share of the two party vote, not just binary outcome…It’s much easier to evaluate predictive skill with a quantitative outcome like that!
Chris, beyond that, it’s share of 2 party vote in each state so it’s a 50 dimensional vector.
I know some concentration of measure, but am not a statistician. Is there an accessible online explanation of why a (simplified model of) non-random samples are a valid source evidence in some mathematical sense, e.g., finite sample bounds on bias and variance, under some assumptions that approximately hold?
Daniel:
Real-world samples of humans are always nonrandom. It’s ok to perform inference assuming random sampling–we make this assumption all the time–but it’s good to remember that the assumption is false. The lack of actual random sampling is a big motivation for fitting more complicated models that we hope are more realistic.
Depends on what level you intend your question, but ‘conditional exchangeability’ is the property we adduce to do causal inference, hierarchical modeling and, if you’re into this sort of thing, motivating Bayesian statistics altogether from De Finetti’s ‘Representation Theorem’.
Chris:
In MRP, we refer to this as the assumption that the data are a simple random sample within each poststratification cell. In recent work we’ve been going beyond that assumption to allow unequal sampling probabilities within cells, as indicated by survey weights.