Skip to content

Answers to your questions about polling and elections.

1. David Callaway writes:

I read elsewhere (Kevin Drum) that the response rate to telephone polling is around 5%. It seems to me that means you are no longer dealing with a random sample, what you have instead is a self selected pool. I understand that to an extent you can correct a model for data problems, but 5% response? How do you know what to correct for? Take another poll to determine what errors to correct for and how much? Use the time honored “listen to my gut” method?

Instead of going to a lot of trouble to gin up a random list sized to leave an adequate-sounding sample after 95% hangups pollsters might instead be honest and just say the current polling methods don’t return usable data, take their ball and go home.

My reply: I’ve heard that the response rate is closer to 1% than to 5%. As to your question of whether pollsters should just “go home” . . . I don’t think so! This is their business. And, hey, the polls were off by about 2.5 percentage points on average. For most applications, that’s not bad at all. I do feel that we have too many polls, but you have to remember that the main economic motivation for polls is not the horse-race questions. Those are just the way that pollsters get attention. They make their money by asking questions for businesses. And if you’re a business trying to learn what percentage of people have heard of your product or whatever, then an error of 2.5 percentage points is not bad at all.

Also, if pollsters were all gonna quit just because their polls are off by more than their stated margin of error, then they should’ve already quit years ago.

2. Another person writes:

I respectfully suggest that you owe it to readers of The Economist (and others) to comment on how you figure it is that your forecasting model erred so badly re the electoral count and the popular vote for the presidency as well.

For a quick answer, we forecast Biden to receive between 259-415 electoral votes and it seems that he’ll end up with about 300, so I wouldn’t say the model “erred so badly.” Similarly for the popular vote. But there were some issues, which we discuss here and here. I do think our model had problems, but there’s a big difference between having problems and “erred so badly.” If you think that what happened was “erred so badly,” I think the problem is that your expectation is too high: the point of the wide uncertainty interval is that our model can’t make a precise prediction.

3. Ricardo Vieira asks what I think about this post, which stated:

Over this and past USA presidential elections my memory says that many states have been won by <<1% between the major parties. I’ve heard it suggested that this is due to parties modifying their platforms to appeal to just enough voters to win in relevant swing states. It makes sense that they try to rebalance this way, but as a mechanism for the near perfect splits we often observe it is insufficient. If any party had the techniques to know where 50% appeal is, pollsters would too, so we’d also have much more accurate polls. And the “omniscience party” would surely give themselves enough cushion to rarely ever lose.

The alternate explanation is chance, i.e.: the results of each state’s election are a random sampling over the distribution of possible % splits; in our age the mean of that distribution happens to have shifted to be near 50%; and the “swing states” are those that fall very close to the mean and are thus often decided by <<1%.

That’s fine in principle, but some of the margins are mind-bogglingly small. The most notable example is Florida in 2000 (wikipedia):

After an intense recount process and the United States Supreme Court’s decision in Bush v. Gore, Bush won Florida’s electoral votes by a margin of only 537 votes out of almost six million cast and, as a result, became the president-elect.

Georgia this year is currently <1500 votes difference (<.1%) with >99% votes counted. PA probably won’t be that close in the end but it’s still gonna pretty damn close, within a few tenths of a point.

Having grown up in this era and never having been a student of political science, how normal is this? Can anyone link a study where folks have looked at the distributions and shown how likely it is for a vote to be decided by 1k or less margins this frequently over 50 states?

Actually Gore won Florida by 30,000 votes, or he would have if all the votes had been counted. Getting to the larger question: I haven’t looked at the data carefully, but, yes, I think the explanation for the occasional very close election in a state is just that there are a fair number of states that could end up close, so every once in awhile you get something very close.

We’ve had enough election posts for now so I’ll put this one on lag. Once it appears, maybe no one will care. I hope in the future we can move to a system where all votes get counted, but that’s probably too much to hope for. Recently it seems we’ve moved toward a position where whether the votes should be counted is itself subject to partisan debate.


  1. Dale Lehman says:

    Just to expand on #1 a bit more: if you view being within a margin of error of 2.5% as moderately successful, then 5 of the past 12 presidential elections fell within that margin based on the naive assumption that the vote percentage would be evenly split (between Republicans and Democrats). This is not exactly a stellar demonstration of polling accuracy, but the more disaggregated the view (e.g., looking at vote totals by state) the better the polls look, at least compared with the low bar of just assuming a 50/50 split.

    I was inclined to agree that a 5% (or 1%) response rate makes a poll almost worthless – but the alternative would be to just assume the vote would be evenly split (given the extreme polarization we have been seeing), but that doesn’t match the polls performance – especially when viewed at a state level. I don’t like polling – but it is not due to their inaccuracy. I actually dislike them more when they are more accurate (there was a post about this a couple of months ago).

    • Dale Lehman says:

      I expanded my analysis a bit. Using the 1976-2020 presidential elections, a more sophisticated “naive” view might be to predict the same Republican/Democratic split as in the last election. I propose this as a potential alternative to polling. Using that naive model, and a 2.5% margin of error, on average the following election would fall within the margin of error 37% of the time. Washington DC did this 73% of the time, while at the other extreme, this naive model would have worked for Delaware, Montana, and Vermont less than 10% of the time. Of course, the margins of error probably vary across states and the 2.5% figure is arbitrary. But, it is clear that the polls do much better than this slightly less naive approach (assuming the vote split will be the same as in the prior election, rather than assuming 50%). Not that this should surprise anyone.

  2. brent hutto says:

    What if poll modelers are simply very, very good guessers based on decades of accumulated experience with election polling?

    Because to me that’s a plausible “prior” (colloquially speaking) when I see that they can start with highly non-representative 1% or lower response rates yet still quite often come up with predictions not far from the actual outcome.

    Does anyone remember Jack Palance doing one-armed pushups at the Oscars 20 years ago? He said something like, “I can keep this up all night, after a while it doesn’t matter if the girl is there or not”

    That’s what election poll analysis reminds me of. The actual poll responses are slowly vanishing like the Cheshire Cat and pretty soon all we’ll have left are the modeler’s guesses forming the cat’s grin.

    • Eric B Rasmusen says:

      There’s a lot to be said for expert subjective predictions. It was said back in the 70’s that economic forecasters had their models, but the real value was in the tweaks the Wise Man added at the end to make it come out like he thought it should. In academic rankings of departments and journals, people use all kinds of numerical metrics like cite counts, but I’d trust subjective opinion much more–though in both these examples, “subjective” opinion really just means some wise expert has paid attention to lots of evidence and combined it by some process he can’t explain to us clearly enough to turn it into a formula a dummy can use.

      Thus, I like the idea of just having political junkies making their predictions and establishing track records, especially if they give verbal descriptions of why they predict as they do. Leiter Reports’ rankings of philosophy departments used to do this— Brian Leiter had his one-man rankings show, with notes on which professors had changed departments in the past year, etc. Then he went and muddied it up by getting opinions from lots of other people to make it more objective.

  3. Eric B Rasmusen says:

    If presidential race polling is accurate, that would affect the distribution of state outcome closeness. We would expect a number of states to be very close to 50-50, because campaigns would spend on ads to try to win the state by 51% to economize on spending. We would expect other states to be extreme– 70-30, for example– because both sides would give up on spending there and let them go naturally. We would have fewer states in between.

    Is there some way to test this theory?

    It is complicated by the fact that campaign strategy is not maximizing risk neutrally for number of electoral votes. Rather, you want to get a majority, and it doesn’t matter how much you lose by if you lose, which can result in apparent risk-loving—McGovern in 1972 needed to throw a Hail Mary pass, though I don’t remember him doing so, perhaps because his campaign wasn’t smart enough to see the need.

  4. Zhou Fang says:

    Really what this election should put paid to is the idea of using political betting spreads as an indicator of win probability.

  5. It depends on how much you are willing to invest on the survey. Telephone survey response rates have dropped over the last decades. In Polls this introduces considerable biases, because the most “radical” people are more likely to answer. You can implement corrections, but they are always subject to more errors and researchers need to make more assumptions and build models that include a higher dose of subjectivity.

    This also happens in some business surveys. However, most marketing research companies sell their product to firms that are truly interested in unbiased accurate results (this is less important for pollsters). In the marketing research context, firms usually provide incentives to increase the response rate, they could send a pre-notification letter, maybe a reminder, call backs, there are relevant economic incentives, and follow up controls. This increases a lot the response rates, they can be near 80%.

  6. Joshua says:

    I’ve asked before… but maybe someone will answer this time.

    Can’t anyone think of a better sampling methodology?

    What about a random selection of a group of people, that you interview at intervals in person, where you pay them to participate, coupled with a thorough analysis of who drops out and why (pay them to supply data to support that analysis).

    It’s obviously not perfect but might it not be better?

    I know that some polling does keep polling the same group of people people rather than ongoing calling people to collect the sample and as I recall that methodology isn’t notably less reliable – but how sophisticated is the non-participant analysis in that kind of sampling?

  7. Dzhaughn says:

    …in which Andrew continues his campaign to subvert the legitimacy of elections in the US.

    Here, he posits a counterfactual world where the election systems were competently run in certain Florida localities where the Democratic Party dominates. Then, taking ballots from the real world, models a count in the counterfactual world, and then zips back to the real world to conclude that this shows Bush to be a “Wrong Man.”

    Of course, the counterfactual is a fantasy world where the Democratic Party was inclined to make political appointments based on competence rather than other factors. One can dream, but would we in this world recognize such a thing as the Democratic Party? (Look how quickly the Republicans became unrecognizable, in fact!)

  8. Michael Nelson says:

    Consider a hypothetical world where public polling is almost always incredibly accurate in describing the present state of the race, even in very close races. Consequently, individual campaigns’ polling is even more accurate: candidates and parties can spend more money on more frequent polls in more targeted areas asking more granular questions. Candidates adjust their strategies accordingly, in terms of messaging, spending, and where they physically campaign. When election results come in, the public polling projections look inaccurate, since polls don’t (and probably can’t) account for the feedback loop of their own results and can’t compete with internal campaign polling results. Without putting too fine a point on it, either we already live in that world or, as pollsters’ methodology gets better, they move our world closer to that one.

    If that sounds crazy, this will really blow your mind: If pollsters’ methodology gets worse (less effective), as many suspect it is, then so do campaigns’ ability to react to polls and shift future public sentiment, which makes the projections less inaccurate. It’s the Red Queen’s race!

    • Jonathan (another one) says:

      In economics, this is succinctly phrased as the Lucas Critique.

      • Michael J says:

        Is there a more general name for it? This situation comes up a lot in many other fields and it would be nice to have some common language to talk about things. Using “Lucas critique” seems to be no good since the name itself is not descriptive and it was originally in specific reference to macroeconomics which might confuse people searching it.

      • Michael Nelson says:

        Thanks for sending me down a rabbit hole on Wikipedia. :) Seriously, I do see the parallel, and models that incorporate “micro-foundations” of campaigning may do better. But the interesting twist here is that where Lucas was talking about the intentions of public policy being subverted, campaigns aren’t subverting anything. They’re using inferential methods to optimize outcomes in a perfectly rational and intuitive way, and they’d do that regardless of whether there were public polls. Naturally, better polling methodology would serve campaigns better. There’s nothing surprising there.

        What does surprise is the realization that satisfying the public’s curiosity isn’t what polls are fundamentally good for, so doing a poor job of predicting election outcomes isn’t a fault in the polling, and improving the polling methods won’t result in better satisfaction of the public’s curiosity. To the contrary, if public polls made perfect predictions, then campaigns wouldn’t matter, and that’s contrary to the public good. Polls are primarily good at providing a means to alter what they predict, secondarily good at promoting those who conduct them, and tertiarily good at providing content for media. They’re doing those things perfectly. So what at first seems like a paradox is really just a natural consequence of people mistaking polling for a public good.

  9. fogpine says:

    For those interested in #3, here is a histogram of all state results for presidential elections 2000-2020.

    Republicans and Democrats run neck-and-neck in some states, but not so much more often than would be expected from the rest of the state results distribution. In other words, there is a spike in two-party vote shares near 50%, but it does not stand out all that much from the rest of the histogram.

    Over this and past USA presidential elections my memory says that many states have been won by <<1% between the major parties. I've heard it suggested that this is due to parties modifying their platforms to appeal to just enough voters to win in relevant swing states. It makes sense that they try to rebalance this way, but as a mechanism for the near perfect splits we often observe it is insufficient

    Since the spike at 50% is not so large, I don’t agree with the above quote. In particular, it doesn’t seem clear that a new explanation is needed for the number of state results near 50%, even in the presence of polling inaccuracy.

    So if close state results are not too disproportionately common, why do they FEEL so common? They feel very common to the OP and to me too!

    I think this is because of the electoral college. Use of the electoral college means that close state results frequently determine the presidential winner, and therefore we hear about them often. So results near 50% are more present in our minds, even if they are not actually so much more common than results at nearby percentages.

    Here is an article that nicely explains how use of the electoral college means that close state results are unduly likely to determine the presidential election winner:

    However, I haven’t checked any of this for congressional elections, which would probably be a better data source for evaluating whether election results are disproportionately likely to be close.

    P.S. In the histogram, the bin width is half a percentage point. Results can be very sensitive to bin width in this kind of bump hunting, but when I reduce bin width further, this doesn’t produce a more disproportionate spike at 50%.

Leave a Reply

Where can you find the best CBD products? CBD gummies made with vegan ingredients and CBD oils that are lab tested and 100% organic? Click here.