A journalist sent me a bunch of questions regarding problems with polls. Here was my reply:
In answer to your question, no, the polls in Brexit did not fail. They were pretty good. See here and here.
The polls also successfully estimated Donald Trump’s success in the Republican primary election.
I think that poll responses are generally sincere. Polls are not perfect because they miss many people, hence pollsters need to make adjustments; see for example here and here.
I hope this helps.
This notion that polls for Brexit failed completely mystifies me. They were clearly indicating that the result was too close to call. I don’t know if people are just saying that because they want to undermine polls, or if they just lack the basic understanding of how polls work.
Phil:
Yeah, this deserves its own post, something like The Myth of Brexit Poll Failure. The short story is that the polls did not fail but the prognosticators failed. The experts said that Brexit was going to lose. It was a lot like the pundits’ failure with the Donald Trump candidacy in the Republican primaries: the polls were accurate but the experts were telling us to not listen to the polls.
Here is a proposal for measuring intentional bias in the presidential polls. It rests on the assumption that pollsters will eliminate their intentional bias before the election because they want their final numbers to be as close to the actual outcome as possible for reputational reasons.
Background:
1. The range of current polling results is startlingly large. See, e.g., Real Clear Politics at http://www.realclearpolitics.com/epolls/2016/president/us/general_election_trump_vs_clinton_vs_johnson_vs_stein-5952.html
2. Some polls are systematically higher or lower than the average. The Rasmussen and LA Times polls jump out in this regard.
3. Systematic deviations may be due to honest differences in assumptions and methodology, or they may be due to intentional bias.
4. For reputational reasons, pollsters will want to say after the election that their polls were close to the election results. Therefore, the last polls will reflect little or no intentional bias.
5. Therefore, poll movements just before the election can be used to estimate intentional bias in the prior as pollsters eliminate intentional biases in the last days before the election.
Measuring this elimination of intentional bias is tricky:
1. A simple test of aggregate intentional bias is to look at the dispersion of the poll numbers. Elimination of intentional biases should reduce dispersion and the amount of the reduction could be used to estimate average intentional bias across polls. This would not, however, tell us which polls were intentionally biased.
2. If we assume that voter’s minds are made up already and there are no significant developments before the election, then simple deltas in the polls would measure intentional bias. This is not a bad approach in this election because voter attention has been very high for a long time, and the choices are stark. But of course, other factors are always present even when there are no clear exogenous events. For instance, the effects of scandals may dissipate over time.
3. Exogenous developments could be controlled for by looking at how much each poll moves toward the average of all the polls. This adjustment is reasonably good if only a small number of polls are intentionally biased, or if the distribution of intentional biases is mean zero. If not, the average may be affected in annoying ways by the elimination intentional bias.
4. Looking at poll deltas toward the actual election outcome could be used, but that risks conflating intentional bias with unintentional bias due to honest, incorrect assumptions.