Skip to content

“If the horse race polls were all wrong about Trump, why should his approval rating polls be any better?”

A journalist forwarded the above question to me and asked what I thought.

My reply is that the horse race polls were not all wrong about Trump. The polls had him at approx 48% of the two-party vote and he received 49%. The polls were wrong by a few percentage points in some key swing states (as we know, this happens, it’s called non-sampling error), but for the task of measuring national opinion the polls did just fine, they were not “all wrong” as the questioner seemed to think.

P.S. Some polls during the campaign did have Clinton way in the lead but we never thought those big bounces were real, and based on the fundamentals we were also anticipating a close election.


  1. Jonathan says:

    I’m more aghast at the thought behind the question: it compares projections of who will win a contest in which there is an absolute winner with some generalized response which doesn’t relate to a contest at all. It’s the continuing extension of the “horse race reporting” which journalism has succumbed to: they look for things they can treat as a horse race but without any sense of irony, as though they actually believe everything is a win or lose contest at each moment or week or story. I could label this the “IsPa effect”: in the Israel/Palestine conflict, every moment, every act or non-act, every statement or non-statement, is “contested”, meaning treated as though each of these insignificant moments were literally the embodiment of the entire conflict. From what I’ve been seeing over the past weeks, this is exactly what is happening in America: we’re approaching the point where no point is uncontested and that means every difference tends to be magnified – at least on the poles (so if you invert the space, then you get a normal-ish distribution centered on the core activist beliefs, etc.) – and that not only makes compromise difficult but can allow one side to do a McGovern or Goldwater and go too far away from the center.

    • Andrew says:


      I think you’re being too hard on McGovern and Goldwater. Both candidates lost in landslides, but under political and economic conditions such that I expect just about anyone would’ve lost in a landslide. And, sure, they seem like extremists but in part this is because they lost. Reagan was seen as an extremist but then he won. Both McGovern and Goldwater offered platforms that at least in theory could’ve appealed to many millions of disaffected voters.

  2. Any idea whether those few swing regions were primarily electronic polling machine locations? The computer programming industry has been talking about the crap security of all kinds of products including polling machines for decades. It was a common topic on Slashdot back in the late 1990’s. Given the revelations from Edward Snowden it seems totally banally plausible that some smart Russian hackers added a few votes here and there in a clever pattern adding in just enough to swing the key states.

    Now, it’s totally plausible as well that this didn’t happen. But I get the sense that people outside the computer industry would have found the idea of large-scale voting machine hacks pretty implausible before it became common knowledge how easily computer systems are compromised. In say 2007 few would have rated it “plausible” that say the Chinese could systematically shut down the US power grid via remote hacks. But, it shouldn’t seem too implausible to anyone who’s been paying attention in the last decade.

  3. Josh says:

    Another point of difference is that election polls are trying to estimate both the composition of the electorate and also who that electorate will likely support. In contrast, public opinion polls that attempt to assess public approval are usually either based on a general population or registered voter sampling frame and this removes potential errors that can be introduced by the need to estimate who “likely voters” are.

  4. Phil Koop says:

    “Some polls during the campaign did have Clinton way in the lead …”

    I doubt that your journalist was thinking of these polls. I think it is more likely that the essence of the matter is reasoning backwards from effect to cause: if the effect is large, then surely so must be the cause. Not that different from confusing statistical significance and clinical effect size, really.

  5. Rob MacCoun says:

    The debate about the polls has been confusing because of a failure to distinguish between (a) the national poll estimates, which did pretty well; (b) the state poll estimates, which did less well; and (c) the Upshot and 538 probability-of-winning forecasts — modeling not polling — which gave HRC such strong chance of winning. Polling didn’t do nearly as bad as forecasting. And forecasting did poorly because of correlated error; the states that were off were all off for similar reasons, creating a cascade of error — something Nate Silver did warn about.

  6. satoshi says:

    The critics of the recent popularity polls were saying that pollsters “oversampled” one group or another. What do you think about that objection?

    • Andrew says:


      I followed this link and it’s all wrong. It’s not appropriate for them to focus on a poll that showed a 12-point Clinton lead. As I wrote in my P.S. above, those polls were highly variable and we always thought the election was close.

      Also, I’ve been suspicious of the “Zero Hedge” site ever since encountering this story from several years ago in which they garbled data in order to make a ridiculous propaganda-style claim (of which both Tyler Cowen and Kaiser Fung were, rightly, suspicious).

      • satoshi says:

        >> I followed this link and it’s all wrong. It’s not appropriate for them to focus on a poll that showed a 12-point Clinton lead.
        I am sorry if I a missing something obvious, but cold you comment on the oversampling more specifically? Is it a real danger generally? Is it something they are right about with respect to the latest Trump popularity numbers?

  7. Thomas B says:

    By definition and design, the electoral college is a separate and distinct mechanism from the popular election. Given that, it’s worth noting that the polls and prediction markets both got the popular election results right: Clinton won the popular election by a margin of over 2.5 million votes. Except by extrapolation from the popular vote, no one “predicted” the results of the electoral college which is where Trump “won” the election.

  8. Nadia Hassan says:

    Your recommended approach to conceptualizing election surveys is % of 2-way vote +/- 2. What would you recommend for approval?

Leave a Reply