Skip to content
 

Remember that paper we wrote, The mythical swing voter? About shifts in the polls being explainable by differential nonresponse? Mark Palko beat us to this idea, by 4 years.

So. The other day I came across a link by Palko to this post from 2012, where he wrote:

Pollsters had long tracked campaigns by calling random samples of potential voters. As campaign became more drawn out and journalistic focus shifted to the horse race aspects of election, these phone polls proliferated. At the same time, though, the response rates dropped sharply, going from more than one in three to less than one in ten.

A big drop in response rates always raises questions about selection bias since the change may not affect all segments of the population proportionally . . . It also increases the potential magnitude of these effects. . . .

Poll responses are basically just people agreeing to talk to you about politics, and lots of things can affect people’s willingness to talk about their candidate, including things that would almost never affect their actual votes . . .

[In September, 2012] the Romney campaign hit a stretch of embarrassing news coverage while Obama was having, in general, a very good run. With a couple of exceptions, the stories were trivial, certainly not the sort of thing that would cause someone to jump the substantial ideological divide between the two candidates so, none of Romney’s supporters shifted to Obama or to undecided. Many did, however, feel less and less like talking to pollsters. So Romney’s numbers started to go down which only made his supporters more depressed and reluctant to talk about their choice.

This reluctance was already just starting to fade when the first debate came along. . . . after weeks of bad news and declining polls, the effect on the Republican base of getting what looked very much like the debate they’d hoped for was cathartic. Romney supporters who had been avoiding pollsters suddenly couldn’t wait to take the calls. By the same token. Obama supporters who got their news from Ed Schultz and Chris Matthews really didn’t want to talk right now.

The polls shifted in Romney’s favor even though, had the election been held the week after the debate, the result would have been the same as it would have been had the election been held two weeks before . . .

So response bias was amplified by these factors:

1. the effect was positively correlated with the intensity of support

2. it was accompanied by matching but opposite effects on the other side

3. there were feedback loops — supporters of candidates moving up in the polls were happier and more likely to respond while supporters of candidates moving down had the opposite reaction.

The above completely anticipates the main result of our Mythical Swing Voter paper, which is based on the Xbox polling data we collected in 2012, analyzed in 2013, wrote up in 2014, and published in 2016, and which was picked up in the news media in time for the 2016 campaign.

I’m not saying our paper was valueless: we didn’t just speculate, we provided careful data analysis. The thing is, though, that the pattern we found, that big swings in Obama support could mostly be explained by differential nonresponse, surprised us. It wasn’t what we expected, it’s not something we thought about at all in our 1993 paper, and it took us awhile to digest this finding. But Palko had already laid out the whole story, all the way including the feedback mechanism by which small swings in vote preference are magnified into big swings in the polls, with all this connecting to the rise in survey nonresponse.

I probably even read Palko’s post when it came out back in 2012, but, if so, I didn’t get the point.

There’s something wrong with the world that his blog (cowritten with Joseph Delaney) doesn’t have a million readers.

P.S. Doug Rivers (one of my coauthors on the Mythical Swing Voter paper) was also talking in 2012 about differential nonresponse; see the last three paragraphs here.

16 Comments

  1. John R Minter says:

    This is one of the reasons I continue to respect and follow your work. When you discover you missed something, you explain what happened and what you learned from the process.

  2. D Kane says:

    Well, one reason that blog does not have a million readers is nonsense like this:

    The Republicans’ chances of holding anything more than an entrenched court and a few statehouses are very small and dependent on doing two things: slowing these trends and keeping their coalition completely intact.

    Is he really offering odds (like 20:1) that the Republicans won’t have at least one of the House/Senate/Presidency for at least a part of 2020 to 2030?

    Please, let’s bet on this!

  3. zbicyclist says:

    Off Andrew’s specific point, but likely of interest to those concerned about response rates, is this time trend of phone survey response rates by Pew, since 1997 (36%) to 2018 (6%).

    http://www.pewresearch.org/fact-tank/2019/02/27/response-rates-in-telephone-surveys-have-resumed-their-decline/

    Pew is now switching from phone to online surveying, so this series will likely end. They haven’t gone to XBox yet.

    https://www.pewresearch.org/fact-tank/2019/02/27/what-our-transition-to-online-polling-means-for-decades-of-phone-survey-trends/

  4. Dalton says:

    Since voting itself is just another poll, wouldn’t differential nonresponse affect the actual vote as well? If voters are so demoralized that they won’t pick up the phone, then why would they actually get off the couch and vote? How is it the the actual vote is consider stable so much so that “had the election been held the week after the debate, the result would have been the same as it would have been had the election been held two weeks before”?

    Perhaps people may not be persuadable about their vote preference. But is not turnout persuadable?

    • Andrew says:

      Dalton:

      We discuss this in the paper. Short answer is that voter turnout rate in presidential elections is 60% while survey response rates are typically below 10%. So I think survey response is much more contingent on situation, than is voter turnout.

      • Gary says:

        …Voter-Turnout is well below 20% for Republican & Democratic Presidential Primaries.

        Since multiple candidates compete in Primaries… the Primary winner typically has less than 10% formal voter support.

        Thus, the usual two main candidates (a Republican & a Democrat) presented on the national Presidential ballot to the American electorate … severely lack voter endorsement as candidates. Majority Rule democracy is thwarted from the getgo.

        U.S. Congress and state legislatures all have strict quorum requirements — American elections need them too.

        (“I don’t care who does the votin’ as long as we do the nominatin’ !”) -NYC Tammany Hall motto

  5. Ignazio Ziano says:

    Has nonresponse bias been systematically investigated from a psychological point of view?

  6. Nat says:

    Where can I download the Xbox polling data?

    • Andrew says:

      Nat:

      Private data, can’t be downloaded, sorry!

      • Nat says:

        Andrew:

        Could you point me towards any public data for a similar panel survey where I might observe the individual temporal trends in non-response?

        Thanks!

        • Andrew says:

          Nat:

          I’m not sure. We refer to a couple other surveys in our paper, so that would be a start. I expect you’d see this pattern in just about any panel survey. I’m not quite sure what’s out there but there must be a lot, and some of it should be publicly available. The Roper Center has individual-level data on a lot of old polls from many sources; sometimes once the survey is a year old, the polling organization will make the data generally available.

Leave a Reply