Remember that paper we wrote, The mythical swing voter? About shifts in the polls being explainable by differential nonresponse? Mark Palko beat us to this idea, by 4 years.

So. The other day I came across a link by Palko to this post from 2012, where he wrote:

Pollsters had long tracked campaigns by calling random samples of potential voters. As campaign became more drawn out and journalistic focus shifted to the horse race aspects of election, these phone polls proliferated. At the same time, though, the response rates dropped sharply, going from more than one in three to less than one in ten.

A big drop in response rates always raises questions about selection bias since the change may not affect all segments of the population proportionally . . . It also increases the potential magnitude of these effects. . . .

Poll responses are basically just people agreeing to talk to you about politics, and lots of things can affect people’s willingness to talk about their candidate, including things that would almost never affect their actual votes . . .

[In September, 2012] the Romney campaign hit a stretch of embarrassing news coverage while Obama was having, in general, a very good run. With a couple of exceptions, the stories were trivial, certainly not the sort of thing that would cause someone to jump the substantial ideological divide between the two candidates so, none of Romney’s supporters shifted to Obama or to undecided. Many did, however, feel less and less like talking to pollsters. So Romney’s numbers started to go down which only made his supporters more depressed and reluctant to talk about their choice.

This reluctance was already just starting to fade when the first debate came along. . . . after weeks of bad news and declining polls, the effect on the Republican base of getting what looked very much like the debate they’d hoped for was cathartic. Romney supporters who had been avoiding pollsters suddenly couldn’t wait to take the calls. By the same token. Obama supporters who got their news from Ed Schultz and Chris Matthews really didn’t want to talk right now.

The polls shifted in Romney’s favor even though, had the election been held the week after the debate, the result would have been the same as it would have been had the election been held two weeks before . . .

So response bias was amplified by these factors:

1. the effect was positively correlated with the intensity of support

2. it was accompanied by matching but opposite effects on the other side

3. there were feedback loops — supporters of candidates moving up in the polls were happier and more likely to respond while supporters of candidates moving down had the opposite reaction.

The above completely anticipates the main result of our Mythical Swing Voter paper, which is based on the Xbox polling data we collected in 2012, analyzed in 2013, wrote up in 2014, and published in 2016, and which was picked up in the news media in time for the 2016 campaign.

I’m not saying our paper was valueless: we didn’t just speculate, we provided careful data analysis. The thing is, though, that the pattern we found, that big swings in Obama support could mostly be explained by differential nonresponse, surprised us. It wasn’t what we expected, it’s not something we thought about at all in our 1993 paper, and it took us awhile to digest this finding. But Palko had already laid out the whole story, all the way including the feedback mechanism by which small swings in vote preference are magnified into big swings in the polls, with all this connecting to the rise in survey nonresponse.

I probably even read Palko’s post when it came out back in 2012, but, if so, I didn’t get the point.

There’s something wrong with the world that his blog (cowritten with Joseph Delaney) doesn’t have a million readers.

P.S. Doug Rivers (one of my coauthors on the Mythical Swing Voter paper) was also talking in 2012 about differential nonresponse; see the last three paragraphs here.

18 thoughts on “Remember that paper we wrote, The mythical swing voter? About shifts in the polls being explainable by differential nonresponse? Mark Palko beat us to this idea, by 4 years.

  1. This is one of the reasons I continue to respect and follow your work. When you discover you missed something, you explain what happened and what you learned from the process.

  2. Well, one reason that blog does not have a million readers is nonsense like this:

    The Republicans’ chances of holding anything more than an entrenched court and a few statehouses are very small and dependent on doing two things: slowing these trends and keeping their coalition completely intact.

    Is he really offering odds (like 20:1) that the Republicans won’t have at least one of the House/Senate/Presidency for at least a part of 2020 to 2030?

    Please, let’s bet on this!

      • I am happy to read smart people with whom I disagree as long as they take the best arguments of their opponents seriously. This isn’t what Palko is doing:

        What matters is that the conservative movement has spent more than a quarter century using distorted news and disinformation to cultivate a base motivated by anxiety bordering on panic and anger bordering on rage.

        Good summary!

        Andrew: Is this how you would describe the data about Republican voters? I doubt it!

        But I agree that the take-down of hyperloop nonsense is excellent.

  3. Off Andrew’s specific point, but likely of interest to those concerned about response rates, is this time trend of phone survey response rates by Pew, since 1997 (36%) to 2018 (6%).

    http://www.pewresearch.org/fact-tank/2019/02/27/response-rates-in-telephone-surveys-have-resumed-their-decline/

    Pew is now switching from phone to online surveying, so this series will likely end. They haven’t gone to XBox yet.

    https://www.pewresearch.org/fact-tank/2019/02/27/what-our-transition-to-online-polling-means-for-decades-of-phone-survey-trends/

  4. Since voting itself is just another poll, wouldn’t differential nonresponse affect the actual vote as well? If voters are so demoralized that they won’t pick up the phone, then why would they actually get off the couch and vote? How is it the the actual vote is consider stable so much so that “had the election been held the week after the debate, the result would have been the same as it would have been had the election been held two weeks before”?

    Perhaps people may not be persuadable about their vote preference. But is not turnout persuadable?

    • Dalton:

      We discuss this in the paper. Short answer is that voter turnout rate in presidential elections is 60% while survey response rates are typically below 10%. So I think survey response is much more contingent on situation, than is voter turnout.

      • …Voter-Turnout is well below 20% for Republican & Democratic Presidential Primaries.

        Since multiple candidates compete in Primaries… the Primary winner typically has less than 10% formal voter support.

        Thus, the usual two main candidates (a Republican & a Democrat) presented on the national Presidential ballot to the American electorate … severely lack voter endorsement as candidates. Majority Rule democracy is thwarted from the getgo.

        U.S. Congress and state legislatures all have strict quorum requirements — American elections need them too.

        (“I don’t care who does the votin’ as long as we do the nominatin’ !”) -NYC Tammany Hall motto

      • Andrew:

        Could you point me towards any public data for a similar panel survey where I might observe the individual temporal trends in non-response?

        Thanks!

        • Nat:

          I’m not sure. We refer to a couple other surveys in our paper, so that would be a start. I expect you’d see this pattern in just about any panel survey. I’m not quite sure what’s out there but there must be a lot, and some of it should be publicly available. The Roper Center has individual-level data on a lot of old polls from many sources; sometimes once the survey is a year old, the polling organization will make the data generally available.

  5. If I get Gelman et al Q J political sci, campaigns spend billions of dollars based on totally flawed ideas of how voters operate
    A related thing is here
    https://www.nytimes.com/2022/04/08/opinion/russian-gas-germany-acid-rain.html

    also: when smoking bans in bars/restaurants were 1st proposed, the bar/rest owners went ballistic, saying they would rapidly go out of biz cause no one would go out to a bar or restaurant if they couldn’t smoke

    upshot:
    most people running large businesses or campaigns have no idea what they are doing; this isn’t an isolated thing but a widespread thing

    • I remember that smoking ban thing. I took a business trip to DC and stopped in to an Irish pub for lunch (must have been 1997 +-), was like the only one there at 12:30 pm or whatever, the bartender heard I was on a trip from CA and they had just passed the CA ban, he was shocked and sure that it’d destroy his business… Within 6 months or so it was obvious: attendance at brewpubs and restaurants and bars was UP fairly dramatically in CA and stayed up for a long time. I haven’t looked at it since the early 2000’s but yeah. This is one of those things where bias isn’t recognized. Yes the bartenders know that the customer’s they currently have would be upset about not being able to smoke, but what they don’t see is the much larger group of people who just choose not to come because the air is polluted and terrible.

Leave a Reply to Dalton Cancel reply

Your email address will not be published. Required fields are marked *