How much of Trump’s rising approval numbers can be attributed to differential nonresponse? P.S. With more analysis of recent polls from Jacob Long

Josh Marshall writes:

If you follow polling you know that over the last couple weeks President Trump’s approval numbers have been trending up. . . . 538’s composite average has notched up a couple points over the second half of January and this morning Gallup released the highest approval rating of Trump’s presidency: 49%. Why is this happening?

Marshall goes through several natural explanations (economic prosperity, trade agreements, acceptance of Trump’s behavior), and then continues:

But there is another plausible explanation. Pollsters call it differential response. When one side gets enthused or energized their numbers go up but in an ephemeral fashion. The pumped-up side is a bit more eager to answer the phone or fill out the survey. The demoralized side is a bit less eager. This is a real and demonstrated phenomenon, not just a concept or speculation. It’s not necessarily an error per se in the polling. It’s picking up something real. It’s just ephemeral.

That’s right. It’s what Sharad Goel, Douglas Rivers, David Rothschild, and I found in our analysis of polling data from 2012, and the same pattern appeared in 2016, as illustrated in this graph from Alan Abramowitz.

image001

Returning to today, Marshall continues:

There are good reasons to think that at least some of that is happening today — the President’s impending acquittal and Republican unity have been the driving news of the last two or three weeks. Republicans are energized and enthused by the certainty of President Trump’s acquittal. Many Democrats are demoralized by seeing an overwhelming and exacting case made for the President’s guilt and seeing it simply not matter.

This is something we should be able to see in the data. It’s testable.

Most simply, if there’s differential nonresponse, then when the president’s approval is going up, we should also see an increasing percentage of self-declared Republicans in the sample.

Let’s take a look at that Gallup poll where Trump has an unprecedentedly high 49% approval:

Additionally, the poll finds 48% of Americans identifying as Republicans or leaning toward that party, compared with 44% Democratic identification or leaning. Recent Gallup polls had shown a fairly even partisan distribution, after the Democratic Party held advantages for much of 2019.

So, yeah, some of that shift can be explained by differential nonresponse: more Republicans and fewer Democrats answering the poll. This explanation for the change is not mentioned in the Gallup report, but we can read between the lines and see it.

For another example, let’s look at the Harris poll, which puts Trump at 46% approval:

Results were weighted for age within gender, region, race/ethnicity, marital status, household size, income, employment, education, political party and political ideology where necessary to align them with their actual proportions in the population.

They adjust for party ID, so differential nonresponse should not be an issue. And from the CBS News poll, where Trump has 43% approval, we see that the weighted number of respondents of each party are 298 Republicans, 362 Democrats, and 542 Independents: that’s 25% R, 30% D, 45% I.

Marshall concludes:

I’m pretty confident that at least some of this is ephemeral differential response. But I am not confident that’s all of it. We just don’t know and won’t know for a while.

I agree. Not just about the “we just don’t know” etc. but also that I think that some, but not all, of the recent trend is attributable to differential nonresponse. Again, that’s consistent with what we saw in 2012.

To answer the question more quantitatively, you’d want to look at a large number of polls and track both presidential approval and party identification. It shouldn’t be hard to do this.

P.S. Jacob Long did what I asked!

Here are his graphs:

And here’s his summary:

Using a statistical method called multilevel modeling, I [Long] can estimate how big an impact differential nonresponse has on the polling results after accounting for several other things known to affect the outcome . . . The model suggests that for every 1 percentage point increase in the difference between Democrats and Republicans in the sample, you should expect Trump’s net approval to drop by about 0.7 percentage points.

The model also suggests that even after accounting for differential nonresponse, Trump may have gained somewhat in net approval since August, perhaps 2 percentage points (or roughly half the increase detected by FiveThirtyEight’s tracker). But that upward trend goes away when I restrict the analysis to polls that measure partisanship best, by including “leaners” who call themselves independent but later disclose a clear preference for one party over the other. . . .

That said, as Long notes, “maybe with an election coming, Trump is bringing in more support, as his challengers become better known. Incumbent presidents typically become more popular during election years. Barack Obama’s approval went from a similar level to Trump’s to over 50 percent as he ramped up his reelection campaign in 2012.”

17 thoughts on “How much of Trump’s rising approval numbers can be attributed to differential nonresponse? P.S. With more analysis of recent polls from Jacob Long

  1. When I eyeball a poll aggregator like 538, it seems to me like the rises and falls in Trumps’ approval rating are driven by the latest Rasmussen Reports/Pulse Opinion Research. Every time it comes out, Trump’s numbers improve, and then they gradually drop. I would love for someone with real knowledge to comment on that.

    • I’ve noticed the same thing. Rassmassen most recently has posted 2 outlier polls that have bumped down the disapproval a bit. None of the other polls are showing any major movement: his approval remains stuck in the low forties, his disapproval in the low 50s, with a pretty consistent 10 point differential.

    • Their model tracks bias for each pollster. So no, in short, if a pollster says what the model thinks the pollster will say, then there shouldn’t be a rise/fall in their model output.

      • The thing though is, those of us who would agree with Steve, I think, have a gut feeling that Rasmussen’s outliers are getting more outlier-ly. Like they’ve cranked the bias up to 11 or something.

        But I’m not sure that can actually be proven in real time. It’s also possible the other polls are moving in the opposite direction.

      • Andrew:

        I appreciate the response. I am concerned though because I thought that their tracking poll for 2016 was much closer than many of the other polls. I am going off memory. But, my actual question was isn’t the truth that Trump’s numbers are super stable and every time we see a trend over the last several years, it is in large part driven by Rasmussen or the polling bias of the various polls. When I look at each individual pollster, I just see the same basic numbers over and over. But the aggregators show movement that seems to have more to do with which set of polls are the latest to come out, and of course, Rasmussen is the biggest outlier. Am I right that there has been an amazing amount of stability in Trump’s numbers?

        • Steve,

          What with polarization, there has been striking stability in Trump’s approval numbers. But they’ve increased in the last couple of months. I suspect some but not all of this is from differential nonresponse.

        • I’ve been trying to gauge the direction of the election by aggregating polls and I, too, rate current data higher than old data. Makes senses, doesn’t it?

  2. I would hypothesize that non-differential response for an approval rating is a feature not a bug, as far a predicting election outcomes; it will correlate with voter turnout. It will prove at least as important as rates of shark attacks.

    Is the claim that non-differential response rates are more ephemeral than the approval rating is ephemeral? That’s pretty ephemeral! Such fine tools are what enable us to know future outcomes with such precision.

    • Dzhaughn:

      We discuss this in our 2016 paper. The short answer is that turnout in presidential elections is about 60%, but response rates in polls are less than 10%. Survey response is much more volatile than turnout in the general election.

  3. Steve, thanks. I’ve felt this way about Rasmussen’s effects on 538 for some time. When you watch the 538 aggregate every day, Rasmussen appears to have at least an annoying effect. Assuming 538’s bias adjustment is good, this is only an impression. However, I think 538 ought to re-evaluate. Rasmussen has plenty of statisticians too. They could be tailoring to 538. They could see holes 538 doesn’t. Quite simply, Rasmussen could simply realize the bias adjustment is just weak enough for Rasmussen to have influence on 538.

  4. “Trump is bringing in more support, as his challengers become better known. Incumbent presidents typically become more popular during election years.…”

    Well it’s fun to track approval ratings but does this have any direct bearing on the election? Does “approval rating” equate to “election support”? I don’t know about statistically, but taken at face value it doesn’t seem appropriate to equate them. Ultimately people will between Trump and X Democrat. Without knowing who “X Democrat” is, people can’t make that choice. The country is strongly partisan but most elections are decided by less than 2%, and with the extreme range of positions on the Dem’s side, who becomes “X Democrat” probably will swing the election – as it did in 2016.

  5. Tracking true party identification might be difficult. Trump’s behavior is bad, along with nearly all Replicans supporting him. It is my understanding a significant percentage of former Republicans do not want to be identified as Republicans, swelling the number of claimed independents. Trumps support among remaining Republicans may appear to increase simply because the remainers are increasingly a more nazified sample.

Leave a Reply to Andrew Cancel reply

Your email address will not be published. Required fields are marked *