This is a great example for a statistics class, or a class on survey sampling, or a political science class

Under the heading, “Latino approval of Donald Trump,” Tyler Cowen writes:

From a recent NPR/PBS poll:

African-American approval: 11%

White approval: 40%

Latino approval: 50%

He gets 136 comments, many of which reveal a stunning ignorance of polling. For example, several commenters seem to think that a poll sponsored by National Public Radio is a poll of NPR listeners.

Should NPR waste its money on commissioning generic polls? I don’t think so. There are a zillion polls out there, and NPR—or just about any news organization—has, I believe, better things to do than create artificial headlines by asking the same damn polling question that everyone else does.

In any case, it’s a poll of American adults, not a poll of NPR listeners.

The other big mistake that many commenters made was to take the poll result at face value. Cowen did that too, by reporting the results as “Latino approval of Donald Trump,” rather than “One poll finds . . .”

A few things are going on here. In no particular order:

1. Margin of error. A national poll of 1000 people will have about 150 Latinos. The standard error of a simple proportion is then 0.5/sqrt(150) = 0.04. So, just to start off, that 50% could easily be anywhere between 42% and 58%. And, of course, given what else we know, including other polls, 42% is much more likely than 58%.

That said, even 42% is somewhat striking in that you might expect a minority group to support the Republican president less than other groups. One explanation here is that presidential approval is highly colored by partisanship, and minorities tend to be less partisan than whites—we see this in many ways in lots of data.

2. Selection. Go to the linked page and you’ll see dozens of numbers. Look at enough numbers and you’ll start to focus on noise. The garden of forking paths—it’s not just about p-values. Further selection is that it’s my impression that Cowen enjoys posting news that will fire up his conservative readers and annoy his liberal readers—and sometimes he’ll mix it up and go the other way.

3. The big picture. Trump’s approval is around 40%. It will be higher for some groups and lower for others. If your goal is to go through a poll and find some good news for Trump, you can do so, but it doesn’t alter the big picture.

I searched a bit on the web and found this disclaimer from the PBS News Hour:

President Trump tweeted about a PBS NewsHour/NPR/Marist poll result on Tuesday, highlighting that his approval rating among Latinos rose to 50 percent. . . .

However, the president overlooked the core finding of the poll, which showed that 57 percent of registered voters said they would definitely vote against Trump in 2020, compared to just 30 percent who said they would back the president. The president’s assertion that the poll shows an increase in support from Latino voters also requires context. . . .

But only 153 Latino Americans were interviewed for the poll. The small sample size of Latino respondents had a “wide” margin of error of 9.9 percentage points . . . [Computing two standard errors using the usual formula, 2*sqrt(0.5*0.5/153) gives 0.081, or 8.1 percentage points, so I assume that the margin of error of 9.9 percentage points includes a correction for the survey’s design effect, adjusting for it not being a simple random sample of the target population. — AG.] . . .

[Also] The interviews in the most recent PBS NewsHour/NPR/Marist poll were conducted only in English. . . .

They also report on other polls:

According to a Pew Research Center’s survey from October, only 22 perfect of Latinos said they approved of Trump’s job as president, while 69 percent said they disapproved. . . . Pew has previously explained how language barriers and cultural differences could affect Latinos’ responses in surveys.

That pretty much covers it. But then the question arises: Why did NPR and PBS commission this poll in the first place? There are only a few zillion polls every month on presidential approval. What’s the point of another poll? Well, for one thing, this gets your news organization talked about. They got a tweet from the president, they’re getting blogged about, etc. But is that really what you want to be doing as a news organization: putting out sloppy numbers, getting lots of publicity, then having to laboriously correct the record? Maybe just report some news instead. There’s enough polling out there already.

32 thoughts on “This is a great example for a statistics class, or a class on survey sampling, or a political science class

  1. OK, one poll is just one poll, it has sampling variability, and is emerging from the ocean of polls because of its eye-catching finding. All true, but surely whatever we thought before this poll, this data should move our estimate of Trump’s support among Hispanics upward, right? Maybe not a whole lot, given all the variability and the wealth of data besides this, but not zero either. And isn’t that newsworthy? Would you be OK if the headline was “new poll shifts best estimate of Hispanic support for Trump up a bit”? Definitely less eye-catching, but still important given popular perceptions.

    • Would “new poll shows support of English-speaking Hispanics for Trump is relatively high” be newsworthy? Because this seems to be the right headline. No irony intended, I’m not American, so I just don’t know about the US new cycle and I’m curious.

    • Not necessarily, I mean, upward from where? And after adjustment how?

      If we pretend this result is from a random sample of the population of Latino voters, it should move our estimate towards 50%… but pretending that would be wrong. The real question is given the current polling methodology, do polls have much of any meaningful content? Especially sub-polls. I think polls adjusted using MRP by careful researchers probably can tell us +- 5 to 10 percentage points what the approval is… But +- 5-10% is pretty close to the same amount of information that’s already in our prior based on just what we know about our neighbors opinions and things… If you just asked me what was Trump’s approval, I’d have guessed it was say between 35 and 50% So is the individual poll adding much given its true uncertainty?

    • How about something like “Variability in polls on Hispanic support for Trump” — focusing on a serious discussion of variability in polling results. (Yeah, I know — not sexy; but what we really need.)

  2. I guess that polls are published because people view elections like horse races. There will be a winner, and while the race is on we listen to see who is ahead at the present time. Polling requires sampling, and this entails the sources of error, bias, and plain statistical noise that are Dr. Gelman’s concerns.

  3. Andrew, serious question: What is the minimum sufficient number of polls on presidential approval? If all polls used a standard, best-practice methodology, and especially if they all used true random samples, this would be a simple math question, and the answer would be “the number of polls that will reduce the margin of error to the desired level.” However, due to real life constraints including but not limited to money and time, all polls do not use the same methods, and few if any use random samples. A variety is approaches is actually better. We want different polls to use different methods (or, less realistically, each poll would use all reasonable methods) because there is no one best practice. Online polls, phone polls, automated polls, live-interviewer polls, multi-lingual polls, registered-voter polls, likely-voter polls, etc., all reach slightly different sub-populations. Things like question phrasing may also be better off varying across polls, if we believe that item wording is sampled from a population of questions that may be perceived differently by different voters. In the aggregate, a variety of polls is more likely to give a full(er) picture, which is one argument for more polls even of generic but interesting questions.

    Another argument is that most polls have significant bias–either because the poll is conducted/sponsored by ideological organizations, or due to simple house effects at ostensibly neutral polling companies. If you have sufficient archival data of an organization’s past poll results, you can generate weights based on estimated past bias (as Nate Silver does), though that’s another source of uncertainty. Ideally, you will (in the long run) have about as many left-leaning polls as right-leaning polls so their political bias cancels out, as well as plenty of polls that try to stay non-ideological whose smaller house effects are approximately randomly distributed around the true proportion. But if the point of polling this question is to take a snapshot, we won’t have the opportunity to look at the long run over time, but we can look at a larger number of polls in the short run.

    One last factor: As I said, you’d like to have many polls sponsored by organizations without an overt political agenda, and NPR seems like an ideal candidate. News organizations have an incentive to sponsor polls because it gives them content, which as you point out is not the best motivation, but it is better than the motivation of moving the very public opinion you are measuring.

    Given all this, are you still confident that there are enough polls out there that news organizations should stop sponsoring them, even if it means the remaining polls will be significantly more ideologically biased?

  4. “Margin of error” is irrelevant because this is non-probability sampling. So…that ends that discussion right there.

    The headline should reflect what is actually going on: “Out of the 153 Latinos who chose to participate in our study, 50% expressed some degree of approval of Trump”

    • Alan:

      Just about all polls of humans are non-probability samples. Response rates in telephone polls are below 10%. But the point still is to learn about the population. That’s why we do statistical adjustment and analysis.

      • I think the point still kind of holds. Basically whatever they are quoting as uncertainty intervals is meaningless *unless* the adjustment mechanism is well understood, documented, and has a good track record. I don’t think that’s true for most polls these days.

      • ” But the point still is to learn about the population. ”

        … the point here is that faulty/misleading opinion polls do NOT increase our knowledge of the population!

        Disinformation ain’t good for learning

        • Dongan:

          I agree that faulty polls should be adjusted or, if necessary, just set aside if not enough information is available to do the adjustment. I have no reason to think that the poll discussed in the above post is hopeless in that way. It’s not enough just to dismiss a poll on the grounds that it is “non-probability sampling”: On that grounds you’d dismiss all polls, and that would be a mistake, as we can learn a lot from polls.

      • I agree the aim is to learn about the population

        However, we should stick to observables as much as possible – or back up “what people say” with what they do.

        We saw this happen in the 2016 election. Remember all those polls (e.g. NYT) – Clinton’s chance of winning is 97%…

        I’d rather have data on the % of Latinos who voted…which brings up another good point: “approval of” ⧣ will vote for.

        The reliability, validity, and utility of these data are so far off…it’s best to assume it’s disinformation.

        (BTW keep up your work. I’ve got your book and read your blog)

        • Alan:

          We did some research on polling errors (see here) and more specifically on what happened with certain state polls in 2016 (see here). The short answer is that there is an intermediate step between “treat a poll as a random sample of balls from an urn” and “don’t trust the polls at all.” We can do quantitative adjustment and assess non-sampling errors quantitatively.

          P.S. Nobody serious gave Clinton a 97% change of winning the electoral college. I agree that her chances of winning the electoral college were overestimated because people did not fully account for nonresponse errors in certain state polls. But that 97% thing was never right.

        • If by nobody serious you mean like academic statisticians who specialize in polling and public opinion maybe…. but most people take NYT pretty seriously… An appropriate model using an unobserved common bias parameter should have given Clinton something like a 50-60% chance. So anyone reporting better than 65-70% wasn’t really serious, which was EVERYONE

        • Daniel:

          Sure. Actually, I even promoted an analysis on Slate that gave Clinton a 90% chance, and I was wrong—this analysis was using state polls without properly adjusting for possible errors. So, yeah, lots of serious people got this wrong.

          I was reacting specifically to the 97% number. I don’t recall the NYT or any other serious outlet giving 97%.

        • To me, 90 or 97 are both so wrong that quibbling over the 7 percent is less important. The big news is that professional polling organizations and poll aggregators didn’t include in their models the possibility of substantial bias on the order of 5-10 percentage points even though they had 10-15% response rates on most of these things.

          Of course, why is this? Because you can get within 10 percentage points by just asking 5 friends what they think will happen. I’d have told you clinton will get 50% +- 10% of popular vote if you’d asked me, and I don’t follow politics at all. It’s a no-brainer, we have a voting system that rigs the system into a 2 party system, and one party is never SO far off that it gets only say 30% of popular vote… so you’re guaranteed to be somewhere between 40 and 60% for whichever of the two major candidates you ask about… Close elections are about 53 vs 47 or whatever.

          The electoral college system is more complicated, but indeed to predict it you need to have accurate predictions on a *state by state* basis, and doing that requires *even more sophistication* or much bigger sample sizes and much higher response rates.

          So doing all this “sophisticated” polling of thousands of people without *seriously sophisticated models* that compensate for 90% non-response isn’t adding anything at all. But it sure sells clicks and garners attention!

          To me, the day when you can just “commission a poll” and learn anything at all about a binary question where our prior is already 50 +- 10 is gone. You have to do something dramatically more sophisticated than whatever people have been doing over the last 20 years.

        • Andrew:

          In this nice summary of election forecast performance, 3 of 9 evaluated forecasts gave Hillary more than a 97% chance of winning the electoral college: HuffPo, Princeton Election Consortium, and DeSart and Holbrook.

          https://www.buzzfeednews.com/article/jsvine/2016-election-forecast-grades

          Of course one can argue about what is and isn’t a serious prediction, but if your favored Slate / Kemp and the PredictWise models are to be considered serious, then these others probably should too.

  5. Andrew,

    At what point do you think the field (survey research) will collectively throw in the towel and declare that it is no longer possible to derive any value from surveys? At 5%? or 1%? One in a thousand?

      • Interesting.

        I wish I could recall the book but there was a science fiction novel I read several years ago where everyone’s head had a little networking device and certain people earned a living by constantly being polled about whatever questions someone wanted answered. It was all using algorithms where the better your poll answers were at predicting things, the more you got paid and the more often you got polled.

        Might have been something by Alastair Reynolds. Not necessarily the Demarchist series but one of the standalone novels.

        • I think this was one of the societies in “The Prefect”? As I recall, they were one of the groups that ended up being exterminated by their robot servitors. :-(

          But with regard to paying people to be survey participants, I wonder what that demographic distribution looks like? It might be somewhat more representative than phone surveys, but presumably it also attracts people who would value the extra cash more, sort of like people who are active on Mechanical Turk.

  6. Where did the 0.5 come from in the standard error calculation? Why assume standard deviation is 0.5? (pardon my lack of statistics knowledge here)

    Formula: Standard error = sigma / sqrt(n)

    Quote from article: “A national poll of 1000 people will have about 150 Latinos. The standard error of a simple proportion is then 0.5/sqrt(150) = 0.04”

  7. Andrew writes: “The other big mistake that many commenters made was to take the poll result at face value. Cowen did that too, by reporting the results as “Latino approval of Donald Trump,” rather than “One poll finds . . .””

    I’m not so convinced of this. I think you’re right that Cowen should have been at least somewhat critical of this poll when he posted it, but I’m not sure that just posting it how he did constitutes an endorsement of it or uncritical belief in it. I could be wrong here, but I get the sense that Cowen just comes across things and posts them on his blog without comment just for the sake of sharing things that he’s reading with people. My takeaway from this is: maybe Cowen is posting this because he believes it uncritically and wants to use it as evidence for some preexisting argument (as you seem to be implying), or maybe he just wants to post something every day and he found this an easy/topical thing to post.

    • I think Andrew errs badly in reading Cowen’s headline. Cowen is simply reporting a poll with a rate of approval among Latinos, not claiming to have discovered “(an) approval,” whatever that might mean. After all, Cowen doesn’t do more than higlight one poll result and link to the full poll. Why that merits this dissection, I have no idea.

      Although, I note that I would avoid casually mentioning that poll result to many of my friends, because I wouldn’t want to shock them. It would be just as shocking to them if the approval rate among Latinos was 39% instead of 50%.

      • I don’t think the real message is about Cowen at all, it’s really about how people in general take polling at face value, including the published margins of error which are probably close to meaningless given the poor response rates and soforth. Paid panel polls are probably better but also need statistical adjustments that few people really understand. Basically polling went from a reliable indicator to a highly problematic indicator in around 20 years

      • Dzhaughn, Daniel:

        Polling is great, and to some extent it’s possible to quantify the scale of non-sampling errors. My big problem with Cowen’s post is selection. It’s fine to just report the polls that are surprising, but then you should do some sort of partial pooling, or at least address the fact that you’re picking out one surprising result from one poll.

        The analogy would be if I want to Vegas, watched lots and lots of casino games for several days, and then reported that, in a period of time, 70% of the spins landed on Red.

        • Polling is definitely useful if you are doing a good job of it. But I don’t think it’s common knowledge who is actually doing a good job, nor how good that job is. It seems like commercial polls are largely attention getters… maybe organizations like Pew who actually care about the answers rather than just about attracting media attention are doing a good job? I honestly don’t know. Who would you trust to have accurate polls with meaningful uncertainty quantification?

      • One would assume a senior PhD Economist with Cowen’s extensive background & experience would well understand the basics of Survey Research.

        However, he is a bit eccentric and likes to play Straussian mind games with his audience.
        He does not always say what he means nor mean what he says.

        Most of his posts seem casual blog bait with items that strike his fancy, but he does occasionally state his non-endorsement of external articles/research that he posts. He is fond of the labels “speculative” or “I don’t fully agree” attached to articles he does not trust.
        But absent such disclaimers he indeed accepts the basic validity of his blog posts IMO.

Leave a Reply to S Cancel reply

Your email address will not be published. Required fields are marked *