A retired economist pointed me to this document and remarked that it’s really hard for him to read Appendix B and not despair about the methodologies used by organizations not nearly so forthcoming about their adjustments.
I asked, Why despair? It’s good that they’re open, no?
My correspondent replied: “Sure, Pew is open about it, but what about Rasmussen? Or Morning Consult? Or are you making the Gelmaniacal point that Honesty and Transparency Are Not Enough? I fully grant that Pew’s openness hides about 642 researcher degrees of freedom they aren’t telling us about.”
I replied: No, I’m pro-Pew. Surveys are hard. Why do private companies not share these details? My guess is that (a) it’s embarrassing to see how the sausage gets made, (b) trade secrets, (c) it takes work to write such a document.
To which my correspondent wrote:
I’m pro-Pew as well. This whole document is thoughtful and ought to be really helpful to consumers of polls; but polling is, as you say, hard. And fitting in to what you wrote today, the credulity with which poll results find themselves reported is troubling. (Unless of course the poll finds a result you don’t like, in which case every possible objection to the result is marshalled.) But if polling is hard, we ought to be much more skeptical of what it is polls are telling us. I am somewhat more heartened by poll aggregation, though that is no panacea. And I think that changes in polls, holding methodology constant, have much more informational content than the poll levels.
I’ll just echo the point we’ve made before, that polling for a close election is particularly challenging because if you’re off by 3 percentage points you can get the outcome wrong. For survey questions ranging from Joe Biden’s approval to support for a minimum wage increase, attitudes are far enough from 50/50 that a 3% shift would not really change our interpretation of the polls. The key uncertainties in these settings are not in any particular poll, but in what might change in the future, and even a perfect random sample can’t handle that concern. In short, yes, I recommend some skepticism in what polls are telling us, but it’s more about skepticism in the implications, not so much in the polling results themselves.
“it’s more about skepticism in the implications, not so much in the polling results themselves.”
I’m not sure what you mean here. The only reason anyone even takes a poll is to draw implications. But sure… some implications are better-founded than others. There are two separable issues here: (a) does the poll do what it purports to do, ie. instantiate a random sample from a universe with known sampling probabilities of each unit; and (b) is the result stable, are the standard errors correctly calculated, are the respondents answering honestly… the sorts of things that make the implications of the poll reliable.
What I think Appendix B makes clear is that it isn’t just (b) — there’s a whole set of calculations, corrections and guesses required just to generate a poll result, and poll designers hide the ball, or at least handwave vigorously to get people to use the common understanding of (a), Doing this transparently, as Pew does, is good. And the problem, for example, of going from poll results to voting behavior (which is a problem under (b)) is discussed a lot. But there are a lot of other things under the general rubric of “adjustments” which, while almost certainly necessary, are not designed in a mechanical fashion, but with some dark arts that are entitled to be judged skeptically, particularly when some of the pollsters themselves might be less than completely objective.
I am reminded here of economic forecasting models. Ray Fair used to argue, with some justification, that his model was the best performer among all models where the model designers didn’t tweak the parameters when they got results they didn’t like for whatever reason. It’s not that the results of these other models weren’t better than Ray’s — they were. But the real model they were using — “here are my equations.. (in this iteration)”, was in an important way unscientific. Were they more reliable? Maybe. But *users* of those models were made to feel the results were more exclusively data-based than they were. I think there’s an analogy here to polling.