There’s been a lot of discussion of how the polls performed well in the recent midterm elections. Response rates are so low that pollsters need to do lots of adjustments to aim to approximate attitudes in the population. We don’t know that they’ll succeed so well in the future, but they get blamed for their mistakes so it’s only fair to give them credit when they succeed.
We learn more from mistakes than successes
We learn from mistakes, though, so let’s look for examples where polling went wrong. Jay Livingston shares an example:
Maybe the usual question — “Who are you going to vote for?” — is not the best way to predict election results.
The most recent episode of NPR’s Planet Money explored this question and in the end tried a less direct approach that some polls are now using. They went to the Marist College poll and got the directors to insert two questions into their polling on local House of Representatives races. The questions were:
– Who do you think will win?
– Think of all the people in your life, your friends, your family, your coworkers. Who are they going to vote for?
At the time, the direct question “Who will you vote for?” the split between Republicans and Democrats was roughly even. But these new two questions showed Republicans way ahead. On “Who will win?” the Republicans were up 10 points among registered voters and 14 points among the “definitely will vote” respondents. On the friends-and-family question, the corresponding numbers were Republicans +12 and +16.
What’s interesting about this NPR report is that, first, it came out on 4 Nov, just a few days before the election; and, second, that it was soooo far off. Sure, people were talking about a so-called red wave, but nobody—nobody—was predicting that the Republicans would win by 12 or 16 percentage points.
So this is an interesting example of a poll that anyone could tell was way off, even before the election actually occurred.
Oddly enough, NPR just reported these numbers with a straight face, without saying anything about how off they seemed to be. But there’s nothing stopping us from trying to figure out what went wrong.
What happened?
I’ll discuss this in three parts, corresponding to each of the thee survey questions:
1. “Who do you plan to vote for?”: Respondents were split 50/50 on this one, which matches the election outcomes. So nothing much to explain here: the pollsters did their best to adjust to attain a representative sample, and it seems they succeeded.
2. “Who do you think will win?”: 10-14% more people thought Republicans would win than thought Democrats would win. I’m actually surprised the gap wasn’t higher! The major media outlets were saying that the Republicans were favored, so the real question is why more than 43-45% of respondents thought the Democrats were favored. I’m guessing that these were hard-core Democrats who were answering the question as a sort of surrogate, “Who do you plan to vote for?” question, but I don’t really know. In any case, as Julia Azari and I wrote back in 2017 (see section 4 of this article), it’s not at all clear how the responses to the “Who do you think will win?” question is giving us any useful information.
3. In that 2017 article, we wrote:
In future studies, we recommend studying information about networks more directly: instead of asking voters who they think will win the election, ask them about the political attitudes of their family, friends, and neighbors.
The good news is that this is what the above-linked NPR survey did! The bad news is that it didn’t go so well: “On the friends-and-family question, the corresponding numbers were Republicans +12 and +16,” numbers which were implausible at the time and even more so once the election results came out.
So what happened? One possibility is that Republican family/friends/coworkers were more public about their political views, compared to Democratic family/friends/coworkers. So survey respondents might have had the impression that most of their contacts were Republicans, even if they weren’t. Another way things could’ve gone wrong is through averaging. If Republicans in the population on average have larger family and friend groups, and Democrats are more likely to be solo in their lives, then when you’re asked about family/friends/coworkers, you might be more likely to think of Republicans who you know, so they’d be overrepresented in this target group, even if the population as a whole is split 50/50.
My recommended next step is to break down those responses to the family/friends/coworkers question: look at the proportion of Republicans and Democrats in the social network of respondents for different ethnic groups, age groups, also compare by state, urban/rural, education, etc. And party ID of respondents.
Also it would be good to see exactly how that question was worded and what the possible responses were. When writing the above-quoted bit a few years ago, I was imagining a question such as, “Among your family and friends who are planning to vote in the election, how do you think they will vote?” and then a 6-point scale on the response: “all or almost all will vote for Republicans,” “most will vote for Republicans,” “slightly more will vote for Republicans,” “slightly more will vote for Democrats,” “most will vote for Democrats,” “all or almost all will vote for Democrats.” I might have even put a question like this on a survey at some point; I can’t remember! Anyway, I’m not committed to that question wording; I’d just like to know the wording that the NPR survey used.
In any case, I hope they look into what went wrong. It still seems to me that there could be useful information in the family/friends/coworkers question, if we could better understand what’s driving the survey response and how best to adjust the sample.
“Think of all the people in your life, your friends, your family, your coworkers. Who are they going to vote for?”
I’d say this question is doomed from the start. “All” the people includes people I may know well and people I hardly know anything about their political views. So, I may believe my family and close friends will vote like me (probably true in my case), but I don’t think the same of my coworkers, and definitely don’t feel that way about “all the people” in my life. Why ask the question that way? Why not limit it to a well defined group that might be reflective of a person’s likely voting pattern? I’m not sure exactly what that group would be, or how to word it, but I most assuredly would not word it as above.
Dale:
I’m not sure that was the exact question wording. I couldn’t find any details from the NPR link or elsewhere on the web, but it might be somewhere.
So, I listened to the podcast from Planet Money (clearly I have too much time on my hands). At 22:15 they talk about the wisdom of crowds questions. They never do say exactly what the wording was. They paraphrase the verbiage above, also say “who the folks in their social circle will vote for,” and at another point refer to is as who their “social contacts” will vote for. None of the polls they provide details for included these “wisdom of crowds” questions. Frustrating to not see exactly what was asked, particularly when the language keeps changing when they discuss these questions. In any case, I think there is a problem about asking about social contacts when the size of someone’s social network probably varies (probably by order of magnitudes) across people. I have some idea how my family and close friends will vote but not if this extended very far. I suspect most people have larger social networks than me (for example, I have a Facebook account, but use it perhaps twice each year), and I’d guess that the larger the social network the more the response will mirror what they hear on the media. If I don’t know someone well, then I’m likely to guess that they will vote the way I hear the media reporting likely voting patterns.
I’m not sure I’d call this “wisdom of crowds” questions. Perhaps the opposite.
Dale Lehman refers to the famous 1906 “wisdom of crowds” and I presume this is a reference to Galton’s famous “weight of an ox when it was butchered and dressed”
https://www.all-about-psychology.com/the-wisdom-of-crowds.html
But, as https://www.all-about-psychology.com/the-wisdom-of-crowds.html points out,
“However, to benefit from the wisdom of crowds several conditions must be in place. First each individual member of the crowd must have their own independent source of information. Second they must make individual decisions and not be swayed by the decisions of those around them. And third, there must be a mechanism in place that can collate these diverse opinions.”
And, the participants know something or other about oxen and how heavy a “pound” is. The crowd would have a whole lot less wisdom if asked about the fine structure constant which was first revealed in 1916.
Paul: That description make it sound like just asking people who they will vote for is a much better “wisdom of the crowds” question than asking them the other two queries.
Livingston certainly acted as though he were giving the exact question wording. If he wasn’t, he owes us some re-wording of his own.
Another thing that caught my eye is the ambiguity of asking “Who do you think will win?” We do have two national legislative chambers in the U.S., you know. If the Democrats win one and the Republicans win the other, who’s really won? If the Republicans gain seats but fall short of capturing a majority, does that count as “winning” in a sense? Some respondents might even have interpreted the question as referring to a particular House or Senate race wherever they happened to live.
Finally, this survey seems to offer no way of distinguishing respondents who thought a bare majority of their friends would vote Democratic from those who thought almost all of their friends would vote that way. Surely that could make a major difference.
Kevin:
Livingston was quoting from the NPR report, but the NPR report did not give question wordings, nor did it give a link to a document with exact question wordings. Unfortunately it is common when surveys are reported for the details not to be given. One of the most notorious examples of this was the Iraq survey by Burnham et al. from around 2006.
Most of the people in my life live in different countries let alone jurisdictions. So even if I knew how they were going to vote, it would not help me predict local elections. Is this another example that you can get anything in the papers if you say its based on a survey?
Clearly that doesn’t apply and for maybe 99% of correspondents this is not going to be the case. If they polled you they would just throw the results out.
How on earth would the pollers know? I doubt it was a polling question.
Likewise, in the places where I have lived its not customary to talk a lot about politics or political news with casual face-to-face acquaintances. Some subcultures in some places are clearly different.
I have no idea how many people’s lives mostly involve living people within an hour or two’s travel radiusl I am commenting on an academic blog run by a statistics geek after all :)
Andrew begins with pointing to
https://www.grid.news/story/politics/2022/11/09/dont-blame-polls-for-the-red-wave-that-never-happened-blame-pundits/
and it makes for excellent reading. In particular, it says
“The whole ‘red wave’ was manufactured by partisan polls and Super PACs. The independent polls were calling a lot of those races as dead even. And they were.”
I live in a personal left-wing bubble, surrounded by a wife and daughter, each of whom is way left of even me. So given that Herschel (Father of the Year) Walker was slated to be elected Senator from Georgia, I was prepared for the worst. Took a sleeping pill early in the evening so as to avoid agony and reality, only to wake up to find that Democrats unexpectedly swept everything in Minnesota (all Statewide offices, both houses) despite the dire predictions of my exceedingly personal paranoia.
Moral of the story (to me): Especially when it comes to future events, forecasting is especially difficult.
Unfortunately, forecasting past events is becoming more difficult (at least, more controversial) than ever. Who really won that 2XXX election?
it appears that there was a blue wave two or three month ago, when both nyt and 538 predicted blue houses and senate. do you think that bygone blue wave was also manufactured?
When did 538 predict Democratic control of the house? According to the forecast at https://projects.fivethirtyeight.com/2022-election-forecast/house/, it looks to me like the highest probability they assigned was 32 out of 100 (in late September/early October).
Good question! I _think_ there really was a big surge towards the Democrats when Roe v Wade was overturned. Democrats were appalled and energized by the ruling and people who are (somehow) on the fence or previously apathetic moved a notch towards voting Democratic. But why do I think this? Largely because of polls, and if we decide we can’t trust the polls then how strong is the evidence, really?
Anyway one possibility is that the center of balance shifted a big step towards the left a couple of months ago and has since drifted back towards the right. But I guess it’s also possible it never really moved much in either direction at any point, and what we see is poll results swinging back and forth while underlying voter intent and enthusiasm stays about the same. That second possibility seems unlikely to me but I’m not sure I can disprove it with available data.
> So what happened? One possibility is that Republican family/friends/coworkers were more public about their political views, compared to Democratic family/friends/coworkers.
I think this question is confounded by a feedback loop with polling and other exogenous factors.
Consider the “shy Trump voter” theory. I don’t think it really holds up because whether or not someone is a “shy Trump voter” would depend on whether they live and isn’t some kind of universal mechanism. I live in a rural area not far from a gun club. If I fly a Biden flag it could well be Swiss cheese before too long. I might be a “shy Biden voter.” But down the road from me there are billboard-sizes “Fuck Biden” flags flying. Not exactly shy Trump voters where. Down the road in the other direction is Woodstock, which goes heavily Dem. Not to many “Fuck Biden” flags flying in Woodstock and they wouldn’t be up long id there were.
Anyway, back to that question – I think it that self-report data would necessarily be confounded by polling data in the news (which would affect the level of observation bias, recall bias, etc.) and demographics of where people live, where they work, etc.
You might get lucky and I find that question predictive in one election, it find it pretty useless
in the next.
” If I fly a Biden flag it could well be Swiss cheese before too long. I might be a “shy Biden voter.” But down the road from me there are billboard-sizes “Fuck Biden” flags flying. Not exactly shy Trump voters where. Down the road in the other direction is Woodstock, which goes heavily Dem. Not to many “Fuck Biden” flags flying in Woodstock and they wouldn’t be up long id there were.”
I think this crystalizes what is actually our most serious political problem.
“You might get lucky and I find that question predictive in one election, it find it pretty useless
in the next.”
If you always answer “heads,” that might be pretty predictive in one flip, but pretty useless in the next. :)
The idea that asking people indirect questions to discern their likely vote will be more predictive than direct questions seems outlandishly hopeful if not downright dubious, given the well-known problems with surveys. Probably the only thing that has made election polls more accurate over the years than surveys in general is that they have asked a direct question about a simple action where there’s not much room for psychological inference on the part of the subjects.
>whether or not someone is a “shy Trump voter” would depend on *whether* they live…
Fun typo; should be *where*. I don’t think it depends their short term life expectancy! ;)
I don’t understand why the difference between Q1 and Q2 is at all surprising. I definitely voted for a D, and I still think Rs might win the house. Whoever has the survey data could show us the crosstab (and maybe they did). I see no reason to think there’s anything wrong with either question.
Ken:
There’s nothing wrong with asking any of these questions. I just that question 2 is kind of a mess, as it’s a mix of people’s personal preferences and their reading of the news.
This is an interesting issue for me, because I used ecological questions in surveys I created for child labor research in the early 00’s. My assignment was to investigate empirically whether child workers were more exploited than adults in paid labor where they were concentrated. I had a set of occupations/locations spread across four countries, not a big enough sample to come to any firm conclusions, but at least to arrive at where-to-dig-type findings. (And the data I collected definitely called into question the neoclassical econ assumption that w = MP across the board.)
I had to estimate wage rates and productivity for child laborers, and I was concerned at the outset that, since most of the work I would be looking at (in some cases literally) was illegal and usually frowned upon in the community, respondents would either clam up or dissimulate. I was surveying both the kids and their employers, and the literature shows that kids are relatively unreliable informants for questions of this type, so I needed the employer responses too.
My solution was to ask wage and productivity-related questions (I was constructing primitive child labor production functions) of employers directly and also ecological questions about what they thought others in their industry did, experienced or believed. In the end, the problem I feared didn’t materialize: most employers felt they were doing good by taking on children — helping their families — and weren’t reticent at all to report on it. As for the ecological questions, they matched up very well, with correlations in the .6 – .8 range. I took this as at least a hint of cross-validation.
Now, comparisons to ecological questions in 2022 polling: (1) They were operating in a context strongly influenced by media reports about a red wave. There were, to my knowledge, few if any community priors regarding the topics I was asking about. (2) My questions were all Likert, so they required at least a bit of refinement in responding. (3) This is probably the most important: employers of child labor probably have only the vaguest idea what the others are actually doing. This is what I assumed going in, that their responses would to a large extent project their own practices on others while providing a layer of “moral separation”. Not so with the political polling questions. In that case, asking respondents about the expected voting behavior of close others really is asking them to be informants about their network, where relatively small shifts in percentages matter a lot, and they probably don’t have the information or skills to finetune their observations. (No reason to assume errors will cancel out, either.) If I really wanted to know about networks, which of course is hugely important in political preference formation and transmission, I’d investigate them directly, a political version of the Framingham heart study. Maybe this has been done. Good!
I’d like to cherry-pick three lines from above:
First: “We learn from mistakes, though, so let’s look for examples where [X] went wrong.”
Second: “In that 2017 article, we wrote: In future studies, we recommend studying information about networks more directly: instead of asking voters who they think will win the election, ask them about the political attitudes of their family, friends, and neighbors.”
Third: On “Who will win?” the Republicans were up 10 points among registered voters and 14 points among the “definitely will vote” respondents. On the friends-and-family question, the corresponding numbers were Republicans +12 and +16.
So, essentially, if you want to learn from your mistakes, perhaps you can use the recent NPR polling results to show that your 2017 advice was off. Don’t necessarily view this as a failing of NPR’s polls, but rather of NPR’s poll showing that your 2017 advice might not be quite so clear cut (or at least needs to be more detailed/specific.)
Eric:
Here’s what I wrote in my above post after describing our 2017 recommendation:
So, yeah, I did say that our recommendation was off! Or, at least, that following our recommendation would not be so easy. Regarding the NPR poll, I still don’t know exactly it was worded, so it’s hard to say much more about it than what I wrote above.
Real Clear Politics projected Republicans getting 53 Senate seats.
https://www.realclearpolitics.com/epolls/2022/senate/elections-map-rcp-projection.html
Didn’t Galesic and colleagues study these kinds of questions a while back? https://www.nature.com/articles/s41562-018-0302-y
But after asking about a person’s own intentions their first question about social circle was about expected turnout (below is from USC/LA Times poll):
Social-circle questions: “Now we would like you to think of your friends, family, colleagues and other acquaintances of 18 years of age or older that you have communicated with at least briefly within the last month, either face-to-face, or otherwise. We will call these people your social contacts. (1) What percentage of your social contacts is likely to vote in the upcoming election for President? For instance, 0% means that you think none of your social contacts will vote, and 100% means that all of your social contacts will vote. If you are not sure, just try to give your best guess. (2) For the next question, please consider only those of your social contacts who are likely to vote in the upcoming election for U.S. President. Of all your social contacts who are likely to vote, what percentage do you think will vote for Clinton, Trump, or someone else? For instance, 0% would mean that you think no voters in your social circle will vote for that candidate, and 100% means that all voters in your social circle will vote for that candidate. Again, if you are not sure, just try to give your best guess.”