A colleague writes:
Have you seen this article by Nate Cohn at the New York Times?
A few things in it seemed weird. For one, he writes:
The tendency for recall vote to overstate the winner of the last election means that weighting on recall vote has a predictable effect: It increases support for the party that lost the last election.
Is this always true? I think I have some small algebraic examples where it is not. Furthermore, his table here seems to contradict that?
I was curious so I sent the following message to Nate Cohn:
A colleague pointed me to an article of yours and had a question; see below. Did you make a mistake in your article?
Also, on the general topic of the benefit of adjusting for party identification, I recommend these articles:
from 2001: https://stat.columbia.edu/~gelman/research/published/aprvlRv1.pdf
from 2016: https://stat.columbia.edu/~gelman/research/published/swingers.pdf
from 2016: https://www.nytimes.com/interactive/2016/09/20/upshot/the-error-the-polling-world-rarely-talks-about.html
from 2016: https://slate.com/news-and-politics/2016/08/dont-be-fooled-by-clinton-trump-polling-bounces.html
No reply! I have a horrible feeling that my message had too many links and it got caught in his spam filter. So maybe blogging this is the best way to communicate it.
Anyway, I haven’t looked into this particular question of adjusting for past vote; there could well be subtleties here of which I’m unaware. In general, I think that it’s a good idea to adjust for some measure of partisanship (recall Lohr and Brick’s reanalysis of the famous Literary Digest poll from 1936), whether that be party identification, party registration, or recalled past vote, because we do have relevant information on these variables at the state level. But, yeah, these measures themselves have errors, so the best adjustment will not be a simple “weighting.”
P.S. My colleague adds this explainer:
I think even if measures like recall vote have errors, the questions is whether those errors negate the gains due to the adjustment. They are the ideal type of weighting variable: correlated with both outcome (vote choice) and nonresponse/selection/coverage, the prime combination per Little & Vartivarian (2005)
Raphael:
The measures definitely have errors! It should be possible to account for these possible errors when doing the adjustment. I guess I should demonstrate this with a simple example in Stan. The key is to model the problem and not get stuck in a narrow “weighting” formulation.
Sorry, I didn’t mean to say that the measures did not have errors. What’s interesting is that pretty much most weighting variable has errors as well. For me, it’s a question of how much that measurement error matters to the point to negate the gains due to weighting. Anyway, that’s my survey statistician perspective. I’m sure you can tackle this problem from other perspectives as well. I tend to think from a “weighting” formulation, because that’s usually what gets used in large-scale survey production. Not that other approaches don’t have its own merits.
I don’t think Nate Cohn is mistaken. I think he’s making 2 very reasonable assumptions here:
1. The fraction of respondents who claim to have voted for the winner in 2020 is greater than the fraction of voters who actually voted for the winner in 2020. (i.e., more survey respondents say they voted for Biden than the actual fraction of voters who voted for Biden.)
2. Respondents who claim to have voted for the winner in 2020 are more likely to say they will vote for the 2020 winning party in 2024 than respondents who claim to have voted for the loser in 2020. (i.e., people who say they voted for Biden skew more towards Harris than people who say they voted for Trump.)
The handwritten explainer uses hypothetical values that I find unusual: only 50% (green) or 51.7% (yellow and red) of respondents even claim they voted for Biden in 2020, even though in actuality 52% of voters voted for Biden in 2020, which is still larger than all 3 cases of recalled votes for Biden. So that violates assumption 1, which is how it manages to get those counterintuitive results.
Ben:
But what about the screenshots of the polls in different states? These patterns go in the opposite direction as hypothesized by Cohn, no?
I agree that the screenshots do appear to contradict Nate Cohn’s statement about the directional impact of the recalled vote weighting adjustment! But the screenshot isn’t showing the same polls on both sides, using recalled vote weighting on one side but not the other. My understanding is Nate Cohn is showing completely different polls, presumably with completely different pollsters, samples of respondents, and estimation methodologies. With differences in outcomes that small, it wouldn’t surprise me if sampling noise could completely explain away the effect, but even beyond that, it’s easy to imagine that other (systematic) differences in polling methodologies between the polls that use recall weighting and those that don’t could explain the effects that are in the opposite direction of Nate’s statement.
“other (systematic) differences in polling methodologies between the polls that use recall weighting and those that don’t could explain the effects that are in the opposite direction of Nate’s statement.”
systematic differences that go in opposite directions by state ?
Shira:
As I understand, it’s not the case that every pollster polls every state. So the main thing I think is that each “cell” in the screenshot is going to be a different mix of polling methodologies.
(But, yes, I also think it’s very possible that systematic differences might go in opposite directions by state. Suppose one poll weights by education, and another doesn’t, and then you look at two states that have very different educational mixes relative to who tends to respond to surveys.)
the hypothesized effect is about changing only the weighting, but in the observed polls with/without weighting by recalled vote, plausibly also lots of other factors differ: lots of possible confounders in this analysis. This would be an issue especially if pollsters are “herding”
To be clear, that concern suggests the table simply isn’t meaningful: you’d want to do it within poll, which Nate Cohn does farther down in the article:
How recent Times/Siena polls would have changed
Pennsylvania: Harris +4 (without recall vote) —> Trump +1 (with recall vote)
Michigan: Harris +1 —> Trump +1
Wisconsin: Harris +2 —> Trump +1
North Carolina: Trump +3 —> Trump +6
Arizona: Trump +5 —> Trump +3
Georgia: Trump +4 —> Trump +6
So, here we see (Arizona) that this shift towards the party who lost the previous election is by no means mechanical, but it seems to occur in many instances.
A simple model for recall bias (due to it being attractive to claim to have voted for the winner):
(1) Everyone who voted winner says they voted for the winner.
(2) Some who voted for the loser say they voted for the winner.
(3) Suppose everyone says this time they will vote for the party that they actually voted for last time.
Then the data will look like the previous winner will lose some support compared to the previous election once one weights by recalled vote (some of the winner’s supposed voters now vote for the other party, but not vice-versa).
To me, that seems like a not completely unreasonable baseline model for a first approximation. In practice, both (1) and (2) don’t hold exactly. There are also lots of covariate cells and exactly how one does the re-weighting in practice, I don’t know.
was meant to be (1) and (3) not holding exactly. Also, of course, there are non-voters and 3rd party voters who may recall having voted, etc.
I continue to be confused by all of this – but I’ll address your comment since it is a place for me to start. I don’t understand how (1) and (2) are consistent with assumption (3). (2) seems to indicate that some people will say they voted for someone other than they actually voted for, but (3) says they will all vote for the party they actually voted for? Isn’t this a contradiction, or is there something hidden in the change in wording from winner/loser to party?
I am also not clear on how people are referring to the word “weighting.” Can someone spell out exactly what the weighting entails? Also, since Harris was not the presidential candidate in 2020, are we implicitly assuming that Harris and Biden are to be treated as equivalent in this exercise? (again, that party vs candidate confusion for me).
I get the idea that the recall vote may well differ from the actual vote, and in predictable ways (people are more likely to recall they voted for the winner, even if they did not). But there is a disconnect for me: how does this translate into how they answer polls today about the coming election? It seems like an implicit assumption that the bias in the recall vote is the same thing as a bias in a poll about the upcoming election. If a person that voted for Trump in 2020 now recalls it as having voted for Biden, how does that relate to the possible bias in their response to a poll about the 2024 election?
As I’ve said, I’m confused. And if I am missing something obvious, I’m going to blame it on the COVID shot I just got.
Dale,
A simple example below, but quick answers to your question:
No contradiction, just a difference in time: who they voted for (past) and who they say they will vote for (present).
The weighting I assume is that you form groups of people based on observable characteristics, then calculate average survey responses within each group, then assign each group the weight you think those characteristics will make up in the population this time around — in practice, pollsters probably can’t do quite that (because there aren’t enough people to calculate averages for every subgroup of age, education, gender, recalled vote, etc.), but it is probably what they’d want to do if they could?
For how that all translates into a bias, see the example below.
Suppose there are five people, A, B, C, D, and E. We survey them in 2024, but don’t know who they truly voted for in 2020.
In 2020, A, B, and C voted for party1, while D and E voted for party2. So party1 got 3/5 = 60% of the vote, party2 got 2/5 = 40%.
In 2024, we survey all of them. A, B, and C correctly remember that they voted for party1. D “misremembers” and claims to have voted for party1. E correctly remembers to have voted for party2. In the survey, A, B, and C say they will vote for party1, while D and E say they will vote for party 2.
If we DON’T adjust for recalled vote, we will say that 3/5 = 60% of people say they’ll vote for party1, and 2/5 = 40% of people say they will vote for party2.
If we DO adjust for recalled vote, we calculate:
3/4 of people who say they voted for party1 say they will again vote for party1; 1/4 say they will vote for party2 now.
0/1 of people who say they voted for party2 say they will vote for party1 now; 1/1 say they will again vote for party2.
And we know that in 2020, 60% of people voted for party1, so the first group (says they voted for party1 in 2020) should make up 60% of the population, while the second group (people who say they voted for party2 in 2020) should make up 40% of the population.
So, now we think that 0.6 * 3/4 + 0.4 * 0/1 = 45% will vote for party1, and 0.6*1/4 + 0.4 * 1/1 = 55% will vote for party2. The weights 0.6 and 0.4 “adjust” for the fact that it looks like people who voted for party1 were more likely to respond to our survey (4/5 = 80% of respondents claim to have voted for party1, while in reality only 60% did) than people who voted for party2.
Andrew, you stated: “I think that it’s a good idea to adjust for some measure of partisanship…whether that be party identification, party registration, or recalled past vote, because we do have relevant information on these variables at the state level.”
I’ve tried to source good data on party identification at the state level (+DC) many times but haven’t found any super reliable sources. I was wondering where you think there is good information on party identification at the state level. Also, I imagine it is harder to get this information for gen pop as opposed to registered voters. I have previously ended up modeling this as its own MRP outcome as opposed to relying more heavily on an external source, but I’d be keen to know of any good sources to look into!
For party registration, I assume this information is available in official documents from each state.
Jamie:
One measure we’ve used for party identification is past survey responses, as in our 2001 paper referenced above.
Thanks! I hadn’t come across this paper and will take a look. I’d previously been using something along the lines of your and your colleagues’ extra census tutorial here: https://bookdown.org/jl5522/MRP-case-studies/mrp-with-noncensus-variables.html
A fun related analysis making the rounds, by Josh Clinton, whom you might know
https://goodauthority.org/news/election-poll-vote2024-data-pollster-choices-weighting/
The range of vote margins one gets in a single poll using different seemingly plausible specifications/weighting-schemes can be 8 percentage points (e.g. D by 1 all the way up to D by 9)
Very reminiscent of the garden of forking paths…
Are pollsters at least consistent across time?
Do models take into account the particular weighting done by pollsters? It seems like the “polls could be off by x even on election day” partially captures this as “pollsters could weighting choices could, on average, yield a result that’s off by x”
But would the perfect world be one where each pollster shows estimates based on a number of different weighting choices such that models/aggregators can say “if weighting choice w is right this year, we’d see x, but if weighting choice w’ was right this year, we’d see y”
I wonder if being concrete about the source of uncertainty and range of results that yields would be something that perhaps people would find more clearly articulates a large chunk of the underlying uncertainty than simply slapping random errors on things or starting with a prior for how large the miss of polls might be in the end.
Without going through the NYT paywall, I don’t know if the article makes this clear or not – but I don’t really understand the table that is shown. I see that weighting polls based on recalled vote will give different results than polls that don’t do such weighting. But how does the 3rd column (the actual 2020 results) relate to that? Don’t we want to see how the recalled votes compare to the actual votes? I guess that his hidden behind the scenes from the table, but I can’t make sense out of the table without seeing that. Also, given that many of the examples in the table are probably within reasonable margins of error, then the “analysis” should involve the probability that a sufficient number of state results behave in some predictable pattern: in other words, if the election is close, then the polls presumably say either candidate could win. If all states showed that the predicted winner with and without weighting in relation to the actual vote had the same pattern, then (if we assumed independence, which may not be a good assumption) we might find that an unlikely pattern. But I find it hard to make anything out of polls that show a 1% difference and an actual vote difference of 1%. For that matter, I don’t see that a comparison of the 2020 results to the difference between 2024 polls with and without weighting tells us anything other than the directional impact that weighting would have.
I guess I don’t understand the table at all.
No, it’s not mistaken. I will repeat most of the article (did you read it?):
– Historically, past vote is biased toward the party that won the last election (ex: Bush+7 ’00 recall in pre-election polls in 2004)
– As such, weighting on past vote has a predictable effect. It tends to help the party that lost the last election (ex: giving more weight to self-reported Gore ’00 voters, which would have meant polls showing Kerry ahead).
– The article does not contend that *every* recall vote weighted poll will be biased toward the party that lost the last election. There are even two examples in the article: a recent Times/Siena poll in Arizona, the ABC/Post poll of Wisconsin in 2020. In the case of the ABC/Post poll, the underlying survey data was *even more* skewed toward the party that lost the last election then the recall vote measure itself, and as a consequence the poll would have been better (Biden+12 rather than +17) if it had been weighted on recall vote. The Arizona poll (or a recent Times/Siena poll of Florida) could have other explanations, like the changing makeup of the electorate since 2020 or the possibility that recall vote is biased in harder to identify ways (could be correlated with changing vote choice, for ex).
– The article also makes a distinction between the effect of recall weighting (the table you’ve published) on individual polls and the effect of recall vote weighting on the averages in 2024. There is a long section in the article on this point. The puzzle: recall vote will should most individual polls to the right (the articles shows that the Times/Siena polls would find a clear Trump lead), but the average of recall vote weighted polls neatly aligns with the 2020 result. This is likely a selection effect: the pollsters employing the technique have more Democratic samples, they’re worried about missing the result again, and they’re using the measure to bludgeon their results toward plausibility (ie: the 2020 result). The subgroup of pollsters that’s likeliest to use recall vote weighting — online polls with highly engaged samples — may be especially prone to inducing this pattern, as highly engaged respondents are more likely to vote and less likely to report flipping from the last election. As such, a recall vote weighted poll of, say, only primary voters (all 2020 voters + no flips) would likely just yield a 2020 repeat.
Nate:
Interesting; thanks for the breakdown. This is super-helpful. As I wrote in my above post, I think it makes sense to adjust for some measure of partisanship, and if recalled past vote is that measure, then I think it’s important to adjust for it. But, for the reasons you discuss, the best adjustment won’t be a simple weighting; it should account for what is known and expected regarding measurement error. It’s a research topic!
Finally, some of this is getting through my thick skull – but I am finding Nate’s analysis unconvincing. First, from your (Andrew) comment, I now see that party and candidate are being treated equivalently. I agree that adjustment for partisanship is important, but I’m not enamored with equating candidates with parties. Recall asks about particular candidates who happen to be running for a party, but I think it is a leap to go from recall about candidates to that being a good measure of partisanship.
Nate’s explanation above says that not every recall weighted poll will result in the same directional bias. Using Arizona as an example, he says it “could have other explanations, like the changing makeup of the electorate since 2020 or the possibility that recall vote is biased in harder to identify ways (could be correlated with changing vote choice, for ex).” This sounds like forking paths to me. When the evidence doesn’t support your theory, then there are always other factors that could account for that. Perhaps this is what you (Andrew) are referring to when you say that “the best adjustment won’t be a simple weighting; it should account for what is known and expected regarding measurement error.” Nate’s theory just seems incomplete to me – too many implicit assumptions required to connect the dots.
However, I’ll admit that they may not be “implicit” assumptions at all – because I have not read the article. I don’t do paywalls.
It is a research topic!
As a survey statistician and methodologist, for me, the interesting point is how much measurement error negates the gains due weighting by a covariates strongly correlated to the outcome as stated in Little & Vartivarian (2005), for example.
And this is much broader than just recall vote. Pretty much most variables used in calibration adjustments have some level of measurement error, as they tend to be reported by the respondents in the survey. But I think we just assume that this measurement error is pretty much negligible to the point it does not really matter for the potential gains due to calibration. However, how much measurement error these weighting variables can have to the point that this is true? I’ll talk to Brady West, Yajuan Si and Mike Elliott here at SRC to see if they would be interested in doing some research in that topic.
Andrew –
Nate Silver also referring to the 1936 Literary Digest poll:
That brings us back to the Literary Digest poll. In 1936, the magazine surveyed almost 2.3 million voters, a colossal number. They had Alf Landon, the Republican, beating Franklin Roosevelt 57-43, an epic landslide. The election was a landslide — but for Roosevelt, who won by 24 points, including all but two states.
Just a mere 38-point polling error. What happened?
Of course the answer to the intriguing question is behind a paywall (and Silver certainly has enough money without my help).
https://www.natesilver.net/p/the-early-vote-doesnt-reliably-predict?utm_medium=web&triedRedirect=true
Joshua:
There’s an interesting story about the Literary Digest poll. It’s not as simple as is usually told. I talk about it in Active Statistics; also see this post from a few years ago, The Xbox before its time: Using the famous 1936 Literary Digest survey as a positive example of statistical adjustment rather than a negative example of non-probability sampling.