Carl Gaspar writes:
Given the apparent shortcomings of forecasting for the 2020 US elections, and the 2016 elections, have you considered that it might be fruitful for polling companies to include alternative questions, based on the collective recognition heuristic, for example?
The collective recognition heuristic: Basically asking people how many other people they think might recognize the candidate. This is something based on Gerd Gigerenzer’s work. And it does seem promising for at least local-level politics. See Gaissmaier, W., & Marewski, J. N. (2011). Forecasting elections with mere recognition from small, lousy samples: A comparison of collective recognition, wisdom of crowds, and representative polls. Judgment and Decision Making, 6(1), 73-88.
Yes, it seems like a total shot in the dark. I mean, everyone recognizes Biden right? Well, maybe not at the beginning of the campaign; especially in those deep dark pockets of America that everyone likes to speculate about. A large and representative enough sample, especially at early and intermediate points in campaigning might be very informative. Part of campaigning for the challenger seems to be about catching up with simple exposure?
And of course, statistical models can include both conventional polls of intention as well as alternative questions based on projected winner and candidate recognitions. I’m no expert at such things – I’m a vision scientist – but I have never seen a hierarchical model of candidate recognitions (or projected winners) before and I’d love to see how that would play out state-wise. Perhaps it exists but I don’t think it does.
One problem with recognition – whether respondent’s recognition, or a respondent’s assessments of others’ recognition – is that, these days we supposedly live in online bubbles or echo chambers. In theory, recognition is less affected by sampling biases than voting intention but, given our little bubbles, that may not be true. Nonetheless, a liberal college student’s ideas of others’ recognitions could be additionally informed by the disappointing choices and ignorance of their friends and family who may be less inclined to respond to polls. So there does appear to be some added information there, which would be hidden if we only asked directly about a person’s voting intentions.
It feels like statisticians stick to polls simply because there’s an existing infrastructure in place for collecting loads of data, and an existing statistical framework for aggregation.
Which leads to my next question: Do prominent statisticians such as yourself ever work with polling companies to consult on question types? If so, I’d love to learn about how that works. If not so much, then why not? I feel like that should be a thing.
Gaspar continues:
Just to be sure, I [Gaspar] am not very confident about any specific alternative to direct queries of voter intention. I’m simply wondering aloud about the exciting possibilities of alternatives filling in gaps in our knowledge. I mention the Gaissmaier & Marewski reference because they also ask respondents who they think will win, which does just as well as recognition (estimates of others’ recognitions), and both much better than intention-queries when matched for sample size . . . 34 students. Too good to be true? Even if predicting only local politics, seems too good maybe. Figure 2, scatter plots on far right.
I have no idea, but I thought I’d share this because it’s good to circulate ideas that are off the beaten path but can make some sense.
P.S. Julien Marewski points to this paper from 2013, Name Recognition and Candidate Support, by Cindy Kam and Elizabeth Zechmeister.
While I’m not convinced about the specific question the correspondent mentions (and, in fairness, neither are they!), I do think it makes sense to do more sophisticated modeling and question construction for polls.
A response on any polling question is the outcome of some decision process applied to some internal representation of information related to the question. There are thus two challenges: First, what is the form in which the respondent represents that relevant information? Second, what are the processes that transform that representation into a response?
If you want to use someone’s responses to predict their future voting behavior, you also need to know what information they use to make their voting decisions and how that information gets transformed into a vote choice (or a choice not to vote at all).
None of these challenges are insurmountable, and all I’m doing is rephrasing what Thurstone said in more cognitive terms. Indeed, to the extent that we have decent theory in cognitive psychology (and we do in areas like memory, perception, and categorization, especially for relatively low dimensional stimuli) it is because we were able to use models to understand how multidimensional psychological representations were transformed into different responses depending on different task demands. So the correspondent’s general point of using a broader range of question types and more sophisticated modeling makes sense and has a good track record.
Lance Rips (on the cognitive psychology side), along with Roger Tourangeau and Ken Rasinski (and to some extent Roy D’Andrade from cognitive anthropology) started a research program called Cognitive Aspects of Survey Responses in the 90s. I think there were a few conferences on this combining perception/memory research with public opinion and this book is a reasonable summary: https://www.amazon.com/Psychology-Survey-Response-Roger-Tourangeau/dp/0521576296
It seems that this research has died down to some extent more recently. Would be great for more cognitive scientists to collaborate with political scientists!
Thanks for pointing to this work! I was totally unaware of it, but it sounds exactly like what I was thinking about.
As you say, it is a shame that this work hasn’t continued or, evidently, influenced polling in any substantial way. It looks like some of the same ideas got used, or at least re-discovered, as part of the “wisdom of crowds” craze about 10 years ago, but I’m not sure if that ended up having any lasting impact.
Agreed, it seems like it would be mutually productive for cognitive science and political science to have more and lasting collaborations.
I don’t know if they include it in their forecasts specifically, but 538 has written a number of pieces about adjusting for name recognition.
I had some experience with ecological questions, and they worked well. They take the form of “What do most people in your group think about X”, where “group” is defined according to the purposes of the survey. I was working with employers of child labor, so this was the group. I can imagine that this kind of question could be reworked in various ways to get at what we might call political subcultures: the group could be a town or neighborhood, a workplace, a church, etc. Of course, a lot of effort would have to go into validation. One of the nice byproducts is that we might not only get more refined measures of voting intention or other political variables but also some insight into the networks through which influence spreads.
Probably not news to anyone here, but somewhat related:
https://www.bbc.com/news/magazine-36803544
I think that it would be wise for the readers of this blog to look at the voting turnout literature. One could do a great job with better question wording, intentions, etc., but voting turnout is likely the elephant in the room when accounting for polling errors in predicting election outcomes.
“Polling using the collective recognition heuristic to get a better sense of popularity?”
Even better: “polling using the election heuristic to get a better sense of popularity?”
Here’s something we rarely consider when developing items: face validity is not necessarily correlated with predictive validity, particularly when the theoretical basis for item design is weak or changes over time. Take the MMPI, which was designed with all sorts of questions that have been empirically found to be predictive but completely lack face validity. This was especially important when the test was originally developed because there was very little in the way of strong theory on psychological disorders, and many popular but wrong theories (e.g., psychoanalytic). One might say political polling is in a similar pickle now: the controlling factors (demographics, candidates, messaging, economy, wars, turnout, etc.) change from election to election, PLUS previous polling results influence future polling responses (supporters of the “winning” candidate tend to be more enthusiastic responders), PLUS the best media/methods for polling different subpopulations are still up in the air. An atheoretical approach might help us to stay afloat until we have sufficiently dynamic theories about polling and voting.
What you and others participating in this post don’t seem to be interested in is why better measures of preferences are important may be of only limited utility in predicting election outcomes. As I pointed out above, there is an rather extensive body of research on voting turnout. There is a significant gap between what people tell you about their voting preferences and what they actually do, both with respect to voting turnout and actual behavioral voting choice. Many of you are simply assuming, without any real evidence, that better measurement of preferences will solve the problem and reduce the gap. Please don’t “shoot from the hip.” Do a little research to learn what others have found.