A couple days ago, Raphael asked in the comment thread:
I have a question about a hypothetical scenario. If one of the candidates were to leave the ticket and be replaced by another candidate, would you recalibrate the model to match the new candidate? I mean, throw out the old Biden vs. Trump polls and wait for new ones? Or would you say, along the lines of your argument about 538’s high uncertainty, that the model allowed for the possibility of a candidate switch? Or, third option, try both (old calibration and new calibration) to see if the model output changes in any meaningful way at all?
Here was my reply of what to do with the forecast if one or both candidates were to be replaced on the ticket:
I don’t know what the plan is with the Economist. My quick recommendations would be as follows:
1. Replace the fundamentals-based model as appropriate. For example, if Biden is replaced, the Democrats are no longer running an incumbent candidate, but they are still the incumbent party, so the prediction should account for that. And fix any home-state or home-region effects.
2. Keep the old polls but downweight them in some way, probably by adding an error term corresponding to an unknown candidate effect.
That’s all that comes to mind right now but maybe some other things would need to be done too.
And I’ll again point to Rajiv Sethi’s thoughtful and informative post on election prediction markets. As I wrote a couple days ago, a tempting solution is to just ignore the data-based forecasts and just go with the prediction market, but that’s a form of circular reasoning: The prediction markets are informed by economic conditions and by the latest polls, but the latest polls are noisy, which motivates some amount of poll aggregation, which then raises the question of how exactly to aggregate, which then motivates forecasting models—and there we are! Turtles all the way down.
I think this question has a much wider application than just this one electoral race, important as it is. For me it gets to, what is the point of empirical analysis in conditions of radical uncertainty? If one is faced with a slew of unknowables that make it impossible to set forth even a tentative probability distribution of outcomes (e.g. Keynes, copper and revolution), there is little practical value in forging ahead anyway. This is where we are right now in the presidential race, with too many imponderables in play.
More useful is pulling back a bit and assessing the evidence for potentially influential mechanisms. For instance, what does the recent experience at the state level tell us about the potential for the succession battle within the Democratic Party to fragment it along race and gender lines? And what factors in the past seem to have influenced approval numbers for Harris; in particular, have her episodes of greater visibility improved or diminished her net approval? (Or both?) Then we have some empirical basis for forecasting the impact of particular developments on ultimate November outcomes, which is not the same as forecasting the outcomes themselves, but not nothing. More generally, in conditions of radical uncertainty, there is a lot of value in being able to assess the effect of developments, themselves unpredictable, in close to real time—and also in knowing which developments to encourage or resist.
The nominee is almost certainly going to be Harris, given the party have left it too late to really go with anyone else. I would assume that you would need to factor in her popularity and prior polling as somehow relatively integral to the model. There have probably only been a handful Harris v Trump polls historically, but potentially plenty of ones facing her off against Pence.
That actually raises a question in my mind at least – how valuable are polls just describing popularity of a candidate in a vacuum (e.g., what is your opinion of Harris) vs their popularity against other candidates (e.g. who do you prefer between Harris and Trump)?
Quote from above: The nominee is almost certainly going to be Harris, given the party have left it too late to really go with anyone else.”
Who could have seen that coming? The o so subtle signs were on the wall concerning Biden’s possible cognitive decline, but still. Man, what a storyline! Some last minute sudden changes and developments are always exciting and keep people watching. Can you imagine if Biden would have stepped down earlier what would have happened. This is way more exciting and it gives the opportunity to pick Harris, who could have predicted that!!
I can’t wait for the new promo-videos with even more story-telling, character development, and emotional connection concerning the new candidate! May I propose an entrance song for Harris if she becomes the nominee: “Puppet on a string” by Sandy Shaw!
p.s. I simply love this post, including the comments, the timing, and the whole prediction market thing and blogpost where it is questioned if there can be “manipulation”. It’s just all so absurd to me it actually fits perfectly with people like Biden and Harris, the state of parts of the world, and social science and all that stuff. It’s like watching a movie.
Quote from above: “It’s like watching a movie.”
A subtle possible underlying message of Harris in relation to Biden (‘s state) is also that it almost literally is an example of the indoctrination and/or manipulation and/or view that “old” and other features are “bad” and that “young” and other features are “good”.
This does, of course, not need any explanation or have to make sense because we’re talking about politics, and social science, and that might all be more about manipulation and direction and stupidity. But the entire thing about changing Harris for Biden might have this subtle undertone to it which is why I think the writers of this particular script may have also consciously or unconsciously opted for this all.
I just thought about this while sitting on the toilet after my 1st morning coffee, and I think in light of this all they could pick an even better entrance song that is in line with the above subtle undertones and themes. Might I suggest something like “Changes” by David Bowie, or “The Times They Are A-Changing” by Bob Dylan. Given the possible age group Harris might aim for, I would guess they might want something as recent as possible, so of the two options above I think they might choose Bowie.
Question:
Do you “computer-nerds”, “social scientists”, and “data-analysts” ever think about how all this attention given to things like “prediction markets” might somehow harm actual political discourse?
Or how things like “polls” and “prediction markets” might actually be open to manipulation to paint an inaccurate picture or steer things in a certain direction?
It’s no wonder perhaps why some folks are messing with “prediction markets” in social science…
Anon:
It seems a bit weird to use the term “Computer-nerd” as an insult, given that it’s no secret that I’m a professional statistician and write computer programs all the time.
Anyway, yes, I’ve indeed thought about problems with polls prediction markets (or, as you call them, “polls” and “prediction markets”). Here’s an example from not long ago, and over the years I’ve written other things on the topic.
Some notes:
1- the term “computer-nerd” was not necessarily meant for you alone, I was/am talking to all the readers of the blog
2- the term “computer-nerd” is often used by “computer-nerds” themselves if I am not mistaken, for instance to come across as “quirky” and “unique” (while often being the exact opposite of that). It may also not necessarily be an insult
3- I used the term “computer-nerd” as part of an experiment I am conducting (e.g. see other recent posts with similar language). I am wondering whether the use of such language might tap into some personal issues or might be “shocking” to certain people in academia which might make them temporarily “snap out” of their possible elitist ivory towers and/or brainwashing and/or manipulation. The underlying hypothesis is that there might be personal, psychological reasons for academics and/or scientists to behave non-scientific or non-ethical or non-rational, etc.
Thank you for thinking and writing about prediction markets, I quickly browsed through your example and it’s not what I hoped for but that all doesn’t matter. I made my point here in a very concrete manner I reason, and I think that folks can ponder those types of questions if they want to. Or not.
What is your experimental design?
Quote from above: “What is your experimental design?”
It’s been a while since I read about study designs and all that. It’s not a proper experiment to be clear, I hope that was clear. I just have thought about stuff for a while and I am more and more convinced that there might be a big problem with certain types of people in academia/science (e.g. psychopaths). One way to try and possibly solve these problematic issues is trying to make this all clear, and one way to do this might be to make people take a step back and really observe things and really think about things. That’s where the “snap out” stuff comes in, by stating things in a perhaps un-academic and/or unusual way.
I think my experiment can best be described as a mix of several things, where I attempt to make things clear in a perhaps not so typical “academic” manner. I may have an unusual writing style and/or point of view which I have grown into more and more over the years while writing manuscripts and while participating on this specific blog actually. I really enjoy trying to state things in a certain way, or use ideas and sentences by others in a different way. Sort of like a musician might take samples and create something new.
In fact, as a result of the experience with scientific writing and not wanting to be part of science/academia anymore I have recently focused on writing song lyrics and poems. I shared some of that on this blog as well, also very recently. Both the scientific writing and poetry and lyric writing sometimes looks like a puzzle to me. I try and find connections with stuff, and “fit” things together, and sometimes something comes out of that all that’s way better than I could have ever thought I could write. I also sometimes feel like I didn’t even write it myself, it sort of ended up like that. Whether the writing is any good I don’t know, and I probably never will know, and perhaps that’s fine. I wrote a poem on here a while ago on a post about cherry blossoms, and
someone replied with “that’s lovely” or something like that and I really appreciated that.
Perhaps all I can do is sing my song. Maybe there is someone who hears it, maybe there isn’t. But perhaps all I have to do is sing my song once in a while. That’s why I came back here after a break of 6 months or so, just to sing a few songs because I felt the time might be right to do so. I don’t know if it is though, and I probably never will know, and perhaps that’s fine.
I suppose polls could be manipulated. However, I don’t think prediction markets can be (at least not for long periods of time). If someone is consistently manipulating prediction markets, they will eventually go bankrupt, and they won’t be able to manipulate them anymore. Also, others will correct the market almost instantly to squeeze profit out of these manipulators. At least with Manifold Markets’ market-maker, wealthy users (who usually have a history of being correct) are able to sway the markets more than poor users who are worse at predicting, so it’s difficult for impoverished manipulators to even make much of an impact anyway.
The comments make me (again!) express my distaste for polls. Peter raises good points about the difficulties – perhaps even impossibility – of estimating the effects of Biden withdrawing. I question the whole idea of polls here. Are they useful? I believe they are, and that is why I dislike them. Will the polls cause any real changes in policies for either party? I don’t believe so – I do believe it will change their messaging. How is it a good thing for the results of the polls to change messaging but not the actual policies that parties or candidates will pursue? It is one thing to use polls to try to gauge how the public feels about issues – I think that is important for democratic processes. But polls to determine who people might vote for, or whether they will vote at all, are a different matter. I actually don’t see how that is socially beneficial (although I can easily see the private benefits that can accrue to the candidates).
I feel the same way about most advertising. If businesses use their market research to design products that are closer to what people want, then I think that is socially beneficial (and I know some market research is used that way). But market research aimed to developing ads that make people more likely to purchase a product – without any effect on its quality – may be privately beneficial and not socially so.
No doubt the Democrats will look intensely at polling data to determine what set of candidates has the best opportunity to beat Trump. The Republicans are already doing the same kind of analysis – hence the messaging from Trump to position himself as a centrist and distance himself from the extreme right of his party. Does this mean he will do anything different if elected? I doubt it.
Dale:
I don’t agree with you fully about polls, but I’ve been on record for a long tim that I think there’s way too much polling being done. Regarding your comment, I’ll just give the quick answer that people will be polling, and others will be interpreting these polls, and perhaps the most important role of poll aggregation is not to forecast elections but rather to shoot down naive readings of the polls based on short-term swings. I think it’s mildly socially beneficial to provide tools that can improve reporting and understanding of public opinion.
I can agree that polling research is valuable if it reduces the misuse and misinterpretation of polls. It has further value in the methodological advances it provides. What I object to is its social value in terms of democratic processes. Even there, it is polling about likelihood of winning or feelings about a candidate that I object to, not polling about public views about issues. The whole “probability of winning” equates politics with sport – an equation that I believe has become dangerously entrenched in the American psyche.
I agree, though you could replace “polls” with “television” (conceptually, if not syntactically) and make the same statement about its effect on how people engage with politics.
Andrew:
If I am not mistaken, here you seem to imply that the main utility of polls is as evidence for some kind of narrative about public opinion and perhaps on how it can be influenced. But the discussion in your post is about “modeling” with all its concomitant assumptions and pseudoscientific machinery. The models invariably are used to produce some kind of statement like “if the election were held today the probability of candidate X winning is 85%” or some such. Nate Silver is a particular purveyor of this kind of useless and unjustified nonsense.
I know that you have generally taken a more sober view of polls and their applicability, but seem often to backslide into suggesting mathematical modeling, where all the work is being done by implicit assumptions, as a way to interpret polls in a world where “we really don’t know what will happen, but these histograms show what people seem to be thinking now” is the only sensible analysis.
No statistical workflow nor super-duper auto-differentiating probabilistic programming should lend credence to a forecast, which will give entirely different answers when for example I assume that the current situation is more similar to the 1972 election than to the the 1980 election or vice-versa.
Harsha:
Statistical modeling of human behavior is not “pseudoscientific machinery” (to use your term). It’s real science: we form hypotheses, gather data, make inferences and predictions, . . . the whole deal. It’s social science, not physical science: we’re asking people their opinions, not measuring the speed of chemical reactions. But it’s still the scientific process, at least as I see it.
Regarding your point about assumptions: One of the main virtues I see in mathematical/statistical modeling, rather than just asking pundits to give their hunches, is that the model has clear assumptions, and you can trace the logic leading from the assumptions and data to conclusions; if the conclusions don’t make sense you can reassess your priors, etc.
One additional issue is worth considering: polls may be self-fulfilling. If enough people hear candidate X is ahead, then they start believing that and even might increase their support for candidate X. Of course, it could work in reverse, where they shift their support to the underdog, but given the American public’s seeming preference for winners, I suspect the polls might impart some sort of momentum effect. Perhaps this has been studied?
Dale:
In multi-candidate elections, I agree that there can be bandwagon effects with polls. In a straight-up two-candidate election where the candidates are far apart on the issues, I don’t think so. It’s not zero—there’s turnout, third parties, etc.—but I don’t think it’s much at all.
While I do think reading too much into prediction markets seems naive (ie you don’t need there to be manipulation to reach a conclusion that there is no way those markets are efficient) I don’t understand your statement about using them is a form of ‘circular logic’.
One could reason that these markets are efficient and that you learn nothing more from looking at polls than is already revealed in prices. One could also reason that even with inefficient even irrational markets the translation of information in polls is superior to anything g someone could do with a model based on raw polling data.
One point in the linked Sethi post is that at least the prediction markets gave nonzero probability to Kamala Harris. But I’d say this just reveals a weakness in the models built on polling data and not an inherent advantage of a prediction market.
Also, though, I think Sethi’s post contains a problematic claim:
“ As discussed in my last post, statistical models perform well only when the past is a reasonably good guide to the future.”
He sounds like Taleb trying to claim that statisticians can’t build models that account for “black swan’s”.
Jyd:
I think prediction markets add value as an information aggregator. The information still needs to come from somewhere, and the polls are a key source of this information. That’s what I meant about circular logic: not that prediction markets are useless, just that if someone wants to rely on prediction markets and ignore the polls, that’s fine—but the prediction markets are not ignoring the polls.
You say, “one could also reason that even with inefficient even irrational markets the translation of information in polls is superior to anything someone could do with a model based on raw polling data.” I don’t think so. Before there were poll aggregators, there were polls, and people did the aggregation in their heads, as it were. Now that there are a zillion polls, you want to aggregate them in some way. If you’re not using a formal statistical model, you’re still doing some procedure, and some procedures are better than others.
To put it even more simply, just as the markets make use of the polls—bettors are aware of the polls—, markets also make use of public forecasts (and maybe also of private forecasts, to the extent that there’s anyone out there doing Economist-style forecasting or whatever using private polls). So, sure, I can believe that the markets can do better than the forecasting models—but a lot of this is that bettors will, quite rationally, use outputs from forecasting models in their decision making.