My new article, “Failure and success in political polling and election forecasting” . . . and the tangled yet uninteresting story of how it came to be

Here’s the article, which is scheduled to appear in the journal Statistics and Public Policy:

Polling got a black eye after the 2016 election, when Hillary Clinton was leading in the national polls and in key swing states but then narrowly lost in the Electoral College. The pre-election polls were again off in 2020, with Joe Biden steady at about 54% of the two-party vote during the campaign and comfortably ahead in all the swing states, but then only receiving a 52% vote share and winning some swing states by narrow margins. The polls also overstated Democratic strength in congressional races. In other recent elections, the record of the polls has been mixed . . .

There is more to political polling than election forecasting: pollsters inform us about opinion trends and policy preferences. But election surveys are the polls that get the most attention, and they have a special role in our discussions of polling because of the moment of truth when poll-based forecasts are compared to election outcomes . . .

The recent successes and failures of pre-election polling invite several questions: Why did the polls get it wrong in some high-profile races? Conversely, how is it that the polls sometimes do so well? Should we be concerned about political biases of pollsters who themselves are part of the educated class? And what can we expect from polling in the future? The focus of the present article, however, is how it is that polls can perform so well, even given all the evident challenges of conducting and interpreting them. . . .

And here are the sections of the article:

1. A crisis in election polling

2. Survey errors and adjustments

3. Polls and election forecasting

4. Explanations for polling errors

5. Where polls get it right

There’s nothing particularly new in this article but I found it helpful to try to organize all these thoughts. The reviewers for the journal were helpful in that regard too.

How the article came to be

In early December 2020 I received the following email:

Dear Professor Gelman,

I hope this email finds you well.

With this email, the editors of Inference: International Review of Science, would like to introduce you to a quarterly online journal, one whose remit is the sciences, from anthropology to zoology.

With this in mind, the editors of Inference would like to invite you to author a critical essay on the reliability of polling, and the statistical inference that polling relies upon. We feel that there are few people better placed to write on such a topic, and would be honored and grateful were you to accept our invitation.

The editors encourage authors to determine the length of an essay according to their own sense of the space needed to address the topic at a suitable level of depth and detail. By way of a general guide, the average length of the essays of this type that we publish is approximately 3,500 words.

Inference is a fully independent and properly funded journal. We have no ideological, political, or religious agendas. We remunerate our authors appropriately for their contributions.

Please do not hesitate to contact me if you have any questions.

With our best wishes.

Sincerely,
**

I’m as susceptible to flattery as the next guy, also I don’t really need the money but I’m happy to take it when it comes in, and in any case I’m always looking for new audiences.

I was curious about this magazine Inference that I’d never heard of, so I googled and found this article from 2019, “Junk Science or the Real Thing? ‘Inference’ Publishes Both.” A five-year-old quarterly review called ‘Inference’ mixes science and political ideology. It is funded by Peter Thiel.” This seemed kinda bad. I mean, sure Peter Thiel is free to fund a magazine and publish what he wants—but an article on polling that “mixes science and political ideology” . . . that didn’t seem like such a good idea. On their webpage, they say, “We have no ideological, political, or religious agendas whatsoever.” But that might not be true!

Then again, the above-linked article was published in the fake-MIT science magazine Undark, which I don’t trust at all! So I wasn’t really sure what to think. A magazine that I don’t trust tells me that this other magazine is untrustworthy. On balance, I liked the idea of a general-interest article on the reliability of polling—it was a topic that was on a lot of people’s minds—so I replied that, sure, I could give it a shot. Writing for a magazine that also dabbles in evolution denial: sure, why not?

They asked for something within 6-8 weeks, and it was about 5 weeks that I sent in my article.

The editorial assistant replied with some helpful suggestions, first off to give citations to all my claims. That was a good idea: Often when writing for non-specialized audiences I’m discouraged from citations, and I appreciated the push in this case to provide a link for each of the detailed statements in the article. The assistant also forward a note from the editor that they wanted more sophistication and more explanation about the statistical models we used. I was like, cool!, and I sent back a revision the next day. The subject was topical so no point in delaying, right? I few weeks later I heard back from the editorial assistant:

Thanks for your email, and apologies for not getting back to you sooner—this has been a busy period for us.

While your most recent work is most certainly heading in the right direction, there remain several points that the editors of Inference would like to see reinforced, in order to shore up the essay’s central argument.

Attached is a copy of this essay with comments from the editors. These comments are there to guide you on which parts of this piece they feel need to be expanded on, or which need to be clarified.

Do let me know how this sounds to you. We enormously appreciate all of the work that you have already put into this piece, and look forward to seeing what you come up with once these revisions have been made.

Many thanks, and with all our best wishes,
**

The additional suggestions were not too much so I revised and sent it back, then a couple weeks later got this:

The editors have looked through the changes made to the piece and have decided that this is something that we will not be pursuing. Unfortunately, it is their feeling that this piece does not meet Inference’s usual standards of precision.

Many thanks for all of the work you have put into this. It is our regret that we cannot move forwards with it.

With all our best wishes.

Fair enuf; it’s their magazine and they can do what they want. It just seemed weird. I sent an email expressing puzzlement but got no reply. The funny thing was that they went to all the trouble of soliciting the article and editing it. It kinda makes me wonder what they were expecting from me in the first place.

I mean, sure, you could easily make the case that this article (see above link; this is the version to be published in the statistics journal but it’s not very different from what I’d sent to the magazine) was too boring for a general-interest magazine—but (a) the magazine editors never said anything about readability or accessibility or interestingness or general interest or whatever, and (b) what were they expecting from me in the first place? The “usual standards of precision” didn’t really fit into their specific comments on the paper. The whole thing was a waste of time for all concerned, but really more of a waste of time for them than for me, as they had to go back and forth with the editing, ending up with nothing, whereas I at least had this article I could publish somewhere else. So I remain baffled. My best guess is that they were interested in polling in December 2020 back when everyone was interested in polling, then they lost interest in polling in February 2021, around the time the everybody else was losing interest. As to the disconnect between the oleaginous initial email and the curt send-off . . . I guess that’s just their style. In retrospect I guess they should’ve thought through more what they were looking for.

Anyway, after this had all happened, it was February and I had this article so I sent it to my contacts at various general-interest outlets: the Atlantic, Slate, New York Times, etc., but nobody wanted it. This made me kinda mad, as I think back in December there was more interest in the topic! I eventually thought of Statistics and Public Policy, which is one of the American Statistical Association’s more obscure journals, but it’s a journal I kinda like! I know its editors, and I’ve published some papers there before:

The twentieth-century reversal: How did the Republican states switch to the Democrats and vice versa?

The 2008 election: A preregistered replication analysis (with Rayleigh Lei and Yair Ghitza)

19 things we learned from the 2016 election

When all is said and done, more people will read my new article on polling from following the link in this post (here it is again) than ever would’ve read it through the Inference website (or, for that matter, the Statistics and Public Policy website); still, I have some wistfulness about the article not being published in that magazine. I’m always trying to reach new audiences. I still kinda wonder what they were originally looking for from me. I could’ve given them some hardcore MRP, but “the reliability of polling” is not a mathematical topic at all; it’s much more about who gets polled, who responds to surveys, and who turns out to vote.

Publishing

Hmmmmm . . . let me put it this way: I write stuff I want to write, typically because I think I have some important or interesting point to make, and then I figure out how to give people the opportunity to read it. Sometimes the idea for the article to write comes from the outside. For example, a few months ago I was asked by an editor for the Journal of Indian Institute of Science to write an article on a certain Bayesian topic. I wasn’t so interested in the particular subject that was suggested, so I proposed the following alternative:

The Development of Bayesian Statistics

The incorporation of Bayesian inference into practical statistics has seen many changes over the past century, including hierarchical and nonparametric models, general computing tools that have allowed the routine use of nonconjugate distributions, and the incorporation of model checking and validation in an iterative process of data analysis. We discuss these and other technical advances along with parallel developments in philosophy, moving beyond traditional subjectivist and objectivist frameworks to ideas based on prediction and falsification. Bayesian statistics is a flexible and powerful approach to applied statistics and an imperfect but valuable way of understanding statistics more generally.

This seemed like it would be a fun article to write and potentially useful to readers. I’d never heard of the Journal of Indian Institute of Science, but who cares? What matters is the article, not where it appears. I guess I was going into the polling article with the same idea: Here’s a good topic and a publication that’s evidently interested in it, so let’s go. If the editors of Statistics and Public Policy had been the first to suggest the topic to me, I probably would’ve sent it to them first, sparing everyone concerned lots of hassle. The article would’ve been slightly different, though.

20 thoughts on “My new article, “Failure and success in political polling and election forecasting” . . . and the tangled yet uninteresting story of how it came to be

  1. > There is also the practical issue that any survey adjustment method (whether by weighting, MRP, or some other approach) requires data from the survey and the population. But often all we have are the “toplines”: the summaries from the poll with no raw data and often with incomplete information on how the summaries were obtained. Indeed, for our Economist forecast we performed no MRP at all—no modeling and no poststratification. Instead, we just worked with the toplines publicly released by the polling organizations, with the hope that individual pollsters did a good job of adjusting for nonresponse (and including in our model the possibility that they did not).

    How much better do you think your predictions would be if you had the raw data?

  2. It sounds like Inference did you a favor and helped you dodge a bullet. I’m wondering if you regret having agreed to write this for them to begin with – not because of the wasted time or opportunities, but because it perhaps was not a wise place to try to publish to begin with. I appreciate your interest in novel outlets and attempts to reach new audiences, but in hindsight is sure looks like that journal has a political agenda. Regardless of what that agenda might be, if you had published there, you would have tied your reputation to it. I guess I’m asking if you think you should have turned down the offer.

    • Dale:

      I’m not sure. I’ve written a bunch for Slate, which started out center-left in the mold of its original editor, Michael Kinsley, but has moved pretty far to the left in recent years—but I don’t think this ties my reputation to their agenda. Maybe writing for a right-wing publication would improve my reputation by making me look more nonpartisan? Ultimately I don’t think it matters much, because my article didn’t have any partisan content. Which is not a brag on my part—I fully respect people who write partisan articles—it’s just the way it is.

      If, however, I’d tried to make the editors of this magazine happy by playing to what I saw as their agenda—perhaps by advancing spurious claims of election fraud, maybe in a “just asking questions” mode—then, yeah, that could’ve damaged my reputation, as would’ve been appropriate. One complicating factor is that I have no idea who actually reads that magazine. I feel like I have a much better sense of who reads Slate and who reads Statistics and Public Policy. Indeed, in writing for Inference, the “new audience” I’m reaching might not be the readers of the magazine (I have no idea if they have any readers at all, actually!) but the editors, who are perhaps influential in the conservative moment? I have no idea at all, really.

      • In terms of due diligence, I’m not sure what the lesson is here. Wasnink, Ariely, et al coauthors trusted the source of the data, ignored potential warning signs, and ended up having to say they did not question enough. Your case is certainly very different – and I’m not trying to portray your decision to try to write the article for them as comparable to these other cases. But in terms of how much due diligence to do, I think there is a common element. I recall that case last fall with the mathematics professor who agreed to do a bogus analysis of the election results, using data that he did not carefully question. We all engage in actions where we ultimately “trust” sources of data, intentions of researchers and journals, etc.

        For myself, I have often testified using data I did not collect, sometimes not asking too many questions about how it was collected. While I am careful not to vouch for the accuracy of the data, I am still loosely tying my reputation to it – just as those coauthors did. In your case, had the article been published, you would have loosely tied your reputation to a number of things about the journal that you do not really know (who is the audience? how is editorial policy decided? how might the journal promote your work?). I know you can defend your work against claims that might arise (I accept your characterization of its lack of partisan content), but I’m wondering whether your decision to write the article is different in kind than the decision that some of these coauthors had made.

        What advice do you have about how much due diligence should be exercised when making such decisions? Does your advice differ in accepting an offer to write for a particular publication and deciding to coauthor a paper for which you will not get to see the data?

        • Dale:

          I’m not sure. To me, writing for a publication is not the same as analyzing someone’s data. A publication can be a loose collection of articles without there always being an organizing theme. I’ve published in PNAS and would continue to do so, even though I think they have shown terrible editorial judgment and even though I think they do a sort of logrolling which is one of the worst things about contemporary science. I’ve published in Slate and would continue do to so, even though I disagree with many articles they publish.

          That said, there are some lines I would not cross. For example, I don’t think I’d publish at the site that features articles such as “Pork-Stuffed Bill About To Pass Senate Enables Splicing Aborted Babies With Animals.” And I don’t think I’d ever give a Ted talk. In either of these cases, perhaps I could be persuaded to do so, but I doubt the persuasion would work—also I doubt that anyone associated with either of these organizations would want a contribution from me.

          For the math professor who did the bogus analysis of election results: yeah, there’s a long tradition of math professors being asked to teach classes in probability and statistics, and then thinking they’re experts in the topic. Back when I was at the University of California, there were about 15 of those sorts of guys in the statistics department. I have no patience for people like this. You could be a math genius and prove Fermat’s last theorem, whatever. That doesn’t mean you can go into an applied stat problem with no preparation and solve it. Michael Jordan couldn’t hit the curveball. I think the problem with that Williams College math professor was not that he had too much trust in others (the people who supplied him with data); I think his problem was that he had too much trust in himself.

    • The journalis available seems to be available online:

      https://inference-review.com/

      Just looking over some of the articles, it didn’t seem to push an overtly ideological position, although I thought a review of Chomsky’s Syntactic Structures was too uncritical. It does publish responses to the articles, however, as well as author’s response to the comment. I don’t think there would be anything to be concerned about: it looks legit to me.

      • Joe:

        I agree that it looks mostly reasonable, although if you look around a bit, you can find some evolution denial. It’s hard for me to get too worked up by evolution denial, though, as even the hardcore anti-Darwinists seem to accept that covid can mutate etc. At this point, anti-Darwinism seems less like science denial and more like a non-actionable religious stance, analogous to believing that Moses parted the Red Sea or that the Mormon dude really discovered those tablets of gold or believing in Thor or whatever.

        It’s still kind of a mystery to me who reads the magazine, given that it seems to have a reasonably large budget but nobody’s heard of it. I’m guessing they send emails about it to some group of elite influencer types, I dunno.

        • David Berlinski is a contributing editor, so it’s not surprising an anti-evolutionist stance will be found. I guess this might be part of a question about whether you’re going to get these things if you advertise yourself as non-ideological. This came up in the Social Text discussion as well. There used to be journals where people published a lot of different things, and they didn’t have a peer-review process precisely because they didn’t see themselves as a “scholarly” journal (in the narrow sense of that term). It’s a modern form of the “little magazine” format, but it’s largely irrelevant now given blogs.

          I agree that I can’t imagine who the audience is for this, and that your interactions with them were extremely odd. I just was responding to Dale’s comment. Looking at the other articles, I don’t think there’s much risk of tying yourself to a particular agenda, but I don’t see you expanding your readership either: as you say, I’m not sure who the readership is.

        • Joe:

          Yeah, it was all mysterious to me. One of the baffling parts of the story was the way in which the editorial assistant kept referring to “the editors” without saying who they are. It was a strange series of email exchanges where I was corresponding indirectly with these faceless people.

          My usual experience with scientific journals is to get a correspondence from the editor or associate editor directly, and my usual experience with magazines is to be corresponding directly with whoever’s editing the piece, so it was unusual to be held at arms length in this way.

          I just noticed that the editorial board is given on this webpage. I wonder who it was who was editing my article? It’s a bunch of physicists and some other people too . . . I don’t see anyone who does political science or sociology or survey research or anything like that. There’s Noam Chomsky—maybe he didn’t like my paper because it wasn’t left-wing enough, or he heard that I was skeptical of the research of his buddy Marc Hauser? Jay Bhattacharya of Stanford—maybe he was mad about that covid stuff? But then why would they have invited me to write the article in the first place? There’s Martin Nowak—I’ve heard of him, he’s the Jeffrey Epstein associate. It’s an odd cast of characters, actually an unusual collection of eminences for a magazine that nobody has ever heard of!

          In general, one of the challenges of writing any article is matching it to the audience. In this case, this was a problem because I had no idea who the audience was.

          Next time someone asks me to write an article for their magazine, I guess I’ll ask them to let me know who I’m writing it for. Or else get payment in advance.

  3. I love this:

    > We have no ideological, political, or religious agendas whatsoever.”

    Funny. What a ridiculous statement.

    There’s so much out there about there about how cognitive biases work, and yet so many who are very focused on cognitive biases seem to know absolutely nothing about it.

    It’s so common for people to think that those that disagree with them ideologically or politically are infected with bias, whereas those that they align with politically or ideologically are bias-free.

    Joe Rogan’s whole brand is that he has no political agenda and people who align with him ideologically and/or politically just suck that up.

    A nice example of this phenomenon just the other day on this blog:

    https://statmodeling.stat.columbia.edu/2022/01/03/the-gullibility-connection-how-does-joe-rogan-and-more-generally-the-joe-rogans-in-the-media-decide-where-to-be-skeptical-and-where-to-be-credulous/#comment-2041908

    • One of the points made by another commenter there was that Rogan himself isn’t providing an agenda, he just lets guests promote their own on his show. Rogan himself presumably didn’t have much in the way of ideas about sleep, but he’ll let Matthew Walker spout nonsense which Rogan himself doesn’t know to be nonsense (because he doesn’t know much about the science of sleep nor is he going to “do his own research” on whether some academic’s published work is actually junk).

      • Wonks:

        My idea was that Rogan should have Alexey Guzey on his show to shoot down Matthew Walker, or he could invite Guzey and Walker on together. That would be good theater, right?

        But, after reading this comment thread, I get the impression that it would not actually be good theater, or good podcast, for Rogan to have this sort of battle on his show. It sounds like what people like to hear on podcasts is not a clash of different views but rather a single take, presented sympathetically. I guess that a Joe Rogan show featuring Guzey, or Guzey and Walker, would not be so popular. Walker gives good stories (even if they’re false, maybe especially if they’re false) and a sense of certainty. Guzey gives boring truths and a sense of uncertainty, which isn’t so much fun to as many listeners.

        • > Guzey gives boring truths

          Well, to your original good theater point, the Walker takedowns at least were entertaining truths

        • Ben:

          Guzey’s takedown was entertaining to the audience of Statistical Modeling, Causal Inference, and Social Science, maybe not so entertaining to the audience of Rogan.

          Conversely, some bullshit Ted talker enthusiastically citing nonexistent or fatally flawed studies, saying how this can Change Your Life, that could be lots of fun to the Rogan audience but boring to us.

      • Wonks –

        Rogan doesn’t have to express his agenda through each and every podcast in order to nonetheless have an agenda.

        I think he clearly has an agenda overall. That’s not a bad thing. We all have agendas.

        Related, beyond the Rogan-specific context, I think it’s pretty clueless and contra a large body of literature for Rogan and his huge supporting cohort to claim that unlike pretty much everyone else, they’re agenda-free.

        Further, the literature would predict that differentially finding an agenda in those who disagree with you ideologically, and finding those you’re in agreement with ideologically as agenda-fee, is EXACTLY what you’d find among pwolw who have an agenda!

        But sure, I could be wrong. Maybe I only see an agenda with Rogan because of my own agenda.

        So from a bird’s-eye view, how could this be settled? How could it be determined in some objective fashion whether he really is in some kind of exclusive class of the agenda-less?

        • And btw –

          Imo, if you listen to a good bit of Rogan, you can see him applying double standards all over the place.

          Imo, that’s a pretty good marker of having an agenda. Of course, someone else might dispute the examples where I see a double standard, but I think that whether he applies double standards could possibly be productively subjected to an objective interrogation.

  4. Why do you, Andrew, think of Statistics and Public Policy as an obscure journal? I’m not so familiar with it but I recently looked into it (because of your articles) and it seems like it has a great editorial board.

Leave a Reply

Your email address will not be published. Required fields are marked *