A note to John

Jeff the Productivity Sapper points me to this insulting open letter to Nate Silver written by pollster John Zogby. I’ll go through bits of Zogby’s note line by line. (Conflict of interest warning: I have collaborated with Nate and I blog on his site).

Zogby writes: Here is some advice from someone [Zogby] who has been where you [Silver] are today.

Sorry, John. (I can call you that, right? Since you’re calling Nate “Nate”?). Yes, you were once the hot pollster. But, no, you were never where Nate is today. Don’t kid yourself.

Zogby writes: You [Nate] are hot right now – using an aggregate of other people’s work, you got 49 of 50 states right in 2008.

Yes, Nate used other people’s work. That’s what’s called “making use of available data.” Or, to use a more technical term employed in statistics, it’s called “not being an idiot.” Only in the wacky world of polling are you supposed to draw inferences about the U.S.A. using only a single survey organization. I do agree that it wasn’t particularly impressive to get 49 states out of 50. What made Nate’s name was not his routine election-night forecast but his exceptional insight during the primary election season, followed up by strong and timely analyses during the following months. If you think that’s so easy, fine: You do it. Until then, how about some division of labor, where analysts such as Nate make use of polling data and pollsters such as John respect that their polls will be used in all sorts of interesting ways once their data go out the door.

Zogby writes: Hey, I have been right within a few tenths of a percent – but you are a probabilities guy and even a 95% confidence level and a margin of sampling error are not enough for some.

A basic understanding of sampling and nonsampling error will tell you that being “right within a few tenths of a percent” is luck, not skill. If you’re within a percent or two, great. A few tenths of a percent on a poll with a 3% margin of error . . . sure, that’ll happen sometime, but perhaps it’s a good idea for you as a pollster to explain to people how randomness works.

Zogby writes: You [Silver] take other people’s polls, compare records for predictions, add in some purely arbitrary (and not transparent) weights, then make your own projections and rankings.

There’s that “other people’s polls” thing again. What’s your problem? Would it make you happier if Nate only used his own polls? Even if Nate conducted his own polls—heck, maybe he’s doing that right now, I have no idea—he’d be a complete an utter idiot to make forecasts from them and not use others’ polls as well. As a forecaster, you’re not goinna go to heaven because you never used anybody else’s data.

Zogby writes: We pollsters are data-based problem-solvers. We work with clients to solve problems, plan the future, project trends, and test effective messages and models. This involves lots of people skills, a passion to get it right and do right by people who trust us. We are so much more than where we stand on election day. Your [Silver’s] ratings come with and generate a lot of vitriol. How does that make our world a better place?

Hey, John. The good news is you’re getting paid for those services! Now take the perspective of those of us who analyze “other people’s polls” (as you put it). You wouldn’t want us to treat all polls equally, would you? To mix a high quality poll such as yours with some crappy robopoll or some discredited partisan hack job? Once we’re using data, it’s our duty to evaluate the quality of our data. Sure, Nate could rate the pollsters and keep his rating a secret, but I think it actually will “make our world a better place” (as you put it) for Nate to be completely open about his procedures and release his poll ratings publicly, where they can be shared, challenged, and improved upon. Earlier in your note you criticized Nate for using weights that are “not transparent.” Releasing poll ratings is a way to increase transparency, no?

Zogby writes: You [Nate] are a statistician – a very good one – but you are not a pollster. You should conduct some polls and learn that the rest of us good pollsters survey people, not statistics.

Haven’t you ever heard about the division of labor? Actually, I agree that Nate would learn a lot by being involved in a survey operation (and, as noted above, maybe he’s already doing so). In the meantime, I have no particular interest in discounting his work because of his lack of experience in this area. Any more than I’d discount Bill James’s 1982 Baseball Abstract because of his lack of direct experience in major league baseball. An outsider’s perspective can be useful.

Zogby writes: The numbers tell the story; preconceived ideologies and fuzzy-math statistical models do not.

Huh? Say again?

P.S. As I think is clear from the above, I think John went just a bit over the top in his criticism of Nate. In all seriousness, though, I think pollsters are extremely important in forming our understandings about politics, and I agree that there are all sorts of skills that good pollsters have. There’s no way I could ever conduct a good poll myself, and I rely in so much of my research on the expertise of pollsters in the field. There’s no reason for John to be so defensive: Nate Silver, Andy Gelman, Larry Bartels and all the rest of us rely crucially on the efforts of public opinion professionals from Gallup to Zogby to gather the information we use in our analyses.

P.P.S. Much more discussion in the comments, in particular here.

24 thoughts on “A note to John

  1. This is a weird and defensive post not in line with your usual posting. Why did this bug you so much?

    Using the example that either you or Phil gave before on climate sensitivity estimates — imagine a snarky letter from James Hansen to an author who only published meta-analyses, arguing against reading too much into the meta-analysis without doing the underlying derivations. (This works quite well as Hansen maintains that climate sensitivity is 4-12 degrees versus the current papers that put it at 2.5-3.5) There is a point there, though a bit abrasive.

  2. Reading between the lines of what Zogby's complaining about, he has a point.

    How does Nate Silver achieve his generally high quality results?

    Answer: by performing meta-analyses of other people's polls.

    But here's the problem that Zogby is pointing toward: it's always easier (or should be easier, if the underlying theory supporting meta-analyses makes sense — and it does) to get better results using meta-analyses than it is to rely on the data and analysis directly available to those who perform the individual polls and/or studies.

    Yet it's simultaneously true that generating that original data is a vastly more demanding effort — certainly in terms of resources — than any meta-analysis.

    And the real problem for individual polling organizations such as Zogby's and Gallup's is that they are pretty much prohibited by institutional competitive demands from presenting a meta-analysis including the results of other organizations.

    I can understand why members of these other organizations see Nate's achievements as cheap exploitations of their own work, given these facts.

    The point here is not that what Nate is doing is not valuable; better information is better information.

    But in terms of credit, it's hardly fair to compare the accuracy of his results directly with those of the individual organizations without taking into account the constraints and responsibilities under which each operates.

  3. Sean:

    Zogby's note pissed me off because he was basically criticizing Nate for building a reputation on using other people's polls. As if it would be better to just use his own. But this is madness. There are zillions of pollsters out there. Of course it is a good idea to combine them; it would be stupid not to.

    If James Hansen wrote a snarky letter to a statistical analyst, I might very well feel the same way. But, as I've discussed several times on this blog, I don't have the expertise in climate analysis that I do in survey analysis.

    You write that "There is a point there." As usual, I'm perfectly willing to believe that I've missed something. I don't see Zogby's point at all. I respect Zogby's polling expertise and I can see that he's upset about some things that Nate's written, but I don't see the merit in Zogby's comments highlighted above.

    Again, I'm happy to learn that I've missed something important. I tried to write the above blog in a serious but lighthearted way in a reflection of Zogby's note, and it would be great if the discussion could move forward.

  4. Liberalbiorealism: I agree (and I emphasized in my blog) that the pollsters who gather the original data deserve most of the credit. But, y'know, nothing is stopping Zogby, or Gallup, or whomever from hiring a "Nate Silver" of their own and doing their own meta-analyses. If they choose to only present their own numbers, that's their choice–but it is a choice. If Nate's achievements are really such "cheap exploitations," they can feel free to duplicate it themselves. Until then, though, I think Nate's adding value.

  5. Andrew,

    I think it's pretty unrealistic to think that, say, Gallup is going to present meta-analyses including the polls of others. Imagine, for example, what those polls might imply as to the perceived relative value within Gallup itself of its competitors.

    On the other hand, I haven't the slightest doubt but that Gallup performs such meta-analyses internally to improve their own modeling.

    Nate achieves his status in no small part, I think, because he's perhaps the best of the amateurs. But I would hardly be surprised if any number of the better polling organizations harbored any number of statisticians/modelers every bit as good — or better — than Nate.

  6. Liberalbiorealism: I don't know about that. Nate's pretty good. Let me put it this way. When it comes to formal qualifications, I'm more credentialed than Nate. And, in some ways, when I look at what Nate does, I think that if I worked full time on this stuff, I could do as well as Nate does with the poll aggregation, poll rankings, etc. But . . . I haven't done this stuff. It's not so easy. You have to care about getting it right, you have to put in a lot of work. And, truth be told, maybe if I tried to do it, I wouldn't do such a good job. Nothing is stopping Gallup from trying to hire Nate, but my guess is that the amount they would pay for such an analyst wouldn't be enough to lure Nate (or, for that matter, me) from our day jobs. Sure, they can hire someone. But "someone" isn't necessarily enough. Even if they find a statistics or economics or poli sci M.A. or Ph.D., that person might not have the skills and sensibility to do a good job. Maybe they do have someone there in the background doing it. Actually, the Gallup reports I see on the web are pretty good–they do seem to have lots of smart people working for them. Still, Gallup (or, for that matter, Zogby) doesn't do what Nate does. Nothing was stopping Gallup or Zogby from starting up their own fivethirtyeight.com-like site back in 2008, and nothing's stopping them from doing it now. Until they do so, I see no sense in disparaging Nate's work by saying that somebody else could be doing better in secret.

  7. I agree, Zogby's note is really confusing if you read it as a sincere personal attack on Nate.
    But looked at another way, it's probably not meant to be taken personally… Zogby has a company to run, and this note seems to be mostly an excuse to advertise his company's expertise without paying to take out an ad:

    "you did rate my telephone election polls in 2008 as one of the three most accurate in a speech to Fordham University"
    "We work with clients to solve problems, plan the future, project trends, and test effective messages and models. This involves lots of people skills, a passion to get it right and do right by people who trust us."
    "We, at Zogby International, have been testing and perfecting new models and methods for years."

    …and so on. So I assume he's not really targeting Nate as such, but prospective clients: *Let me scare you into hiring us instead of a cheaper solo statistical consultant!*
    That would explain why he tries to play down Nate's strengths as if they're weaknesses (using other people's polls, complex statistical modeling).
    (Of course, Nate's "Worst Pollster in the World" headline probably did contributed a bit to the snarky tone.)

    Like you said, "There are zillions of pollsters out there. Of course it is a good idea to combine them; it would be stupid not to." The accusations are clearly silly to anyone who knows the subject, Zogby included, but he's not being sincere — he's just targeting readers who *don't* know the subject.
    Maybe some day he'll try to set up their own Nate-like poll-aggregating service, and just be honest about the pluses and minuses of each approach; but for now he's clearly decided on the strategy of scaring customers off from that route.

    It might be a good public service to write an editorial in response, clarifying that:
    * of course sometimes it really is best to hire a pollster directly;
    * but in other situations you'd rationally hire someone like Nate instead;
    * and it's a shame the paper agreed to run Zogby's fear-mongering self-promotion.

  8. I see nothing but boring turf war. Silver and Gelman take note: you don't get taken seriously by publically mocking your detractors.

  9. Andrew,

    Certainly my speculation that, say, Gallup might harbor statisticians/modelers every bit the equal or better than Nate is, of course, speculation.

    But I don't see how one might reasonably expect Gallup to engage in a fivethirtyeight.com-like site. Just consider only two of the problems with doing so:

    1. They'd have to make implicit comparisons between their polls and those of others — potentially downgrading the relative value of their own polls, or those of others. If they downgraded the value of other's polls, they'd expose themselves to lawsuits.

    2. They'd have to admit their own paid for polls are less accurate than the meta-analysis they are offering up.

    Now I simply must believe that Gallup and other polling organizations have considered the possibility of doing such meta-analyses and publishing them. Really, do you not expect that they aren't aware of the inherent advantages of meta-analyses in terms of greater accuracy? Could they be competent and not have come to this realization? My expectation, then, is that there is a very good business reason indeed for them not to do so — so good, in fact, that, even though accuracy is the touchstone of pollster value, they willingly leave the easy pickings of the inherent greater accuracy of meta-analyses to amateurs.

  10. As a poor individual trying to make sense of the world I would note that all of us, almost all the time, make use of other people's results to determine the truth about a given matter. I for instance am required to use this post, Zogby's letter, and all the comments, to come to some tentative conclusion (or not). But if, as Zogby hints, relying on all this data gathered and analyzed by others, is somehow morally remiss, then why even read anybody, including Zogby himself?

  11. I'm not sure the relevant question is whether or not Gallup could/should/would make use of the polls of other polling firms.

    To me, the Zogby letter smacks of desperation. Nate has, for better or worse, built up a reputation, particularly among the "Netroots." We can see this in how his recent denigration of Research 2000 is looking like it will seriously damage that polling firm. Nate has been doing a series rating pollsters on their accuracy lately, and Zogby will be one of those commented on. And, while Zogby's live-person polls seem to have performed well over many years, Nate has indicated many times that Zogby's internet-based polling indicates serious flaws. While I haven't double-checked Nate's work, that doesn't matter. What mattered to Research 2000 was the accusation of shadiness. That is also likely to matter to Zogby.

    Thus, Zogby's letter. It is an open letter because Zogby is seeking to head this one off at the pass. Regardless of whether Nate is right or not (I happen to think he is), it seems very likely that he will soon end up trashing Zogby's internet-based polling, and it also seems likely that such a trashing would hurt Zogby's business.

    This would be true if Nate was right, or if he was wrong. The truth of the situation won't really matter to Zogby's business, because the accusation will be damaging enough.

    Now, if Nate is right, this would seem to be great. If Nate is wrong, this would be bad. Either way, though, it's bad for Zogby.

  12. Great blog post and conversation. Reminds me of the general trickiness of social epistomology when we discuss incentive barriers to Gallup/Zogby doing meta-analyses of their polls.

    Someone should be policing and ranking the pollsters anyways, right? Academia, science media, think tanks, nonprofits -there seem to be enough funded and vested players around that the only recourse wouldn't need to be amateurs.

    But kudos to Mr. Silver (and also to Prof. Gelman for shining the spotlight analyses of at genius on the mundane). For that matter, kudos to Mr. Zogby for making a boring but important topic more interesting by making it more soap opera-y.

  13. Consider this hypothetical situation though: Gallup generates its own polls, but then secretly conducts a meta-analysis and increases its accuracy by including appropriately and untransparently weighed data from other pollsters – and publishes this as its result! Then Zogby does the same, secretly tweaking its polls with some Gallup goodness. Then all the others do the same thing as well! etc, etc…
    What happens if everyone is secretly and continuously adjusting their data with everyone else's!? A black hole?

  14. Andrew,

    Are you familiar with the Bayesian modeling in horse-race tracking that has been done over the last decade in a series of papers by Jackman, Lewis, Rivers, Pickup, and Strauss?

    From my point of view, it seems superior to what Nate does. From what I understand, it's what the big campaigns use for resource decisions.

    What are your thoughts?

  15. Why do you want to engage with Zogby in a content-free debate about who has insulted whom? It would be much more interesting to hear what you think of Silver's pollster rating methodology. Do you consider his method of estimating a 'reversion parameter' to be reasonable? Not whether shrinkage is a good idea, but the way he does it? What do you make of his standard errors, which seem to be pulled out of a hat? Since he is, rightly or wrongly, associated with you, a lot of people assume that you think his analyses must be state-of-the-art. He is, after all, a famous statistician.

  16. Thanks, all, for the comments. To respond briefly:

    Jerzy: I agree that Zogby might feel threatened by bad publicity from Nate's rankings. From this perspective his attack on Nate is a defensive maneuver. This might all be so; I just don't know enough about the business side of all this so I prefer to respond to Zogby's overt arguments. I have not looked at Nate's pollster rankings and so have nothing to add in that regard.

    Fraac: A wonderful thing about reading a blog is that you can skip the boring entries. I suspect that about half the readers of this blog skip everything in the Political Science category in any case.

    Liberalbiorealism: I'm not saying that Gallup etc. should do their own 538.com-like sites. My point is that Gallup does one thing and Nate does another. As noted above, I think pollsters are great and that Nate is great. My problem with Zogby was his putdown of Nate's work. Nate does something different from Zogby. It's not better or worse, it's just different. Hence my emphasis on the concept of "division of labor."

    J.J.: Indeed. If Zogby wants to argue that Nate's pollster rankings are crappy, I'd prefer he do that directly rather than trying to score points off the fact that Nate uses data collected by others.

    Maja: Your scenario is not so far-fetched. Two incentives that any new pollster has are (a) to be reliable and to give answers comparable to the big boys, (b) to distinguish themselves by get it right when others get it wrong. Zogby touches on both of these in his note. And I'd guess that pollsters adjust their results all them time based on other polling results and based on their expectations and political analysis. Then again, I think there's way too much horserace polling anyway, so this "black hole" scenario doesn't bother be so much.

    David: For reasons discussed in my 1993 paper with King, I'm not a big fan of horserace tracking in any case. I haven't looked in detail at the papers by Jackman, Lewis, etc., but I wouldn't be surprised if they are superior to Nate's method in some aspects. I do think Nate did a good job, day to day, on all the details. Doug Rivers knows more about polling that Nate or I will probably ever know, but ultimately I don't see Nate competing with Doug any more than I see him competing with Zogby: each is playing a different role in the data/analysis/communication process.

    One thing I discussed in 2008 regarding Nate's site is the inevitable tension between the goals of (a) forecasting the election accurately, and (b) providing news every week. During the election season, each new poll–even each new week of polls–provides very little information about the ultimate election outcome. See my paper with Kari Lock (to appear in Political Analysis) for further exploration of this point. There is room in the world for researchers such as Jackman, Lewis, etc., and myself to do rigorous studies and for analysts such as Silver to put things out on the web in real time. And political scientists can definitely learn from outsiders such as Nate Silver in the politics domain or Felix Salmon in economics. We don't have to trust every one of their analyses 100% to recognize that they often supply creative insights.

    Anonymous: I don't think the debate with Zogby is ocntent-free: see all the interesting discussion above. In any case, the short answer is that a friend pointed out this silly thing by Zogby that irritated me, so I blogged on it. I agree that studying Nate's pollster rating methodology would be interesting, but analyzing it would take work! Writing this sort of blog comment is much less effort and, as such, feels more relaxing to me. I'm happy to be associated with Nate Silver and I hope people realize that I am comfortable with Nate's general approach even if I haven't looked at the details of much of what he's done. I've collaborated with Nate on a couple of things, but the pollster ratings are all his own, and I'll leave it to him to defend them.

  17. Is Nate Silver really a statistician? He uses statistics, but so do I and almost every quantitative scientist. Isn't analyst a better term for him? I think it's a disservice to great statisticians who develop methods to call him and other analysts statisticians.

    n.b.: I will gladly call myself a statistician, if the consensus is that Nate Silver is a statistician.

  18. A statistician is a person who works with theoretical and applied statistics. Nate Silver's work in the political realm is exclusively related to predicting political phenomena from statistical data. Nate developed a system for forecasting Major Leage Baseball player performance called PECOTA prior to beginning his psephological work. He is clearly a statistician.

  19. With all due respect, when did it become an issue in the U.S. (or for that matter, the free world) for somebody to find an undeveloped, legal market and exploit it?

    Nate was atracted to this field of endeavor, had the skills to make something of his interest and has been succesful in what he's done.

    I thought that in the U.S., if you built a better mouse trap, the world would beat a path to your door.

  20. Andrew–

    While I think your response was perhaps a bit disjointed, here's where I think Zogby errs, and why his comments are in some cases representative of the pollster industry in important ways:

    What Zogby seems to be resisting is the notion that a pollster should simply be a 'just the facts' data-collector. Being measured purely on data performance seems to be the main objection of his thrust.

    Specifically, Zogby seems to resent the notion of his data being measured against actual election results, especially data far removed from the election. But he also doesn't seem to have actually read Nate's methods in weighting for the effect. Zogby seems to prefer RCP's much cruder method of averaging polls, comparing polls to contemporary (for the poll) competitors.

    Effectively, the John Zogbys of the world (and apparently the R2000's of the world) want the freedom to put their own spin on the numbers. They don't seek to become just raw data gatherers, but rather to gather data and then to present it based on their 'feel' for the data. Which makes sense for Zogby specifically– as one of the few pollsters with individual fame for his profession, Zogby's celebrity is reliant on the notion that a pollster should be more than simply a technician writing down the hard numbers as they gather them, something more than a bean-counter.

    But in a statistic-driven polling future, where data is aggregated across multiple sources, public databases are kept of public reports indefinitely, and transparency of methods becomes the new norm, there is less room for pollster to interpret their data. In the polls-as-raw-data (as opposed to polls-as-end-products), the point is for pollsters to be as impersonal and just-the-facts as possible. And that's a world John Zogby can't really compete with the Nate Silver's of the future in.

  21. From a business standpoint it is easy to explain Zogby's fear. The idea of "commoditizing your complements" is well known in software (see a timeless post here, e.g.,: http://www.joelonsoftware.com/articles/StrategyLe… ). Simply put if you create a technology that works on a higher level of abstraction, and is able to take other existing technologies and use them at inputs you can gain an advantage by pitting those input providers against each other, reducing the price/value they command, while increasing the price/value you can command.

    So what Zogby fears is that the hard job of conducting individual polls is becoming less valuable than the job of creating good models that synthesize multiple polls. And he's right to be worried, as pretty much by definition there should be more signal in more polls, if as Silver does, you track the signal/noise ratio (accuracy) of each pollster over time.

    There is no magic in Silver's work (nor would he claim there is), but the demand for his approach is clearly high. And the ready availability of data makes the work manageable and timely to traditional customers of polling companies. The likely next step is that Zogby and other's will start to try to make legal claims about their ownership of their data, and will stop sharing internals and crosstabs of polls, or at the very least attempt to delay making such info available whenever possible.

  22. Why all the deferential props to Zogby as a pollster? He had a few big hits some years ago, but his many whiffs since sought that the former were more chance than skill.

    Zogby has essentially admitted that he adjusts his results based on his "gut," meaning the result he intuits will make him the winner of the poll heat. But we don't turn to pollsters for gut instincts and pseudoscience. Hence Nate's well-earned reputation, and Zogby's spent cred.

  23. As for whether a polling company could set up an aggregator web site using others data – this was done by YouGov before the 2010 UK general election. You can see it here, no one seemed to mind.

Comments are closed.