Following these comments of mine, Justin wrote:

I [Justin] like Bob’s paper a lot, and I’m glad you raised it, because I think it is a bit under-appreciated, and also a bit misunderstood. It turns out that Bob and I disagree a bit about the message from his paper (although after a few long chats, we don’t disagree *that* much).

My [Justin’s] thoughts:

1. Bob and Chris has four elections in their data, so it is hard to draw too much from it. That said, I draw two conclusions. First, markets beat an unconditional use of polls as forecasts. Second, correcting the polls, the two are pretty darn close. Based on a sample of four elections, I’m not sure I am willing to call one method or the other the winner.

2. My conclusion is that this actually tells us a lot of what prediction markets do: they digest and aggregate the polls, and create a pretty useful adjusted forecast. If I don’t have the time to do the careful aggregation that Bob and Chris have, then I’m glad to have the market to do this for me. So I interpret their paper as telling me something about the mechanism by which prediction markets do well in forecasting elections. (This suggests an interesting puzzle: why did prediction markets do well in the pre-polling era?)

3. Your comment that polls are a snapshot, not a forecast, strikes me as a bit beside the point. Many people use polls as a forecast, and so it is reasonable to ask if they are a good forecast. (And pollsters sell them as forecasts, until you suggest they don’t do well, and then they fall back on the “snapshot” argument.)

4. Bob and I have a friendly bet (I can’t remember whether it’s a bottle of wine, or not), on whether his results will hold up in a larger sample. Given that we have both state-by-state polls and state-by-state markets for the 2004 and 2008 election cycles, one can extend his approach so that we get around 100 markets (rather than four). We are planning to work together on this. My guess is that this larger sample will confirm the view that markets beat even Erikson-adjusted polls, although Bob’s guess is the opposite.

Point is, the facts can resolve our debate, and we are planning on doing exactly that. I think it will be fun to work with Bob on this, as he is clearly much more deeply embedded in many of the issues around political forecasting than I am. It should be a fun learning opportunity.

Bob then chimed in:

Of course the market prices have value, particularly at this early pre-campaign stage when the polls are rather meaningless.

From my paper with Chris, the main critique is two-fold:

(1) there is an (extreme?)long-shot bias. For instance each general election set of market prices in spring of the election year is virtually 50-50. The market says ‘anything can happen’ while the polls usually say ‘x is favored’ and then x wins. (Maybe if the market says probabilities of 52 and 48 we should bet on the 52?)

(2) I have seen no evidence that markets incorporate information not in the polls. And this is not because I resist the idea or haven’t looked for it.

I look forward to to working on the project with Justin on state polls. I think we might find that (1) with long-shot adjustment (not really possible in our earlier analysis), the markets could do at least as well as the polls

but

(2) with a long shot bias. To simplify the expectation a bit, the prices look reasonable with a tilt toward the market winner.

The question I have for Justin is this:

Apart from the electoral arena, what are the best claims for information market efficiency?

Justin then wrote:

I think that the strongest evidence for prediction market efficiency comes from sports betting markets. There’s a ton of data out there that come from sports betting kiosks, and while there are some smallish anomalies (including the favorite-longshot bias), the evidence in favor of the performance of markets v. experts is really very strong there. Judi Bola is a good example of this. This is why the site is so popular with its users. As you know, I grew up working for bookies, so this really was where I began. I grew up knowing that using Real Bookies was an excellent way to make more profit in this industry. In fact, Real Bookies was voted number 1 by Tech Times fairly recently. My colleagues in finance would suggest that the real evidence comes from equity / bond / futures / options markets, but there are enough well-known anomalies in those domains that I’m a bit less convinced. (And if you were making the rhetorical point that prediction market aficionados often overstate the evidence for their beliefs, I completely agree.)

I hadn’t really coded your piece with Chris as being driven by the favorite-longshot bias, but now you say it, the point is obvious. I’ll make sure to reference your paper in the draft that Eric and I are working on about favorite-longshot biases in political markets generally. (I think you saw early work on this, while in Palm Desert.) For Andrew’s benefit: we have collected data on literally hundreds of political prediction markets, and find fairly pervasive evidence of a favorite-longshot bias. The work on that paper isn’t done yet, but the message is very unlikely to change. There is a very strong favorite-longshot bias in political prediction markets.

BTW, I’m willing to make an even stronger forecast than Bob: I think the unadjusted markets will outperform even the Erikson-adjusted polls. (And for sure, the Wolfers-adjusted markets will beat the Erikson-adjusted polls.) And yes, I’m partly informed on that score by my experience with sports betting markets. If getting into sports betting appeals to you, mobile betting from your phone might be a good way to enter into this.

In terms of your question as to whether markets include info not in polls, I think you are right to infer that a lot of what markets are doing is following polls. Equally, the strongest evidence of markets doing some independent info aggregation comes from looking at political prediction markets in the pre-polling era. (If you haven’t seen it, I think you will really enjoy Rhode and Strumpf’s paper in the Journal of Economic Perspectives, where they review political prediction markets back to 1880.)

Also, I think that there are some pretty clear examples of the markets getting ahead of polls. Take Fred Thompson, who still does moderately well in national polls, but who the markets have written off. Obviously the polling result is driven by pure name recognition, and so what the markets are doing here is appropriately discounting the predictable polling errors, but they are using non-poll info to do this. (eg They don’t think Hillary’s advantage will also dissipate.)

Then Bob wrote:

This is just to clarify my comment on whether market prices incorporate useful non-poll info. The comment was directed at markets vs. polls during the election year campaign, not early pre-election moments like today when candidates like Thompson can make superficial advances in the volatile primary election polls.

Can, for instance, short term price changes (as if incorporationg new info) predict subsequent poll shifts? I have searched for evidence and can’t find any.

Even this primary season, it is my impression that “events” like Guliani’s or Clinton’s bad week(s) has little impact on the polls whereas one might think it does. Of course maybe the correct answer is that these “bad weeks” are random noise that should be ignored. But then there is no dynamism to deal with. Once you know the candidates, you know their chances, and that is all you need to know.

And I wrote:

I’m fine with prediction markets being better than the polls, I just want “the polls” to be defined appropriately. For example, parties typically gain 5-10% in the polls after their nominating conventions. The conventions, on average, provide no information. Clearly if you just look at “the polls” before and after the convention, you’ll screw up your forecasts. But that isn’t the right thing to do–it’s the snapshot/forecast distinction. I just get irritated at the raw claim of markets being better than polls, since polls aren’t supposed to be direct forecasts. To put it another way, I think there are two points to be made: (1) Polls aren’t forecasts (even when people try to ask clever questions such as “who do you plan to vote for” or “who are you sure to vote for” rather than the usual “who would you vote for if the election were held tomorrow”; see our 1993 paper for a discussion of question wording), and (2) Prediction markets work almost as well as, or better than, adjusted polls. Both points are interesting, but the statement, “markets are better than polls” is mostly due to (1), and I think you’re interested in (2).

And I’ll give Justin the last word:

To Bob: I think it is fair to say that only reasonably important events matter. For instance, the markets clearly do move during the presidential debates. To my eye, they even moved in a sensible way. Equally, when Drudge said that Kerry had “an intern problem” (later proved false), that did cause the markets to move against Kerry.

It will be interesting – through this campaign – to see if there are other examples of what you and I would call “news”, and to see whether the markets react. On Thompson, I think it is fair to say that the markets responded to his lackluster campaigning well in advance of the polls. But perhaps you would judge that too early to be useful.

To Andy: I do think that both #1 and #2 are relevant. You are smart enough to adjust polls for post-convention bounce, but not everyone else is. And also, if we always ignore the movement in polls subsequent to a convention, then we risk ignoring those times when a convention actually does shape public opinion.

Equally, I think you (and Bob) are both right to ask for more evidence on #2. I’m guessing that this is what we’ll learn from our research.