Aaron Strauss provides more evidence that, compared to forecasts based on fundamentals, early polls give almost no information about election outcomes. Strauss writes the following about allocation of campaign resources:
The key to a effective strategy is determining, ex ante, which states will be pivotal on Election Day. Existing Bayesian election models are inappropriate for this game, as they estimate the current standings of the candidates or states rather than the final outcomes. I [Strauss] develop a Bayesian dynamic linear forecasting model that incorporates informative priors from historical regressions, updates based on in-cycle state and national polls, and accounts for the uncertainty of events that take place between the polls’ issuance and Election Day. National and state shocks are modeled as a reverse random walk beginning with the final outcome and moving backwards through time. Uncertainty about the final outcome is calculated by combining the random walk’s linearly decreasing variance over time, natural poll measurement error, house effects of national polls, and historically stable trends of election results. Using the resulting estimates of the states’ standings relative to each other, I simulate electoral vote outcomes and determine the probability of each state being pivotal. I find that early polls can be misleading to such an extent that putting any weight on them produces worse forecasts than solely relying on historical trends.
How is this any different from 538’s analysis?
This is basically the part of Nate Silver’s model that he hasn’t discussed much publicly at all (at least, from what I’ve seen reading the blog and the FAQs). He says that there is a probability model, and constantly talks about running simulations, but doesn’t say how individual simulations are run. Do they involve 51 random walks, or are there random walks for each demographic group, or something else?
(It’s never too late to respond to blog comments, right? I just saw these…)
First, credit to Nate Silver for popularizing these methods. I would point out that I posted/presented this paper before 538 was in existence.
Though my method and Silver’s are similar in many respects, the one important distinction is that I use historical regressions to downweight state-specific information contained in polls. (Silver downweights national leads by candidates early in the cycle, but even in this case, he regresses the lead to a somewhat-arbitrary mean of zero rather than using national historical regressions.)
North Dakota is the most striking example of the difference between our models. Silver, in August, looked at the impressively pro-Obama polls in ND and did not discount them–ND was projected to be only 3.8 percentage points below the national Obama vote share (estimate from 8/11/08). The historical regression, using only past results and current demographic and economic forecast ND to be 10.9 percentage points below the Obama average. My paper helps inform how much importance you should put on the historical regression vs. the polls (in August, about 2 parts regression, 1 part polls; this ratio reverses later in the campaign). So, in August, combining the historical regression and the polls, I projected ND to come in 8.6 percentage points below Obama’s vote share. (I have email evidence of this projection.)
On Election Day, Obama received 53.7 percent of the 2-party vote nationally and 45.6 percent of the two-party vote in North Dakota, for a difference of 8.1 percentage points. And that’s why Nate Silver should discount spring/summer polls more than he does now.
Caveats: 1) You might say I cherry-picked ND; I did, but I cherry-picked it in August. Silver’s projections and mine lined up well in many states in August; North Dakota was the biggest discrepancy between our models so I kept my eye on it over the course of the election. A more in depth analysis is forthcoming in early January now that (nearly) all the ballots have been counted. 2) While my model is more sophisticated in the above manner, Silver’s model out-does mine when it comes to things like pollsters’ house effects. And I tip my cap to him for that.
Aaron,
I agree with you. I’d also say that Nate put himself in somewhat of an impossible position, because the goals of poll aggregation and election forecasting are different. A convention bounce is meaningful in terms of short-term public opinion but does not say much about the ultimate election outcome.
I just wanted to follow-up with results of the complete analysis, which is easily reported with the sum squared errors of state-level predictions (relative to the national prediction).
538.com, predictions on 8/11/08: SSE of 0.072
Historical regression, using economic data available in mid-August plus the August Palin pick: SSE of 0.075
From that, you might say “Nate wins!”, but I never claimed that one should rely completely on historical trends, just that one should incorporate their information. As I mentioned in comment #3 above, I recommended 2 parts historical, 1 part polling in August (with this ratio shifting toward polling later in the campaign).
2 parts historical, 1 part 8/11 polls: SSE of 0.065, much better than either of the two alone.
I could have provided slightly better info to the Obama campaign by recommending 1 part each, which had a SSE of 0.064.
I used the 2004 polls (and historical regressions) to inform the 2:1 August ratio, and since that’s an n-size of one campaign, I admit that I didn’t (and still don’t…now n-size is two) have a good grasp of the correct ratio.
I see two reasons that this ratio is more weighted toward polls in 2008 than 2004. First, in my 2004 calculations I used the Bayesian model developed by Simon Jackman, which is less sophisticated than Silver’s model. So, Silver is probably teasing more information from the polls. Second, in 2004 an incumbent was running leading toward more stability in terms of vote changes from the previous election to the current one.
Also, yes, I agree that Nate put himself in a tough position. I would have preferred he focus solely on forecasting, but that may have resulted in fewer web hits.