House effects, retro-style

Check out this graph of “house effects” (that is, systematic differences in estimates comparing different survey organizations) from the 1995 article, “Pre-election survey methodology,” by D. Stephen Voss, Gary King, and myself:

houseeffects.png

(Please note that the numbers for the outlying Harris polls in Figure 1b are off; we didn’t realize our mistake until after the article was published)

From the perspective of fifteen years, I notice two striking features:

1. The ugliness of a photocopied reconstruction of a black-and-white graph:

2. The time lag. This is a graph of polls from 1988, and it’s appearing in an article published in 1995. A far cry from the instantaneous reporting in the fivethirtyeight-o-sphere. And, believe me, we spent a huge amount of time cleaning the data in those polls (which we used for our 1993 paper on why are campaigns so variable etc).

3. This article from 1995 represented a lot of effort, a collaboration between a journalist, a statistician, and a political scientist, and was published in a peer-reviewed journal. Nowadays, something similar can be done by a college student and posted on the web. Progress, for sure.

Also, to return to a recent discussion with Robin Hanson, yes, this was a statistics paper that was just methods and raw data and, indeed, I think my colleagues in the Berkeley statistics department probably gave this paper zero consideration in evaluating my tenure review. This work really was low-status, in that sense. But this project felt really really good to do. We had worked so hard with these data that it seemed important to really understand where they came from. And it had an important impact on my later work on survey weighting and regression modeling, indirectly leading to our recent successes with Mister P.

1 thought on “House effects, retro-style

  1. Andrew – you did notice that the comment by pingo is spam? The newer type of blog spam – analyze the topic of a thread and copy&paste something relevant from somewhere else (in this case the abstract from the collection of Freedman essays that just came out from Cambridge UP). Then add a spam link at the bottom.

    It is pretty amusing that the spam-bot picked Freedman, though ;-).

Comments are closed.