Opportunity for publishing preregistered analyses of the 2016 American National Election Study

Brendan Nyhan writes:

Have you heard about the Election Research Preacceptance Competition that Skip Lupia and I are organizing to promote preaccepted articles? Details here: http://www.erpc2016.com. A number of top journals have agreed to consider preaccepted articles that include data from the ANES. Authors who publish qualifying entries can win a $2,000 prize. We’re eager to let people know about the opportunity and to promote better scientific publishing practices

The page in question is titled, “November 8, 2016: what really happened,” which reminded me of this election-night post of mine from eight years ago entitled, “Election 2008: what really happened.”

I could be wrong, but I’m guessing that a post such as mine would not have much of a chance in this competition, which is designed to reward “an article in which the hypotheses and design were registered before the data were publicly available.” The idea is that the proposed analyses would be performed on the 2016 American National Election Study, data from which will be released in Apr 2017. I suppose it would be possible to take a post such as mine and come up with hypotheses that could be tested using ANES data but it wouldn’t be so natural.

So, I think this project of Nyhan and Lupia has some of the strengths and weaknesses of aspects of the replication movement in science.

The strengths are that the competition’s rules are transparent and roughly equally open to all, in a way that, for example, publication in PPNAS does not seem to be. Also, of course, preregistration minimizes researcher degrees of freedom which allows p-values to be more interpretable.

The minuses are the connection to the existing system of journals; the framing as a competition; the restriction to a single relatively small dataset; and a hypothesis-testing framework which (a) points toward confirmation rather than discovery, and (b) would seem to favor narrow inquiries rather than broader investigations. Again, I’m concerned that my own “Election 2008: what really happened” story wouldn’t fit into this framework.

Overall I think this project of Nyhan and Lupia is a good idea and I’m not complaining about it at all. Sure, it’s limited, but it’s only one of may opportunities out there. Researchers who want to test specific hypotheses can enter this competition. Hypothesis testing isn’t my thing, but nothing’s stopping me or others from posting whatever we do on blogs, Arxiv, SSRN, etc. There’s room for lots of approaches, and, at the very least, this effort should encourage some researchers to use ANES more intensively than they otherwise would have.

7 thoughts on “Opportunity for publishing preregistered analyses of the 2016 American National Election Study

  1. I know that NHST isn’t your thing but is that the same as “Hypothesis testing isn’t my thing”? I thought hypothesis testing, in the general sense, was core to the scientific method?

    Or do you mean it in the division of labor sense: You prefer to do the hypothesis generation & let others do the testing?

    Or do we mean that we can go about the scientific enterprise with no real need for hypothesis testing?

  2. This seems to me to be a massive case of multiple comparisons and, with the large amount of designs they are obviously hoping for, some results will turn out “significant” by random chance. I mean, in the normal case I guess you would use some kind of correction, right? In this case though, you need to submit your design before hand, without knowing what other designs and hypothesis are submitted, so you can only design with corrections for any multiple comparisons in your own analysis. I guess you could apply a Bonferroni correction to all alphas when submission closes but that may be way too conservative depending on what is actually submitted.

    Is my interpretation reasonable or is this a non-issue due to something I’ve missed?

Leave a Reply

Your email address will not be published. Required fields are marked *