Brendan Nyhan writes:
Have you heard about the Election Research Preacceptance Competition that Skip Lupia and I are organizing to promote preaccepted articles? Details here: http://www.erpc2016.com. A number of top journals have agreed to consider preaccepted articles that include data from the ANES. Authors who publish qualifying entries can win a $2,000 prize. We’re eager to let people know about the opportunity and to promote better scientific publishing practices
The page in question is titled, “November 8, 2016: what really happened,” which reminded me of this election-night post of mine from eight years ago entitled, “Election 2008: what really happened.”
I could be wrong, but I’m guessing that a post such as mine would not have much of a chance in this competition, which is designed to reward “an article in which the hypotheses and design were registered before the data were publicly available.” The idea is that the proposed analyses would be performed on the 2016 American National Election Study, data from which will be released in Apr 2017. I suppose it would be possible to take a post such as mine and come up with hypotheses that could be tested using ANES data but it wouldn’t be so natural.
So, I think this project of Nyhan and Lupia has some of the strengths and weaknesses of aspects of the replication movement in science.
The strengths are that the competition’s rules are transparent and roughly equally open to all, in a way that, for example, publication in PPNAS does not seem to be. Also, of course, preregistration minimizes researcher degrees of freedom which allows p-values to be more interpretable.
The minuses are the connection to the existing system of journals; the framing as a competition; the restriction to a single relatively small dataset; and a hypothesis-testing framework which (a) points toward confirmation rather than discovery, and (b) would seem to favor narrow inquiries rather than broader investigations. Again, I’m concerned that my own “Election 2008: what really happened” story wouldn’t fit into this framework.
Overall I think this project of Nyhan and Lupia is a good idea and I’m not complaining about it at all. Sure, it’s limited, but it’s only one of may opportunities out there. Researchers who want to test specific hypotheses can enter this competition. Hypothesis testing isn’t my thing, but nothing’s stopping me or others from posting whatever we do on blogs, Arxiv, SSRN, etc. There’s room for lots of approaches, and, at the very least, this effort should encourage some researchers to use ANES more intensively than they otherwise would have.
I know that NHST isn’t your thing but is that the same as “Hypothesis testing isn’t my thing”? I thought hypothesis testing, in the general sense, was core to the scientific method?
Or do you mean it in the division of labor sense: You prefer to do the hypothesis generation & let others do the testing?
Or do we mean that we can go about the scientific enterprise with no real need for hypothesis testing?
Rahul:
Sure, I have hypotheses all the time. But I guess I don’t really see my research as testing hypotheses. I think of my research as using models to make predictions and then seeing where the predictions fail, so as to develop better models.
Can you expand (or have you already expanded elsewhere?) on how this differs from hypothesis testing? Maybe give a concrete example of what you would do compared to what a hypothesis tester would do?
Shea:
Take a look at my applied research papers; that is, the applied subset of these.
Andrew:
What you described sounds like hypothesis testing to me.
Maybe special issues or special sections in journals would work better…
This seems to me to be a massive case of multiple comparisons and, with the large amount of designs they are obviously hoping for, some results will turn out “significant” by random chance. I mean, in the normal case I guess you would use some kind of correction, right? In this case though, you need to submit your design before hand, without knowing what other designs and hypothesis are submitted, so you can only design with corrections for any multiple comparisons in your own analysis. I guess you could apply a Bonferroni correction to all alphas when submission closes but that may be way too conservative depending on what is actually submitted.
Is my interpretation reasonable or is this a non-issue due to something I’ve missed?