April 20, 8pm CET Deadline for answering the pre-market survey.

April 22, 8pm CET Prediction markets open.

May 3, 8pm CET Prediction markets close.

Less than a month from A to Z? That’s suspicious on its own. Particularly when there is ZERO output reported 2+ months later – even though it’s just 18 papers. (Smarter people would have realized that 18 papers is too small a number, IMO).

]]>While the efforts to replicate high profile research should definitely be applauded, I don’t really see the point of using prediction markets to forecast the results. Other than generating interest in the replication efforts, what difference does it make whether ‘peers’ can accurately forecast the results? (And yes, I understand that it may provide some information about the credibility of specific studies.)

]]>Of course, prediction market design requires that event outcomes can be clearly said to have either occurred or not. Therefore, we define a result as being replicated if the statistical method used in the original paper yields a p-value 0.05.

How were sample sizes chosen?

All planned sample sizes were chosen so that if the replication produces an effect of the same strength as in the original study, the probability of p<0.05 in the replication is 90% (the tests have 90% power). In some cases, larger samples than those with 90% power were chosen because the lowest multiple of the typical original session size was somewhat above the 90% power sample."

http://sciencepredictionmarkets.com/repdetails.html

I suspect this is unclear and they mean "significant in the same direction". So if the null is always false there is 50% chance of winning with very large sample size. Looking at http://sciencepredictionmarkets.com/studies.html it appears they have usually doubled the sample size of the original study. So 0.9*0.5=0.36, or 36% percent would replicate if there was nothing to the theories at all?

]]>Betting against counter-intuitiveness, social psychology and small samples was an okay strategy for me (I was a student then and had almost no inside knowledge), I think I tripled my initial seed fund, some even quadrupled it. One problem they had is that the studies took so long to conclude, so they settled 7 on last observed market prices.

Participants have been paid, but I don’t know how good the predictions were, I assume this will be in the big reproducibility project paper.

They repeated this again with the ManyLabs 2 project, though I think that’s not that far along yet.

]]>