In 2000, Francis Tuerlinckx and I published our paper on Type M and Type S errors.
In 2014, John Carlin and I published a paper on the topic with a more applied focus, including an R function “retrodesign” for computing these error rates given assumptions.
In 2019, Andy Timm improved the function and put it on CRAN.
Since then, Timm and Martijn Weterings found some small problems. They fixed and updated the retrodesign package, and it’s right here on CRAN again.
I hope you find it useful.
Why did this technique develop so slowly and not gain more attention?
Jay:
I think part of the problem is confusion with traditional retroactive power analysis which is done conditional on the point estimate of the effect size. It’s well known that this sort of retroactive power analysis is bad, and several papers have been published over the years explaining the problem. When we do retrodesign, we condition on an assumed effect size. That’s the key difference.
The first plot in the vignette seems to have a constant standard error that is not dependant on the sample size. Alternatively, the effect size of 0.5 is defined in terms of difference/se (and not difference/sd). What am I missing?
The sample size is fixed. The chart shows the sampling distribution of estimates for a given true value and standard error. We see 5000 points following a normal distribution and that could also have been represented by the density as in https://nicebrain.files.wordpress.com/2015/05/fourth_plot1.png
I wonder whether Andrew and collaborators imagined this as an actual design tool, and how much it was imagined as an pedagogical and/or persuasive bit of maths to highlight the issues with NHST in certain circumstances? Surely anyone looking at plausibly small effects in a high noise environment would studiously ignore this tool as the main conclusion would be “don’t bother”!
Well, yes indeed. Of course the name ‘retrodesign’ signals the main intent. In my mind at least it’s intended to be a tool for reflecting retrospectively on conclusions that you might make if you are still inclined to do that dichotomization thing (statistical significance!). If you’ve moved on from dichotomization then you probably don’t need it…
Thanks John, appreciate the response!
It seems the package has been removed from CRAN now, will it be added back again?
The link https://cran.r-project.org/web/packages/retrodesign/index.html still doesn’t work…Is there an alternative link to cite?