I (Lauren not Andrew writing) will be speaking at an upcoming online workshop on reproducibility (free and open). More details here. Looking at the talk outlines, I’m really looking forward to it. I think we can generally agree that reproducibility is a good thing, and something we want to strive for, but in practice there’s a lot of complexity to a real world reproducibility workflow. I’m by no means an expert, so I’m hoping to pick up so new tips, tricks and reproducible perspectives!
The Faculty of Information and the Department of Statistical Sciences at the University of Toronto are excited to host a two-day conference bringing together academic and industry participants on the critical issue of reproducibility in applied statistics and related areas. The conference is free and will be hosted online on Thursday and Friday 25-26 February 2021. Everyone is welcome, you don’t need to be affiliated with a university, and you can register here.
The conference has three broad areas of focus:
- Evaluating reproducibility: Systematically looking at the extent of reproducibility of a paper or even in a whole field is important to understand where weaknesses exist. Does, say, economics fall flat while demography shines? How should we approach these reproductions? What aspects contribute to the extent of reproducibility.
- Practices of reproducibility: We need new tools and approaches that encourage us to think more deeply about reproducibility and integrate it into everyday practice.
- Teaching reproducibility: While it is probably too late for most of us, how can we ensure that today’s students don’t repeat our mistakes? What are some case studies that show promise? How can we ensure this doesn’t happen again?
We intend to record the presentations and will add links here after the conference. Again, the conference is free and online via Zoom, everyone is welcome – you don’t need to be affiliated with a university. If you would like to attend, then please sign up here.
> Does, say, economics fall flat while demography shines?
Interesting and again a good argument for randomly auditing studies conducted in different fields.
And thanks for the heads up on this.
Hi Keith,
I wrote that. By way of background, my phd is in economics, while my wife’s is in demography. So it was kind of a tongue-in-cheek aside.
We’ve got a nice collection of speakers, and I hope that you can join for any that you’re interested in. The plan is to record them and post to YouTube, so you can also get to any with bad timing later.
Rohan
This situation only happened because people replaced replication and predictive skill with peer review and p-values.
Simply return to requiring independent replication and testing predictions on new data as the standard.
> predictive skill
This hasn’t been replaced at all! Indeed, the evidence is irrefutable that we all have the skill to predict where pleasant, unpleasant, and/or erotic pictures might be in the next few seconds (Bem, 2011).
The real shame is that we have not exploited this skill to improve the quality of our science.
Gec:
Alan Turing would be so disappointed to learn how little progress has been made in ESP in the past 70 years!
Not sure why my post triggered a comment about ESP. Making predictions about future data is *the* gold standard for a scientific theory/model. And independent replication is *the* gold standard for what data can be trusted.
Coming up with a model that explains data you already have is great, that is abductive reasoning. It is a crucial first step. But then you need to correctly predict what new data will show too. And ideally your predicted outcome would be surprising if the other theories floating around were true. This is all hundreds of years old successful science, that has only been abandoned since 1940 or so.
https://en.wikipedia.org/wiki/Experimentum_crucis
https://en.wikipedia.org/wiki/Abductive_reasoning
Please make a distinction between reproducibility, repeatability and replicability. Different meaning, different ways to assess. I can send references on request….