Skip to content
Search results for replication

Why did it take so many decades for the behavioral sciences to develop a sense of crisis around methodology and replication?

“On or about December 1910 human character changed.” — Virginia Woolf (1924). Woolf’s quote about modernism in the arts rings true, in part because we continue to see relatively sudden changes in intellectual life, not merely from technology (email and texting replacing letters and phone calls, streaming replacing record sales, etc.) and power relations (for […]

Megan Higgs (statistician) and Anna Dreber (economist) on how to judge the success of a replication

The discussion started with this comment from Megan Higgs regarding a recent science replication initiative: I [Higgs] was immediately curious about their criteria for declaring a study replicated. In a quick skim of the info in the google form, here it is: In the survey of beliefs, you will be asked for (a) the probability […]

Why We Sleep—a tale of non-replication.

Good to have a non-coronavirus post that I can put on delay . . . After reading our recent post, “Why We Sleep — a tale of institutional failure”, David Shanks wrote: You may be interested to know that a little while ago we were completely unable to replicate a key result by Walker and […]

Social science and the replication crisis (my talk this Thurs 8 Oct)

My talk at the WZB Berlin Social Science Center 3pm (Central European Time): Social science and the replication crisis The replication crisis is typically discussed in the context of particular silly claims, or in terms of the sociology of science, or with regard to controversies in statistical practice. Here we discuss the content of unreplicated […]

“Postmortem of a replication drama in computer science”

Rik de Kort writes: This morning I stumbled across a very interesting blog post, dissecting some drama related to a non-replicating paper in computer science land. The question the paper tries to answer is whether some programming languages are more error prone than others. For a paper in computer science I would expect all their […]

Some thoughts on another failed replication in psychology

Joe Simmons and Leif Nelson write: We report our attempt to replicate a study in a recently published Journal of Marketing Research (JMR) article entitled, “Having Control Over and Above Situations: The Influence of Elevated Viewpoints on Risk Taking”. The article’s abstract summarizes the key result: “consumers’ views of scenery from a high physical elevation […]

Let’s do preregistered replication studies of the cognitive effects of air pollution—not because we think existing studies are bad, but because we think the topic is important and we want to understand it better.

In the replication crisis in science, replications have often been performed of controversial studies on silly topics such as embodied cognition, extra-sensory perception, and power pose. We’ve been talking recently about replication being something we do for high-quality studies on important topics. That is, the point of replication is not the hopeless endeavor of convincing […]

Different challenges in replication in biomedical vs. social sciences

Summary – In biological sciences, it might be reasonable to expect real effects to replicate, but carrying out the measurement required to study this replication is difficult for technical reasons. – In social sciences, it might be straightforward to replicate the data collection, but effects of interest could vary so much by context that replication […]

This study could be just fine, or not. Maybe I’ll believe it if there’s an independent preregistered replication.

David Allison sent along this article, Sexually arousing ads induce sex-specific financial decisions in hungry individuals, by Tobias Otterbringa and Yael Sela, and asked whether I buy it. I replied that maybe I’ll believe it if there’s an independent preregistered replication. I’ve just seen too many of these sort of things to ever believe them […]

Instead of replicating studies with problems, let’s replicate the good studies. (Consider replication as an honor, not an attack.)

Commenter Thanatos Savehn pointed to an official National Academy of Sciences report on Reproducibility and Replicability that included the following “set of criteria to help determine when testing replicability may be warranted”: 1) The scientific results are important for individual decision-making or for policy decisions. 2) The results have the potential to make a large […]

More on why Cass Sunstein should be thanking, not smearing, people who ask for replications

Recently we discussed law professor and policy intellectual Cass Sunstein’s statement that people who ask for social science findings to be replicated are like the former East German secret police. In that discussion I alluded to a few issues: 1. The replication movement is fueled in large part by high-profile work, lauded by Sunstein and […]

Replication police methodological terrorism stasi nudge shoot the messenger wtf

Cute quote: (The link comes from Stuart Richie.) Sunstein later clarified: I’ll take Sunstein’s word that he no longer thinks it’s funny to attack people who work for open science and say that they’re just like people who spread disinformation. I have no idea what Sunstein thinks the “grain of truth” is, but I guess […]

You are invited to join Replication Markets

Anna Dreber writes: Replication Markets (RM) invites you to help us predict outcomes of 3,000 social and behavioral science experiments over the next year. We actively seek scholars with different voices and perspectives to create a wise and diverse crowd, and hope you will join us. We invite you – your students, and any other […]

Oh, I hate it when work is criticized (or, in this case, fails in attempted replications) and then the original researchers don’t even consider the possibility that maybe in their original work they were inadvertently just finding patterns in noise.

I have a sad story for you today. Jason Collins tells it: In The (Honest) Truth About Dishonesty, Dan Ariely describes an experiment to determine how much people cheat . . . The question then becomes how to reduce cheating. Ariely describes one idea: We took a group of 450 participants and split them into […]

Bayes, statistics, and reproducibility: “Many serious problems with statistics in practice arise from Bayesian inference that is not Bayesian enough, or frequentist evaluation that is not frequentist enough, in both cases using replication distributions that do not make scientific sense or do not reflect the actual procedures being performed on the data.”

This is an abstract I wrote for a talk I didn’t end up giving. (The conference conflicted with something else I had to do that week.) But I thought it might interest some of you, so here it is: Bayes, statistics, and reproducibility The two central ideas in the foundations of statistics—Bayesian inference and frequentist […]

Hey—take this psychological science replication quiz!

Rob Wilbin writes: I made this quiz where people try to guess ahead of time which results will replicate and which won’t in order to give then a more nuanced understanding of replication issues in psych. Based on this week’s Nature replication paper. It includes quotes and p-values from the original study if people want […]

The replication crisis and the political process

Jackson Monroe writes: I thought you might be interested in an article [by Dan McLaughlin] in NRO that discusses the replication crisis as part of a broadside against all public health research and social science. It seemed as though the author might be twisting the nature of the replication crisis toward his partisan ends, but […]

“The idea of replication is central not just to scientific practice but also to formal statistics . . . Frequentist statistics relies on the reference set of repeated experiments, and Bayesian statistics relies on the prior distribution which represents the population of effects.”

Rolf Zwaan (who we last encountered here in “From zero to Ted talk in 18 simple steps”), Alexander Etz, Richard Lucas, and M. Brent Donnellan wrote an article, “Making replication mainstream,” which begins: Many philosophers of science and methodologists have argued that the ability to repeat studies and obtain similar results is an essential component […]

Ambiguities with the supposed non-replication of ego depletion

Baruch Eitam writes: I am teaching a seminar for graduate students in the social track and I decided to dedicate the first 4-6 classes to understanding the methodological crises in psychology, its reasons and some proposed solutions. In one of the classes I had the students read this paper which reports an attempt to reproduce […]

Why is the replication crisis centered on social psychology?

We had a post on this a couple years ago, but the topic came up again, and here are my latest thoughts. Psychology has several features that contribute to the replication crisis: – Psychology is a relatively open and uncompetitive field (compared for example to biology). Many researchers will share their data. – Psychology is […]

Where can you find the best CBD products? CBD gummies made with vegan ingredients and CBD oils that are lab tested and 100% organic? Click here.