Skip to content
Archive of posts filed under the Zombies category.

This study could be just fine, or not. Maybe I’ll believe it if there’s an independent preregistered replication.

David Allison sent along this article, Sexually arousing ads induce sex-specific financial decisions in hungry individuals, by Tobias Otterbringa and Yael Sela, and asked whether I buy it. I replied that maybe I’ll believe it if there’s an independent preregistered replication. I’ve just seen too many of these sort of things to ever believe them […]

Researcher offers ridiculous reasons for refusing to reassess work in light of serious criticism

Jordan Anaya writes: This response from Barbara Fredrickson to one of Nick’s articles got posted the other day. Alex Holcombe has a screenshot of the article on Twitter. The issue that I have with the response is that she says she stands by the peer review process that led to her article getting published. But […]

“Repeating the experiment” as general advice on data collection

Izzy Kates points to the above excerpt from Introductory Statistics, by Neil Weiss, 9th edition, and points out: Nowhere is repeating the experiment mentioned. This isn’t the only time this mistake is made. Good point! We don’t mention replication as a statistical method in our books either! Even when we talk about the replication crisis, […]

Intended consequences are the worst

I saw this news story by Jesse Drucker and Eric Lipton, headlined, “Meant to Lift Poor Areas, Tax Break is Boon to Rich.” The news article is informative, and the story it tells is horrifying. Beyond all that, I’m bothered by the headline, as it seems that the scamtastic aspect of this tax benefit was […]

The hot hand fallacy fallacy rears its ugly ugly head

Funny how repeating the word “fallacy” reverses the meaning, but repeating the word “ugly” just intensifies it . . . Anyway, Josh Miller points us to this article by what must be the last person on the planet to write uncritically about the so-called “hot hand fallacy.” It’s in the blog of the Society for […]

MRP Carmelo Anthony update . . . Trash-talking’s fine. But you gotta give details, or links, or something!

Before getting to the main post, let me just say that I’m a big fan of Nate Silver. Just for one example: I’m on record as saying that primary elections are hard to predict. So I don’t even try. But there’s lots of information out there: poll data, fundraising numbers, expert opinion, delegate selection rules, […]

The fallacy of the excluded rationality

Malcolm Bull writes: Thanks to the work of behavioural economists there is a lot of experimental evidence to show what many of us would have suspected anyway: that people are not the rational, utility-maximisers of neoclassical economics, but loss-averse sentimentalists who, faced with even the simplest cognitive problem, prefer dodgy short cuts to careful analysis. […]

Forget about multiple testing corrections. Actually, forget about hypothesis testing entirely.

Tai Huang writes: I am reading this paper [Why we (usually) don’t have to worry about multiple comparisons, by Jennifer, Masanao, and myself]. I am searching how to do multiple comparisons correctly under Bayesian inference for A/B/C testing. For the traditional t-test approach, Bonferroni correction is needed to correct alpha value. I am confused with […]

“It’s not just that the emperor has no clothes, it’s more like the emperor has been standing in the public square for fifteen years screaming, I’m naked! I’m naked! Look at me! And the scientific establishment is like, Wow, what a beautiful outfit.”

I happened to come across this post while writing on another topic (the ever-popular “critical positivity ratio”) and I thought the title was so great I just had to post it again. Someone still needs to build that bot for me that reposts all the old posts from this blog, starting at the beginning in […]

Don’t talk about hypotheses as being “either confirmed, partially confirmed, or rejected”

Kevin Lewis points us to this article by Paige Shaffer et al., “Gambling Research and Funding Biases,” which reports, “Gambling industry funded studies were no more likely than studies not funded by the gambling industry to report either confirmed, partially confirmed, or rejected hypotheses.” The paradox is that this particular study was itself funded by […]

The latest Perry Preschool analysis: Noisy data + noisy methods + flexible summarizing = Big claims

Dean Eckles writes: Since I know you’re interested in Heckman’s continued analysis of early childhood interventions, I thought I’d send this along: The intervention is so early, it is in their parents’ childhoods. See the “Perry Preschool Project Outcomes in the Next Generation” press release and the associated working paper. The estimated effects are huge: […]

How to get out of the credulity rut (regression discontinuity edition): Getting beyond whack-a-mole

This one’s buggin me. We’re in a situation now with forking paths in applied-statistics-being-done-by-economists where we were, about ten years ago, in applied-statistics-being-done-by-psychologists. (I was going to use the terms “econometrics” and “psychometrics” here, but that’s not quite right, because I think these mistakes are mostly being made, by applied researchers in economics and psychology, […]

Progress in the past decade

It’s been a busy decade for our research. Before going on, I’d like to thank hundreds of collaborators, including students; funders from government, nonprofits, and private industry; blog commenters and people who have pointed us to inspiring research, outrages, beautiful and ugly graphs, cat pictures, and all the rest; all those of you who have […]

“Why we sleep” data manipulation: A smoking gun?

In his post, Matthew Walker’s “Why We Sleep” Is Riddled with Scientific and Factual Errors” (see our discussions here, here, and here), Alexey Guzey added the following stunner: We’ve left “super-important researcher too busy to respond to picky comments” territory and left “well-intentioned but sloppy researcher can’t keep track of citations” territory and entered “research […]

Horns! Have we reached a new era in skeptical science journalism? I hope so.

Pointing us to this news article from Aylin Woodward, “No, we’re probably not growing horns from our heads because of our cellphone use — here’s the real science,” Jordan Anaya writes: I haven’t looked into it, but seems like your basic terrible study with an attention grabbing headline. Pretty much just mention cell phone use […]

Response to criticisms of Bayesian statistics

I just happened to reread this article of mine from 2008, and I still like it! So I’m linking to it here. Enjoy.

Judith Rich Harris on the garden of forking paths

Ethan Ludwin-Peery writes: I finally got around to reading The Nurture Assumption and I was surprised to find Judith Rich Harris quite lucidly describing the garden of forking paths / p-hacking on pages 17 and 18 of the book. The edition I have is from 2009, so it predates most of the discussion of these […]

“Inferential statistics as descriptive statistics”

Valentin Amrhein​, David Trafimow, and Sander Greenland write: Statistical inference often fails to replicate. One reason is that many results may be selected for drawing inference because some threshold of a statistic like the P-value was crossed, leading to biased reported effect sizes. Nonetheless, considerable non-replication is to be expected even without selective reporting, and […]

“Deep Origins” and spatial correlations

Morgan Kelly writes: Back in 2013 you had a column in Chance magazine on the Ashraf-Galor “Out of Africa” paper which claims that genetic diversity determines modern income. That paper is part of a much large literature in economics on Persistence or “Deep Origins” that shows how medieval pogroms prefigure Nazi support, adoption of the […]

Unquestionable Research Practices

Hi! (This is Dan.) The glorious Josh Loftus from NYU just asked the following question. Obviously he’s not heard of preregistration. Seriously though, it’s always good to remember that a lot of ink being spilled over hypothesis testing and it’s statistical brethren doesn’t mean that if we fix that we’ll fix anything.  It all comes to […]