Skip to content
Archive of posts filed under the Zombies category.

Unquestionable Research Practices

Hi! (This is Dan.) The glorious Josh Loftus from NYU just asked the following question. Obviously he’s not heard of preregistration. Seriously though, it’s always good to remember that a lot of ink being spilled over hypothesis testing and it’s statistical brethren doesn’t mean that if we fix that we’ll fix anything.  It all comes to […]

What comes after Vixra?

OK, so Arxiv publishes anything. But some things are so cranky that Arxiv won’t publish them, so they go on Vixra. Here’s my question: where do the people publish, who can’t publish on Vixra? The cranks’ cranks, as it were? It’s a Cantor’s corner kinda thing.

In short, adding more animals to your experiment is fine. The problem is in using statistical significance to make decisions about what to conclude from your data.

Denis Jabaudon writes: I was thinking that perhaps you could help me with the following “paradox?” that I often find myself in when discussing with students (I am a basic neuroscientist and my unit of counting is usually cells or animals): When performing a “pilot” study on say 5 animals, and finding an “almost significant” […]

A reduction in error rate of 400-600%: Pretty good, huh?

In comments to the previous post, Alexey Guzey points to this bit from his post on sleep legend Matthew Walker: In The Lancet, Walker writes: pilot studies have shown that when you limit trainee doctors to no more than a 16 h shift, with at least an 8 h rest opportunity before the next shift, […]

“Whether something is statistically significant is itself a very random feature of data, so in this case you’re essentially outsourcing your modeling decision to a random number”

I happened to come across a post of mine that’s not scheduled until next April, and I noticed the above line, which I really liked, so I’m sharing it with you right now here. The comment relates to a common procedure in statistics, where researchers decide exclude potentially important interactions from their models, just because […]

Break out the marshmallows, friends: Ego depletion is due to change sign!

In a paper amusingly titled, “Ego depletion may disappear by 2020,” Miguel Vadillo (link from Kevin Lewis) writes: Ego depletion has been successfully replicated in hundreds of studies. Yet the most recent large-scale Registered Replication Reports (RRR), comprising thousands of participants, have yielded disappointingly small effects, sometimes even failing to reach statistical significance. Although these […]

In research as in negotiation: Be willing to walk away, don’t paint yourself into a corner, leave no hostages to fortune

There’s a saying in negotiation that the most powerful asset is the ability to walk away from the deal. Similarly, in science (or engineering, business decision making, etc.), you have to be willing to give up your favorite ideas. When I look at various embarrassing examples in science during the past decade, a common thread […]

Why do a within-person rather than a between-person experiment?

Zach Horne writes: A student of mine was presenting at the annual meeting of the Law and Society Association. She sent me this note after she gave her talk: I presented some research at LSA which used a within subject design. I got attacked during the Q&A session for using a within subjects design and […]

Zombie semantics spread in the hope of keeping most on the same low road you are comfortable with now: Delaying the hardship of learning better methodology.

Now, everything is connected, but this is not primarily about persistent research misconceptions such as statistical significance. Instead it is about (inherently) interpretable ML versus (misleading with some nonzero frequency) explanatory ML that I previously blogged on just over a year ago. That was when I first become aware of work by Cynthia Rudin (Duke) […]

Should we mind if authorship is falsified?

In a typically thought-provoking piece, Louis Menand asks, “Should we mind if a book is a hoax?” In his article, Menand (whose father taught the best course I ever took at MIT, in which we learned that eternal vigilance is the price of liberty) focuses on imaginative literature written by white people but attributed to […]

Instead of replicating studies with problems, let’s replicate the good studies. (Consider replication as an honor, not an attack.)

Commenter Thanatos Savehn pointed to an official National Academy of Sciences report on Reproducibility and Replicability that included the following “set of criteria to help determine when testing replicability may be warranted”: 1) The scientific results are important for individual decision-making or for policy decisions. 2) The results have the potential to make a large […]

Battle for the headline: Hype and the effect of statistical significance on the ability of journalists to engage in critical thinking

A few people pointed me to this article, “Battle for the thermostat: Gender and the effect of temperature on cognitive performance,” which received some uncritical press coverage here and here. And, of course, on NPR. “543 students in Berlin, Germany” . . . good enuf to make general statements about men and women, I guess! […]

Epic Pubpeer thread continues

Here. (background here and here)

The incentives are all wrong (causal inference edition)

I was talking with some people the other day about bad regression discontinuity analyses (see this paper for some statistical background on the problems with these inferences), examples where the fitted model just makes no sense. The people talking with me asked the question: OK, we agree that the published analysis was no good. What […]

“The paper has been blind peer-reviewed and published in a highly reputable journal, which is the gold standard in scientific corroboration. Thus, all protocol was followed to the letter and the work is officially supported.”

Robert MacDonald points us to this news article by Esther Addley: It’s another example of what’s probably bad science being published in a major journal, where other researchers point out its major flaws and the author doubles down. In this case, the University of Bristol has an interesting reaction. It’s pulled down its article praising […]

The climate economics echo chamber: Gremlins and the people (including a Nobel prize winner) who support them

Jay Coggins, a professor of applied economics at the university of Minnesota, writes in with some thoughts about serious problems of within the field of environmental economics: Your latest on Tol [a discussion of a really bad paper he published in The Review of Environmental Economics and Policy, “the official journal of the Association of […]

The 5,000 Retractions of Dr. E

Rigor, of course, but put a lid on the aggression & call off the social media hate mobs.

Kool-aid != Dogfood, and Certainty is no substitute for knowledge.

Palko quotes from this news article by Shirin Ghaffary quoting Marcelo Claure, the new executive chairman of the recent Ponzi scheme WeWork. Here’s Claure, in a speech to WeWork employees: [W]e got to drink our own Kool-aid, we got to make sure that if we’re selling this magic to others, we got to have this […]

“Any research object with a strong and obvious series of inconsistencies may be deemed too inaccurate to trust, irrespective of their source. In other words, the description of inconsistency makes no presumption about the source of that inconsistency.”

Nick Brown and James Heathers write: We have seen two documents from the Scientific Integrity Officer at the University of Rennes-2 . . . The first of these dates from June 2018 and is entitled (our translation from French), “Preliminary Investigation Report Regarding the Allegations of Fraud against Nicolas Guéguen”. . . . We would […]

Social science plaig update

OK, we got two items for you, one in political science and one in history. Both are updates on cases we’ve discussed in the past on this blog. I have no personal connection to any of the people involved; my only interest is annoyance at the ways in which plagiarism pollutes scientific understanding and the […]