Kris Hardies points to this new article, Impossible Hypotheses and Effect-Size Limits, by Wijnand and Lennert van Tilburg, which states:
There are mathematical limits to the magnitudes that population effect sizes can take within the common multivariate context in which psychology is situated, and these limits can be far more restrictive than typically assumed. The implication is that some hypothesized or preregistered effect sizes may be impossible. At the same time, these restrictions offer a way of statistically triangulating the plausible range of unknown effect sizes.
This is closely related to our Piranha Principle, which we first formulated here and then followed up with this paper. It’s great to see more work being done in this area.
For whatever reason, I always have a hard time remembering exactly what the piranha principle is. I followed the link to remind myself, and this quote stood out to me:
“The analogy is to a fish tank full of piranhas: it won’t take long before they eat each other.”
I’m no fish expert, but based on the fact that piranhas generally do their cow skeletonizing in large groups, I got to wondering if they would, in fact, eat each other. A quick Google search led me to a guide for keeping pet piranhas that said that they should be kept in groups (in large aquariums, since they get pretty big).
Anyway, I guess I’m just offering the feedback that “the piranha principle” is maybe not the most memorable or accurate name for this phenomenon.
Noah:
Agreed that the piranhas-eating-each-other thing is what we refer to in our article as a “folk belief,” not a real thing. Indeed, last year we went to the zoo and we saw a fish tank full of piranhas. We took a picture to share on the blog but it somehow didn’t get saved.
I find the idea of you going to the zoo with Christopher Tosh, Philip Greengard, Ben Goodrich, Aki Vehtari and Daniel Hsu quite charming but I guess that it might have been a different “we”.
How about “The Colinvaux Principle”, after the author of the classic ecology book “Why Big Fierce Animals Are Rare”? But I guess that has the significant drawback that “Colinvaux” doesn’t mean anything to most people.
Nice. I can keep using the term “burn in,” secure that it’s just based on a folk belief.
Bob:
I’d be up for changing the name of the piranha principle if a better alternative is on offer. Our paper on the topic has been accepted but has not been yet published in the journal. I do think that the piranha image is vivid, and on those grounds I prefer it to Phil’s suggestion above, for example. But I’m open to an alternative.
I used to use the term “burn-in” for iterative algorithms but I switched to “warm-up,” which seems to be to be better in all aspects: it’s more accurate and also less obscure.
I guess we could rename “the piranha principle” to “the cannibal principle”—but maybe that’s a bit too vivid.
Even setting aside the question of accuracy regarding actual piranha behavior, the “piranha principle” name isn’t perfect, as it would seem to imply that the different hypothesized effects are eating each other, which is not the same thing as what we’re really saying, which is that they cannot all coexist. Maybe a better analogy would be something like a cacophony of musical instruments, all playing different tunes.
But I don’t think the name “piranha principle” is terrible. It’s not as good a name as “the no-U-turn algorithm” or “the garden of forking paths” (or, for that matter, “warm-up” or “posterior predictive checking”), but it’s better than the unwieldy “potential scale reduction factor” (which gets reduced to the mysterious “R-hat”) or “jackknife” (more an advertisement than a descriptive name), and I think it’s better than all the algorithms and ideas that are named after people (“Bayesian inference” included).
https://www.youtube.com/watch?v=OFr74zI1LBM
In
https://www.stat.columbia.edu/~gelman/research/published/piranhas.pdf
they write:
“When conducting systematic reviews of evidence, it could be appropriate to consider the potential interactions among various hypothesized causal factors, rather than attempting to combine separate estimates using meta-analysis.”
Don’t meta-analysis usually combine separate estimates of the same causal factor (i.e. treatment) ?
Shira:
There are three big problems with meta-analysis of published studies in social science, even on the same treatment: (1) the treatment isn’t really the same in the different studies, (2) different studies consider different interactions, and (3) selection in what results are reported in a study. See our discussion here for example.
Thanks, Andrew !
That makes sense. I like this part of your discussion:
“instead of focusing on average effects, we need to understand when and where some nudges have huge positive effects and why others are not able to repeat those successes”
So in the Piranhas paper, when you say “rather than attempting to combine separate estimates using meta-analysis” you are referring to a focus on average effects ? A meta-analysis could theoretically try to understand interactions among various factors and the full distribution of effects rather than just the average ?
Shira:
The piranha paper is not really about how to do better data analysis. It’s more of an exploration of a problem with weakly-theorized studies of causal effects. There’s something we’ve seen in various areas of the social sciences, where a group of researchers have a vague theory that can be taken to imply that some general input X can affect some general output Y. For example, the time in a woman’s monthly cycle could affect behavior, or subliminal messages could affect attitudes, or politically-irrelevant but mood-affecting events could affect voting, or the labeling of food could affect dietary choices. Many of these ideas have some general plausibility—for example, lots of money is spent on advertising, and presumably it has some effect. The problem arises when these theories are studied using noisy experiments that show, or attempt to show, that some specific input (call it X_j for study j) affects some specific output Y_j under some specific conditions Z_j. The individual studies have problems, but what the piranha paper is addressing is that the different studies j will have different treatments X_j, different outputs Y_j, and different conditions Z_j. A treatment working under some conditions and not others is a form of interaction effect. Mathematically, it’s difficult to have all these interactions swimming around in the same tank without interfering with each other, and that is the issue that we deal with, in various simplified forms, in the piranha paper.
The piranha paper is a contribution to meta-science to the extent that it helps us understand the failures of this shotgun-style research paradigm in which researchers study lots of (X, Y, Z)_j’s and report large, statistically-significant results.
Thanks, Andrew !
I get that the piranha paper isn’t focused on advising how to do analysis. But the phrase at the end is about analysis and I do not yet understand it: “rather than attempting to combine separate estimates using meta-analysis”.
Is the piranha paper claiming we shouldn’t use information across studies ?
Shira:
Any analysis is what it is, and there’s nothing wrong with meta-analysis, as long as you recognize that it’s just shuffling the numbers that go into it. If the separate study estimates that are going into the meta-analysis are full of selection bias and are all measuring different things, then the resulting meta-analysis is going to give you some kind of average which is more descriptive of the biases and problems with the literature than with the underlying object of study. For discussion of these points in the context of specific examples, see here and here.