Skip to content
Search results for multiple comparisons

Of multiple comparisons and multilevel models

Kleber Neves writes: I’ve been a long-time reader of your blog, eventually becoming more involved with the “replication crisis” and such (currently, I work with the Brazilian Reproducibility Initiative). Anyway, as I’m now going deeper into statistics, I feel like I still lack some foundational intuitions (I was trained as a half computer scientist/half experimental […]

Using multilevel modeling to improve analysis of multiple comparisons

Justin Chumbley writes: I have mused on drafting a simple paper inspired by your paper “Why we (usually) don’t have to worry about multiple comparisons”. The initial idea is simply to revisit frequentist “weak FWER” or “omnibus tests” (which assume the null everywhere), connecting it to a Bayesian perspective. To do this, I focus on […]

Multilevel models for multiple comparisons! Varying treatment effects!

Mark White writes: I have a question regarding using multilevel models for multiple comparisons, per your 2012 paper and many blog posts. I am in a situation where I do randomized experiments, and I have a lot of additional demographic information about people, as well. For the moment, let us just assume that all of […]

Analyze all your comparisons. That’s better than looking at the max difference and trying to do a multiple comparisons correction.

[cat picture] The following email came in: I’m in a PhD program (poli sci) with a heavy emphasis on methods. One thing that my statistics courses emphasize, but that doesn’t get much attention in my poli sci courses, is the problem of simultaneous inferences. This strikes me as a problem. I am a bit unclear […]

Bayesian inference completely solves the multiple comparisons problem

I promised I wouldn’t do any new blogging until January but I’m here at this conference and someone asked me a question about the above slide from my talk. The point of the story in that slide is that flat priors consistently give bad inferences. Or, to put it another way, the routine use of […]

In one of life’s horrible ironies, I wrote a paper “Why we (usually) don’t have to worry about multiple comparisons” but now I spend lots of time worrying about multiple comparisons

Exhibit A: [2012] Why we (usually) don’t have to worry about multiple comparisons. Journal of Research on Educational Effectiveness 5, 189-211. (Andrew Gelman, Jennifer Hill, and Masanao Yajima) Exhibit B: The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis […]

Correcting for multiple comparisons in a Bayesian regression model

Joe Northrup writes: I have a question about correcting for multiple comparisons in a Bayesian regression model. I believe I understand the argument in your 2012 paper in Journal of Research on Educational Effectiveness that when you have a hierarchical model there is shrinkage of estimates towards the group-level mean and thus there is no […]

Multiple comparisons dispute in the tabloids

Yarden Katz writes: I’m probably not the first to point this out, but just in case, you might be interested in this article by T. Florian Jaeger, Daniel Pontillo, and Peter Graff on a statistical dispute [regarding the claim, “Phonemic Diversity Supports a Serial Founder Effect Model of Language Expansion from Africa”]. Seems directly relevant […]

That xkcd cartoon on multiple comparisons that all of you were sending me a couple months ago

John Transue sent it in with the following thoughtful comment: I’d imagine you’ve already received this, but just in case, here’s a cartoon you’d like. At first blush it seems to go against your advice (more nuanced than what I’m about to say by quoting the paper title) to not worry about multiple comparisons. However, […]

Data exploration and multiple comparisons

Bill Harris writes: I’ve read your paper and presentation showing why you don’t usually worry about multiple comparisons. I see how that applies when you are comparing results across multiple settings (states, etc.). Does the same principle hold when you are exploring data to find interesting relationships? For example, you have some data, and you’re […]

Sort of multiple comparisons problem

Nick Allum writes:

Multiple comparisons: my talk at the London School of Economics on Monday

Monday 2 Nov, 5-6:30pm at the Methodology Institute, LSE. No link to the seminar on the webpage, so I’ll give you the information here: Why we (usually) don’t worry about multiple comparisons Applied researchers often find themselves making statistical inferences in settings that would seem to require multiple comparisons adjustments. We challenge the Type I […]

Multiple Comparisons in Linear Regression

Hamdan Yousuf writes: I was reading your Kanazawa letter to the editor and I was interested in your discussion of multiple comparisons. This might be an elementary issue but I don’t quite understand when the issue of multiple comparisons arises, in general. To give an example from research I have been involved in, assume I […]

Why we don’t (usually) worry about multiple comparisons

Here’s the paper (with Jennifer and Masanao), and here’s the abstract: The problem of multiple comparisons can disappear when viewed from a Bayesian perspective. We propose building multilevel models in the settings where multiple comparisons arise. These address the multiple comparisons problem and also yield more efficient estimates, especially in settings with low group-level variation, […]

More on multiple comparisons and inference for small effects

I told Phil Stark about this paper on Type S error rates and this paper on statistical challenges in estimating small effects, and he replied with these references on Type S errors and multiple comparisons: Benjamini, Y. and Stark, P.B., 1996. Non-equivariant simultaneous confidence intervals less likely to contain zero, J. Am. Stat. Assoc., 91, […]

Why I don’t (usually) care about multiple comparisons

Statisticians often get worried about multiple comparisons and have various procedures for adjusting p-values. It’s only very rarely come up in my own research, though. Here’s a presentation explaining why, from a workshop of educational researchers to which I was invited by Rob Hollister, an economist and policy analyst at Swarthmore College. The punch line […]

Bayesian inference and multiple comparisons

Gregor sent another question:

Forget about multiple testing corrections. Actually, forget about hypothesis testing entirely.

Tai Huang writes: I am reading this paper [Why we (usually) don’t have to worry about multiple comparisons, by Jennifer, Masanao, and myself]. I am searching how to do multiple comparisons correctly under Bayesian inference for A/B/C testing. For the traditional t-test approach, Bonferroni correction is needed to correct alpha value. I am confused with […]

Causal inference and within/between person comparisons

There’s a meta-principle of mathematics that goes as follows. Any system of logic can be written in various different ways that are mathematically equivalent but can have different real-world implications, for two reasons: first, because different formulations can be more directly applied in different settings or are just more understandable by different people; second, because […]

Data concerns when interpreting comparisons of gender equality between countries

A journalist pointed me to this research article, “Gender equality and sex differences in personality: evidence from a large, multi-national sample,” by Tim Kaiser (see also news report by Angela Lashbrook here), which states: A large, multinational (N = 926,383) dataset was used to examine sex differences in Big Five facet scores for 70 countries. […]