José Iparraguirre writes:
Just finished “Understanding Statistics and Experimental Design. How to Not Lie with Statistics”, by Michael Herzog, Gregory Francis, and Aaron Clarke (Springer, 2019). Near the end (p. 128), I read the following regarding “optional stopping”:
…suppose a scientist notes a marginal (p = 0.07) result in Experiment 1 and decides to run a new Experiment 2 to check on the effect. It may sound like the scientist is doing careful work, however, this is not necessarily true. Suppose Experiment 1 produced a significant effect (p = 0.03), would the scientist still have run Experiment 2 as a second check? If not, then the scientist is essentially performing optional stopping across experiments, and the Type I error rate for any given experiment (or across experiments) is unknown.
Indeed, the problem with optional stopping is not the actual behavior preformed by the scientist (e.g., the study with a planned sample size gives p = 0.02) but with what he would have done if the result turned out differently (e.g., if the study with a planned sample size gives p = 0.1, he would have added 20 more subjects). More precisely, if you do not know what you would have done under all possible scenarios, then you cannot know the Type I error rate for your analysis.
It is the last statement, “if you do not know what you would have done under all possible scenarios, then you cannot know the Type I error rate for your analysis”, that kept me wondering and prompted me to write to you and ask you for your comments.
My reply:
1. Yes, they’re correct that if you do not know what you would have done under all possible scenarios, then you cannot know the Type I error rate for your analysis. We make this point in section 1.2 of our Garden of Forking Paths paper and this is part of the definition of these error rates; the statement should not be controversial.
2. I’m pretty much not interested in type 1 or type 2 error rates; mostly the only reason I think it’s worth thinking about them is to respond to confused researchers who think that a low p-value represents strong evidence for their preferred theories.
3. I think that stopping the experiment based on the data is just fine in practice and is not cheating, as some people might think. See my discussion here.
I also sent to Greg Francis, who wrote:
I too don’t think the statement is controversial, even though it is surprising to many practicing scientists. In part I think that is because textbooks on hypothesis testing give examples with very specific settings (always with a fixed sample size). Even though (usually) the textbooks properly describe theorems (e.g., for defining a sampling distribution), they ignore some realities of data collection (a fixed sample size is not the norm) and so do not consider what happens when you deviate from the theorems.
Contrary to Andrew, I think a lot of scientists genuinely do care about Type I and Type II error rates. However, getting control of those error rates is more difficult than many people realize, and I think that difficulty is good motivation to consider Bayesian approaches.
For a bit more discussion about error rates across experiments, you might look at this paper on a “reverse Bonferroni” method. I’m not sure I would actually recommend the method for any practical situation, but it highlights what to consider when you try to control error rates across experiments.
If only people were taught Bayesian statistics, they would stop caring about Type I and Type II errors. I recommend
Inference for a Bernoulli Process (A Bayesian View)
by D. V. Lindley and L. D. Phillips
The American Statistician, Vol. 30, No. 3 (Aug., 1976), pp. 112-119
https://www.jstor.org/stable/2683855
and the book “The Likelihood Principle” by James O. Berger and Robert L. Wolpert.
David:
Sure, but just remember that it not necessary that Bayesian methods conform to the likelihood principle.
I wasn’t suggesting that we replace teaching hypothesis tests with teaching the likelihood principle. I was suggesting that people who have been taught the usual school (frequentist) statistics read the article and book.
Change
“I’m pretty much no interested in type 1 or type 2 error rates;”
to
“I’m pretty much not interested in type 1 or type 2 error rates;”
As to “optional stopping,” what sporting event allows that? Does there exist a sporting event which keeps on going only until my side is eventually triumphant? In other words, my side never loses because the game, by definition, may never end.
Typos fixed; thanks.
You changed the wrong typo… it was better as ‘Scottish’… “Ah’m no interested in type 1 or type 2 error rates”. Credentials: Ah’m Scoattish!
Pure dead brilliant. But ‘sgotty be “Type Wan” ‘n’ “Type Twa”, dya no hink?
It depends which side you’re on. In college softball there is the rule limit – a lead of 8 runs ends the game after 5 innings. In golf, there is “stroke control” which limits your score on a hole to 10 strokes. So, these are forms of stopping rules that limit how bad things can get (or, alternatively, they limit the possibility of a comeback in the case of softball: unfortunately, there is no way to lower a golf score on a hole by taking additional strokes).
Poker is a perpetual game of optional stopping by both sides, so long as somebody will advance you credit.
Fergie Time. Refers to a perception in English football that referees allowed a game to extend longer if a particular team needed to score.
Check out this paper about the pioneer anomaly: https://arxiv.org/abs/1204.2507
The scientists bend over backwards to find ways to not see a significant effect, because the null hypothesis corresponds to their hypothesis. This is exactly opposite the NHST incentive.
Most here have read this paper already, but Paul Meehl called it a paradox:
https://meehl.umn.edu/sites/meehl.umn.edu/files/files/074theorytestingparadox.pdf
How can there be something we call “science”, but the exact opposite procedures are also called science? It is like bizarro-science.
So getting control of Type 1 error just requires pre-specifying all possible experiments/studies you might want to run over the course of your career? Sounds easy!
> the scientist is essentially performing optional stopping across experiments, and the Type I error rate for any given experiment (or across experiments) is unknown
I don’t get it. How do “across experiments” decisions affect the type I error rate for any given experiment?
Or does the “(or across experiments)” remark indicate that “for any given experiment” doesn’t mean each experiment taken independently?
If I throw a die the probability of getting a 6 is 1/6. If I can throw again, and stop when I want, I will eventually get a 6. But the probablility of getting a 6 for any given throw is 1/6 even if I perform optional stopping across throws.
But if your hypothesis is that the probability of getting a 6 is at least 1/5*, and you choose when to stop, most experiments will confirm your hypothesis.
*or some such number larger than but sufficiently similar to 1/6.
> suppose a scientist notes a marginal (p = 0.07) result in Experiment 1 and decides to run a new Experiment 2 to check on the effect
In this case each throw would be a different experiment. It would not correct to say in this case that “most experiments will confirm your hypothesis.”
Or maybe when they say “the Type I error rate for any given experiment is unknown” they don’t mean “the Type I error rate for Experiment 1 is unknown” and “the Type I error rate for Experiment 1 is unknown” .
Carlos, you seem to be assuming the experiments are independent. They are not because whether the second experiment happens depends on the outcome of the first experiment.
It’s like deciding to flip a coin until it comes up heads and then for the first flip that does come up heads saying, this is a fair coin so this was a fifty-fifty result.
I’m not assuming anything, I’m trying to understand what they wrote: “Suppose Experiment 1 produced a significant effect (p = 0.03), would the scientist still have run Experiment 2 as a second check? If not, then the scientist is essentially performing optional stopping across experiments, and the Type I error rate for any given experiment (or across experiments) is unknown.”
If that’s not wrong at least it seems very ambiguous because after discussing “Experiment 1” and “Experiment 2” it’s far from obvious that the claim “for any given experiment …” must not be interpreted as “for Experiment 1 …” and “for Experiment 2 …”.
This is the problem with frequentist analysis: You have to decide what the plan was rather than just look at what data you have. Were they always planning to do a second experiment to confirm their results? (This reminds me of the recent intent to treat discussion.)
For a clear example of how ridiculous it is to let textbook statistics box you into this corner, see the article ” Inference for a Bernoulli Process (A Bayesian View)” by D. V. Lindley and L. D. Phillips, The American Statistician, Vol. 30, No. 3 (Aug., 1976), pp. 112-119, https://www.jstor.org/stable/2683855
And for people who think you can ignore priors, consider the example of the lady tasting tea.
The conundrum presented by the scenario just shows how badly the Neyman–Pearsonian hypothesis test approach marries with scientific approaches to gaining knowledge. The accept/reject dichotomy is necessary to the accounting required for type I and type II error calculations, but is inimical to efficient learning from experimental results.
If you get a P-value from one experiment that is neither small enough nor large enough to convince you to discontinue a line of investigation, then you SHOULD continue in one way or another. A fresh dataset cannot not influence the evidential meaning of the original dataset and so a fresh experiment analysed independently of the original dataset would constitute good science whether the accountants of error like it or not.
Type I and type II error rates are related to the properties of the analysis, not to the evidence in the data. You can read a longer account here: https://link.springer.com/chapter/10.1007/164_2019_286
+1, and thanks for the reference.
Actually, your figure shows a four-way relationship. Alpha (significance threshold) is also important.
What happens in practice is that sample size is a function of how expensive it is to run the experiment. Eg, you will rarely see studies of more than 10 primates per group but this is common for rodents. For data like microarrays or particle collisions it can run into the hundreds, thousands, or more.
Then alpha is chosen so that there are enough “discoveries”, but not so many that it seems too easy. This corresponds to a power of 25-50%, so you need to at least try out a few different things before getting something to publish.
Ie, alpha is a function of the typical effect size for that type of data, along with the achievable sample size. It caps out at 0.1, and below that gets rounded to some easily remembered number. Obviously 0.05 is a very popular one, most likely what gets considered “too expensive” by funding agencies is also a function of that standard choice.
So… what about a replication from another lab done because the initial study was statistically significant. What’s the Type I error rate for that? What about the many labs studies replicating an experiment many times?
I deny that correctly reporting the (approximate error rate–and it’s always approximate–requires you to “know what you would have done under all possible scenarios”. This makes it appear all but impossible to satisfy, and it’s not the case. It suffices, for example, to predesignate the sample size or stopping rule to be followed. It is always acknowledged that if things change, and there are grounds to violate the plan, then this will be reported. In short, it suffices that you would not treat it the same way as if the predesignated stopping plan had been followed.
Rule 1: collect 100 data points, if p less than 0.05 under the null of mean 0 then report the finding, if p less than 0.1 collect 100 more data points if p less than 0.05 report finding otherwise give up.
Rule 2: collect 100 data point, if p less than 0.05 … report the finding, otherwise collect another 100 repeat until p less than 0.05 or 1000 data points collected, report finding at end.
two separate labs have these two separate protocols. They both collect 100 data points from a process with mean 0, they both get the same initial dataset, they both find p = 0.041 against the mu = 0 hypothesis and report their findings… Neither lab reported their protocol. What is the probability of error that the reader should calculate each has? (here I’m assuming “error” means they report that mu doesn’t equal 0 when in fact it does).