The p-value is one of the most common, and one of the most confusing, tools in applied statistics. Seasoned educators are well aware of all the things the p-value is not. Most notably, it’s not “the probability that the null hypothesis is true.” McShane and Gal find that even top researchers routinely misinterpret p-values.
But let’s forget for a moment about what p-values are not and instead ask what they are. It turns out that there are different meanings of the term. At first I was going to say that these are different “definitions,” but Sander Greenland pointed out that not all are definitions:
Definition 1. p-value(y) = Pr(T(y_rep) >= T(y) | H), where H is a “hypothesis,” a generative probability model, y is the observed data, y_rep are future data under the model, and T is a “test statistic,” some pre-specified specified function of data. I find it clearest to define this sort of p-value relative to potential future data; it can also be done mathematically and conceptually without any reference to repeated or future sampling, as in this 2019 paper by Vos and Holbert.
Definition 2. Start with a set of hypothesis tests of level alpha, for all values alpha between 0 and 1. p-value(y) is the smallest alpha of all the tests that reject y. This definition starts with a family of hypothesis tests rather than a test statistic, and it does not necessarily have a Bayesian interpretation, although in particular cases, it can also satisfy Definition 1.
Property 3. p-value(y) is some function of y that is uniformly distributed under H. I’m not saying that the term “p-value” is taken as a synonym for “uniform variate” but rather that this conditional uniform distribution is sometimes taken to be a required property of a p-value. It’s not a definition because in practice no one would define a p-value without some reference to a tail-area probability (Definition 1) or a rejection region (Definition 2)—but it is sometimes taken as a property that is required for something to be a true p-value. The relevant point here is that a p-value can satisfy Property 3 without satisfying Definition 1 (there are methods of constructing uniformly-distributed p-values that are not themselves tail-area probabilities), and a p-value can satisfy Definition 1 without satisfying Property 3 (when there is a composite null hypothesis and the distribution of the test statistic is not invariant to parameter values; see Xiao-Li Meng’s paper from 1994).
Description 4. p-value(y) is the result of some calculations applied to data that are conventionally labeled as a p-value. Typically, this will be a p-value under Definition 1 or 2 above, but perhaps defined under a hypothesis H that is not actually the model being fit to the data at hand, or a hypothesis H(y) that itself is a function of data, for example from p-hacking or forking paths. I’m labeling this as a “description” rather than a “definition” to clarify that this sort of p-value is used all the time without always a clear definition of the hypothesis, for example if you have a regression coefficient with estimate beta_hat and standard error s, and you compute 2 times the tail-area probability of |beta_hat|/s under the normal or t distribution, without ever defining a null hypothesis relative to all the parameters in your model. Sander Greenland calls this sort of thing a “descriptive” p-value, capturing the idea that the p-value can be understood as a summary of the discrepancy or divergence of the data from H according to some measure, ranging from 0 = completely incompatible to 1 = completely compatible. For example, the p-value from a linear regression z-score can be understood as a data summary without reference to a full model for all the coefficients.
These are not four definitions/properties/descriptions of the same thing. They are four different things. Not completely different, as they coincide in certain simple examples, but different, and they serve different purposes. They have different practical uses and implications, and you can make mistakes when you use one sort to answer a different question. Just as, for example, posterior intervals and confidence intervals coincide in some simple examples but in general are different: lots of real-world posterior intervals don’t have classical confidence coverage, even in theory, and lots of real-world confidence intervals don’t have Bayesian posterior coverage, even in theory.
A single term with many meanings—that’s a recipe for confusion! Hence this post, which does not claim to solve any technical problems but is just an attempt to clarify.
In all the meanings above, H is a “generative probability model,” that is, a class of probability models for the modeled data, y. If H is a simple null hypothesis, H represents a specified probability distribution, p(y|H). If H is a composite null hypothesis, there is some vector of unknown parameters theta indexing a family of probability distributions, p(y|theta,H). As Daniel Lakeland so evocatively put it, a null hypothesis is a specific random number generator.
Under any of the above meanings, the p-value is a number—a function of data, y—and also can be considered as a random variable with probability distribution induced by the distribution of y under H. For a composite null hypothesis, that distribution will in general depend on theta, but that complexity is not our focus here.
So, back to p-values. How can one term have four meanings? Pretty weird, huh?
The answer is that under certain ideal conditions, the four meanings coincide. In a model with continuous data and a continuous test statistic and a point null hypothesis, all four of the above meanings give the same answer. Also there are some models with unknown parameters where the test statistic can be defined to have a distribution under H that is invariant to parameters. And this can also be the case asymptotically.
More generally, though, the four meanings are different. None of them are “better” or “worse”; they’re just different. Each has some practical value:
– A p-value under Definition 1 can be directly interpreted as a probability statement about future data conditional on the null hypothesis (as discussed here).
– A p-value under Definition 2 can be viewed as a summary of a class of well-defined hypothesis tests (as discussed in footnote 4 of this article by Philip Stark).
– A p-value with Property 3 has a known distribution under the null hypothesis, so the distribution of a collection of p-values can be compared to uniform (as discussed here).
– A p-value from Description 4 is unambiguously defined from existing formulas so is a clear data summary even if it can’t easily be interpreted as a probability in the context of the problem at hand.
As an example, in this article from 1989, Besag and Clifford come up with a Monte Carlo procedure that yields p-values that satisfy Property 3 but not Definition 1 or Description 4. And in 1996, Meng, Stern, and I discussed Bayesian p-values that satisfied Definition 1 but not Property 3.
The natural way to proceed is to give different names to the different p-values. The trouble is that different authors choose different naming conventions!
I’ve used the term “p-value” for Definition 1 and “u-value” for Property 3; see section 2.3 of this article from 2003. And in this article from 2014 we attempted to untangle the difference between Definition 1 and Property 3. I haven’t thought much about Definition 2, and I’ve used the term “nominal p-value” for Description 4.
My writing about p-values has taken Definition 1 as a starting point. My goal has not been to examine misfit of the null hypothesis with respect to some data summary or test statistic, not to design a procedure to reject with a fixed probability conditional on a null hypothesis or to construct a measure of evidence that is uniformly distributed under the null. Others including Bernardo, Bayarri, and Robins are less interested in a particular test statistic and are more interested in creating a testing procedure or a calibrated measure of evidence, and they have taken Definition 2 or Property 3 as their baseline, referring to p-values with Property 3 as “calibrated” or “valid” p-values. This terminology is as valid as mine; it’s just taking a different perspective on the same problem.
In an article from 2023 with follow-up here, Sander Greenland distinguishes between “divergence p-values” and “decision p-values,” addressing similar issues of overloading of the term “p-value.” The former corresponds to Definition 1 above using the same sort of non-repeated-sampling view of p-values as favored by Vos and Holbert and addresses the issues raised by Description 4, and the latter corresponds to Definition 2 and addresses the issues raised by Property 3. As Greenland emphasizes, a p-value doesn’t exist in a vacuum; it should be understood in the context in which it will be used.
My thinking has changed.
My own thinking about p-values and significance testing has changed over the years. It all started when I was working on my Ph.D. thesis, fitting a big model to medical imaging data and finding that the model didn’t fit the data. I could see this because the chi-squared statistic was too large! We had something like 20,000 pieces of count data, and the statistic was about 30,000, which would be compared to a chi-squared distribution with 20,000 minus k degrees of freedom, where k is the the number of “effective degrees of freedom” in the model. The “effective degrees of freedom” thing was interesting, and it led me into the research project that culminated in the 1996 paper with Meng and Stern.
The relevant point here was that I was not coming into that project with the goal of creating “Bayesian p-values.” Rather, I wanted to be able to check the fit of model to data, and this was a way for me to deal with the fact that existing degrees-of-freedom adjustments did not work in my problem.
The other thing I learned when working on thaT project was that a lot of Bayesians didn’t like the idea of model checking at all! They had this bizarre (to me) attitude that, because their models were “subjective,” they didn’t need to be checked. So I leaned hard into the idea that model checking is a core part of Bayesian data analysis. This one example, and the fallout from it, gave me a much clearer sense of data analysis as a Popperian or Lakatosian process, leading to this 2013 article with Shalizi.
In the meantime, though, I started to lose interest in p-values. Model checking was and remains important to me, but I found myself doing it using graphs. Actually, the only examples I can think of where I used hypothesis testing for data analysis were the aforementioned tomography model from the late 1980s (where the null hypothesis was strongly rejected) and the 55,000 residents desperately need your help! example from 2004 (where we learned from a non-rejection of the null). Over the years, I remained aware of issues regarding p-values, and I wrote some articles on the topic, but this was more from theoretical interest or with the goal of better understanding common practice, not with any goal to develop better methods for my own use. This discussion from 2013 of a paper by Greenland and Poole gives a sense of my general thinking.
P.S. Remember, the problems with p-values are not just with p-values.
P.P.S. I thank Sander Greenland for his help with this, even if he does not agree with everything written here.
P.P.P.S. Sander also reminds me that all the above disagreements are trivial compared to the big issues of people acting as if “not statistically significant” results are confirmations of the null hypothesis and as if “statistically significant” results are confirmations of their preferred alternative. Agreed. Those are the top two big problems related to model checking and hypothesis testing, and then I’d say the third most important problem is people not being willing to check their models or consider what might happen if their assumptions are wrong (both Greenland and Stark have written a lot about that problem, and, as noted above, that was one of my main motivations to doing research on the topic of hypothesis testing).
Compared to those three big issues, the different meanings of p-values are less of a big deal. But they do come up, as all four of the above sorts of p-values are used in serious statistical practice, so I think it’s good to be aware of their differences. Otherwise it’s easy to slip into the attitude that other methods are “wrong,” when they’re just different.
I’ll just throw it out there that there’s a difference between Bayesian Independent and Identically Distributed Errors, and Frequentist IID errors.
BIID means we give the same credibility to each error value. FIID means we model the physics of the world as if the past doesn’t affect the future.
One of these is strongly incorrect, and as a hint it’s the FIID one.
One can easily find situations where we need to reject a frequentist IID model because the physics is wrong and therefore the frequencies are wrong.
For example measure the flight of a falling ball using round off error. The ball will be a little high compared to the nearest integer value for several time points, and then be a little low for several time points and then be a little high compared to the next lower integer roundoff… Etc there’s nothing independent about these errors. Using a test of independence you will easily reject the hypothesis.
Using a Bayesian model with BIID errors you instead are saying that you do not USE the information from the previous error measurement to describe the information about the next error. Since the Bayesian model is about our knowledge and the description of it, rejecting a p value here tells you only that there is information you could have used, not that your description of the physics is wrong.
P values mean different things in these two cases because the models are of different things.
“Sander also reminds me that all the above disagreements are trivial compared to the big issues of people acting as if “not statistically significant” results are confirmations of the null hypothesis and as if “statistically significant” results are confirmations of their preferred alternative.”
Naive comment, but isn’t this the Neyman-Pearson instruction for working under definition 2? That is, *act* as if your theory is true if passes the “test”? I totally accept the Greenland descriptive approach to P-values, but, if I understand correctly, the above logic isn’t necessarily wrong if someone has declared as NP and works entirely and strictly within that framework. Would that be fair to say?
Correct. The way I’d word the problem of my concern is the persistent reporting of p>0.05 or a 95% CI containing the null as “no difference between groups” or as supporting “no difference” (a nullistic fallacy). That’s a problem I see constantly in med journals, yet ignored by “replication crisis” discussions (which grew out of the psych and social-science literature, where reporting of false discoveries is apparently the dominant problem).
As you note, in his purest moments Neyman said his theory was only about how to behave, not about what to believe or claim; he also would emphasize that alpha should be chosen based on error costs, which I never see done in the med literature. I regard his approach as unrealistic for most medical research because it forces each study to pretend it is the sole decision node in treatment evaluation, which is patently absurd: There is far more information that goes into treatment approval, warning and withdrawal decisions than that in a P-value or CI, including informal considerations of multiple endpoints, seriousness of outcomes, other studies, availability of other treatments, and so on.
100% right there with you. It makes no sense to use p values to drive decisions, it completely ignores “utility” or “importance” tradeoffs.
My wife asked me to give a lecture on stats to her bio students once and I set up a “gotcha” on this… We came up with good (simulated) evidence that in a bioreactor drug mfg problem treating with treatment B produced more yield of some drug per batch than treatment A, and I asked the students which one we should choose to use? After they all said B I moved to the next slide in which it was revealed that treatment A was a generic growth hormone that cost $0.10 per batch and treatment B was a patented drug with a sole source provider which cost $10,000 per batch. The difference in yield was something like 30% which was highly statistically significant but still very small compared to a factor of 100k cost difference.
I think it drove the point home really well.
There is a reason that the $10,000 drug was approved, most likely a niche patient population. The production process is not relevant to them.
You’ve missed the point entirely. The example is made up. The goal is to produce something in a bioreactor. You can run the reactor without any additive, you can add additive A which is a cheap growth factor that speeds up the production, or you can add growth factor B which is an expensive fancy drug that is a growth factor as well. The students don’t have the price information, so they look at fancy drug B results increasing the yield 30% or whatever and it’s a tiny p value and so they immediately assume they should use that. The gotcha is that without information on how much it costs you can’t make the decision. Yet people make decisions like that all the time.
Many thanks for the explanation Sander
I don’t think this is the Neyman-Pearson “instruction”. I rather think that for Neyman-Pearson this prescription holds if you have a decision problem and your alternative actions are to either act as if the theory is true or not. This means you first have to ask “when should I act like this”, and the NP say, *if* you want a decision rule that fulfills the NP conditions, *then* this is the prescription. So NP address a request, they don’t tell you that this is what you should ask.
Fair enough, although my query seems to me valid and clear regardless of whether I used the word instruction or prescription
True, that wasn’t my point. It was that the instruction/prescription is not general but for situations where something of this kind is required and requested by the “user”.
What I often see is that researchers think that a significant p-value is confirmation not necessarily of their “preferred alternative”, but rather of the particular point estimate that they happened to obtain in their analysis. E.g. if the point estimate of the treatment effect in their RCT is X and the p-value is less than 0.05 then that can be taken as definitive evidence that the treatment had an effect of X.
> I find it clearest to define this sort of p-value relative to potential future data; it can also be done mathematically and conceptually without any reference to repeated or future sampling
“A single simple random sample of n individuals from a population creates a statistical ensemble where the possible states consist exactly of the possible samples of size n from the population. The conceptualization of a statistical ensemble differs from repeated sampling in that a large number is considered all at once and this idea avoids several pitfalls associated with repeated sampling.”
Potential alternative data doesn’t seem very different from potential future data – and if we are going to focus on the differences I don’t see why the latter would be clearer.
Actually I’m not sure if I understand your first definition
p-value(y) = Pr(T(y_rep) >= T(y) | H)
Does it also apply if y_rep are past data under the model instead of future data?
You write that H is “a generative probability model” but you also write later that “a p-value can satisfy Definition 1 without satisfying Property 3 (when there is a composite null hypothesis ….”.
If composite means that the distribution of the parameters is not completely specified by H is that still a generative probability model?
You refer to a paper about “posterior predictive p-values”. I’m not sure if your definition above would be more precisely written as in that paper as
p-value(y) = Pr(T(y_rep) >= T(y) | y, H_0)
Some of the things you wrote make me suspect that your H is not just H_0 but has a dependency on the observed data y. But you also write later that “a hypothesis H(y) that itself is a function of data” would fall under 4 rather than 1.
I would say that y_rep must apply equally to past and future datasets, and that fact makes Andrew’s definition 1 a little misleading. A better way to talk about the y_rep would be to acknowledge that the statistical model brings with it a distribution of possible y_reps rather than thinking about future or past events. The model does not need any arrow of time and so avoiding time-loaded terms is good.
If we start with the model entailing a distribution of y_rep test statistics then we can define the p-value as the fractional rank of the observed test statistic value among that distribution. That leads directly to Andrew’s definition 1.
Carlos, Michael:
For further explanation, see the diagrams on page 739 of our paper from 1996 on posterior predictive checking.
I identify definition 1 as “Fisherian”, although Fisher called those P-values “significance levels” (apparently derived from usage of “statistically significant” that began with Edgeworth and Venn in the 1880s). This tail-area P-value can be traced back to the 18th century and was labeled “value of P” by K. Pearson in 1900.
As far as I know, definition 2 doesn’t appear until the ascendancy of Neyman-Pearson hypothesis testing and statistical decision theory in the mid-20th century. Some of this decision-frequentist literature takes property 3 along with another property as characterizing even if not quite defining legitimate P-values. In that view, a P-value is a statistic which
a) is uniform or at least dominates a uniform distribution under the tested hypothesis or model, and
b) becomes increasingly concentrated toward 0 as the actual data-generating distribution gets farther from the tested model according to some targeted, relevant measure of distributional distance (such as the difference in means), and also as the sample size increases.
Power or efficiency is then evaluated by looking at downward concentration as a function of distance.
As others have noted, the resulting NP formalization of these notions using definition 2 generates a lot of complexities (which are either fun or frustrating, depending on one’s taste) that are absent from Bayesian decision theory. Most are also absent from the Fisherian approach as well.
For example, uniformly most powerful unbiased tests do not exist in the Fisherian treatment in which a P-value is simply a continuous piece of evidence or measure of compatibility. UMPU tests and their inversions into CI correspond instead to definition 2, and generate paradoxes or incoherencies of NP tests and CIs that some writers (e.g., Lakens) mistake as applying to all P-values and CI. See the first 2023 Scand J Stat article that Andrew linked for a detailed analysis of the subset-incoherency of UMPU P-values described by Schervish (TAS 1996).
One “misunderstanding” that bugs me is when I read a statement like, “This difference is statistically important”. I assume that what was going on in the author’s mind is that in ordinary language, “significant” and “important” are synonyms. However, in a technical definition, you can’t rationally substitute an ordinary-language synonym for a word in the definition.
Right. I maintain however that to complete the picture, observations like yours need to be paired with their complement, as seen in the confusion of claims such as “the difference is not statistically important” or worse, “there was no statistical difference”.
The field of statistics bears the responsibility for having co-opted ordinary words to label technical concepts only distantly related to the ordinary language meanings, then turning the misleading labels loose on the vast majority of users who do not understand the disconnect. Yet to this day some defend the labels as if the labeled concepts capture the essence of the ordinary word meanings, thus inflating the apparent importance of the concepts in real-world inference. To those like me, that’s just promoting deceptive statistics sales practice. As Arthur Bowley (an old-school “inverse probability” Bayesian) said in 1934 when Neyman presented “confidence” intervals, labeling the intervals that way looks like a confidence trick.
I thus advocate harm reduction by using words whose ordinary meanings are less distant from the technical concepts and thus more modest and accurate, such as replacing “confidence” and “significance” with “compatibility” and “incompatibility” (whose usage can be traced back to Fisher), and replacing “null hypothesis” (Fisher’s malapropism) for any targeted statistical hypothesis) with “tested hypothesis” (Neyman’s preferred term). Such modesty would however undermine the confusion of “statistical inference” with scientific inference, so it’s only to be expected that there will resistance from segments of the statistical community who fear the loss of status that the distinction might entail for statistics in research and policy formation.
No surprise then that it’s an uphill battle to change terminology, especially when the jargon has become a religious tradition. No surprise too when experts dismiss the problem as “mere semantics”, as if semantics is less than crucial for communication. You can refute such dismissal instantly today by using any common term for racial categories from the time deceptive statistical jargon took hold in the early 20th century. If society at large can recognize the need to change harmful terms and descriptions but the statistics community can’t, it indicates that the statistics is suffering from a degree of ossification that should be treated as an illness of the field. The only ray of hope may then be in the maxim “statistics progresses funeral by funeral” (which means I am not sanguine about personally witnessing extensive reform).
I don’t think lack of reform on these points can be laid solely at the feet of the statistics community. I am an applied statistician who is fully onboard with your ideas about interpreting “confidence intervals” as compatibility intervals, etc. When I’m tasked with writing up results sections, I often try to frame the results in those terms. I also make a point of doing this for all analyses that we did, regardless of statistical significance. The response I get from my scientific collaborators is typically confusion about why I’m bothering with all that, especially if the p-value is the least bit greater than 0.05. Paraphrasing the pushback I got from a PI on one of my recent projects: “The p-value was 0.07 so I feel like I need to say there was no effect.” So even when their statistician tells them not to just look at whether the p-value is “statistically significant”, even when their statistician gives them the alternative language to describe their results in a richer way than just “was it significant”, they still want to default to that reductive NHST framework. At that point, what can a statistician do?
The comparison of renouncing offensive racial language is helpful. As it took a collective effort across diverse racial groups, statistical reform might similarly require collaboration between statistician and applied researchers.
As someone who came from a typical applied science program, I was not initially aware of the issues surrounding null hypothesis significance testing until I stumbled upon related literature a few years ago. However, gaining this knowledge came at a cost: my previous investments became irrelevant, and my career suffered. For mid-career researchers who lack the necessary mathematical background to become statisticians, it may seem like there is no way out. Therefore, why bother addressing this issue? It may be more important to produce more papers through PhD students to sustain one’s career. In my attempts to discuss this issue with colleagues whom I believed were more open-minded, they quickly changed the subject. People simply avoid discussing it. I suspect that when users of these methods see no alternative, technical solutions alone may not be enough to bring about change. The issue is partially social, and addressing it concurrently with technical solutions is necessary for meaningful reform.
All this talk is merely conjecture until we can identify crucial social factors and develop practical plans. It’s difficult to determine where to begin, but I’ve been pondering a curious question: Is it possible to conduct a project similar to the Open Science Framework replication project to demonstrate that a substantial portion of applied literature is statistically unsound? Such an endeavor could attract attention, particularly from younger individuals. I’m uncertain about its feasibility. It would undoubtedly require the involvement of statisticians. But will applied journals welcome technical feedback from statisticians? Is there a role for applied researchers who have reservations about their practices? These questions may be political, but attempting to answer questions like these could reveal something about the social structure that sustains the current situation.
Samuel: you are of course correct that lack of reform on these points shouldn’t be laid solely at the feet of the statistics community, as my phrasing overstated. In fact some leading members of the community have tried to push for reforms – e.g., see Wasserstein, Schirm, & Lazar, TAS 2019. If my description was overheated, it is because of the counter-reformation movement by other, sometimes more influential members of the community.
Your dilemma is one I’ve heard about and experienced over many decades. While I’ve had the luxury of saying change the wording or remove my name, I know few have that as a practical option. I suppose it comes down to doing whatever you can, including teaching students about what they ought to be doing so that someday, after enough funerals, it won’t be so hard to do the right thing.
You can also have students and encourage resistant colleagues to read some of the many articles written to explain misconceptions to users, and the reforms that have been proposed to stem abuses. These articles (especially the 2016 one below) can also be used to respond to resistant reviewers and editors. Here’s some from over a dozen I’ve been on (all open access):
Amrhein, V., Greenland, S., and McShane, B. (2019). Retire statistical significance. Nature, 567, 305-307
Amrhein, V., and Greenland, S. (2022). Discuss practical importance of results based on interval estimates and p-value functions, not only on point estimates and null p-values.
Journal of Information Technology, 37(3), 316-320.
Greenland, S., Mansournia, M., and Joffe, M. (2022). To curb research misreporting, replace significance and confidence by compatibility. Preventive Medicine, 164.
Greenland, S., Senn, S.J., Rothman, K.J., Carlin, J.C., Poole, C., Goodman, S.N., and Altman, D.G. (2016). Statistical tests, confidence intervals, and power: A guide to misinterpretations. The American Statistician, 70, online supplement; reprinted in the European Journal of Epidemiology, 31, 337-350.
Rafi, Z., and Greenland, S. (2020). Semantic and cognitive tools to aid statistical science: Replace confidence and significance by compatibility and surprise. BMC Medical Research Methodology, 20, 244.
Sander:
Rather than “Science progresses one funeral at a time” (according to wikipedia, this is a shortened version of Max Planck’s statement, “A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die and a new generation grows up that is familiar with it”), I prefer the statement, “The house is stronger than its foundations.” New methods come up and make old theories obsolete, or people adapt the theories to make some kind of sense. One example is false-discovery-rate theory, which I think is based on silly foundations (hypothesis testing, type 1 error rates, etc.) but is flexible enough to give reasonable answers and be a very useful practical tool in a way that traditional 1950s-style multiple comparisons corrections were not. Another example is lasso, which I’ve argued has been very useful despite being based on incoherent foundations. Arguably one could say the same of hierarchical Bayes, that it’s a great problem-solving tool even though it does not make sense to try to model all uncertainty using a joint distribution. All that is one reason that in our article on the most important ideas in statistics in the past fifty years, Aki and I emphasized methods rather than foundational theories. Anyway, my point here is that we don’t need to wait for the funerals. The old theories are still there, and lots of influential people believe in them, but meanwhile they’re using modern methods. Yes, it’s kind of ridiculous to try to get Neyman-Pearson confidence levels or Fisher p-values from densely-parameterized neural nets, but in the meantime people are fitting densely-parameterized neural nets, which solves all sorts of problems, and attachment to old-fashioned foundations isn’t slowing them down too much.
Andrew: I think your comment reflects a very different literature focus than mine. You focused on what sounds like a technically adept segment using methods like lasso and neural nets, presumably by statisticians or similar quantitative professionals who head analyses. I focus on an inept segment including researchers misinterpreting basic statistical outputs for general readers in the health sciences and professions.
Those misinterpretations and misuses have not been extensively reigned in by statisticians, and are even encouraged by some; see for example McNaughton’s book “The War on Statistical Significance”. Among other things that encourages maintaining the editorial practice of using a p-threshold as a publication criterion, despite the well-documented and massive literature distortion that or similar outcome-based criteria produce. And if you think Fiske and Wansink were harmful, imagine the harms brought by similar researchers dominating whole segments of medical research – this time with massive financial stakes to drive their abuses and a much more byzantine system to protect them. For an example, recall Duke Medical School response to what was uncovered by Baggerly and colleagues.
Even when statisticians don’t encourage bad practices, they become part of the problem if they remain silent about abuses they witness or fail to systematically teach about the abuses in their classes. In contrast, you have been a bold critic in your fields and have developed and promoted reforms. I simply think we need far more reform in basic statistics.
My thesis is that, as with automobiles and medical care, statistical methods are essential but dangerously vulnerable to operator abuse. Thus they need an extensive safety redesign like automobiles underwent in the 1960s-1970s and which medical care tries to achieve through curriculum change and continuing education. The reforms I favor will take away time spent on traditional formulas and interpretations, which were developed for highly idealized settings and which are far easier to teach and grade.
Those reforms would emphasize throughout how traditional approaches assume settings in which all players are heroic scientists concerned only to uncover the truth no matter how that might impact their grants or funders, cast doubt on their previous conclusions, or raise liability issues about their past practices. It would then go on to discuss how this assumption is unrealistic and what that implies for statistical interpretation, using case studies to illustrate. It would also involve revamping terminology and methods descriptions to accommodate how most users and readers understand language, rather than expecting them to make sense of math beyond their training and jargon that never had any well-conceived justification.
Finally, it has been said that the originators could not have anticipated the abuses their jargon encouraged. But by 1906 (only 20 years after “statistically significant” began appearing in the literature) K. Pearson was already lamenting that “the absence of significance relatively to the size of the samples is too often interpreted by the casual reader as a denial of all differentiation, and this may be disastrous” (‘Note on the Significant or Non-Significant Character of a Sub-Sample Drawn from a Sample’, p. 183). Yet examples still abound today.
If the “hypothesis” H in definition 1 is in fact an ex-post model H(y) it would be clearer to write explicitly as
p-value(y) = Pr(T(y_rep) >= T(y) | H_0, y)
and maybe add a definition 0 were the “hypothesis” H is an ex-ante model H_0
p-value(y) = Pr(T(x) >= T(y) | H_0)
Also, the “it can also be done mathematically and conceptually without any reference to repeated or future sampling” seems misleading in that case. Conditioning on the observed data is intrinsically coupled with having previous knowledge of that one sample.
There is a conceptual difference between “how surprising would it be to observe y, given H_0” and “how surprising would it be to observe y, given that we have actually observed y and we combine that additional information with H_0”.
Carlos:
The difficulty is that the theory is set up for continuous data and a point null hypothesis, in which case the different senses of p-value coincide. Once the model has unknown parameters, there’s no generally agreed-upon approach, and people don’t always realize that the different sorts of p-values answer different questions.
I fully agree that there are different things that we can call p-value.
That’s why it’s worth making clear when you calculate a p-value for some observation whether you’re using a model conditional on that observation or not.
To be honest, I still don’t know if your intention with definition 1 was to cover these two conceptually distinct cases or not.
Maybe it’s better to start informally saying that a p-value is the probability of something happening under H that counts as much or more against H then what was actually observed. This captures the idea that small p means that something has happened that indicates rather strongly against H. Thi smay require different ways of formalising depending on whether H has just one point or more, and how “counting against” is actually measured.
I don’t think Property 3 is helpful, as this requires continuity, which we often don’t have.
By the way, on misinterpretation that bugs me and seems to come up often is the idea that there is a “true” unobserved p-value potentially different from the observed one in case the model doesn’t hold. Not so. The p-value measures the relation between the data and a specified model, and it does this regardless of whether the model is “true” or not.
Yes! Allow me to refine that slightly to “A P-value measures one relation (of many) between the data and a specified model, and it does this regardless of whether the model is correct or not.” That description holds even if the model is visibly absurd and the P-value nearly 1, or if we know the model is correct (as in a simulation) and the P value is nearly zero. Examples like those may drive home the idea that a P-value is not some continuous version of a truth indicator, as some treat it, and how compatibility is much weaker than validity.
For more technical users I’d add that, more generally, a P-value measures one relation between a more restricted (more precisely specified) model family M and a less restricted model family A within which the more precise family is nested or embedded. That is a more accurate way to describe a P-value for a regression coefficient, where M sets that coefficient to zero and A leaves it unspecified. To subsume the previous description, “the data” is replaced by a saturated model family serving as A.
Finally, of course I’d add my hobbyhorse that the scaling of the P-value is poor (as seen in how the difference between p of .999 and .900 is trivial while the difference between p of .001 and .100 is not), and we get a more equal-interval measure in the S-value or surprisal s = log(1/p) = -log(p), where base-2 logs enables translation of s to bits of Shannon information.
I highly recommend David’s Cox 4 approaches to p – values https://www.youtube.com/watch?v=txLj_P9UlCQ
https://www.annualreviews.org/doi/abs/10.1146/annurev-statistics-031219-041051
see also https://www.youtube.com/watch?v=2mWYbcVflyE
The web page below gathers some of my attempts to get students over the hurdles that so often occur in teaching and learning frequentist statistical methods:
https://web.ma.utexas.edu/users/mks/CommonMistakes2016/commonmistakeshome2016.html
This is an interesting conversation — wish I had seen this earlier. The published version of Vos and Holbert appeared in Synthese in 2022 (https://link.springer.com/article/10.1007/s11229-022-03560-x).
I am happy to learn that Andrew agrees that p-values can be defined and interpreted without repeated or future sampling. This means that when the p-value is criticized in terms of future samples, the problem is not with the p-value itself, but with that particular interpretation.
Interpretations that use future sampling are fraught. In addition to introducing the notion of time (which is better avoided as Michael Lew indicates), the p-value is not designed to address future data. While it could be used for this, this requires not only hypothetical future samples but samples that are from a model for the population — a model that may very well be a poor fit to the population.
For me, there is just one sample, the one observed from the population. There is a space of models for the population and for each of these there is a sampling distribution. We choose a test statistic that provides an ordering of the sample space for each model. For the null model the p-value is the percentile (or 100-percentile) indicating where the observed sample is in the ordering provided by the test statistic.
Calculation of this percentile does not require randomization; the tail area can be obtained for any point in the sample space. The role of randomization is to justify treating samples that are extreme as though they are unlikely to be observed. A single randomization provides this justification. Details are in Vos (2023) (https://doi.org/10.1111/sjos.12647).
While there are many interpretations, there is just one definition – at least I find one is sufficient. Using the notation from Definition 1. p-value(y) = Pr(T(y_rep)>=T(y)|H) where Pr is a measure on the sample space that depends on H. H specifies a distribution (model for the population) and a sampling plan. The argument of Pr is shorthand for the subset of all samples for which the test statistic is at least as large as that obtained from y: S_tail = {y_rep:T(y_rep)>=T(y)}. The notation y_rep suggests that this value is chosen repeatedly (and randomly) but this is not part of the definition. y can be any point in the sampling distribution, when y is the observed sample Pr(S_tail|H) is the p-value for the null hypothesis H.
Part of the confusion with p-values comes from the fact that terms such as ‘random variable’ and ‘generative probability model’ evoke concepts that are not part of the definition. Introducing randomness and hypothetical samples where they are not required does not aid understanding, and it distracts from the original motivation of the definition.
It’s too late to join this discussion, but none of these definitions of p-values is correct. Incorrect definitions like these are increasing confusion.
As for current critics of statistical significance, my views of the current “abandoners” are in a paper I wrote with David Hand in 2022.
“Statistical significance and its critics: practicing damaging science, or damaging scientific practice?”
https://errorstatistics.com/2022/05/22/d-mayo-d-hand-statistical-significance-and-its-critics-practicing-damaging-science-or-damaging-scientific-practice/
Deborah:
None of these definitions or properties are “correct” and none are “wrong.” “P-value” is just a word, and the meaning of a word depends on how it is used. You might as well ask if a tomato is a vegetable or a fruit. The point of the above post is that these are four different definitions and properties which are all used at different times to characterize p-values. And whatever you think about any one of these definitions or properties, it’s not possible in general to have all four of them.
Andrew,
Can you explain what you mean here: “None of these definitions or properties are “correct” and none are “wrong.” “P-value” is just a word, and the meaning of a word depends on how it is used.”
Why are these definitions (or properties) neither correct nor incorrect though, as you note at the beginning of this post, some definitions are not correct, e.g. a p-value is the probability that the null hypothesis is true? Thanks
Josh:
The four definitions or properties in the post all correspond to actual calculations that have been given the name “p-value,” and they all include as a special case the p-value for the special case of a point null hypothesis and for the special case in which the distribution of the test statistic does not depend on theta.
In contrast, “the probability that the null hypothesis is true” does not correspond to anything that anyone has used and has given the label “p-value”; it is just a misunderstanding. If there had been a history of people performing Bayesian analyses, computing the probability that the null hypothesis is true, and calling it a p-value, then, yes, I’d have to include that definition also.
There is a special case of a one-sided p-value under certain models that can be interpreted as the probability that the null hypothesis is true, but this is so different from the other uses of the p-value, and it does not work with the point null hypothesis, so I did not include it.
Thanks, Andrew. Clear.
I really can’t figure out what you mean.
From your article “Statistical significance and its critics: practicing damaging science, or damaging scientific practice?”
The observed significance level or p-value associated with d(x0) is the probability of getting at least as extreme a value as d(x0) computed under H0, where x0 is the observed sample.
p-value = Pr(d(X) >= d(x_0 ; H_0)
In words, the p-value is the probability that the test would have produced a result differing from H0 at least as much as the one observed, if H0 is the case.
That seems pretty much identical to definition 1 above
p-value(y) = Pr(T(y_rep) >= T(y) | H), where H is a “hypothesis,” a generative probability model, y is the observed data, y_rep are future data under the model, and T is a “test statistic,” some pre-specified specified function of data.
The only space on between these definitions is the reference to y_rep as “future data”, while you leave your random variable X for data vaguely specified. I struggle to imagine what you might mean by X that is incompatible with the Vos and Holbert link left for elaboration.
> That seems pretty much identical to definition 1 above p-value(y) = Pr(T(y_rep) >= T(y) | H),
Not really. The H there is not H_0 (independent of the observation) but depends on y.
Carlos:
No, in definition 1 in my above post, H does not depend on y.
What about the references to “posterior predictive checking” then?
You linked to a paper about that mentioning “Bayesian p-values that satisfied Definition 1” in the post – and directed Michael and me to the same paper when we asked whether y_rep applied equally to past and future observations or not.
Is the following relevant for your definition 1 or not?
Is the sampling distribution conditional on y or not?
“Because θ is unknown, but assumed to have the same value that generated the current data y, we simulate from its posterior distribution given y.”
Carlos:
H is a model. In some cases it can include unknown parameters theta, in which case we can do Bayesian inference, p(theta|y,H). H itself is a model that does not depend on y.
Carlos:
In the bayesian p-value paper, the hypothesis is not restricted to be a point hypothesis (i.e. theta = 0), but rather a model + prior. The example in section 2.4 tests the hypothesis that data is chi squared distributed with a prior with support in a restricted subspace of R^n. Thus, posterior inferred theta depends on the data, but the hypothesis being tested does not.
> H itself is a model that does not depend on y.
H is a parametric model and the distribution of the parameter in the model used to calculate p-value(y) depends on y. The sampling distribution for the statistic conditional on H depends on y.
Whether that means that H depends on y or not may be a fruitless semantic discussion. But hopefully we can all agree that this definition is not pretty much identical to the classic one where the sampling distribution of the statistic depends only on H_0 and doesn’t depend on y at all.
By the way, you write explicitly that dependence in equation (5) in that paper: p_b(y) = P_A(T(y_rep)≥T(y) | H, y). If it had been also included in definition 1 above it would be more evident that Pr(T(y_rep) >= T(y) | H, y) is not pretty much identical to a classical p-value conditional on H_0 only.
If I’m not mistaken, if we have for example a location model with a flat prior and known variance and the statistic chosen is (y-θ)^2 then p-value(y) would be the same for every value with definition 1. While you may find that this definition is better – it would be difficult to argue at the same time that it’s pretty much identical to the classic one.
Carlos:
Yes, the posterior probabilities of theta and y_rep conditional on H also depend on y; that’s why we use notation such as p(theta|H,y) and p(y_rep|H,y). H does not depend on y. H is defined before y is even observed. There is a special case in which H is a point null hypothesis, in which case theta does not exist (or, if you prefer, the value of theta is known). The definition of the p-value conditional on H includes the special case of a point null hypothesis and also the special case in which the distribution of the test statistic does not depend on theta, as well as the more general form.
> that’s why we use notation such as p(theta|H,y) and p(y_rep|H,y). H does not depend on y.
Unfortunately you didn’t use that notation here
“Definition 1. p-value(y) = Pr(T(y_rep) >= T(y) | H)”
and that made everything more confusing that it needed to be.
My recent attempt to reconcile the |H) notation given with the |H, y) notation needed was to incorporate the dependency on y into the H – to have a posterior model H(y).
Of course writing the dependency on the posterior model as a dependency on both the prior model (H_0) and the observed data works as well. Thanks for the clarification.
What it’s still unclear to me is under which of those definitions – if any – would you put the classical p-value for a point hypothesis θ and the prior predictive p-value for a hypothesis defined by a density p(θ).
Carlos:
Yes, that expression is implicitly conditional on y, as can be seen because the whole thing is a function of y. I agree that adding the explicit conditioning on y helps, from a Bayesian context.
In answer to your question, the classical p-value for a point null hypothesis satisfies all four of the definitions and properties of the above post. That’s the point of the post: people get confused because they’ll take one of these four definitions or properties and consider it to be the p-value, but the four definitions or properties generalize in different ways once we go beyond the special case of a point null hypothesis and also the special case in which the distribution of the test statistic does not depend on theta.
I made this mistake in my papers on predictive checking by defining “p-value” using definition 1, not recognizing that definition 2 and properties 3 and 4 would generalize in different ways. Other people made this mistake by taking definition 2, or properties 3 or 4, as the definition of “p-value” and not recognizing that their choice was not the only one.
And then we were talking past each other, with me saying that other people were using “u-values” rather than p-values, and other people saying that my p-values were not “calibrated.” None of us fully grasped that there were multiple ways of making the generalization. I almost got there in my 2003 paper talking about p-values and u-values, but I was still making the mistake of privileging one of the definitions and giving it the special name “p-value” rather than being more clear that none of these definitions is inherently correct. Once you have a term with multiple meanings, I think it’s better to avoid it or, when you have to use it, be very clear about its meanings, as I do with so-called fixed and random effects.
> Yes, that expression is implicitly conditional on y, as can be seen because the whole thing is a function of y.
I wouldn’t say that it “can be seen” from your “definition 1” alone. I don’t understand either what does it mean that “the whole thing is a function of y”. Every p-value(y) is a function of y but the “classic p-value” and the “prior predictive p-value” are not conditional on y like the “posterior predictive p-value”.
> In answer to your question, the classical p-value for a point null hypothesis satisfies all four of the definitions and properties of the above post.
Sure, I can see how the point null hypothesis can be written as a prior density with all the mass at some point – and the posterior density is just the same. The “classic p-value” is a particular case of the “posterior predictive p-value”.
However, I asked also about the “prior predictive p-value”. If definition 1 corresponds to the “posterior predictive p-value”, conditional on H and y, it cannot cover as well the “prior predictive p-value”, conditional on H only.
(I would say that the “prior predictive p-value” is the natural generalisation of the classic p-value that addresses the ex-ante question “how surprising would it be to observe y, given H_0” which is different from the question “given H_0 and that we have observed y already, how surprising would it be to observe y again” that the “posterior predictive p-value” would answer.)
I agree with much of what is in the Mayo/Hand article but I find the discussion of “thresholds” schizophrenic. As is pointed out repeatedly, the context of an investigation/analysis is important to determining what the evidence means. It is also stated (and I agree) that abandoning thresholds might exacerbate many of the problems of p-hacking, forking paths, replicability, etc. But this sidesteps the issue of whether using a threshold entails using a single threshold regardless of the context.
What I think is misleading, if not wrong, is that the analysis requires a decision. I’m in favor of reporting p-values (or confidence intervals), but I don’t believe analysts need to conclude that effects are real or not real. Decision makers must make decisions (often repeatedly, so that they may change their minds as new evidence becomes available), but analysts do not need to. Nor do I think they should, regardless of how much decision makers would like them to. I’m sure a busy medical provider would like to point to a paper concluding that treatment X does or does not improve outcomes. But I don’t think that is the right role for the analyst. They should present the evidence without a conclusion. The practitioner, or perhaps a representative body of practitioners, can establish thresholds they find appropriate if they think that is necessary. But I don’t see why the analyst needs to use a threshold at all – in fact, they are not the ones making the decisions about whether or not treatment X should be used.
Aside from the rather lengthy and far reaching scope of the article (including Bayes vs frequentist debates), I think the key issue surrounds the role of analysis. Is it to recommend or make decisions, or just to provide evidence. If it is the latter, then I think much of the debate about p values disappears. It is a piece of evidence, and its value will vary with many factors. Journal editors will be put in the position of deciding whether that particular evidence (and the study that produced it) is useful or not – but isn’t that what they should be doing? The only thing a fixed p value threshold can do is save them from having to do that – and save decision makers from having to consider the context. Why should analysts be making any of these decisions?
“What I think is misleading, if not wrong, is that the analysis requires a decision. ”
I don’t see how that is misleading or wrong at all. If there is any difficulty in determining whether the observed effects of the treatment are real or indistinguishable from uncontrolled variation, then the analyst better damned well provide a decision on that question because no one else can. But once you turn to any statistical inference method as a test for efficacy of a treatment, it’s because the effects of the treatment are difficult to discern from uncontrolled variation, so the question of efficacy of the treatment is by definition a question for the analyst.
This is why, like in every other science that uses mathematical techniques, the analyst using statistical inference must also be a scientist or working collaboratively with a scientist.
Suppose we do our preliminary study on efficacy of handstands for back pain. Then immediately we need to make a yes no decision on whether everyone should do handstands? No. Decisions should be made on the basis of all available evidence and on the cost and benefit of the proposal. An analyst in a preliminary study should not be once and for all deciding the question on the basis of a p value. In fact exactly zero decisions should be made on the basis of p values as they completely are unconnected to costs and benefits
I don’t believe an analyst should be making a “decision on that question [whether the observed effects of the treatment are real or indistinguishable from uncontrolled variation].” They may believe it is real or indistinguishable but no study will absolutely establish that. Should we care what the analyst believes? I’d rather have faith in the analysis they did than their beliefs – and I think I’d have more confidence in their analysis if they avoided telling me what they believe.