Just did this one in class today. Suppose you run 10 independent experiments, each has the same true effect size and each has the same standard error. Further assume that the experiments are pretty good, and the true effect size is 2 standard errors from zero.
What will the p-values look like?
Here’s a simulation:
set.seed(123) J <- 10 theta <- rep(0.2, J) se <- rep(0.1, J) theta_hat <- rnorm(J, theta, se) p_value <- 2*(1 - pnorm(abs(theta_hat), 0, se)) print(round(p_value, 3))
And here's what we get:
0.150 0.077 0.000 0.038 0.033 0.000 0.014 0.462 0.189 0.120
Now imagine this was real life. A couple of experiments are consistent with pure noise, a few of them seem to show weak evidence against the null hypothesis, and a few are majorly statistically significant. It would be natural to try to categorize them in some way. Yeah, the difference between "significant" and "not significant" is not itself statistically significant, but a p-value of 0.46 in one case and 0.0002 in another . . . surely this must be notable, right? No.
OK, this is an extreme case, in that there is no underlying variation, and indeed if you fit a multilevel model you could see the lack of evidence for underlying variation.
The larger points are:
1. The p-value is a statement relative to the null hypothesis of no effect. It doesn't have much of an interpretation relative to a real, nonzero effect.
2. The p-value is super-noisy. It's a weird nonlinear transformation of the z-score (which does have a pretty clear interpretation) with all kinds of nonintuitive behavior.
And:
3. You can learn a lot from a simulation experiment. Even something really simple like this one!
P.S. Bayesian inferences are highly variable too. Any data summary will be variable! The problem here is not so much with p-values as with when they are misinterpreted (point 1 above) and when they are taken to be a strong statement about reality (point 2 above), rather than a noisy summary of a particular experiment. If you misinterpret and overinterpret Bayesian inferences---for example, by fitting model with flat prior, taking the posterior probability that the parameter is greater than zero, and then making a decision based on whether that probability exceeds some threshold---then you're just as bad off.
Remember, the problems with p-values are not just with p-values
P.P.S. There was some confusion about the simulation in comments, so I'll explain some more.
The above is not a simulation of raw data. I could’ve simulated raw data, but it was simpler to just simulate the parameter estimates and cut out the middleman. Instead of simulating y_1,…,y_n ~ normal(theta_1, sigma_1) and z_1,…,z_n ~ normal(theta_2, sigma_1), I simply note that theta_hat = y_bar – z_bar has mean theta = theta_1 – theta_2 and standard error s = sqrt(sigma_1^2/n + sigma_2^2/n), I assume n is large enough that we can treat s as known, and I just directly simulate theta_hat and s. I did a simulation where the true effect is 2 standard errors away from 0, which is realistic for many studies. In medical research, they’re supposed to be powered so the true effect is 2.8 standard errors from 0 (for explanation of that 2.8, see for example chapter 16 of Regression and Other Stories); realistically, though, I think the assumptions made in such designs are overly optimistic.
My colleagues and also done empirical work on this topic: see A new look at p-values for randomized clinical trials (with Erik van Zwet, Sander Greenland, Guido Imbens, Simon Schwab, and Steven N. Goodman) and A proposal for informative default priors scaled by the standard error of estimates (with Erik van Zwet).
Yes, with a uniform distribution, the p-values can be all over the map. In any one experiment, as Andrew has written before, you don’t know the p-value but only an estimate. The variance, if I remember rightly, is 1/12. So the S.D. is near 0.3.
It doesn’t seem wise to base serious conclusions on a statistic of 0.05 +/- 0.3. Of course the value won’t be centered in the range and the p-value can’t go negative, but it still isn’t promising.
In what way if the p-value an estimate?
Not sure where variance 1/12 comes from, but (using test statistics that are non-central chi-squared with 1df) it’s reasonable to expect p-values to fluctuate randomly by at least a couple orders of magnitude. And much more than this, when there are strong signals present;
Code below
getq = function(ncp){ pchisq( qchisq(c(0.025, 0.975), df=1), df=1, ncp=ncp )}
matplot( seq(0,25,l=301), t(sapply(seq(0,25,l=301),getq)), type=”l”, xlab=”ncp”, log=”y”, ylab=”central 95% of p-value distb’n”)
correcting some typos:
getq = function(ncp){ pchisq( qchisq(c(0.025, 0.975), df=1, lower=FALSE, ncp=ncp), df=1, lower=FALSE)}
matplot( seq(0,25,l=301), t(sapply(seq(0,25,l=301),getq)),
type=”l”, xlab=”ncp”, log=”y”, ylab=”central 95% of p-value distb’n”)
> Now imagine this was real life.
Well, I might expect that there is an optimal location for theta in your example, given some se, that provides the maximum amount of ‘real life’ surprisal for this example. For example, try the following code:
par(mfrow=c(3,1))
set.seed(123)
J <- 1000
theta <- rep(0.01, J)
se <- rep(0.1, J)
theta_hat <- rnorm(J, theta, se)
p_value <- 2*(1 – pnorm(abs(theta_hat), 0, se))
sd(p_value)
plot(p_value ~ theta_hat, xlim=c(-0.2, 0.7), ylim=c(0, 1))
theta <- rep(0.2, J)
theta_hat <- rnorm(J, theta, se)
p_value <- 2*(1 – pnorm(abs(theta_hat), 0, se))
sd(p_value)
plot(p_value ~ theta_hat, xlim=c(-0.2, 0.7), ylim=c(0, 1))
theta <- rep(0.4, J)
theta_hat <- rnorm(J, theta, se)
p_value <- 2*(1 – pnorm(abs(theta_hat), 0, se))
sd(p_value)
plot(p_value ~ theta_hat, xlim=c(-0.2, 0.7), ylim=c(0, 1))
The example using theta = 0.2 (middle plot) is particularly compelling because theta_hat concentrates around the value where the curve of corresponding p-values turns sharply toward flattish, which also happens to be around the threshold of interest to most people p=0.05. Thus, you have a wide range of values that span much larger than p=0.05 and much less. In contrast, when you look at either extreme (top or bottom plot in my code) of theta_hat very close to zero or very far away, for given se, then you see p-values that are more homogeneous in terms of people's sense of what is surprising (namely that it fluctuates well above or well below the threshold of 0.05). In other words, being surprised at the result of these toy examples is a function also of the threshold that people are interested in and how they perceive differences qualitatively; for example, people aren't surprised maybe in the difference between p=0.9 and p=0.4 or the difference between 0.01 and 0.001, like they are between say 0.4 and 0.04.
This is a recreation of Geoff Cumming’s “Dance of the p values” and in my mind it promotes the same unhelpful conclusion. If you were to plot the data for each of the experiments you would find that the p-values do offer a reasonable index of the strength of evidence in the data against the null hypothesis. Each replication of the experiment yields stronger or weaker evidence and that is correctly reflected in the resulting p-value.
Of course, the non-linear relationship between eyeballed strength and the p-values can be a problem for the unpracticed, but if you rank the eyeball strengths you will find it matches the ranking of the p-values.
Michael:
The p-value is what it is, and if it’s interpreted directly, then, sure, why not. The problem is when p-values are used for thresholding or as a sort of measure of effect size, which unfortunately is often done. For that reason I think it’s helpful for people to realize how noisy the p-value is. This can also be seen by bootstrapping.
So it’s the thresholding into dichotomous decisions that’s the problem. I agree.
My point was that the p-value is not “noisy”. The evidence in a random sample concerning the null hypothesis is more variable than people might expect (and you could describe that as noisy, I guess) but the p-value is not noisy.
Michael:
I’d call the p-value “noisy” in the sense that if the experiment were replicated under the same conditions, the p-value could be much different. I don’t think that researchers generally realize this point; indeed this was the subject of some of the very first research by Tversky and Kahneman on cognitive illusions. Experienced researchers, who one would’ve hoped had known better, routinely expected an unrealistic level of stability among p-values in replications. The idea that the p-value could be 0.46, or 0.08, or 0.03, or 0.0002, all in different replications of the same experiment . . . the math is clear, but it’s not intuitive to people, perhaps because they’ve been trained to think of 0.46 and 0.08 and 0.03 and 0.0002 as completely different results.
I’m embarrassed to admit I don’t understand why this shows that dichotomizing based on p-values is incorrect. Below is my thought process, but I understand I am missing something and hope someone can help me.
Doesn’t this simulation show the results of an experiment with 50% power, so we would expect the p-value to fall above and below alpha half the time in a simulation? So if we interpret the p-value as a continuous measure, we will over-interpret its importance when we get a very small p-value. If we dichotomize instead, then we can treat the test as evidence for or against the null hypothesis while defining the strength of that evidence in terms of the mathematical properties of the test (e.g. a high-powered test would be considered stronger evidence than a low-powered one).
Reese:
A p-value, dichotomized or otherwise, is not in itself “incorrect”; it’s a summary of data relative to a specific model. The error comes in the interpretation (in the common setting where the specific model being used as a comparison is not itself of direct interest) or in decision making (if results that exceed a threshold, as comparison to others that do not exceed the threshold, are treated as a “discovery”).
I am not sure this is a valid example. If I read the R code correctly, you are using a single observation and then calculating the p-value based on an unmeasured variation. I think this example would be more meaningful if you varied the sample size for each of the 10 experiments. I suggest running a loop (or more efficient vectorized method though it won’t really matter for this simple simulation) to calculate theta_hat for 1 sample per run, 2 samples per run, up to 10+ samples per run. You can explore average, range, sd, or even visualize how the individual p-values converge as the sample size increases.
Jeffh:
No, for each of the 10 experiments I’m simulating a parameter estimate theta_hat with standard error se. I’m not bothering to simulate individual data because, with the central limit theorem, I know the estimate will be approximately normally distributed. I’d get the same result if, for example, I’d simulate 10 experiments, each with 100 people in the treatment group and 100 people in the control group, as long as the sample size is such that the true average treatment effect in each experiment happens to be 2 standard errors from 0.
Andrew:
I’m not quite sure what you’re doing here but if this is the way p-values work in practice then how would we have any effective drugs or treatments at all? P-values are clearly problematic and badly misued in many applications, especially in social science, but the significance value cut-off for medical treatments has provided thousands of highly effective treatments.
What gives?
This apparent “success” is determined by statistical significance, which will then get selectively reported to you in the form of press releases.
Meanwhile life expectancy peaked in 2014.
Even formerly common-sense safeguards like aspirating before IM/SQ injections are being removed (due to lack of significance, a blood vessel is hit only every ten or hundred thousand injections).
There is also “no evidence” that clean drinking water is good for children, or that keeping the elderly from seeing their families and telling them to fear symptoms of a panic attack is bad for them.
Anoneuoid
According to the many views you express, it seems that no vaccines do any good, that all medical research is worthless, and we know nothing about human health that wasn’t known hundreds of years ago (yes, I know I have taken the liberty of paraphrasing your views, perhaps inaccurately). But my point is that your views are always extreme to the point of absurdity to me. You may believe that the correct response to over-reliance on NHST and clinical trials is to go as far in the opposite direction as possible. But for me, it just leads me to reject what you are saying. In this latest comment of yours, I can’t even figure out what you are saying about “clean drinking water.”
So then dale, what gives? Andrew’s similulation implies that for an effect with similar size and variance to his parameters, half the approved treatements would be bogus and half the unapproved treatments would be effective. This is obviously false.
So presumably Andrew’s paramaters aren’t indicative of actual paramters in most medical trails.
Andrew makes claims like this from time to time that are obviously bunk. The question is what is he up to? I presume he knows this is bunk, but it’s possible I’m crediting him with either too much intelligence or insufficient bias. OTOH, if I’m right and he knows this is bunk, why is he doing it? To demonstrate that almost no one knows why it’s bunk? Almost no one cross-checks with reality to make sure what he’s saying makes sense?
Chipmunk:
What are the claims like this that I make that are “obviously bunk”?
I mean there is no clinical trial showing clean drinking water is good for children, or that elder abuse is bad for the elderly. This is “no evidence” in the EBM-speak used by the WHO and similar.
So it isn’t just lack of new effective interventions since 1950 or so, previously cheap and effective practices are being actively disregarded.
The opposite of NHST is science. Testing a null hypothesis rather than your hypothesis inverts all the logic and incentives. Meehl was writing about this since the 1960s.
https://philpapers.org/rec/MEETIP
Its like, how long until a room full of praying monks invent/construct a smartphone?
That is like what you are asking us to believe about NHST and medicine. It is totally implausible. So if effective interventions were developed despite NHST we should study how that happened.
@Chimpmunk can you share evidence that has convinced you of one of these new interventions?
Chipmunk (and others):
I don’t think Andrew’s simulations would imply that “for an effect with similar size and variance to his parameters, half the approved treatements would be bogus and half the unapproved treatments would be effective”. As I understand it, Andrew is just simulating where there is a real effect (not some where there is one, and some where there is not one), and a known standard error (for simplicity). Specifically an effect that happens to be 2 standard errors away from zero (with whatever sample size is used), meaning there is a roughly 50/50 chance that the p-value will be above or below .05 (i.e. 50% power if .05 is the alfa-level, and we test again a zero effect for the null-hypothesis).
Then the simulations show that high p-values are still fairly common, while a naive view might be that they should still practically always be “around” .05. If one thought of p-values as uniformly distributed also when there is an effect, one might expect it to vary around .10 to .00 in this case, for example (from it being below .05 in 50% of all cases). This could make it problematic if one took p = .462 to mean “probably no really meaingful effect, no point in trying to keep researching it”.
Lol you’ve moved onto straight up mathematical denialism. This is true by definition, high school statistics stuff
Mathias:
I get that. Where there is a real effect within the parameters provided, half the time the treatement will be rejected and half the time it will be accepted when using the P-value cutoff. Perhaps my assumption that it would be symmetrical – that no real effect would have an equal accept/reject rate or that the parameters could be similiarly arranged – is incorrect. I didn’t think that through.
But it’s hard to escape the fact that, as Andrew tacitly admits in his response to my comment, if his simulation employs paramenters typical of actual medical trails, then many ineffective treatments would be approved. This is clearly not the case. OTOH, if his similuation doesn’t reflect the parameters of real medical trials then his point is moot.
Reality seems to lie in between but closer to the “moot” end: that occasionally mildly effective treatments are rejected and only very rarely are ineffective treatements approved.
Chipmunk:
You write that if my simulation “employs parameters typical of actual medical trials, then many ineffective treatments would be approved.” That’s not a correct description of my simulation. In my simulation, the treatment does have a positive effect. I’m not simulating any ineffective treatments. If you want to see what happens with ineffective treatments, you’d have to do a different simulation. My 5 lines of code are not intended to simulate all of the medical system, they’re only simulating one scenario, which is a series of treatments that are effective and measured with some error.
Andrew: “If you want to see what happens with ineffective treatments, you’d have to do a different simulation. ”
>OK, yes, I realized that in retrspect, the two would require different simiulations.
‘You write that if my simulation “employs parameters typical of actual medical trials, then many ineffective treatments would be approved.” That’s not a correct description of my simulation. ‘
>Yes, I see in this case I misspoke: your similation specifically indicates that many effective treatments would be rejected, not that many ineffective treatements would be approved.
But from the results of your similulation, you make a more general claim about the noise of p-values. However, that noise doesn’t apparently lead to the routine approval of ineffective treatments so I’m still not clear on why people should replace this system for medical treatments without another system that is demonstrably significantly more effective.
In more general terms, sure, yeah, again I don’t see the real utility of Pvalues, but again in more general terms if people can’t type into a spreadsheet and manage their data, check their calculations, design experiments with reasonable and/or testable assumptions and check their results against reality, it’s hard to see how harping on P-values has any benefit. Unless the baseline goes up not much positive will happen.
Chipmunk:
1. p-values provide some information. Not zero, just less than people sometimes think.
2. Without going full Anon on this one, I don’t have a sense of how much p-values have been helpful in developing or identifying effective drugs and treatments. I agree that there are lots of effective drugs and treatments for many medical conditions, and they are often evaluated using statistical methods. I expect that, to the extent that decisions have been made using p-values (as I put it, “outsourcing your decision to a random number generator”), this has not helped. Ultimately, though, studies are replicated, and I expect that effective treatments are identified. If you wanted to model that in a simulation, you could simulate replicated experiments from each study, and eventually the treatments with large positive effects would produce enough statistically significant p-values to make it through.
“eventually the treatments with large positive effects would produce enough statistically significant p-values to make it through.”
Presumably that’s exactly what the system intends to do: eliminate “treatments” for which the effect is unclear or fuzzy and approve treatments with clearly beneficial effects. By all accounts it seems wildly successful, whatever the vagaries of P-values or the problems in the FDA. Downplaying how successful this system has been is just not very credible. The US system is the worldwide model *because* it has been so successful.
As much as I agree with your condmenation of NHST / p-values in many of their common uses, I think – and hope – you’re going to have a very hard time convincing to people abandon the current system of drug / treatement approval when:
a) you give examples that are clearly at odds with reality;
b) you give examples that are entirely theoretical with no concrete element whatsoever,
c) you offer no credible alternative that has a sharp boundary between approval and non-approval, and even seem to believe that such a boundary is not necessary. It is necessary, because if no boundary exists the system will be badly abused and we **will** be awash in quack treatments and useless drugs.
Also, in this particular example, you give your reasons for not simulating the actual data, which are sound theoretical reasons, but you frequently criticize other research for not showing the data. Seeing the data is beneficial even if its theoretically perfect.
In a purely scientific world, where the only objective is to understand deterministically how something works, I agree with almost all of your criticisms of P-value use. Scientifically it’s dumb. But we live in a world where medicine is still non-deterministic, the range of effects of a given treatement are almost never well understood, and people have competiting interests. This world necessitates a systematic decision-making procedure that gives absolute answers regarding treatments.
If you really want change in that procedure, develop a system and demonstrate it’s effectiveness with real, concrete examples. That’s your challenge. You don’t seem interested in doing that, which I get because it’s a monumental challenge – the work of an entire career and possibly more. But if your not interested in doing it and you think it needs to be done it makes more sense for you to shine the light on the path that will lead someone else to accomplish that and to do that you have to acknowledge what has been successful and work with it.
Chipmunk, medical interventions are discovered in spite of NHST not because of it. Typically there’s a couple decades of mechanistic testing in vitro and animal models before you even start doing clinical trials. And a lot of clinical trials just wind up washing out interventions before they really get evaluated. Keith O’Rourke was fond of saying that we should be spending a LOT more on clinical trials that are a lot bigger to get unambiguous results from stuff that’s getting washed out even though it’s effective. (Hi Keith, I miss you and I never even met you in person)
One thing that happens is, because of that decades of biologists actually doing their real biological sciences thing, some interventions have really big effects, and those don’t really need p values to be obvious. Andrew’s simulation results are dependent on the true effect being 2 standard deviations. But there are effects that are like 10 standard deviations. Despite what Anon thinks, I think the COVID vaccine was similar to that, at least in the 3-5 months post injection. Organ transplants are like that… You can take a person who is in kidney failure and on dialysis every day, and then they don’t need dialysis at all. That’s not a small intervention effect. The chance of a kidney failure patient randomly getting better and needing zero dialysis is ZERO to many decimal places.
There’s all kinds of other stuff which is much more ambiguous. For people with mild hypertension do we know if commonly given drugs are actually that effective at reducing mortality? Like lets say you have 145 over 90 as an avg blood pressure. We give you Amlodipine and it still fluctuates but the average drops to say 135 / 83 is this a win or not? P values are likely not telling us that. However, it’ll all be declared as a win by the industry.
Then there’s stuff like Tamiflu. We bought literally tens of billions of stockpiles on that one, and the data which as I understand it was uncovered after being actively buried by the mfg shows it’s basically useless.
The FDAs opinion of itself is much lower than your assessment:
https://web.pdx.edu/~pcooper/2007FDAScience&MissionatRisk.pdf
Of course, they attribute it to mission creep and lack of funding. But the end result is the same.
I’d really like to see some examples of, what you consider to be, recent (post-1950ish) effective medical interventions.
Chipmunk:
You say that I give examples that are clearly at odds with reality, but I don’t think that’s the case at all. I have lots of experience looking at all sorts of studies, and (a) I think my examples do match reality, and (b) I have no reason why you say otherwise.
Regarding my example that is entirely theoretical with no concrete element whatsoever: Well, yeah, this is 5 lines of simulation code! That’s the point: it’s math, we make it as simple as possible. My colleagues and also done empirical work on this topic: see [2024] A new look at p-values for randomized clinical trials. {\em NEJM Evidence} {\bf 3} (1). (Erik van Zwet, Andrew Gelman, Sander Greenland, Guido Imbens, Simon Schwab, and Steven N. Goodman) and [2022] A proposal for informative default priors scaled by the standard error of estimates. {\em American Statistician} {\bf 76}, 1–9. (Erik van Zwet and Andrew Gelman)
For further discussion, you can read this comment to Dale below.
Anoneuoid
Can you provide more information about the FDA report you linked to? These days, I take nothing for granted and it is a strange document. No authors listed (although I discovered a Barry Leonard as author of this published as a book in 2009). Also, there is supposedly an Appendix A which lists the contributors, but I can’t locate that Appendix anywhere. I basically take no printed document involving a government agency at face value – things are so politicized that it could just be a rant by whatever interest group or political party has an ax to grind. I’m not saying this is the case, but linking to documents like this (that are incomplete and without accountability) leaves me wanting more complete details if I am to put any stock in this report. Certainly citing this as “the FDA opinion of itself” is a stretch without that.
Like this?
https://www.govinfo.gov/content/pkg/CHRG-110hhrg47433/html/CHRG-110hhrg47433.htm
That same document was discussed in the comments here years ago, you can also try searching the blog for that. At first, it was easily available from the FDA site but that doesn’t seem to be the case any more.
Thank you, that helps. I would note that the issues cited really don’t address NHST or statistical methodologies – it involves lack of resources (human and technical), lack of data, and the familiar failures to adequately support public health. I don’t think it is fair to stretch it to mean that the FDA’s statistical models rely excessively on unreliable p values.
Yes, that is what I originally said above. They attribute rhe problem to mission creep.
The reason I brought it up is that (regardless of the reason) chimpunk’s version of reality appears to conflict with the FDA’s.
Of course, the real root of their problem *is* NHST. Ie they are wasting resources dealing with a beastiary of ad hoc methods that kind of work for some toy problems but are worthless for real-life ones.
Daniel, tangentially related to your comment but about blood pressure. The SPRINT trial found both major cardiac and all cause mortality benefit to treating hypertension down to ~120mmHg systolic as opposed to ~140mmHg systolic, so we know that a 20mm drop even in this range is pretty meaningful. The HR was 0.75 with 95% CI (0.64-0.89), so I’d guess a 10mm drop from 145 to 135 would indeed show up…if we take their results at face value :) But more generally, I would much prefer to see a fully Bayesian non-linear risk vs exposure analysis. I hate how with all these things – vaccine dosing, medications for chronic conditions, whatever – it seems our current system wants to pose all these questions “one at a time”, asking for arbitrary significance thresholds rather than partially pooling all our information including reasonable constraints from biology like when to expect a saturating versus linear versus exponential versus logistic response, etc etc
https://www.nejm.org/doi/full/10.1056/NEJMoa1511939
Chris. Interesting. I too would love to see a comprehensive Bayesian analysis of the whole shebang. I mean heart disease is a major problem in the US right? Why don’t we actually know what the F is going on there and instead have one at a time “x seems to cause 30% reduction in y” for 30 different x and 10 different y independent of whatever else occurred etc.
In fact, I’d like to see it so much that I’ve been considering doing some research on what datasets are available. If you’ve got info on what I should look at email it to me!
They also show baseline systolic BP ~140, diastolic BP ~78, and already taking ~2 antihypertensive drugs.
My interpretation is that if you are already taking drugs for hypertension, and no longer meet the criteria for mild hypertension, adding another drug has ~1% chance of making you not die in the next ~4 years. Is that accurate?
Regarding the SPRINT data, I spent a lot of time with that data as part of the competition run by the NEJM a few years ago. The drugs used to lower blood pressure had serious adverse effects (side effects) which made the reduction in primary events of questionable value. What I discovered is that the published study underestimated the benefits of lower blood pressure – for those who followed the protocol. For people who missed scheduled visits, the outcome was actually worse than those in the control group – more primary events and more adverse events. Presumably (but not known for sure, for reasons I could go into a lot of detail about) the people who missed their scheduled visits were probably not taking their meds as prescribed, probably because they had bad side effects. In fact, you could have identified this problem group early in the study.
Did that include individual timeseries of the blood pressure, etc along with the outcomes?
Like, you could see how consistent the BP measurements were and how this correlated with mortality? Seems like they collected some great data here but I’m not seeing it in the paper.
Yeah, where do I get individual level data? (I know damn well the answer is very likely to be “you don’t” but hope springs eternal.
Daniel, would that I knew! I think it would be a good project though. I’d love to help but am already over-committed on like a dozen things.
While we’re at it, another topic of WTF is blood lipids. I’ve been lead to believe that meta-analyses of LDL/ApoB lowering interventions robustly show all-cause and cardiac mortality benefits, in a pretty much linear, dose-dependent manner down to something like an LDL-C of 70 mg/dl in primary prevention settings. Likewise, genetic variants with, say, PCSK-9 knock-outs who have way more LDL receptors on liver, hence drastically reduced circulating LDL, show virtual immunity to atherosclerosis and consequent heart disease. Mendelian randomizations seemt o back this up at population level.
And yet. And yet the observational studies almost invariably have a U-shape mortality relationship with LDL-C, e.g. one of many here https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10982736/
Yes yes everybody knows about confounding and reverse causality and all that, and yet ^nobody ever explains what the reverse causality actually is/might be here^! For instance, in linked study, what is it about the group with very reasonable low-normal LDL-C of 80-99 mg/dl that correlates them to higher mortality than 100-160? Everybody brings up things like cancer cachexia or whatever, but like, this is surely excluded from having an impact on these U-curves when studies do things like exclude everyone who died within 1 year. What are all these other disease states that somehow upregulate LDL receptors or cause faster clearance of LDL or whatever? Why do they do so, and why does this have no biological significance to the point that LDL lowering wouldn’t potentially carry negative consequences?
Anoneuoid
Yes, they released individual data that shows the complete time series of blood pressure measurements during the trial. I did look at the variability of BP readings as a potential effect on the primary event but that never emerged as an important influence. If you search for the SPRINT competition you can probably find the data release somewhere (probably somewhere on GitHub), although I recall that I needed to get IRB approval and have an administrator at my university sign off in order to see the anonymized data that they released! Just part of the story I’m not telling here about the NEJM and the issue of open data.
If you search “cholesterol hypothesis” there are many papers like this:
https://www.tandfonline.com/doi/full/10.1080/17512433.2018.1519391
I also couldn’t find anything about atherosclerosis in PCSK-9KO mice.
In general, the “U-shape” indicates higher mortality in the case of a deficiency, while very high levels indicate some kind of organ failure. This is the same for other substances that travel from the blood to damaged tissue and get oxidized, like vitamin C.
Why are LDL-C levels are so high in some people, liver failure?
> Ultimately, though, studies are replicated
One would assume this is happening, but is it really?
Low p-values are *correlated* with the effect being real. They’re just not very *strongly* correlated.
I’ll admit I am a bit puzzled about the simulation. I don’t know how to exactly replicate it, but I have run a number of bootstrap analyses of various data sets to see how stable the p-values are: presumably, these bootstraps should somewhat match Andrew’s simulation. From my explorations, it seems that in large data sets with highly significant effects, the p values are fairly stable (almost too small to measure). When I look at marginally significant effects (p values between .01 and .10) in fairly small data sets, I get results similar to Andrew’s. What I don’t understand is why the sample size is irrelevant to Andrew’s simulations. In my explorations, it is quite important.
The simulation here is framed relative to standard error, which implicitly fixes a sample size. As sample size goes to infinity, standard error goes to 0 and so does the p value. The relevant point is that you can have very large effect sizes (relative to standard error) and still get really different p values.
Dale:
What Somebody said. I could’ve simulated raw data, but it was simpler to just simulate the parameter estimates and cut out the middleman. Instead of simulating y_1,…,y_n ~ normal(theta_1, sigma_1) and z_1,…,z_n ~ normal(theta_2, sigma_1), I simply note that theta_hat = y_bar – z_bar has mean theta = theta_1 – theta_2 and standard error s = sqrt(sigma_1^2/n + sigma_2^2/n), I assume n is large enough that we can treat s as known, and I just directly simulate theta_hat and s. I did a simulation where the true effect is 2 standard errors away from 0, which is realistic for many studies. In medical research, they’re supposed to be powered so the true effect is 2.8 standard errors from 0 (for explanation of that 2.8, see for example chapter 16 of Regression and Other Stories); realistically, though, I think the assumptions made in such designs are overly optimistic.
OK. I prefer not to cut out the middleman – it just feels more concrete to work from data. I did some simulations and do find results analogous to your more concise simulation. Even in fairly large data sets, with effect sizes around 2, the p value does vary considerably doing a bootstrap. Isn’t this related to the idea that in a large sample, you need much lower p values (far less than .0001 in a sufficiently large data set) for the p value to be at all stable. In other words, if the effect is strong enough, then the p value doesn’t vary nearly as much. Of course, the p value is fairly unnecessary (or useless) in such cases anyway. An effect size of 2 isn’t really very large after all. Hasn’t this led some people to advocate for much lower p value thresholds than the commonly used 0.05 in reaction to criticisms of NHST? I don’t like the idea of thresholds at all, but if they are going to be used, then perhaps a more realistic value (one that exhibits more stability) should be used.
Dale,
Just for clarity: Perhaps we use the term “effect size” in different ways, but the 2 you refer to is for the effect relative to the standard error, not relative to the standard deviation, right? If so, the effect size is not 2 in my book, it is smaller than 2 (how much smaller of course depends on how large a sample we have).
Mathias
Yes, I was using effect size of 2 relative to the standard error. I’m not sure what you mean be the effect size being less than 2 in that case – isn’t Andrew’s simulation also using 2 relative to the standard error?
Hi Dale,
Right, then it makes sense to me!
When I use the term “effect size” I mean some population parameter (as that’s how its used in my field), either an (absolute) effect (e.g. a mean compared to zero, a difference in means between two groups), or a (relative) effect size that compares an effect (e.g. mean) to the spread of the distribution (e.g. through the standard deviation), and which is therefore meant to provide information about probabilities for individual observations (if the variable is normally distributed).
If we have a variable X for one sample, then an effect size (as I use it) might be E(X)/Var(X)^0.5, i.e. the population mean over the population standard deviation. If that had been the parameter put into the simulation, that is E(X)/Var(X)^0.5 = 2, then it would have been strange to me if the p-value kept varying a lot also for larger sample sizes in your simulations.
However, as it is something like E(X)/(Var(X)^0.5/n^0.5) that equals 2, it makes sense. That’s why I said that in my version of an effect size, it would be below 2, as it would equal E(X)/Var(X)^0.5 = 2 / n^0.5 for these simulations.
I see now that Andrew wrote that “the true effect size is 2 standard errors from zero”, although I do not translate that to “the true effect size is 2” from my way of understanding effect sizes.
But they are a great way of selecting variables for a multivariable regression model.
Hi Andrew et. al.,
This is my first comment (or second – I commented on a 10 year old thread a while back). I feel emboldened by the recent post inviting comments of the right sort, but I don’t know if this is it. I’ll post and see whether I get a Ripley-like reply or, worse yet, no response, which will be telling enough. I realize now that my comment is really a question, and That might not be appropriate, but live and learn as they say. Here goes. I wonder where to look for evidence of progress since “The Problems With P-Values are not Just With P-Values was published”? Are there better written books now? Have some statistics programs made an effort to better prepare its students to properly respond to and solve data analytic problems? Has anything be done to shake the notion that statisticians have some kind of magic by which every problem is reduced to N and p, but not much else? I realize also that this is a wide open question, but just a few pointers a links would be great. Any encouraging signs which indicate that we are no longer living in the world of Nail Finders, Edifices and Oz (https://digitalassets.lib.berkeley.edu/sdtr/proof/pdfs/032.pdf) would be most welcome.
Fz:
Thanks for the comment. And, wow, referring to “Ripley-like replies” . . . that brings back memories!
In answer to your question about progress on p-values and the replication crisis since that 2016 article . . . I don’t know! I think we have a much better understanding of what’s been going wrong in science, and I think that Psychological Science and even PNAS publish and promote a lot less of that himmicanes-style crap than they used to. On the other hand, I still have the impression that hypothesis testing is the predominant mode of thinking in quantitative research, and that the world of theoretical statistics remains mired in that perspective. This sort of episode is frustrating, in that even the leaders of the open-science movement seem to be stuck in old-fashioned ways of thinking about statistical evidence.
It’s particularly telling to read Reddit r/Ask statistics where daily students write questions summarized as “What’s the right magic test to show my data is statistically significant?”
It’s like a penance for me these days. Just read them and weep. Keeps me grounded in reality.
Dang it autocorrect… r/AskStatistics
Andrew: Thank you for the memories. Professor Ripley was really quite entertaining, and sometimes helpful. (I might be pushing my luck here..)
In terms of progress in the teaching of statistics I am still looking for a correction to the problem described here (this may sound familiar):
First, in our courses and textbooks (my own included), we tend to take the “dataset” and even the statistical model as given, reducing statistics to a mathematical or computational problem of inference and encouraging students and practitioners to think of their data as given (and that the statistical problem is simply one of finding the best matching method from a textbook, or software library) . Even when we discuss the design of surveys and experiments, we typically focus on the choice of sample size, not on the importance of valid and reliable measurements. The result is often an attitude that any measurement will do, and a blind quest for statistical significance.
You wrote this so I know that you know this is a problem, but I don’t know if you realize how accurately this describes statistical practice in some industries today. Educational programs are in part to blame, but I tend to think that the lion’s share of the blame must be awarded to employers. Deming argued that the blame for the lack of quality in American goods didn’t fall on the shoulders of engineers – engineers are giving CEOs what they want; a quick turn around. Same here. If CEOs (or the VCs the CEOs answer to) needed more than a quick analysis capped with a p-value, we would probably work harder at understanding the sources of variability in the process, and how to harness the randomness to produce solid, lasting solutions that we can trust, and build on. But I digress. My question remains: Is an alternative textbook and teaching format possible? Maybe not and perhaps the graduate consulting seminar class is it, but please say it ain’t so.
I recently came by this BlueSky thread on the related “Uncanny Valley” of p-values by Steve Haroz, filled with some interesting plots: https://bsky.app/profile/steveharoz.com/post/3kqxzcbdtst27