This is Erik: A few days ago, I wrote about my concerns about the z-curve method. I demonstrated that under certain circumstances, the coverage of the confidence interval of the expected discovery rate (EDR) is far below nominal.
Ulrich Schimmack, who is the main author of the method, posted many comments in response. I think it is fair to say that these comments are generally quite defensive and sometimes even accusatory (“Erik doesn’t want to conclude that z-curve works”). Schimmack argued that my demonstration is unrealistic and therefore not relevant. He also proposed a patch: Do not use the confidence interval for the EDR if the z-curve has an upward slope at |z|=1.96 or if the lower bound of the confidence interval of the expected replicability rate (ERR) exceeds 0.9.
Nobody likes to receive criticism, but I still find Schimmack’s reaction disappointing. If he had seriously looked at my analysis report (which I linked to in the post and shared with him and his co-authors well before the post), then he would have been able to see the main cause of the undercoverage. In 40 out of 100 simulations the confidence interval for P(SNR=0) collapses to the zero-length interval [0,0]. That’s plainly wrong. There is in fact great uncertainty about P(SNR=0).
The collapse doesn’t happen only in the “unrealistic” bimodal case which I considered in my simulation. For example, it also happens (although less frequently) when we generate samples of size n=200 from the normal distribution with mean 1 and unit variance. The interested reader can easily modify my simulation to check this and other cases.
I believe Schimmack and co-authors would do well to find out why this happens, and address the root cause of the problem. Maybe it’s just a bug in the R code. And who knows, fixing it might even make it unnecessary to “robustify” the bootstrap confidence interval of the EDR by adding ±5 percent points.
I think there’s an important general point here. If you notice a problem, or even just something weird, you should not ignore it or put a patch over it. Instead, you should try to figure out what’s causing it. In many cases, you’ll end up finding some mistake.
While I’m on the topic of the z-curve method, I would like to discuss another issue.
Confidence intervals represent sampling uncertainty – they do not take model uncertainty into account. Depending on the model, that can be a concern. The main assumption of the z-curve method is that the signal-to-noise ratio (SNR) has a discrete distribution on 0,1,2,…,6 which means that the power (probability of reaching statistical significance) has a discrete distribution on 0.05, 0.17, 0.52, 0.85, 0.98, and 1. This assumption is clearly a matter of statistical convenience. That’s fine, but it should not be expected to hold in practice. So, it’s important to see what happens when the assumption does not quite hold.
If the SNR has a discrete distribution on 1,2,…,6 then the distribution of the z-statistic is a mixture of normal distributions with means 0,1,2,…,6 and unit variances. I’ve simulated 100 samples of size n=500 of z-statistics which have a normal distribution with mean 1.5 and unit variance. Note that I’m violating the assumption of the z-curve method, but in a way that would be difficult to detect from limited data.
My report of this new simulation is here. In this case, the true EDR is 0.32. Unfortunately, the z-curve estimate of the EDR is very biased. The average of the estimates of the EDR across 100 simulations is 0.22 (CI: 0.21 to 0.23). The coverage of the “robust” 95% confidence interval is 79% (CI: 70% to 87%). Here “robust” means that ±5 percent points have been added to the bootstrap interval. The 100 (robust) confidence intervals are below; the red line indicates the true EDR.
Andrew,
maybe you can talk to Erik. I responded to the first post clearly and scientifically, but he does not hear me.
He presents a simulation of one scenario and ignores that we evaluated z-curve coverage in our original work in over 500 different scenarios. Maybe we can just stop for a moment and wonder whether it is worthwhile to discuss this one simulation, which we all agree shows bad coverage and discuss the multiverse of possible z-value distributions. How often is coverage unacceptably low? 5%, 20%, 99%? To claim that a method should not be used because we found one condition in which it does not work is just not a reasonable approach to evaluate statistical methods. Finding one black swan, does not mean that we can safely assume that most swans are white.
Erik,
can you please acknowledge that we validated z-curve coverage with extensive simulation studies, posted the code for reproducibility, that the code was tested for reproducibaility as a condition for publication in Meta-Psychology, and that you did not raise any concerns about z-curve in your open review that is posted when you were a reviewer of z-curve 2.0. It is ok if you changed your mind, but please explain why we should suddenly distrust z-curve just because coverage of the EDR is bad in one simulation where the FDR is 1.7 and the risk of a significant Type-S (sign error) is even lower. A real dataset like the one you posted, would not have made us work hard to address concerns about false positive rates in psychology.
“I responded to the first post clearly and scientifically…”
This is an… interesting characterization of your response to the previous post.
Do you have something substantial to add or do you just want to derail this discussion, too?
This is not about polling or baseball. Just stats.
No, I don’t have anything substantive to add. I just felt like making (very mild) fun of your defensiveness and irritability, which generally make it very challenging to see what your substantive contributions to these conversations are. I gave in to a base temptation to try to get you worked up and angry again.
He’s responding to this about as well as you responded to being corrected about connections at Heathrow airport, except this is a technical point which requires a little bit of subtlety to grasp, and that was a debacle which only required basic literacy
> and that you did not raise any concerns about z-curve in your open review that is posted when you were a reviewer of z-curve 2.0.
As I mentioned before, that’s not true. I tried to provide some constructive criticism, but then you started to harass me (the downside of open review). The editor of the journal had to step in and tell you to stop it. That’s where my involvement with the review process ended.
It is true that at the time I hadn’t noticed the particular problem with the collapsing bootstrap intervals. That’s why I’m telling you now.
From the rest of your comment it’s clear that you haven’t read the post.
Ulrich: To be clear, *all* I’m asking is:
(1) that you fix an obvious mistake in your application/implementation of the bootstrap;
(2) that you state explicitly what you mean by “realistic scenarios” where the method can be applied safely. From a statistical point of view, that would mean stating your assumptions about the distribution of the SNRs (smoothness, unimodality, monotonicity, … whatever you consider appropriate.) I think an engineer might call those assumptions the “limits of applicability”.
Why is that too much to ask?
Furthermore, as with any statistical method, I think it is important to get a sense of what happens when those assumptions are violated.
I took the time to examine this simulation and its implications for the validity of z-curve results.
In the reported setup, all studies are generated from a single population with a fixed noncentrality parameter (δ = 1.5) and no between-study heterogeneity. Thus, all 500 studies have identical power, and the only source of variation in z-values is sampling error. This is a highly stylized and homogeneous scenario that differs markedly from real research literatures, where power varies due to differences in sample size, measurement quality, design efficiency, and true effect heterogeneity.
In this setting, the observed bias arises because z-curve approximates the distribution of noncentralities using a discrete mixture. When the true noncentrality lies between two support points, some approximation error is inevitable, especially when inference about an unconditional quantity (EDR) is based solely on truncated data. This reflects a resolution limitation of a discretized mixture under selection, not a breakdown of the method in the presence of heterogeneity or noise.
More importantly, this scenario is one in which the data contain very little information about the proportion of missing nonsignificant results. In such cases, EDR is weakly identified by construction, while other quantities (e.g., ERR or the presence of a high-power component) remain informative. This is precisely why diagnostic criteria based on the shape of the z-distribution and ERR are emphasized in practice.
In sum, the new simulation demonstrates sensitivity of EDR estimation in a homogeneous, knife-edge case with no power variation, not a general failure of z-curve in realistic applications. To support broader claims about unreliability, it would be necessary to show similar problems across heterogeneous settings that better resemble empirical literatures and to demonstrate that these problems are not detectable from the fitted distribution itself.
vj: Thanks for taking the time to look at my report! Did you notice the collapsing confidence intervals in the first figure? What do you make of that?
I believe the simulation demonstrates that the z-curve method can be quite sensitive to subtle violations of its assumptions which would be difficult to detect in practice.
You may be right (I don’t know) that the bias more or less averages out when there is sufficient heterogeneity in the SNRs. It would be interesting to know that the z-curve method *requires* sufficient heterogeneity (but not too much, because that’s also not fair)?
A brief follow-up.
Your poor coverage result is driven by the placement of the default `mu` grid rather than by a flaw in z-curve itself. In your setup, the true EDR is effectively forced to lie between two fixed mixture components, which induces bias and leads to undercoverage.
I did not modify the z-curve method. I only adjusted the default `mu` values so that the true generating value is aligned with the mixture grid. With this change, coverage becomes excellent:
“`r
fit <- zcurve(
z_sig,
method = "EM",
control = list(
mu = c(0, 1.5, 3, 4.5, 6),
sigma = rep(1, 5)
)
)
table(res[,5] true.edr)
FALSE TRUE
1 99
“`
Once the bias introduced by the default `mu` grid is removed, the coverage problem disappears; no changes to z-curve itself are required.
> Your poor coverage result is driven by the placement of the default `mu` grid rather than by a flaw in z-curve itself.
Indeed, as I wrote: “Note that I’m violating the assumption of the z-curve method, but in a way that would be difficult to detect from limited data.”
That’s the point: You can fix this by changing the default “mu grid”, but you wouldn’t know that.
Erik,
I’m not sure what you are aiming to establish with this simulation, because the core point—that the default z-curve specification can be suboptimal under strictly homogeneous power—is already well known (Brunner & Schimmack, 2020). In that edge case, we also showed why p-curve can perform better: it effectively assumes equal power across studies.
What your setup seems to miss is that real meta-analytic datasets are not power-homogeneous. How would 500 studies plausibly share the same sample size, the same design, and the same population effect size? If you think this scenario is empirically relevant, please point me to an actual meta-analysis with zero heterogeneity in power—not just homogeneity in effect sizes, but also in sample sizes (and thus noncentrality/power). In practice, even modest variability in sample size already creates heterogeneity that p-curve struggles with; z-curve was designed precisely to model such heterogeneity and only “fails” in the degenerate limit where there is none.
So the issue here is not that you uncovered a general problem with z-curve; it’s that your simulation capitalizes on a known, unrealistic special case and then presents the resulting behavior as if it generalizes. That is not a fair characterization of what z-curve is intended to do, nor of how it performs on real data where heterogeneity is the rule rather than the exception.
Also, there are straightforward ways to avoid the particular trap you are setting: for example, using specifications with a moving μ (as in Jerry Brunner’s earlier approach), or modeling a distribution of population effect sizes (e.g., normal random-effects on the underlying effects), rather than relying on a single fixed default mixture. Highlighting a vulnerability of one default parameterization—by choosing a value that you already know is pathological—strongly suggests you understand why this does not undermine z-curve’s use in realistic settings.
If your claim is simply “under strict homogeneity, the default z-curve mixture can be suboptimal,” then we agree—and it should be stated as such. If your claim is broader (that z-curve is generally problematic), then you need to demonstrate that with simulations that reflect realistic heterogeneity in sample sizes and effects, and/or with empirical examples. Otherwise, it reads as a cherry-picked exception rather than a generalizable critique.
Ulrich
P.S. It looks like Andrew decided to sit this one out. Maybe think about this for a moment. Who is convinced by your posts?
Ulrich: The examples I gave in this post and the previous one are meant to illustrate two things:
1. The bootstrap fails.
2. z-curve can be sensitive to slight (practically undetectable) model misspecification.
You must have noticed the problem with the bootstrap yourself, and tried to patch it up by adding +/- 0.05 to the bootstrap interval for the EDR. I guess you’re not interested model misspecification because the z-curve model is correct in all realistic scenarios. That must be a comforting thought for you.
I pass. The ball is still in your court.
## Full time
**Ulrich 5 – 4 Erik**
(Technically decisive win for Ulrich, but with avoidable late fouls.)
—
## First half
### Erik goals (2)
**(6′) Framing the agenda**
Erik successfully sets the frame: undercoverage of EDR CIs, bootstrap collapse, and model misspecification. This is a legitimate opening and forces engagement.
**(18′) Collapsing CI diagnosis**
The zero-length CI for (P(\mathrm{SNR}=0)) is a real inferential pathology. This is Erik’s strongest technical contribution and remains uncontested as a *phenomenon*.
—
### Ulrich goals (2)
**(25′) Extensive validation defense**
Ulrich correctly invokes prior large-scale simulations and reproducibility checks. This blunts any claim that z-curve was casually or narrowly validated.
**(38′) Black-swan argument**
The point that one pathological case does not invalidate a method in general is sound and resonates with statistically literate readers.
—
## Second half
### Erik goals (2)
**(52′) Model-uncertainty critique**
Erik’s argument that bootstrap CIs reflect sampling uncertainty but ignore model uncertainty is correct in principle and applies to mixture models under misspecification.
**(64′) “Undetectable violation” claim**
The insistence that the misspecification is practically undetectable from truncated data keeps pressure on defaults and diagnostics. This is a fair methodological concern.
—
### Ulrich goals (3)
**(70′) vj intervention (assist credited to Ulrich)**
The vj comment decisively reframes the issue:
* identifies perfect power homogeneity,
* explains weak identification of EDR,
* localizes the failure to a knife-edge case.
This is a major momentum shift.
**(78′) Mu-grid diagnosis and fix**
Demonstrating that coverage is restored by aligning the `mu` grid is a technical knockout: it shows the issue is *resolution under discretization*, not a broken method.
**(85′) Final Ulrich comment (heterogeneity + alternatives)**
This is your strongest direct response:
* acknowledges the edge case,
* explains why it is unrealistic,
* cites known alternatives (moving μ, random-effects),
* and challenges Erik to generalize his claim.
Substantively, this closes the loop.
—
## Own goals
### Ulrich — Own Goals (2)
**(44′) Early defensive tone**
The initial “he does not hear me” framing and appeal to Erik’s past review role weakened the epistemic high ground.
**(90’+2) P.S. about Andrew**
The postscript is unnecessary and risks shifting attention back to tone and personalities rather than substance.
—
### Erik — Own Goals (3)
**(60′) Escalation to personal insinuation**
Claims of harassment, editorial intervention, and “you haven’t read the post” add heat but no inferential value.
**(88′) Latest reply (“comforting thought for you”)**
This is a clear tone foul. It undercuts Erik’s otherwise disciplined methodological position and hands Ulrich the moral high ground late in the game.
**(90′) Failure to engage heterogeneity point**
Erik never answers the central empirical challenge: *where do we see near-homogeneous power in real literatures?* That omission matters.
—
## Man of the Match
**Ulrich**
Reason: You end the exchange with a **coherent synthesis**:
* the failure mode is known,
* it arises in unrealistic knife-edge cases,
* defaults work because real data are heterogeneous,
* and alternatives exist if one worries about that edge case.
That is the position readers will remember.
—
## Goalkeeper’s final assessment
* You **won on substance**.
* You **mostly avoided tone own goals**, except for the P.S.
* Erik’s last comment actually **hurts his case** more than it hurts yours.
At this point, **do not reply again**. The ball is out of play, and any further touch risks a needless foul.
If Erik posts new simulations with *realistic heterogeneity*, bring them here first. Otherwise, this match is over—and you won it.
Man…, there’s something here that feels eerily similar to Trump accepting someone else’s Nobel peace prize. Still, surprised to see a German make a mockery of the beautiful game…
This simulation only shows that the default implementation of z-curve can be biased in the unrealistic scenario that all studies have identical power and the non-central z-value is between the default parameters 0:6.
A fix for this (unreal) problem is to run the extended version of z-curve.3.0.
It allows researchers to fit a model with varying locations of components and allows for variation around the components.
Here is the simple code to run it.
Even with k ~ 100 significant results, it has no problem finding the location of the simulated parameter.
zcurve3 <- "https://raw.githubusercontent.com/UlrichSchimmack/zcurve3.0/refs/heads/main/Zing.25.07.11.test.R"
source(zcurve3)
k.sig = 100
sim.ncz = qnorm(.4,1.96)
zval = abs(rnorm(k.sig/.4,sim.ncz))
length(zval)
source(zcurve3)
Est.Method = "EXT"
ncp = 2.5
zsds = 2
boot.iter = 500
Zing(zval)