This is Erik: A few weeks ago, Andrew blogged about a paper by Richard Morey and Clint Davis-Stober entitled “On the poor statistical properties of the P-curve meta-analytic procedure”. Andrew quoted Morey:
We make the point that many of these techniques were never vetted by experts, and often are just “verified” by a few simulations. For tests, this is not good enough, but nevertheless these methods can get popular because (in my opinion) they tell people what they want to hear.
I believe that another meta-analytic method called z-curve (Brunner and Schimmack (2020), Bartos and Schimmack (2022), Schimmack and Bartos (2023)) has similar problems.
Recall that the signal-to-noise ratio (SNR) in statistics is the ratio of the true effect to the standard error of its estimator. If we make the “usual assumptions” then the z-statistic (the estimator divided by its standard error) has the normal distribution with mean SNR and standard deviation 1.
If we have a collection of studies, then the distribution of the z-statistics is the convolution (sum) of the distribution of the SNRs of the studies and the standard normal distribution. If we’ve estimated the distribution of the z-statistics, we can get the distribution of the SNRs by deconvolution. Deconvolution is known to be very unstable. That means that we need very many data points (studies) or very strong assumptions – preferably both – to get an accurate result.
The z-curve method is based on the assumption that the absolute values of the SNRs have a discrete distribution supported on 0,1,2,…, 6. Note that SNR=0 corresponds to “null effects”. To circumvent the effects of selection on statistical significance, z-curve uses only the absolute values of the z-statistics which exceed 1.96 in magnitude to estimate the 7 probabilities. Deconvolution is bad enough, but it gets much worse if only such a small part of the data is used. This makes uncertainty quantification especially important.
The z-curve method as implemented in the R package zcurve provides (among other things) estimates and confidence intervals of the expected discovery rate (EDR) and the expected replicability rate (ERR). I believe these are defined in my terminology as
- EDR=P(|z|>1.96)
- ERR=P(|z_repl| > 1.96 and z_repl × z > 0 | |z|>1.96)
The zcurve package also provides an estimate of “Soric’s FDR” but that is just a simple (monotone) transformation of the EDR.
It should be clear that z-curve’s estimate of P(SNR=0) (i.e. the proportion of “null effects”) will be especially noisy because studies with SNR=0 contribute relatively little to the significant z-statistics. Consequently, the estimate of the EDR will be very noisy too. To quantify this uncertainty, the authors use the bootstrap. By default, the zcurve function provides “robust” intervals by adding 5 percentage points to the confidence interval of the EDR and 3 percentage points to the confidence interval of the ERR. This approach is “verified” by a few simulations. Unfortunately, even the adjusted intervals do not provide correct coverage.
To illustrate the problem, I’ve done a small simulation. I generate samples of size n=100 from the two-component mixture 0.25×N(0,1) + 0.75×N(4,1). In 40 out of 100 simulations, the null component is missed entirely. In other words, P(SNR=0) is estimated to be zero. The problem is easy to see from a typical example (see the figure below). The null component is essentially “invisible” from the observations that exceed 1.96.
The consequence is that across 100 simulations, the coverage of the 95% “robust” confidence intervals is incorrect. In particular,
- The coverage of the EDR is 65% (CI: 55%-74%).
- The coverage of the ERR is 100% (CI: 96%-100%)
I shared my concerns with the authors Ulrich Schimmack, Jerry Brenner and Frantisek Bartos. Bartos responded that he generally agrees with the simulation, but notes that the coverage does come close to nominal when the sample size is increased from n=100 to n=1000. I responded that the zcurve function accepts as few as 10 significant z-statistics, and that most meta-analyses don’t have 1000 studies. Bartos wrote:
To be fair, I agree that we should’ve been explicit about the recommended sample size in the original article (and probably add a warning to the method if used with less than XXX estimates). I didn’t anticipate that people would apply z-curve to small meta-analyses. In my mind, the purpose of the tool (including our examples) is larger-scale meta-epidemiological projects.
Bartos also noted:
With respect to the simulations – although apparently imperfect – I still think that we did actually a much better job than most published methods. (…) The commonly used alternatives for the same purpose at the time were p-curve (for ERR and EDR) and Jager and Leek’s mixture model (for FDR) which both have much worse properties in my opinion. As such, I view this development as a step forward.
In my opinion, statistical methods should be reliable when their assumptions are met. I don’t think unreliable methods should be used because no better methods are available.
To avoid bias, I asked ChatGPT to evaluate this critic of z-curve.
I think the review is fair and accurate.
[rest of comment removed because we try to keep our comment threads an AI-free zone. — ed.]
Ulrich:
If you have a response to Erik’s post in your own words, we’d be happy to see it.
I read it and I approve it. After all this is statistics and my ChatGPT is not some generic one. It has learned a lot about z-curve and the review is high quality. In my own words, better than Erik’s blog post by a human with human biases and a silly comparison of z-curve with p-curve as if all the problems of p-curve apply to z-curve. Is that enough human comment?
Instead of criticizing me for being a human, it would be better if you reflected crititcally on the poor statistical properties of the z-curve method. Do you agree with your coauthor that the method is only suitable for large-scale meta-epidemiological projects? Should the R function zcurve() indeed issue a warning when there are fewer than XXX studies? What should XXX be? 500? 1000?
I tried to be nice, but if you want my own words, here you go.
The post is basically simulation hacking; choose an extreme, implausible scenario in which the method fails and then generalize to realistic scenarios without showing that the method fails in those situations. This approach is just as silly as p-curve DataColada simulation hacking to show that p-curve works with heterogeneity, which it does not (Brunner & Schimmack, 2020).
Here the simulation assumes 25% tests of true nulls and 75% tests with 98% power, a bimodal distribution. That is not a realistic scenario. Most literatures have a unimodal distribution with a mode at p = .05, and a decreasing slop (not an increasing as in this simulation).
Given his setup, it is inevitable that the significant results are dominated by the high-SNR component and the null component becomes nearly invisible after truncation at |z|>1.96. Calling z-curve “unreliable” because it cannot recover a component that the truncated data barely contain confuses lack of information with estimator failure. Any method—p-curve, selection models, mixture models—can be made to look bad under similarly contrived conditions.
The reason for the choice of such an unrealistic scenario is clear. Z-curve performs well in realistic scenarios, but Erik doesn’t want to conclude that z-curve works. So, an unrealistic scenario is found and generalized to all other scenarios. That is a questionable simulation practice, but why would we think that meta-scientists or statisticians are more ethical than applied researchers. Fortunately, we can soon get rid of the human element and let AI evaluate statistical methods. Try it. You will find out that z-curve does pretty well compared to other methods.
Uli’s point that the simulation assumes bimodality while the typical use of these assessments (including z-curve) is to a literature that can be thought of as having unimodality is correct and undermines the critique that it is an unreliable method for its intended use.
Yet, the fact that the Erik can identify an area where the method is unreliable is also important, since it isn’t some crazy unrealistic situation. Sometimes a literature may have multiple modes, and as the method proliferates, we might expect it to be applied to such situations with ill effect.
Dr. Bartos’s response was fair, but I think saying oh just get n=1000 is more like “simulation hacking” than what Erik did.
Ulrich: On the one hand, you like to emphasize that the z-curve method is very general because it makes no assumptions about the mixture weights. On the other hand, you’re upset when I give an “unrealistic” example where the method does not work. But you can’t have it both ways.
In statistics, it’s considered good practice to state clearly the assumptions under which a method may be expected to work. What are the assumptions under which the z-curve method works properly? How can those assumptions be checked in practice?
Erik, you are Duth and I am from Northern Germany,
so we are both familiar with windmills. Do you blame windmills for not working when there is no wind?
The amazing thing about z-curve is that it often can estimate the distribution of data that are not reported based on the data that are reported. This makes z-curve different from other method and we need to examine the performance not based on comparisons of models that draw inferences about the observed data.
Z-curve will not work when the distribution of z-values just below significance is driven by powerful studies to the right rather than the missing studies to the left. I could have told you so, but you figured it out by trying to find an example where z-curve does not work and then generalize from there to all applications.
Warning: If the slop is upwards, there are many high powered studies that are reported and there may be many low powered studies that are not reported. Happy? No find a real datasets that triggers that warning. I am waiting.
PS fact checked by ChatGPT with one correction
You were wrong to say z-curve is the only method that “guesses” missing studies.
Trim-and-fill also explicitly imputes missing studies, in the literal sense of estimating the size of the file drawer and adding hypothetical observations to account for it.
> Warning: If the slop is upwards, there are many high powered studies that are reported and there may be many low powered studies that are not reported.
How do you determine that slope? When should the results of the z-curve method be mistrusted?
> PS fact checked by ChatGPT with one correction: You were wrong to say z-curve is the only method that “guesses” missing studies.
Where did I say that?
The consequence is that across 100 simulations, the coverage of the 95% “robust” confidence intervals is incorrect. In particular,
The coverage of the EDR is 65% (CI: 55%-74%).
The coverage of the ERR is 100% (CI: 96%-100%)
Ok, I will add a warning to z-curve. Ignore EDR when your ERR is 100%
That takes care of your objection.
The statement is about the coverage of the confidence interval for the ERR, not about the estimate of the ERR.
Ok, that explains why it is 100%, With z = 4 it should be 98%. So, yes, don’t interpret EDR when lower CI of ERR is over 90%.
My sensible response was not posted here and the discussion degenerated quickly. Anybody interested in the scientific response to this criticism can read it on my blog that also contains a lot of additional useful information about modeling distributions of z-values that are truncated by selection bias.
https://replicationindex.com/2026/01/13/response-to-van-zwets-critic-of-our-z-curve-method/
Ulrich:
If you had a response that “was not posted here,” that’s your choice, as our comments are open.
https://replicationindex.com/2026/01/27/z-curve-only-works-on-earth/
Erik was given multiple opportunities to post critical comments about z-curve, but these comments did not engage with the substantive concerns I raised about his critique. Andrew, in turn, did not extend the same courtesy to allow a full response explaining when z-curve works well and when it does not.
For readers who are genuinely interested in the strengths and limitations of z-curve, my full responses are available on my own blog, which does not censor contributions. I explicitly offered Erik the opportunity to post his concerns there to enable a substantive and transparent discussion.
At its core, default z-curve is a selection model that assumes the presence of selection for statistical significance. If the full, unbiased set of results is available, z-curve or other models can be fit to all data. However, when selection bias is present—as it demonstrably is in much of psychological research—z-curve is designed to evaluate the credibility of the published significant results by explicitly modeling that selection process. This is precisely the context for which z-curve was developed.
Ulrich: I thought we’d be done with this by now…
You have proposed the “z-curve method” to analyze collections of z-values. This model assumes that what I call the studies’ signal-to-noise ratios (SNRs) have a discrete distribution over the numbers 0,1,…,6 and that, conditionally on the SNR of a study, the z-value has a normal distribution with mean SNR and standard deviation 1. The main parameter of interest is the “expected discover rate” (EDR). To circumvent possible publication bias, you maximize the conditional likelihood of the observed z-values given |z|>1.96.
I’ve provided 3 examples where the estimate of the EDR is very biased, and its (“robustified” bootstrap) confidence interval falls far short of nominal coverage. As far as I can tell, you do not dispute the correctness of these examples but you do claim they are “unfair”. The first one is unfair because it’s a bimodal mixture of studies with low and high SNRs, the second one is unfair because it’s too homogeneous and the third one is unfair because the number of significant z-values is too small (the R package accepts as few as 10).
My examples were meant to indicate that the z-curve method has certain limits of applicability which had not been made explicit. In other words, even when the assumed model is exactly correct, the method can fail. I even explained *why* it can fail. Instead of getting angry, your co-author Frantisek Bartos recognized that the method indeed may fail when the number of significant z-values is small, and added a warning to the z-curve R package.
You have not demonstrated problems of z-curve to evaluate the credibility of published significant results.
Your simulations do not even test performance of z-curve to do so.
I have demonstrated that clearly and I showed that you gave z-curve zero information to detect missing non-significant results in some of your simulations.
You have not responded to my objections. You are still welcome to do so, but to simply maintain that your simulations tell us something that they do not is not very reasonable or scientific.
https://replicationindex.com/2026/01/27/z-curve-only-works-on-earth/
Ulrich: You say: “I have demonstrated that clearly and I showed that you gave z-curve zero information to detect missing non-significant results in some of your simulations.”
I don’t see any demonstration of anything that has to do with my simulations in your linked post. I also don’t know what you mean that I gave z-curve “zero information”. In each of the simulations there are at least 10 significant z-values (as required by the R package) and usually more.
It’s certainly true that the conditional likelihood of the significant z-values has very little information about P(SNR=0). But that’s a problem of the z-curve approach, not of the simulations.
You say: “to simply maintain that your simulations tell us something that they do not is not very reasonable or scientific.” Again, I have no idea what you mean. What do my simulations tell us that they do not?
“your co-author Frantisek Bartos recognized that the method indeed may fail when the number of significant z-values is small”
Frantisek is a brilliant student and I am happy that we developed and validated z-curve 2.0 and the ability to estimate the EDR across a wide range of scenarios with a minimum of 100 significant results (Bartos & Schimmack, 2022).
However, nobody is perfect, and Franisek’s response here was misleading and is already being exploited in biased peer-reviews. There was no need to add a warning to the z-curve package if there are fewer than 300 significant results As we already showed in the peer-reviewed article, the 95% confidence intervals hold in most scenarios, with a few unrealistic exceptions (full homogeneity of power, meaning same population effect sizes, designs, and sample sizes in all studies).
I added a simulation with just 50 significant results that generalized this finding to sizes more common in traditional meta-analysis. Right now, concerns are limited to less than 50 studies and I would not recommend z-curve for those small sets of studies.
Your simulations all have fewer than 50 signficiant results, so we can agree on 50 as a boundary for use of z-curve.
I mainly post this here, so that biased reviewers cannot cite this blog post as evidence that z-curve requires at least 100 significant results.
An ancient proverb in Go (a board game from East Asia) says “don’t go hunting when your house is on fire.” An analogy in soccer would be everybody trying to score a goal, but nobody playing defense.
It seems this situation describes EvZ’s examination of our z-curve method: strong claims that the method is not robust and easily breaks down, demonstrated with a single simulation study.
As I pointed out in my reply, our z-curve (or call it an empirical Bayesian meta-analysis) has been validated in actual simulation studies with a full design that covers many (not all) different shapes of observed z-value distributions.
What has not been mentioned in this discussion, but should be relevant to anyone wondering about the performance of z-curve and related methods, is that EvZ created his own z-curve method (EvZ-curve) and published an article with substantive conclusions based on EvZ-curve. It is also noteworthy that Andrew (Andrew Gelman to be clear) was a co-author of this article. Here is the blog post advocating those conclusions:
Bayesians moving from defense to offense: “I really think it’s kind of irresponsible now not to use the information from all those thousands of medical trials that came before. Is that very radical?” | Statistical Modeling, Causal Inference, and Social Science
What you did there is not fundamentally different from what we are doing. You take an observed distribution of (absolute) z-values, fit a mixture model to that observed distribution, and then use model parameters to make claims about population parameters. You emphasize Type-S error; we allow for true zero effects and make claims about Type-I errors. But both approaches are mixture models for z-score distributions.
Importantly, EvZ-curve has 0 (zero) simulation studies showing that it performs well in general or under plausible deviations from its assumptions, including scenarios that may matter in medical research (e.g., selection/publication bias).
To me this looks like a double standard. Why should we be worried about our z-curve—which was tested across hundreds of simulation scenarios—while EvZ-curve is treated as authoritative with no comparable simulation-based validation?
Maybe before going hunting, check whether your house is on fire.
Ulrich: This is way off topic. It’s clear that you’re taking all this very personally, and that you want to “get back at me.” But whatever you like or don’t like about my work is irrelevant for our discussion about your z-curve method. I will say this:
There is no “EvZ-curve”. I’ve used a specific statistical model to analyze specific datasets where I thought it would be appropriate, see here for the main application to the Cochrane Database of Systematic Reviews (more than 20,000 studies). I have not suggested that others should use this model for other datasets. I have also not made an R package to facilitate its use.
I have not tried to estimate the full distribution of the z-values based on only those that exceed 1.96, and I would not recommend it.
I have not tried to use my method in the context of meta-analysis with, say, 100 studies or less.
Finally, the model I’ve used is due to Matthew Stephens (2017) False discovery rates: a new deal although he used it for a different purpose.
Hi Erik,
of course, this is personal. Are you saying it is not personal for you. Then why do you respond?
You may not like the term EvZ-curve and you may say that you just used a generic method that you fitted to some data, but as a statistician you know that doing so requires assumptions.
The method that you used to make claims about results in clinical trials makes two critically important assumptions.
1. negligible selection bias so that you can use all data and do not have to model selection bias
2. a smooth distribution of varying effect sizes that allows you to model the data with normal distributions centered at zero.
My point is only that you have not tested these assumptions or demonstrated that your application of the model to the Cochran data produces unbiased results.
> You may not like the term EvZ-curve and you may say that you just used a generic method that you fitted to some data, but as a statistician you know that doing so requires assumptions.
I’m not “saying” that, it’s literally what I did.
> 1. negligible selection bias so that you can use all data and do not have to model selection bias
Sure, and we acknowledge this in our paper: “While the Cochrane collaboration puts much effort in collecting all available studies on a particular topic (both published and unpublished), it is likely that the database suffers from at least some publication bias, file drawer effects, fishing, forking paths etc. Unfortunately, this means that the true (achieved) power is likely to be even lower and the exaggeration even greater.”
I’m surprised you worry about this though, because in your re-analysis of our data using your z-curve method, you concluded confidently: “Thus, there is no evidence of publication bias.”
> 2. a smooth distribution of varying effect sizes that allows you to model the data with normal distributions centered at zero.
This is quite evident from the histogram of the absolute z-values, see Figure 1 in our paper.
It’s a real pity that you have to take any criticism of your method so personally. It’s made a reasonable dialogue impossible from the start. I doubt any of the readers of the blog are still interested, so I think it’s best to end this discussion here. Feel free to diss me some more on your own blog.
The real problem is that you think everything you say is “reasonable”, but everything I say is “personal.”
I know you are not a psychologist, but there is no such thing as a reasonable human.
You have not conceded a single point in this discussion even when others have pushed back (Carlos, vj).
Let’s give it a try. Is there anything reasonable that I have said in this back-and-forth?
Also, besides taking this personal, I love debates and I can do this forever. It is like a chess game or a tennis match of the mind. I think I am scoring some points while also making some unforced errors, but so far z-curve looks fine and your edge-cases are either irrelevant, can be diagnosed, or addressed with z-=curve.3.0
You haven’t really moved on from just saying your posts are sufficient to justify claims that z-curve is just another p-curve, edge-cases mean we should never trust z-curve, or that failure with k = 25 significant results mean we need to warn people not to use z-curve unless they have 300 significant results. You can repeat these claims again and again, ignore my responses, and dismiss my comments as unreasonable, but that doesn’t mean your responses are reasonable.
Ulrich: Same thing every time. You come up with some comment to which I respond, and then you move on to something else without even acknowledging my response. I’m not interested in a “tennis match” and “scoring points” — I’ll leave that to Djokovic and Alcaraz. I’m sure you can keep this nonsense up forever, but I’m done.
Hi Erik,
I am a bit confused. I check your article for publication bias and it doesn’t return any hits. Where does this quote come from?
“While the Cochrane collaboration puts much effort in collecting all available studies on a particular topic (both published and unpublished), it is likely that the database suffers from at least some publication bias, file drawer effects, fishing, forking paths etc. Unfortunately, this means that the true (achieved) power is likely to be even lower and the exaggeration even greater.”
This should be something we can agree on. Is this quote in the “New Look” article or not? My bad, if I just can’t find it.
Follow the link, genius.
I mainly post this here, so that biased reviewers cannot cite this blog post as evidence that z-curve requires at least 100 significant results.
Your simulations rely on small sets of studies (sometimes 0, often less than 25). So, the question is how many results are enough for z-curve to estimate the EDR.
Our peer-reviewed article did not just do a few simulations. It actually varied sets of studies, effect sizes, heterogeneity, and false positive rates, and we validated that the conservative confidence intervals have 95% coverage even with just 100 significant results.
I published a new simulation with just 50 significant results on my blog that showed even 50 are enough. So, 25 may not be enough, but 50 are.
Any user of z-curve who has 50 significant results can trust the confidence intervals in the output.
Frantisek is a brilliant student and I am happy that we developed and validated z-curve 2.0 and the ability to estimate the EDR across a wide range of scenarios with a minimum of 100 significant results (Bartos & Schimmack, 2022).
However, nobody is perfect, and Franisek’s response here was misleading and is already being exploited in biased peer-reviews. There was no need to add a warning to the z-curve package if there are fewer than 300 significant results As we already showed in the peer-reviewed article, the 95% confidence intervals hold in most scenarios, with a few unrealistic exceptions (full homogeneity of power, meaning same population effect sizes, designs, and sample sizes in all studies).
I added a simulation with just 50 significant results that generalized this finding to sizes more common in traditional meta-analysis. Right now, concerns are limited to less than 50 studies and I would not recommend z-curve for those small sets of studies.
Your simulations all have fewer than 50 signficiant results, so we can agree on 50 as a boundary for use of z-curve.
Hi Erik,
I have seen some Z-curve analyses, which I thought were interesting and something to potentially look more into, although I haven’t really dug down to understand the method. I’m therefore writing hoping to understand things better. It seems people might use the method for two things. By your example, it seems it is not good at detecting zero-effects when there are also real ones, at least not unless the number of studied effects grow quite large. Consequently, taking an estimate of a low such proportion as proof of a low actual proportion of zero-effects does not seem appropriate. But does it also have problems the other way around? That is, if it estimates a high proportion of zero-effects, and therefore low power even though there are a lot of significant results, is this also problematic, in that it could overestimate such a proportion? Or does it just happen the other way around? I’m asking, because when I’ve seen the first author present it, he seems to more present in terms of the other usage than the one you present. Of course, I don’t really know in what way it tends to be used in the larger literature, so once again, I’m writing hoping to understand it all better, and hopefully to let other readers understand it better as well.
Mathias: It’s obviously very unstable to try to estimate the proportion of studies with zero-effects (or effects that are very small) from only those studies with z-statistics that exceed 1.96. The best you can hope for is a very wide confidence interval that’s at least correct. In some cases, like my simulation, even that goes wrong.
Hi again Erik,
You say “only those studies” but the examples I’ve seen have a large proportion of studies with |z| > 1.96, and few with |z| < 1.96. I was more thinking of if this usage is poor for detecting overreporting of sig. results. Or again, is that not how its used in the larger literature?
If you want to learn about z-curve, please check out the tutorial on replicationindex.com
Erik simulates a bimodal distribution that I haven’t seen “in the wild”. In reality we see unimodal distributions with a mode at z = 1.96 (p = .05). Erik’s concerns do not apply to these realistic scenarios and estimates of EDR and ERR work well. Of course, precision increases with larger sets of studies, but confidence intervals have good coverage even with small sets of studies, they are just wide because – as Andrew would agree – small samples are noisy and unreliable. That is also true for meta-analyses.
Mathias: The z-curve method is trying to estimate the distribution below 1.96 from observations above 1.96. That’s extremely noisy unless the sample size is huge, or you make strong assumptions (for example, monotonicity or unimodality of the distribution of the SNRs). To get a sense of the instability, it’s best to simulate a few times:
rmix = function(n,p,m,s){
d=rmultinom(n,1,p)
rnorm(n,m%*%d,s%*%d)
}
n=100
p=c(0.25,0,0,0,0.75,0,0)
m=0:6
s=rep(1,7)
z=rmix(n,p,m,s)
fit=zcurve::zcurve(z)
plot(fit)
Also change n and p to see what happens. More details are in the analysis report which I linked to in the post.
Reminds me of sampling chess matches to model the game rules. You may get a large sample without ever seeing something like en passant, or exchanging a pawn for queen/knight. But in reality these possibilities are influencing all the moves you do see.
Anon:
I don’t know how relevant your analogy is to the problem of meta-analysis, but it’s a good general point that I’d never thought of before, so thanks for sharing it!
One interesting meta-question is, how can a statistical method with such bad statistical properties still be so evidently useful? This also comes up with other methods we’ve discussed on this blog, such as p-curves, p-values, and Bayes factors: I have lots of bad things to say about such methods both in theory and in practice, but it’s my impression that they can still be helpful to working scientists.
How could this be?
One possible answer is that the statistical criticisms, while valid, are minor. Uri Simonsohn made this case regarding his p-curve method, saying that it failed in some unrealistic edge cases that were not of practical interest. Based on my reading of Erik’s above post, I don’t think this is the case with the z-curve.
Another possible answer is that the method will be used in a contingent or advisory fashion: if the method gives a reasonable result for the problem at hand, the output will be carried forward, otherwise not. I wonder if that’s what’s happening here. The z-curve may be very noisy, and when it gives results that make no sense, the user will know to ignore it.
I’m not quite sure what to say about this line of reasoning, but I think it’s important. On one hand, this sort of researcher flexibility can lead to lots of confusion (p-hacking, researcher degrees of freedom, forking paths, etc.); on the other hand, these methods can be useful data summaries if they’re not used in too restrictive a way.
I’m not sure z-curve is all that useful. I think I would prefer the histogram of the z-statistics without that blue curve.
Erik:
Agreed. Similarly, I think the world would be better if all p-values were replaced by simple z-scores. But, for various reasons, some people want the p-value. Similarly, for various reasons, some people want that blue curve. Conditional on them using that curve, it could still be useful for them–even though maybe they’d be even better off not using it at all.
In our p curve app (https://richarddmorey.github.io/pcurveAppTest) we show CDFs of (log) p values along with a curve showing the expectation under the null (with a 90% band). This seems to me at least to be a better alternative to a histogram (particularly one of p values, which compress all the evidence near 0). Under “Examples” you can find the (in)famous Simmons and Simonsohn power posing p values. The picture is pretty bleak (though this won’t answer any questions over whether these are the “right” p values to put in a graph…).
Richard, I don’t care how slow you are, but offering opinions until you do the science is not part of open science.
Until you actually evaluate z-curve, you are only revealing your human biases. Meanwhile, people interested in z-curve can evaluate it within 20 minutes by asking it good questions. Maybe statisticians can be replaced by AI fairly soon.
“Another possible answer is that the method will be used in a contingent or advisory fashion: if the method gives a reasonable result for the problem at hand, the output will be carried forward, otherwise not.”
This is one possibility we raised on our paper on the p curve – e.g. a good graph of the p values will probably tell whether the result is reasonable. Although, as we said there: if the graph validates the test, why do you need the test? Particularly if there are many possible tests—and they have properties different enough properties—how can they be validated by the same graphs?
The “I know it when I see it in a graph but I’m going to defend this *particular* test of mine” attitude also happens with e.g. Bayes factors vs p values, I suppose. What I like about graphs, though, is their multidimensionality and their humility (i.e. there’s no rule to tell you how to interpret it). It is a weird quirk of human psychology that boiling down a complex set of data to a single number is what seems to induce the *most* confidence in an inference.
Re: the Erik’s post…my impression of the Z curve after having looked at it is similar to Erik’s. I’ve had parallel conversations with the Z curve authors. Estimation of EDR is awful, because you’re effectively trying to find the expectation of a reciprocal of a random variable, and it is volatile. Estimation of ERR seems much better, but I don’t know what to do with this number, particularly when it is an average over hundreds of studies (and if it is less than that, the estimation gets much worse).
Sorry, you have not had any discussion about z-curve with me. I actually invited you to write a critic of z-curve for a serious journal with open reviews. Erik was also invited. So, why don’t you collaborate and write a critical review of z-curve and submit to Meta-Psychology (where Erik was a reviewer of z-curve 2.0 and found no problems). Then we can have a real discussion and not one based on simulation hacked unrealistic scenarios.
> where Erik was a reviewer of z-curve 2.0 and found no problems
That’s simply not true. I tried to provide some helpful criticism, but then you started to harass me. The editor of the journal had to step in and tell you to stop. That’s where my involvement with the review process ended.
For the record (i.e. anyone reading) it is indeed true that I have not had direct discussion with Schimmack. It has been mediated through Bartos. Re: an “invitation”, I don’t recall such a thing but it wouldn’t matter. I don’t like the cage-match-challenge model of academic critique. If I write about this, it will be at my own pace on my own terms (which unfortunately usually means painfully slowly).
“It is a weird quirk of human psychology that boiling down a complex set of data to a single number is what seems to induce the *most* confidence in an inference.”
Maybe social “scientist” psychology. Not hard science (physics, chemistry, molbio, climate). The p-value takes an “advisory role”, as Andrew says.
A single study is not informative. So, some quantitative integration of information is necessary, just like we do not rely on N = 1 studies to make claims about populations.
Andrew, at least read z-curve articles before you comment on it. it is really strange how you diss all meta-analyses of z-values or p-values, but then Erika and you publish an article doing just that to conclude that only 2% of studies in medicine are sign errors. What makes this method superior and why should we trust a false discovery rate of 2%, when other studies that you diss estimate it to be 13%?
Ulrich:
You write that I “diss all meta-analyses of z-values or p-values.” That’s not true. I’ve written lot about meta-analysis–we even have a meta-analysis in chapter 5 of Bayesian Data Analysis. I do think that people can make mistakes with meta-analysis, and I’ve pointed out problems I’ve seen, but that does not mean I think all meta-analyses are bad.
Regarding the question in your comment: I do not think the concept of “false discovery rate” is useful in medical studies. I’m not interested in the question of whether effects are exactly zero.
You reported one in the “z-curve” analysis with Erik. Please check your own work. 2% sign errors, those are false discoveries because a significant result had the wrong sign. Call it Type-S error, it is still a claim that only 2% are false. If you think that is not important, why did you report it?
are those meta-analyses of p or z values or effect size meta-analyses?
And how do you deal with bias in effect size meta-analyses?
Ulrich:
No, false discoveries are not the same as sign errors. The false discovery rate is defined based on the discrete model in which effects can be exactly zero. The type S error rate is defined based on the continuous model of effect sizes.
Andrew, in inferential statistics I make an error when I reject hypotheses. In a directional test, I make a sign error when I reject the null hypothesis in favor of a positive effect, but the effect is negative. Type-S / cond. on sig
If I use a two-sided test, I still make a mistake if I reject H0 and the true effect is 0 or in the opposite direction.
So, Type-S cond. on sign is nested within standard significance testing. 2% sign errors also imply 2% false rejections of H0 if we assume zero effects do not exist.
So you did claim in an published article that no more than 2% of clinical trials in Cochrane meta-analysis produce false significant results based on the distribution of z-values.
I never was interested in the p-curve test. It has always seemed pointless to me. A graph of the distribution of p-values highlights what’s important. There are many shades of grey between passing and failing a test and I would much prefer to just discuss the curve.
psyoskeptic: In most cases, I would recommend a graph of the distribution of the z-statistics instead. Just transform the (two-sided) p-values to (absolute) z-statistics via z=qnorm(1-p/2).
I suppose that would definitely improve things for cases where effects go both ways or you have larger p-value ranges in the lit. However, haven’t run into the issue for my cases.
Instead of criticizing me for being a human, it would be better if you reflected crititcally on the poor statistical properties of the z-curve method. Do you agree with your coauthor that the method is only suitable for large-scale meta-epidemiological projects? Should the R function zcurve() indeed issue a warning when there are fewer than XXX studies? What should XXX be? 500? 1000?
No, that is not he said and if he did, I disagree. Confidence intervals provide information about uncertainty and provide useful information even when they are wide. Should we trust 50 significant results, if the EDR is 10% and the 95%CI is 5% to 100%?
Ulrich,
Despite what you say in this comment, nobody here has criticized you for being a human.
I did not have a direction option to reply to Erik. So, to clarify, Erik posted
“Instead of criticizing me for being a human, it would be better if you reflected crititcally on the poor statistical properties of the z-curve method. Do you agree with your coauthor that the method is only suitable for large-scale meta-epidemiological projects? Should the R function zcurve() indeed issue a warning when there are fewer than XXX studies? What should XXX be? 500? 1000? ”
I responded to Erik:
“No, that is not he said and if he did, I disagree. Confidence intervals provide information about uncertainty and provide useful information even when they are wide. Should we trust 50 significant results, if the EDR is 10% and the 95%CI is 5% to 100%?”
In general, AI thinks this discussion is pretty shallow and without real content. Demonstrating that z-curve does not estimate EDR well in a contrived scenario with two distinct population of studies is not really telling us anything about z-curve. Instead, you may read the 2023 meta-analysis of behavioral priming and tell me how you explain the finding of a robust average effect size of d = .3, if you think meta-analysis works better than z-curve.
https://replicationindex.com/2026/01/11/priming-may-work-dai-albarracin-2023-psych-bull/
Also, still would love to hear why Erik and your’s model of z-value distribution is fine, but z-curve is not.
I wrote: “Instead of criticizing me for being a human, it would be better if you reflected crititcally on the poor statistical properties of the z-curve method.”
in resonse to Ulrich, who wrote:
“After all this is statistics and my ChatGPT is not some generic one. It has learned a lot about z-curve and the review is high quality. In my own words, better than Erik’s blog post by a human with human biases and a silly comparison of z-curve with p-curve as if all the problems of p-curve apply to z-curve.”
which I took to mean that ChatGPT is unbiased and I’m not.
> Warning: If the slop is upwards, there are many high powered studies that are reported and there may be many low powered studies that are ot reported.
How do you determine that slope? When should the results of the z-curve method be mistrusted?
Look at it. Does it go up or down?
> PS fact checked by ChatGPT with one correction: You were wrong to say z-curve is the only method that “guesses” missing studies.
Where did I say that?
I said it in ChatGPT corrected me. Z-curve is not the only method that guesses missing observations. Trim-FILL also does it, but poorly.
Can’t share pictures here, but z-curve of 38 money priming studies looks terrible.
EDR 8%, 5% to 53%
implied FDR: 60%, 5% to 100%
Slope: steep drop from 1.96 to 3. Many missing non-significant results. Estimation problem: No
Is this useful information for readers of a meta-analysis of money priming?
Anoneuoid on January 11, 2026 5:12 PM at 5:12 pm said:
Erik simulates a bimodal distribution that I haven’t seen “in the wild”. In reality we see unimodal distributions with a mode at z = 1.96 (p = .05).
Reminds me of sampling chess matches to model the game rules. You may get a large sample without ever seeing something like en passant, or exchanging a pawn for queen/knight. But in reality these possibilities are influencing all the moves you do see.
Not really. This is more like sampling swans. You see 1,000 White ones. Does not mean Black ones do not exist, but if you want to learn about swans, focusing on White ones is ok.
I would like to highlight the important work of Pek and colleagues on the z-curve forensic meta-analytic procedure forthcoming at Cognition and Emotion:
https://www.researchgate.net/publication/399027184_What_does_a_Z-curve_analysis_tell_us
This work raises concern about the statistical properties of the z-curve similar to those raised by Erik in this blogpost (as well as those raised by Morey and Davis-Stober [JASA 2025] about the highly-related p-curve forensic meta-analytic procedure and discussed on this blog).
It also raises concern about some deeper conceptual / ontological issues about the very purpose of the z-curve and related methods that have gone unaddressed by Schimmack and colleagues.
Your post makes only one false claim, but it is also short, so still F for False.
“have gone unaddressed by Schimmack and colleagues.”
This commentary is riddled with false claims as we will show in our rejoinder (here is a reprint)
https://replicationindex.com/2026/01/01/willful-incompetence-repeating-false-claims-does-not-make-them-true/
Thank you for the failing grade, Prof. Schimmack! More seriously, thank you for making me aware of your 1/1/26 blogpost.
In light of that blogspot, please allow me to amend my comment from “gone unaddressed” to “not been persuasively addressed”–notwithstanding whatever your customized ChatGPT reassures you based on your training of it.
Speaking of being “riddled with false claims,” your blogpost opens with a direct quotation attributed to Pek and colleagues that appears to have been made up out of whole cloth by you. Please apply the same standards to yourself that you seek to apply to others and do not continue to knowingly misattribute to Pek and colleagues something they never said.
If you find an error in the blog post, please let me know, ideally by commenting on the blog post.
All rebuttals start with a direct quote from Pek et al.s’ commentary, but if I didn’t quote them correctly, please let me know.
Ulrich:
Yeah, Pek et al. never wrote, “Zcurve is made by the devil,” despite your claim. More generally, this thread has long reached what Phil has memorably called “garbage time.” Time to move on. Anyone who’s interested can go to the sources and form their own judgments.
As a complete outsider to this debate, but at the same time also a practicing researcher with great concerns regarding the state of scientific inquiry, I’d just like to say that I find that the way this debate is unfolding here is quite disappointing. I get that people are passionate about their viewpoints, but honestly, some of the responses appear incredibly petulant and dogmatic. Do people realize that this blog – including the comment section – is widely read and has quite a bit of impact? And that how people respond to inquiries and criticism likely to have an impact on how their work is perceived more generally? I mean, I try to keep personality and style separated from substance as best as I can, but it does shift my priors.
Maybe it would be helpful to consider the larger issues at stake here.
Related: A ladder of responses to criticism, from the most responsible to the most destructive.
Citing the criticism ladder here is a bit of a low blow. Where are we?
Ulrich: I’ve designed a new tool for tightening and loosening pipes. I call it a pipe wrench.
Erik: It won’t loosen this pipe.
Ulrich: That’s not a pipe, that’s a rock.
Erik is demanding that Ulrich admit his tool is worthless because it doesn’t work on a rock, and Ulrich is now guilty of bad behavior?
How about maybe Erik shows us a distribution of values from the real world that also defeats the z curve? I have tried to think of that myself since Erik seems to be avoiding the question, but I am just not seeing it. Even in the most “bimodal” fields like transgenderism – where everyone seems to be either an activist or a hater – everyone gets p < .05. Maybe in a meta-analysis of Francesca Gino's p values, where she makes up really low p values for the studies that are supposed to support her hypothesis, and then just uses the null results for all the studies that are not supposed to support.
Since we started with the criticism ladder, I have Erik at around 3 or 4 in terms of responding to a perfectly valid rebuttal to his irrelevant criticism.
Matt: I didn’t “demand that Ulrich admit his tool is worthless.” I just pointed out a problem with the coverage of the EDR, and I wrote: “I think I would prefer the histogram of the z-statistics without that blue curve.”
Also, my simulation obeys the z-curve description, i.e. it’s a mixture of normals with means 0,1,…,6 and unit variances. So it’s definitely a pipe – if perhaps an unusual one.
You ask that I show a distribution of values from the real world that also defeats the z-curve. However, a collection of z-statistics from the real world is not enough to assess the performance of z-curve. We would need to know the underlying distribution of SNRs. Even worse, we would need to know that distribution without publication bias.
Erik, I have addressed your concern. I have pointed out that coverage depends on the slope of just significant z-values. I have pointed out that no method can recover the true distribution of missing non-significant results from a distribution of significant results that is determined by the observed significant results with high power (z = 4). Finally, I have pointed out that your simulation is an extreme case that is extremely unlikely to exist in real data, and that generalizing from that one scenario to all other applications of z-curve is not valid (pipes and stones).
It would be great if you would respond to this response to your criticism and explain why your simulation implies that we should be concerned about the use of z-curve. Matt was just pointing out that you haven’t responded to these counterarguments.
> I have pointed out that coverage depends on the slope of just significant z-values. (…)
When the slope is increasing, inference about missing studies is weak and should not be made.
When the slope is decreasing with a mode at the significance threshold, the data are dominated by low-power studies, and the truncated distribution contains information about unreported nonsignificant results.
First, coverage isn’t supposed to depend on recognizable properties of the data.
Second, your rule about increasing/decreasing slopes seems to be something you just came up with in this thread. At least, I don’t recall seeing it anywhere in your publications. I also don’t think it’s a sensible rule. Selection on downward slopes of the z-curve at 1.96 will induce a strong downward bias in the estimate of the EDR.
> I have pointed out that no method can recover the true distribution of missing non-significant results from a distribution of significant results
I never said that.
The issue is very simple. The simulation shows a problem in one specific scenario. It is wrong to generalize from here to all applications of z-curve, and there is a simple way to diagnose when z-curve EDR estimates will be too optimistic. Using this diagnostic criterion, many z-curve applications in psycholoy are valid because low power studies dominate and shape the distribution of significant z-values.
z-curve does not assume that significant-only data are always informative.
The informativeness of the data is revealed by the shape of the significant z distribution itself.
When the slope is increasing, inference about missing studies is weak and should not be made.
When the slope is decreasing with a mode at the significance threshold, the data are dominated by low-power studies, and the truncated distribution contains information about unreported nonsignificant results.
Critiques based on the former regime do not invalidate applications in the latter.
1. Look into the issue and, if you find there really was an error, fix it publicly and thank the person who told you about it.
What we need is a list of responses, where the criticism is false. How should the critics respond to a scientifically valid rebuttal of the critique?
Not like this: “Your post makes only one false claim, but it is also short, so still F for False.”, for example.
Your unwillingness to engage with substance is amazing.
But I guess, if you discuss a dozen random topics a day,
you just don’t have the time to actually read about z-curve.
The only question is why you even care. It is not baseball.
Ulrich:
The commenter “Andrew (yet another one)” is not me. But he might like baseball too. Baseball is cool!
how does Andrew respond to counterarguments?
1. Type I error includes Type S errors.
2. ERR is estimated well.
3. Te critique is only about EDR The slope is diagnostic and the simulation has an unrealistic slope, simulations with other slopes show z-curve estimates of EDR have good coverage.
Examining Andrew’s replies in the comments it is clear that Andrew does not engage those three counterarguments in a technical, point-by-point way. His pattern is: (i) reframe the target concept, (ii) downgrade the inferential ambition of z-curve-type procedures, and (iii) exit the discussion.
Andrew,
Saying “just look at the z-curve” is not a principled alternative to estimation. It is an appeal to informal pattern recognition in precisely the kind of setting where statistical methods exist to replace human intuition.
We do not stop at scatterplots because trained readers can “see” associations; we compute correlations and fit models because human perception is noisy, scale-dependent, and unreliable for quantification and uncertainty. Likewise, z-value plots can be informative descriptively, but they do not provide calibrated estimates of replicability, nor do they communicate uncertainty in a way that scales across users and literatures.
Appealing to “informed readers” quietly restricts the audience to statisticians, even though methods are built for users. If interpretation depends on expert visual judgment (“steep,” “flat,” “looks reasonable”), then the method has been replaced by discretion. Formal estimates like ERR exist precisely to encode what experts claim to see into quantities that are reproducible, comparable, and criticizable.
If visual inspection were sufficient, we would not need models at all. It is like saying we do not need telescopes because we can just squint to see better.
Ulrich:
I never wrote, “just look at the z-curve,” so you’re arguing with someone other than me here!
I misinterpreted your claims. So, are you then saying that z-value plots alone without EDR estimates are also uninformative or is it just the modeling with z-curve that you do not like?
On a positive note,
Andrew’s blog is superior to datacolada because it allows comments to correct false claims, while datacolada has no comment section (not open science). So, thank you for allowing me to rebuttal the arguments against z-curve here.
Ulrich:
No problem. And thank you for commenting here. Disagreement is frustrating but it’s even more frustrating when there’s disagreement that’s not expressed.
Erik:
First, coverage isn’t supposed to depend on recognizable properties of the data.
Second, your rule about increasing/decreasing slopes seems to be something you just came up with in this thread. At least, I don’t recall seeing it anywhere in your publications. I also don’t think it’s a sensible rule. Selection on downward slopes of the z-curve at 1.96 will induce a strong downward bias in the estimate of the EDR.
The issue is not new. We never bothered to put a warning because real data are not like your simulation. I can also simulate cases with outliers, etc. to make any method fail. By your criterion, all methods are flawed because they sometimes fail. Maybe they should teach logic in statistics courses as well. Can you at least acknowledge that we simulated many scenarios and that in these scenarios EDR coverage was good and your simulation in an exception?
Ulrich: When you quote me (the first two paragraphs), please clearly indicate so.
It’s also unclear why you quote me, because you’re ignoring my point that selection on the slope of the z-curve will induce bias.
> I can also simulate cases with outliers, etc. to make any method fail.
Statistical methods may fail when their assumptions are violated. For example, the t-test may fail when the data are not normal (like when there are outliers). The problem with your method is that can fail when its assumptions are met, namely that the distribution of the z-statistics is a mixture of normals with means 0,1,2,…,6 and unit variances.
> Can you at least acknowledge that we simulated many scenarios and that in these scenarios EDR coverage was good and your simulation in an exception?
*Even* in your own selected scenarios, the method doesn’t work properly because you have to add an arbitrary +/- 5 percent points to the bootstrap interval for the EDR.
I am getting concerned that we are talking past each other. It would help if you ran your interpretation by a neutral party (another statistician or an AI assistant) to check whether the disagreement is substantive or mostly about framing. I do this myself, and it helps.
I revisited our simulations, focusing for now on the 576 scenarios in my simulation universe (not František’s simulations). The situation you emphasize is effectively covered, but we did not consider it important to distinguish between 0% and ~1.7% false discoveries among significant results in that corner case. When the H1 component has very high power (around 98%), the published set is dominated by true positives, so the chance of selecting a false positive is already very small. If one focuses on the stronger results, the false-positive risk drops further. On that basis, any EDR bias introduced by approximating 1.7% as 0 seems practically negligible.
If you think that approximation is consequential, the useful next step would be to demonstrate that it changes conclusions materially across realistic regimes, rather than only in the most extreme boundary case.
Andrew, it is perfectly appropriate to host guest posts and invite discussion. My concern is narrower: you appear to accept Erik’s criticism without critically evaluating it, and you have not engaged with the counterarguments that have been raised.
Given your role in convening the discussion, I would suggest either (a) remaining explicitly neutral as the host, or (b) engaging directly with the arguments so readers can see how you assess their merits and limitations. As it stands, the impression is one of endorsement rather than facilitation.
Uncritical acceptance of Erik’s critique may be a defensible personal view, but it functions primarily as an expression of opinion. Given your standing as a leading statistician, I think readers would benefit from a more explicit, reasoned adjudication of the competing claims.
Ulrich,
You have been bullied.
The irony is that Andrew has categorically criticized what he calls “God of the Gaps” type arguments, and you can’t epitomize that concept any better than Erik has here.
This identical scenario played out on Data Colada, minus the piling on from the blog host. There, Uri Simonsohn argued:
“The authors illustrate the four properties with carefully chosen edge cases. For example, in one such edge case they consider combining six p-values that are p < 1/750 trillion, with a seventh study that is p = 0.0499997. These edge cases are good for illustrating that something could possibly happen. These edge cases are less good for assessing whether something is likely to happen or to matter.
What I think is relevant but missing from the critique is any consideration of practical consequences, answering questions like:
How likely is p-curve to get it consequentially wrong?
Does any existing tool make fewer consequential errors?
If we abandoned p-curve, and went back to assuming there is no selective reporting, would inferences be better?"
Can Erik give an affirmative answer to any of these perfectly valid questions?
> How likely is p-curve to get it consequentially wrong?
Who knows – it depends on the underlying distribution of the SNRs in a particular application. My point is that any method should come with a clear set of assumptions under which the results can be trusted. In the case of the z-curve method, that would be a set of assumptions on the distribution of the SNRs (smoothness, unimodality, monotonicity, … whatever). Then a user would be able to consider if those assumptions are likely to hold in their application.
> Does any existing tool make fewer consequential errors?
Maybe, maybe not. Probably again depends on the underlying distribution of the SNRs.
> If we abandoned p-curve, and went back to assuming there is no selective reporting, would inferences be better?”
Why would abandoning p-curve or z-curve mean going back to assuming there is no selective reporting? That makes no sense at all.
I agree. Fortunately, I am actually enjoying how Erik (and to a lesser extent Andrew) are embarrassing themselves, and everybody with a rational mind (some humans and AI) can see it. If you need a crazy scenario like this one to criticize z-curve, it probably means that all the file-drawered attempts to do so failed. Wouldn’t it be nice to have a z-curve to see Eriks’s file drawer. I mean even the same setup with k = 1000 still works, so we had to show k = 100 to claim z-curve never works in this sample of N = 1.
A lot of emotion here, but little substance.
Van Zwet showed that z‑curve’s confidence intervals can fail, but he did it using one very specific kind of data pattern: a situation where the distribution of z‑values between 2 and 4 slopes upward. That pattern means the results are dominated by very high‑power studies, which leaves almost no information about the low‑power studies that might be missing. In that situation, the model becomes weakly identified, and the confidence interval can be too narrow. His point is mathematically correct — if you feed z‑curve that kind of data, the CI can misbehave.
But that’s only half the story.
Bartoš and Schimmack, when they introduced z‑curve, didn’t test one scenario. They tested hundreds. Their simulation design systematically covered the kinds of effect sizes, heterogeneity levels, sample sizes, and selection patterns that actually show up in psychology. Across this broad, realistic space, the confidence intervals behaved well. Failures were rare and mostly confined to extreme corners of the parameter space.
And here’s the crucial practical insight: z‑curve gives you a visible warning sign when you’re in the danger zone. If the slope of the z‑value distribution from 2 to 4 is decreasing — which is the norm in real psychology data — the model is dominated by low‑power studies, and the confidence intervals have good coverage. If the slope is increasing — which is unusual — that’s when the method struggles. Van Zwet’s critique is essentially a simulation of that rare, increasing‑slope case.
So the practical conclusion is simple:
z‑curve’s confidence intervals are not universally valid (no method is), but they work well in the kinds of data researchers typically analyze. The main failure mode corresponds to a pattern that is both rare in real applications and easy to diagnose by looking at the slope of the z‑values.
If the slope goes down, you’re fine.
If it goes up, be cautious.
That’s the whole story.
I agree largely with vj
I think part of the disagreement here is that we’re mixing up two different questions: whether z-curve has a fundamental limitation, and whether van Zwet’s example actually undermines the method in practice.
To me, van Zwet’s simulation shows something fairly obvious: if the significant results contain essentially no information about the non-significant ones—because the literature is dominated by very high-power studies—then z-curve (like any method) can’t recover what’s missing. In that regime, the model is weakly identified and estimates of EDR are necessarily unstable.
What’s important, though, is that this regime is not hidden. It corresponds to a non-decreasing or increasing slope of the z-value distribution above the significance threshold. When the slope goes up, the curve is driven by the right tail (high power), not by selection against non-significant results on the left. In that case, z-curve simply shouldn’t be interpreted.
When the slope is decreasing, however, the significant results do contain information about missing studies, and that’s precisely the situation z-curve was designed for. So I don’t see van Zwet’s example as showing a general failure of z-curve, but rather as illustrating its identification boundary.
The real takeaway, in my view, is that this boundary and the role of the slope should be made explicit to users—ideally as a clear warning—rather than treating z-curve as a tool that can work in all regimes.
> If the slope goes down, you’re fine.
You seem very confident, but it’s actually not true as I demonstrate here.
While the discussion mostly settles the debate, I have one more comment that moves it forward.
Here is a revised version that explicitly incorporates your concrete example, while keeping the tone appropriate for a discussion thread and avoiding any overclaiming:
> I agree that homogeneity of the data-generating process is not identifiable in a strict sense. However, the fitted mixture weights in z-curve are still informative for **model criticism**, in the same way residuals or modification indices are used in CFA. For example, in a homogeneous simulation with a true mean around 1.5, a default z-curve specification may assign nonzero weight to adjacent components (e.g., μ = 1 and μ = 2, with weights ≈ .67 and .33, and zero weight elsewhere). In this scenario, the weight pattern is diagnostic of insufficient grid resolution rather than substantive heterogeneity and naturally motivates model re-specification (e.g., refining the grid around 1–2 or allowing freer components). This diagnose–respecify workflow is entirely standard whenever models have defaults rather than fixed, non-modifiable assumptions—analogous to CFA versus EFA—and should not be conflated with claims about recovering the true underlying distribution.
On further inspection I found that Erik’s simulation with 1.2 false positives on average often simulated scenarios without any false positives that could provide any information about the amount of non-significant results (19 for every 1 significant one with p < .05).
I asked Erik for comments, but he declined. Andrew declined to post my detailed rebuttal to Erik's misleading simulation.
You can find it here:
https://replicationindex.com/2026/01/24/on-the-interpretation-of-z-curve-coverage-in-an-extreme-simulation-scenario/
Ulrich:
I have no idea what you’re talking about when you say I declined to post something of yours. Unless you’re talking about the comment that you generated from a chatbot.
Is this your current email address:
[email protected]
Hi Andrew,
I think this short and funny blog post explains the disagreement between Erik and me and it would be great if you could post it on your blog.
Best, Uli
Z-Curve Only Works on Earth – Replicability-Index
Ulrich Schimmack
Dept. of Psychology
University of Toronto
I sent you an email:
To: Andrew Gelman [email protected]
Sat 1/24/2026 10:31 PM
Hi Andrew,
I found an error in Erik’s simulation that leads to an underestimation of the z-curve coverage even in extreme edge cases. In short, some of this simulations did not include any false positive results. In those simulations non-significant results from tests of a true H0 have no influence on observed data and z-curve is not biased.
I would like to post it as a response post rather than a comment that is ignored.
Best, Uli
On the Interpretation of Z-Curve Coverage in An Extreme Simulation Scenario – Replicability-Index
Maybe all we need to know is that when Erik writes “This approach is “verified” by a few simulations” that a few means over 500 in a full design with different levels of H0, effect sizes, heterogeneity and number of studies.
Let us just compare “those few” to Erik’s 1 edge case.
If this counts as valid criticism of a method on this blog, it should be clear that this is a fun blog to exchange opinions, but not a forum for serious scientific discussion.
Ulrich: The number of simulations (cases, scenarios) you did, says absolutely nothing about how well you’ve explored the parameter space. See also Using simulation studies to evaluate statistical methods
[long string of chatbot output which I’ve deleted. — ag]
VJ: You write: “I think both Erik and Uli are making points that are partly correct” What did I state that isn’t entirely correct?
You also write: “If the claim is that the simulation design fails to explore the relevant space, it would help to specify which dimensions are missing or which ranges are implausible” Easy: small numbers of significant z-values (say less than 100) and irregular (for example, bimodal) distributions of SNRs. It would also be appropriate to explore what happens under model misspecification.
You write: “The more substantive issue concerns the edge case itself.” Why “edge case”? Why “corner case”? Why “extreme conditions”? Even Ulrich agreed that my last example was realistic.
You write: “He also showed how this situation can be diagnosed from the observed distribution of significant z-values, rather than being a silent or hidden failure mode.” Did he? Where are the explicit diagnostic rules that a user can apply? Ulrich has merely *claimed* that he can tell when his method will work.
I think vj is generating his responses using an AI agent tbh.
Hey, I think you’re right! It’s just like that fake match report that declared Schimmack the winner.
Vj:
If you’re submitting chatbot output as blog comments, please stop. I agree with Anon that your comment looks suspiciously like a chatbot.
vj’s first comment looked less like chatbot output, but now that I look at it longer, it looks like one of those marketing linkedin posts, so I would bet it’s AI also. The second one is more obviously chatbot output.
I am real person. I am a graduate student in statistics and working on these models. I did use AI to write my comment because my English is not good, In my own words, I understand Dr. van Zwet’s concerns but also agree with Dr. Shimmack’s response that z-curve is useful in many scenarios and applications to real data.
Vj:
No chatbots here. Please use your own words.
I was just trying to say that I agree with Daniel
https://statmodeling.stat.columbia.edu/2026/01/10/concerns-about-the-z-curve-method/#comment-2408277
and Mary Brown
https://statmodeling.stat.columbia.edu/2026/01/10/concerns-about-the-z-curve-method/#comment-2408722
that the simulated scenario is an extreme bimodal mixture of a few true null hypotheses and mostly high powered studies. We cannot generalize from this example to data that are unimodal with a mixture of low, moderate, and high powered studies to say that the method is not useful in general.
> how well you’ve explored the parameter space
What parameter space? Parameters for what? (I’m not sure if you mean for each observation so we would be talking about hundreds or thousands of parameters.)
Carlos: The z-curve model has 7 parameters, namely the probabilities (“mixture weights”) of the SNR being equal to 0,1,2,…,6.
A further variable to consider in a simulation study is the sample size.
By “model misspecification” I mean (for example) that the SNR does not follow a discrete distribution on 0,1,2,…,6.
Erik, please listen, the model has discrete components.
This does not mean that we assume real data are mixtures of studies with 7 specific power values.
That also means we are aware that the model weights can differ from simulated data.
What matters is that the ERR and EDR estimates fit the true values close enough and coverage of the CI corresponds to the nominal coverage. The advantage of a discrete model is that it avoids distribution assumptions about the true power of studies.
In contrast, your model (EvZ-curve) makes distribution assumptions and you might want to test how it performs when these distribution assumptions are not met in the data.
As I said in another comment “don’t go hunting, when your house is on fire”
If I understand what you mean in this simulation, for example, the SNR does not follow a discrete distribution on 0,1,2,…,6:
https://github.com/UlrichSchimmack/zcurve3.0/blob/main/Tutorial.R.Script.Chapter6.R
sim.ncz = rnorm(200000,2,.5) # simulate the distribution of the non-central z-values
Carlos: Yes, I think “non-central z-values” (ncz) are what I call SNRs. So sampling them according to sim.ncz = rnorm(200000,2,.5) is an example of model misspecification. Of course, n=200000 is very large. I don’t know what would happen in this case when n=100 (I haven’t tried it).
My general impression is that the z-curve method works quite well for estimating the EDR (and ERR) when the sample size is very large. That is, the estimate of the EDR will be approximately unbiased and the confidence interval will approximately have the correct coverage. Of course, it will still be very wide.
When the number of significant z-values is low to moderate (say less than 100 or 200) the method can give misleading results (bias and undercoverage). So, I think it’s very sensible that Frantisek Bartos added a warning to the zcurve() function to discourage users from applying the method to typical meta-analyses.
(I just sent another comment that may be in moderation) I’m not sure about what are exactly your concerns about “model misspecification”. The very first example in https://replicationindex.com/2025/07/08/z-curve-tutorial-introduction/ is for a single value of power (40%). Maybe I’m missing your point but if the distribution is just the point 1.7 it doesn’t follow a discrete distribution on 0,1,2,…,6.
Erik, the simulations are open access on OSF, It covered common effect size ranges d = 0 to .8, no to high heterogeneity, tau = 0 to .6, sample sizes typical for psychology, 40 to 200, etc.
All of this is open access on OSF with code to examine coverage.
Plus, you were a reviewer and did not see any problems with our simulations.
So, your claim “a few” is not a factually accurate description of our simulations.
We simulated a FDR of 0 and 20%, we did not simulate FDR of 1.6%, like in your simulation. So, yes, our model can say 0 wen the true FDR is 1.6%. This may be a problem for statisticians but not for real researchers who are told most published results are false (FDR > 50%).
Hi Carlos.
thank you for your comment that mentioned the z-curve.3.0 tutorial.
https://replicationindex.com/2025/07/08/z-curve-tutorial-introduction/
Z-curve.3.0 addresses many of Erik’s concerns about z-curve.2.0, but the 40% power and z = 1.7 example is a stylized example.
It walks readers through the logic of z-curve and z-curve plots. It does not show that the default specification gives perfect estimates in that scenario. Erik’s second post shows that it does not.
The z-curve.3.0 tutorial addresses this issue by explaining how to diagnose heterogeneity and how to fit models that assume homogeneity. A homogeneous model with a single component matches the assumption in this example. It gives appropriate confidence interval coverage in that setting.
That being said, the simulated scenario that all studies have the same power is interesting, but equal power for all studies is not a meaningful assumption for real literatures. Real data always have variation in true power.
Adding a warning to z-curve.2.0 before the next package update is fine. In the meantime, users can already work with the z-curve.3.0 code.
Erik said he is done. Fine. The fact remains that he used one extreme bimodal simulation to make bold claims about the performance of z-curve and then to say “I’m not sure z-curve is all that useful.”
I responded by pointing out, plainly and scientifically, that z-curve works well in many other scenarios and that this has already been shown in actual simulation studies using realistic designs—not just one extreme and unlikely case. After all, the scenario Erik chose assumes 25% of studies with zero effects, which is not exactly consistent with how Erik and Andrew usually argue about the prevalence of true nulls when the topic is Type-I error or “false positives.”
Based on this single example, Andrew then writes: “One interesting meta-question is, how can a statistical method with such bad statistical properties still be so evidently useful?” The answer is simple: z-curve does not have bad statistical properties. If you rely on one extreme simulation and ignore the rest of the evidence, you can talk yourself into a false conclusion.
Finally, I turned the table using Erik’s (and Andrew’s) own work in the “New Look” paper, where they take an off-the-shelf mixture model developed for genomics and apply it to absolute z-scores in medical meta-analysis. That kind of transfer requires justification and assumption checking. In genomics, selection problems are often much less central. In meta-analysis, publication bias and selective reporting are core concerns. So using a model that effectively assumes no selection, and then proceeding as if there is no selection, is not good practice for meta-analysis—especially in the 2020s, when we have multiple tools to probe and model publication bias.
That really sums it up. It is easy to criticize a method using a single edge-case example. It is harder to apply the same standards—robustness checks and assumption testing—to one’s own analysis.
Cheers, now let’s go back to chatting about billionaires, baseball, and the Epstein files and leave evaluations of z-curve to serious academic efforts.
Ulrich:
Remember–no insults please. I’m referring here to the last paragraph of your post. Erik’s research, and mine, is serious. Just cos you don’t like our conclusions, that doesn’t mean that our work is not serious. And if you don’t like to hear about politics or sports, that’s your problem; I don’t need to be insulted about that either.
It is not about the conclusions. My work is also serious. It is fair for Erik to challenge z-curve, but it is equally fair for me to challenge Erik’s model. I hope you can agree that all models make assumptions that need to be made explicit, justified, and tested. It is a simple fact that this did not happen for the fitting of a mixture model to a distribution of absolute z-values in the “New Look” paper. So, yes, serious, but fair to be seriously examined, too. No insult. Just saying that calling our mixture model of z-values useless (quote) and yours serious, is not a fair comparison.
Sorry, I know we are done, but this quote from your recent blog post is just too good.
“we’re all busy and sometimes we apply models that don’t make sense or don’t fit the data, or both.”
And that is ok, if you are able to acknowledge the mistake when somebody points it out.
Yes, definitely, as here.
Andrew, you write that “Erik’s research, and mine, is serious.”
I’m not seeing that same level of seriousness in how you and Erik have engaged with our work on z-curve.
The one valid concern here is narrow: z-curve can break down when you try to estimate EDR from very small sets of studies. I took that concern seriously and did the obvious thing. I repeated our large simulation design (192 conditions varying effect size, heterogeneity, proportion of true nulls, and sample size) but this time with only 50 significant results per condition.
The takeaway is straightforward. The results are robust. The coverage is below 95% in about 5% of scenarios and occur in extreme cases where the z-value distribution shows a positive slope because the data are dominated by high-powered studies—basically the situation Erik simulated. This can be diagnosed easily and we will add a warning to alert users to this issue. Even then coverage is above 80%, not fatally low.
If you want to be serious about evaluating z-curve, I think the next step is to engage this evidence directly. Does it change your view that z-curve is broadly unreliable based on one edge case?
https://replicationindex.com/2026/02/03/concerns-about-z-curve-evidence-from-new-simulations-with-few-studies/
Ideally, this misleading blog post would be removed, but that is unlikely to happen. Therefore, I can only use the comment section to correct the false claims made by the author.
The blog post starts with an implication that z-curve is just another p-curve with similar problems.
“”This is Erik: A few weeks ago, Andrew blogged about a paper by Richard Morey and Clint Davis-Stober entitled “On the poor statistical properties of the P-curve meta-analytic procedure”. … I believe that another meta-analytic method called z-curve (Brunner and Schimmack (2020), Bartos and Schimmack (2022), Schimmack and Bartos (2023)) has similar problems.”
“This claim is factually incorrect. The criticisms by Morey and Davis-Stober target specific properties of p-curve’s probit-based test combination that do not apply to z-curve, which uses a fundamentally different approach — EM estimation on a mixture of truncated normals. For a detailed scientific discussion of the differences between p-curve and z-curve see this scientific blog post.
https://replicationindex.com/2026/03/14/is-z-curve-just-another-p-curve/