Erik van Zwet, Witold Więcek, and I write:
Effect sizes typically vary among studies of the same intervention. In a random effects meta-analysis, this variation is explicitly taken into account. However, when we have only one study, the heterogeneity remains hidden and unaccounted for. We are left to assume that the effect in the study is a fixed property of the treatment, and to expect that the same effect will be present in another study. However, this is usually not the case. We used the Cochrane Database of Systematic Reviews to estimate the distribution of the heterogeneity across more than 1,500 meta-analyses. We demonstrate how taking this information (and information about the effect sizes themselves) into account can help us better interpret the results of single trials.
And here’s what we find:
Our Bayesian “meta-analyses of single studies” perform much better than naively assuming non-varying effects. The prior on the heterogeneity results in better quantification of the uncertainty. The prior on the treatment effect reduces the mean squared error both for estimating the study-level and population-level effects by about 60% on average. Such a reduction is equivalent to more than doubling the sample size.
Everything is fully reproducible with the R code provided in a GitHub repository. The data are publicly available here.
Good stuff, I think. And these ideas should be applicable to other application areas as well.
This is really cool stuff, thanks Andrew. I’m assuming there’s a deeper connection to Type M and S errors on the effect sizes but I’ll have to take time to read the paper fully.
I’m curious about the scope of this approach, like how you think about the population of effects captured in the CDSR. Are there constraints on the target study one is running that would make using these priors more or less appropriate?
Jessica: Throwing all the trials from the CDSR on a big pile is quite crude, and I’m sure we could do better with some modelling. But I would guess it’s one of those 80-20 things where a small effort already makes the biggest difference. Using a flat prior on the effects and setting the heterogeneity to zero is *so* far removed from reality that any prior that is at least somewhat realistic is already a lot better.
Thanks, makes sense.
Thanks for posting our paper! I like the paper because it’s a nice example of how we may seem to have more certainty with less data. If we have only one trial, we can’t see the heterogeneity between multiple trials and so it’s easiest to just ignore it. When we have 10 trials, we can’t do that anymore. I think it happens quite often that we confront problems only when we can’t ignore them anymore.
This is a really excellent idea! Intuitive, logical, and useful. I think this would be particularly useful for models where you borrow priors/parameters from published research and are concerned about the narrowness of the error terms (e.g., decision analytic models). I’ll give it a try next time I have a good use case.
Robin: Thanks! Let us know if come across a good use case!
On page 2 near the bottom, should it be “… second part
(2) models the uncertainty of the estimates from each individual study”? The current version states “second part (1)”.
I’m slowly making my way through the paper, but is there an observed difference between the binary outcome studies and the numeric outcome studies? I’m not particularly familiar with the CDSR data, but I’d expect that publication bias might be different in the two types of studies.
Second, is there a correlation of the meta-analysis \mu_i and \tau_i values depending on the number of studies in the meta-analysis? I’d expect that the “smaller” meta-analyses could have more larger (in magnitude) \mu_i and \tau_i values. For example, if Figure 1 was restricted to only meta-analyses of 5, 10, 15, etc. studies (ex. if using ggplot2, analogous to using a facet_wrap(~n_studies)), would the plots differ for different sizes of meta-analyses?
Finally, are the results sensitive to the cutoff of only including meta-analyses that have 5+ studies? Obviously, this paper is trying to figure out how to evaluate n=1 “meta-analyses”, but what’s the distribution of sizes of meta-analyses in the CDSR and is n >= 5 reasonable?
Thanks.
Eric: Thank you very much for the detailed comments! I’ll just add to Witold that trials with binary endpoints tend to have lower power than those with numeric outcomes. So, I could imagine that publication bias would affect them differently. Fortunately, the CDSR does seem to suffer too much from publication bias. Maybe that’s because trials are pre-registered and expensive – I’m not sure.
Thanks, we will fix the mistake.
I don’t remember if out results change if we look at continuous vs binary outcomes, but it’s possible that Erik looked into this in the past.
There are indeed some strong and very interesting relationships between mu and tau! We have a whole set of results on that which we did not want to stuff into this one, but we will include them in a follow-up paper; n >= 5 is reasonable for this short paper, but in the future we will have much more to say about this.
Will need to take a closer look at this. Sounds closely related to the idea of using meta-meta-analysis (meta-analysis of meta-analyses) to improve the estimates of mean effect size from individual meta-analyses. Shamelessly self-promotional link: https://onlinelibrary.wiley.com/doi/10.1002/ece3.9521
On page 5, you report the 95% posterior interval for beta as 0.05 to 1.1, but report that the posterior probability that beta is positive as 0.74. Did you mean to say that the posterior interval is -0.05 to 1.1?
Brad: In a meta-analysis there are two effects to keep in mind: The true effect in a particular trial and the mean of the true effect of similar trials. The 95% interval for the true effect in the trial is 0.05 to 1.1. The mean of the true effect of similar trials is more uncertain so the 95% interval is wider; -0.6 to 1.4. The probability that it’s larger than zero is 74%.
Thanks Erik. I think I understand the distinction between beta and mu. What I was getting at in my original comment was that there may be an inconsistency in how the posterior for beta is reported.
The paper states that the 95% posterior interval for beta is [0.05, 1.1]. But then at least 97.5% of the posterior probability mass for beta must lie above 0.05, and therefore above 0. This seems inconsistent with the reported P(beta > 0) = 0.74.
I suspect either the interval should be [-0.05, 1.1], or the probability P(beta > 0) is greater than 0.975.
At any rate, great paper and looking forward to hearing more from you guys on this topic, which I think is super important for clinical medicine.
Sorry, yes, you’re right. It should be -0.5 to 1.1 for the effect in the trial. Thanks!
I am really fascinated by this idea of extending the case to a single study and for the article.
I feel that mu_i and tau_i square will be correlated and assuming independence is a bit rigid assumption, I feel so! Please correct me
Next, I have a basic question: Suppose the OR from a single RCT is 0.68 and after the Bayesian analysis using the priors you suggest, we arrive at a posterior point estimate of 0.76 so should I be using the 0.76 as the effect size in cost effectiveness models as a base-case?
Last question, Are you planning it to submit to a journal?