This one’s important.
Matt Lerner points us to this report by Rosie Bettle, Replicability & Generalisability: A Guide to CEA discounts.
“CEA” is cost-effectiveness analysis, and by “discounts” they mean what we’ve called the Edlin factor—“discount” is a better name than factor, because it’s a number that should be between 0 and 1, it’s what you should multiply a point estimate by to adjust for inevitable upward biases in reported effect-size estimates, issues discussed here and here, for example.
It’s pleasant to see some of my ideas being used for a practical purpose. I would just add that type M and type S errors should be lower for Bayesian inferences than for raw inferences that have not been partially pooled toward a reasonable prior model.
Also, regarding empirical estimation of adjustment factors, I recommend looking at the work of Erik van Zwet et al; here are some links:
– What’s a good default prior for regression coefficients? A default Edlin factor of 1/2?
– How large is the underlying coefficient? An application of the Edlin factor to that claim that “Cash Aid to Poor Mothers Increases Brain Activity in Babies”
– The Shrinkage Trilogy: How to be Bayesian when analyzing simple experiments
– Erik van Zwet explains the Shrinkage Trilogy
– The significance filter, the winner’s curse and the need to shrink
– Bayesians moving from defense to offense: “I really think it’s kind of irresponsible now not to use the information from all those thousands of medical trials that came before. Is that very radical?”
– Explaining that line, “Bayesians moving from defense to offense”
I’m excited about the application of these ideas to policy analysis.
Hey, I’m bummed that nobody commented on this one. It’s an important topic, and it’s missing from most treatments of statistics, even for policy analysis.
Hi Andrew, I am excited to read the blog on the reliability and generalizability of CEA analysis.
There is one aspect of generalizability issue missing in Rosie’s report: the generalization of RCT data to 10-year or longer life horizon. Payers usually ask for a temporal extrapolation to estimate the QALY gain. The health economists use markov model or parametric survival model based on observed data. However, they usually do not explicitly report the assumptions about the waning of treatment effects and sometimes even ignore it.
Another problem in CEA is the implicit bias rooted in conflict of interest. In a CEA analysis, there is plenty of room for model manipulation from the choice of model form to the individual assumption of parameters. For practical reasons, not all modeling details can be prespecified in a SAP, such as whether to include certain type of AE events. For readers, everything in the model seems justified except that the current model may be cherry-picked from hundreds of unreported ones. Sensitivity analyses are done but we know it only evaluated the sensitivity of the pick model not the unreported. Since the majority of CEA analysis of drugs are conducted internally or sponsored by pharmaceutical companies, this problem is widespread. Is there anything we can do from the statistical communities?