Christiaan de Leeuw writes:

I write to you with a question about the construction of informative priors in Bayesian analysis. Since most Bayesians at the statistics department here are more of the ‘Objective’ Bayes persuasion, I wanted some outside opinions as well.

I am now working on my master’s thesis project. My interest is in Bayesian statistics using informative priors, and the goal of my thesis project is to develop (the basis for) a method for constructing such priors using published results of earlier studies (in this case, specifically for linear regression models, where only reported results and not the data sets are available). This seemed like an obvious source of prior information to me, but when I searched through the statistical literature I found virtually nothing on the subject. Though I found numerous instances of researchers using existing literature in some way when specifying priors, this was almost always in a seemingly rather informal and ad hoc way. I could not find any attempts to systematically combine results from several existing studies to obtain informative priors. Searching through the literature on (Bayesian) meta-analysis similarly yielded very little of relevance to this issue.

My question is therefore the following: why is there so little literature on systematically combining existing results from earlier studies (to obtain priors, or more generally as a meta-analysis)? I did quite extensive searches so it seems unlikely that it is there and I just missed it. It also doesn’t seem like a trivial problem to me. Even if the informed consensus is that it is for some reason not worth the effort, I still would expected to find some papers on the subject. Consequently I am thoroughly puzzled as to why I can’t find anything, what I am overlooking.

My reply: I know what you mean. For our 1996 article, Frederic Bois and I constructed prior distributions based on the medical literature, but as you put it, we did it in an ad hoc way. My main advice on this point is that a good parameterization can help: if the parameters mean something, and if their meaning transfers well across people, they it’s a more reasonable task to try to put together a prior distribution.

More generally, yes, there is a literature on meta-analysis–we even have a couple examples in Bayesian Data Analysis. The general idea, as formulated by Rubin and others a few decades ago, is to set up a group-level regression model. I don’t think that there’s any consensus that it’s “not worth the effort” (as you put it). Maybe you’re not looking in the right places.

I think your correspondent might want to look at the power prior literature; he can start with http://ba.stat.cmu.edu/journal/2006/vol01/issue03…

This reminds me strongly of work some of my colleagues in Bristol are doing on with collaborators including Doug Altman, who came up with the acronym BRANDO for the project – "Bias in Randomised AND Observational studies". Rather than simply seeking informative priors for treatment effects and variance parameters, they are also obtaining priors on the degree of bias of studies categorized as having a possible flaw in their conduct, allowing statistically appropriate downweighting and bias-correction of such evidence rather than having to make a binary decision to include or exclude such potentially biased evidence. The work is still ongoing so there's not much published yet, but you can get the idea from these slides of a talk by Jonathan Sterne:

Can we detect bias in the results of trials and meta-analyses? Can we correct it?

– in particular slides 14 onwards.

There's also a paper on the methodology in JRSSA:

Welton, N.J. Ades, A.E., Carlin, J.B., Altman, D.G. and Sterne, J.A.C. (2009) Models for potentially

biased evidence in meta-analysis using empirically based priors. Journal of the Royal Statistical Society Series A.

172:119-136.