As a Bayesian I want scientists to report their data non-Bayesianly

Philipp Doebler writes:

I was quite happy that recently you shared some thoughts of yours and others on meta-analysis. I especially liked the slides by Chris Schmid that you linked from your blog. A large portion of my work deals with meta-analysis and I am also fond of using Bayesian methods (actually two of the projects I am working on are very Bayesian), though I can not say I have opinions with respect to the underlying philosophy. I would say though, that I do share your view that there are good reasons to use informative priors.

The reason I am writing to you is that this leads to the following dilemma, which is puzzling me. Say a number of scientists conduct similar studies over the years and all of them did this in a Bayesian fashion. If each of the groups used informative priors based on the research of existing groups the priors could become more and more informative over the years, since more and more is known over the subject. At least in smallish studies these priors will have an impact on the conclusion, and the impact will increase with time. The worst case might be, that a) there is a form of regression to the mean of outcomes the individual studies and b) the variance of the effect sizes are smaller due to the highly informative priors. In some sense each of the primary studies is boosting its sample size by using informative priors.

While this all makes perfect sense on the level of the primary studies, on the meta-analytic level the studies look as if they had achieved more precise estimates then they actually have and also there might be less heterogeneity observed than there really is. One could even say, that the newer studies are forstalling the meta-analysis.

I am not sure if the above leads to the advice to use non-informative priors on the primary study level, so that primary study level outcomes are not influenced by other studies, or if the above only underlines the need to report outcomes in primary studies for more than one prior.

I would welcome to know your views on this.

My reply:

Yes, I agree that in a meta-analysis you want to have the original data. It is difficult to combine posterior distributions as there is the risk of counting some information multiple times. Sometimes I say, As a Bayesian I want scientists to report their data non-Bayesianly.

14 thoughts on “As a Bayesian I want scientists to report their data non-Bayesianly

  1. Assuming you can get the full dimensional priors and posteriors for each study in the meta-analysis you _can_ divide posterior/prior (with-in the studies) and get the relative surprise that is just a re-scaling of the likelihoods you want (i.e. it’s invariant to the prior as long as you have the full dimension).

  2. As a scientist (and engineer) I want people to simply upload their raw data to a data archive and let me grab it and work with it. I would particularly welcome a SQLite database since it’s a free and powerful format for archiving related data and metadata together. I realize that in settings with privacy concerns this can be difficult to achieve (such as medicine) but it’s a problem we should all be working to solve. Where’s the “ArXive” for datasets?

    • Daniel:

      Don’t worry, I’ll post all my statistically significant results to your database. The other 95% of my experiments are pretty noisy, you wouldn’t be interested in them anyway . . . .

        • Daniel: The only problem is that when I have occasionally obtained both the published results and the raw data, the results cannot not be reproduced.

          Now which is less wrong?

          In my cases, which involved clinical research, our judgement was the published results (due to sloppy data archiving after a bolus of concerted and careful effort to produce the final analysis.) Similar stuff happened in micro-array research field but with a different judgement of which was best.

          Now, what about changing the assumptions (required sufficient statistics) or doing other analyses not reported in the publication?

          But here you do not need to worry because
          1. Andrew’s group is very careful about making their analyses reproducible.
          2. He has told you the selection rule he has used so (albeit inefficient) you can be sure about your selection modeling (that aspect of the model, the selection rule, won’t be wrong.)

        • Having the data lets us ask questions that the original analysis didn’t even imagine.

          In engineering this is perhaps less controversial. For example someone does an experiment on say soil-pile interactions during an earthquake, collecting timeseries data on various loadings during the simulated earthquake, and answers a question that they have about say friction…. later someone grabs the same data and can answer questions about say fluid flow or wave reflections between layers of impedance changes…

          Or, to take a more statistical example, perhaps someone crushes ten thousand concrete cylinders in the process of doing quality control on concrete, and later someone wants to do forensic analysis of bridge failures during an earthquake, and goes back to the concrete cylinders and looks for patterns in the variability of strength… I dunno I’m making it up, but having a lot of data out there is more useful than having a bunch of summaries of what one person thought to ask about their one data set.

  3. I think everyone should publish likelihood functions, not posteriors; no matter what your priors, you want to update your beliefs with the study’s likelihood function. I disagree with KO in the sense that you also want the *value* of the likelihood, not just the shape of it, no? I put some of this into my (exceedingly astronomical) note “Telescopes don’t make catalogs” with Lang: http://arxiv.org/abs/1008.0738 .

    Posteriors are personal. Likelihoods are forever!

    • David (and perhaps Daniel too):

      The likelihood is completely determined by the particular probability model assumed to have generated the observations (i.e. logistic versus probit model) and the multivariate function of the observations that is sufficient (to determine the likelihood FUNCTION!!!). I’ll explain the rudeness in yelling function.

      That function – the likelihood –
      1. Does allow one to test the fit of the probability model to observations.
      2. Does not encode sufficient information to obtain the likelihood for other assumed models.
      3. Any scalar multiple of that likelihood function is sufficient to obtain the likelihood for the assumed model (and most authors define it only up to a scalar multiple anyways.)
      4. Offers little advantage over archiving the data (at the group level if needed for ethical concerns) given today’s technology.

      Now it might be interesting to recall that Fisher recommended the publishing of likelihood functions when there were technical advantages over data archiving but largely avoiding considering the possibility that the assumed model was wrong. On the other hand, Laplace abandoned his Bayesian approach out of concerns the likelihood was often surely too wrong.

      But now the rudeness about yelling “function!”

      It seems, most frequentists only think of likelihood as a means to get a good point estimate and standard error (i.e. first, second and fourth terms of the Taylor series approximation to the likelihood function) but never learn (or forget) about those other was of using the function in inference.

      On the other hand, Bayesians seem think of likelihood as something one needs to simply use in the coding of MCMC samplers – but can be disregarded otherwise. (I once asked someone who was involved in WinBugs where the likelihood could be found and they said it was not directly used. Too bad the “model” statement in WinBugs was not named the “likelihood: statement because that is what it usually is.) In ABC, it’s not needed, but can be recovered by posterior/prior.

      Unfortunately this is confounded by those likelihoodlums, those nominalists who believe everything is relative, who argue that the likelihood provides a measure of relative plausibility and that is all ones needs.

      (Seems similar is ways to subjectivists like Kadane, where posteriors are _relative_ to whoever specified the prior and therefore should not be questioned or checked.)

      • K: No disagreements here; not sure if you were intending to disagree with what I wrote, but I don’t see any conflict.

        A: Notice I said “forever” but not “universal”.

        • oh wait maybe I have a small disagreement with K on the value of the likelihood function; is there really *no* situation in which the overall scale matters? It seems to me that there is, especially if there are multiple function spaces or parameter spaces being tested.

          Okay, enough commenting for me for one day.

Comments are closed.