Petrus Mikkola, Osvaldo A. Martin, Suyog Chandramouli, Marcelo Hartmann, Oriol Abril Pla, Owen Thomas, Henri Pesonen, Jukka Corander, Aki Vehtari, Samuel Kaski, Paul-Christian Bürkner, and Arto Klami write
Specification of the prior distribution for a Bayesian model is a central part of the Bayesian workflow for data analysis, but it is often difficult even for statistical experts. Prior elicitation transforms domain knowledge of various kinds into well-defined prior distributions, and offers a solution to the prior specification problem, in principle. In practice, however, we are still fairly far from having usable prior elicitation tools that could significantly influence the way we build probabilistic models in academia and industry. We lack elicitation methods that integrate well into the Bayesian workflow and perform elicitation efficiently in terms of costs of time and effort. We even lack a comprehensive theoretical framework for understanding different facets of the prior elicitation problem.
Why are we not widely using prior elicitation? We analyze the state of the art by identifying a range of key aspects of prior knowledge elicitation, from properties of the modelling task and the nature of the priors to the form of interaction with the expert. The existing prior elicitation literature is reviewed and categorized in these terms. This allows recognizing under-studied directions in prior elicitation research, finally leading to a proposal of several new avenues to improve prior elicitation methodology.
See the full 60 page paper in arXiv
(This post is by Aki, one of the many co-authors of the paper)
Well, now I know how I’m going to be spending my weekend. :-)
Aki,
From the linked paper: “If you are not choosing your priors yourself, then someone else is inevitably doing it for you.”
Well put! I’d only add that you could change the word “priors” to “models.” It is perhaps merely an accident of history that skeptics and subjectivists alike strain on the gnat of the prior distribution while swallowing the camel that is the likelihood.
Anyway, I’m glad you wrote this paper. I’m embarrassed to say that we discussed this topic very little in any of the editions of Bayesian Data Analysis.
I’m not a Rush superfan by any means, but I’ve always been amazed how well Neil Peart’s lyrics apply to statistical inference…
If you choose not to decide, you still have made a choice!
Jack:
We get the Rush fans on the blog so I don’t feel like such a dork for being a fan of dad-rock REM.
Aki,
Also I have some thoughts on probability elicitation in these slides from 2003; we can discuss more at some point if you’re interested.
This is very valuable. Would be good to wrap these points into a paper. There’s too much confusion around what the decision constitutes and how it relates to inference. As you said, it is not about “false positives”.
There is a big literature in economics on eliciting probabilistic beliefs. See “A penny for your thoughts: A survey of methods for eliciting beliefs” by Schlag et al. (2015).
As I recall, the difficulty of eliciting knowledge from experts also plagued AI people building “expert systems” 3 decades ago.
Yea, but the think out loud protocols did prosper – https://en.wikipedia.org/wiki/Think_aloud_protocol
(My advisor in MBA school was a Phd student with Herb Simon and we did a project together using tapes of women thinking out loud while shopping for clothes. The interest was in setting up online shopping. It was 1982 and as always being ahead of the times ain’t that good for careers.)
I see things differeny. NHST is essentially prior elicitation. It measures the collective prior weighted by funding (required for large sample size and precise measurements).
So, prior elicitation is in fact very widely used. It is actually the primary function of modern research, and has been for some time.
We now need to move on to the next step of collecting reliable measurements. Then we can start guessing models capable of explaining the observations, and finally derive surprising predictions to check the ones that work.
Having heard Jay Kadane in about 1982 describe work on prior elicitation for covariance matrices, and how difficult it was to get subjects to make them positive definite, I concluded that they should be estimated, not elicited. I agree with Andrew’s gnat/camel observation completely. Most of the commonly used priors for smoothing, and hierarchical models seem to me to be just cogs of the model.
I was confused by the excerpted text. The initial paragraph begins by saying why prior elicitation would be useful “in principle,” and ends by listing several very good reasons why prior elicitation isn’t yet useful “in practice.” So far, so good.
Then: “Why are we not widely using prior elicitation?”