David reports,
Boris presented the TSCS paper at midwest and was being accused by Neal Beck for not being a real Bayesian. Beck was making the claim that “we’re not Bayesians” because we’re using uninformative priors. He’s seems to be under the assumption that bayesians only use informative priors. Boris should have just directed him to your book and told him to read chapters 1 and 2! I know you’ve spoken to Beck before, but have you ever had such an exchange with him on this topic? He kept making the claim that if you use diffuse priors, all you’re doing is MLE. It may be true that for many simple anaylses that bayesian inference and MLE can produce similar results, but Bayesian inference can easily be extended to more complex problems (something that MLE may have a harder time doing).
What is Bayesian inference?
My reply: Bayesian inference is characterized by the use of the posterior distribution–the distribution of unknowns, conditional on knowns. Bayesian inference can be done with different sorts of models. In general, more complex models are better (see here, also with some interesting discussion), but a simpler model is less effort to set up and can be used as a starting point in a wide range of examples.
Diffuse prior distributions
Diffuse prior distributions are a type of simplification. Other simplifications we commonly use are conventional data models such as the normal distribution, and conventional transformations such as logit or probit. Bayesian inference with these models is still Bayesian. In the model-checking stage of Bayesian data analysis (see Chapter 6 of our book), you can check the fit of the model and think about how to improve it.
More technically, an improper prior distribution can be considered as “noninformative” if it is a stable limit of proper prior distributions (see Sections 2.2-2.4 of this paper).
Hmmm . . . let me try to put this more aphoristically. Bayesian inference with the right model is better than Bayesian inference with a wrong model. “Improper” models (that is, models without a joint probability distribution for all knowns and unknowns in the model) cannot be right. But Bayesian inference with a wrong model is still Bayesian.
Update (19 Apr 05): Neal says he was misquoted. He also says he’ll reply soon.
I'm glad that I'm only taking introductory econometrics. If I felt that I needed to understand it, that post might have been downright frightening. My professor did refer to Bayesian vs. Frequentist (?) something-or-other a day or two ago, but I was fortunate enough to miss the gist of it entirely. Anyway, the blog makes for interesting reading, even if it's generally far over my poor head.
Andrew, your paper on priors seems appropos.
You write, "Noninformative prior distributions are intended to allow Bayesian inference for parameters about which not much is known beyond the data included in the analysis at hand. Various justifications and interpretations of noninformative priors have been proposed over the years, including invariance
(Jeffreys, 1961), maximum entropy (Jaynes, 1983), and agreement with classical estimators (Box
and Tiao, 1973, Meng and Zaslavsky, 2002). In this paper, we follow the approach of Bernardo
(1979) and consider so-called noninformative priors as “reference models” to be used as a standard
of comparison or starting point in place of the proper, informative prior distributions that would be
appropriate for a full Bayesian analysis (see also Kass and Wasserman, 1996)."
Hmm… I really need to brush up my stats. Unusual blog. Very interesting. I think I might learn something here. Keep up the good work.
Deb – There's a great quote by Peter Grunwald in his introductory chapter to "Advances in Minimum Description Length" (2005, p.16; MIT Press) that talks about parsimony.
It is often claimed that Occam's razor is false — we often try to model real-world situations that are arbitrarily complex, so why should we favor simple models? In the words of Webb [1996], "What good are simple models of a complex world?" The short answer is: even if the true data-generating machinery is very complex, it may be a good strategy to prefer simple models for small sample sizes. Thus, MDL (and the corresponding form of Occam's razor) is a strategy for inferring models from data ("choose simple models at small sample sizes"), not a statement
about how the world works ("simple models are more likely to be true") — indeed, a strategy cannot be true or false; it is "clever" or "stupid." And the strategy of preferring simpler models is clever even if the data-generating process is highly complex
I think that all this comes down to the question of what we are trying to achieve with statistics. If the goal is only to descibe data accurately, then parsimony is irrelevant. If the goal is to describe accurately and concisely, or predict future events in the presence of noise, then parsimony becomes our guard against over-fitting. The earliest formal result that I know of demonstrating this is Akaike (1973), but there have been several variants since then. From an information theoretic point of view, people like Grunwald, Rissanen and Wallace have shown that parsimony is important in the compression of data, while folks like Dawid have talked a lot about predictions of future events (though I'm not as familiar with Dawid's work as I should be, so I might be misinterpreting).