Actually, I think it would be more accurate to describe the trajectory of Bayesian statistics during the past fifty years as “the gradual rise, then rapid rise, then slower rise,” but that doesn’t follow the classic “rise and fall” template.
Also, I’m following up on John Cook’s recent post:
At one time Bayesian statistics was not just a minority approach, it was considered controversial or fringe. . . .
Then somewhere along the way, maybe 20 years ago or so, Bayesian analysis not only became acceptable, it became hot. People would throw around the term Bayesian much like they throw around AI now.
During the Bayesian heyday, someone said that you’d know Bayes won when people would quit putting the word “Bayesian” in the title of their papers. That happened. I’m not sure when, but maybe around 2013? . . .
It’s strange that Bayes was ever scandalous, or that it was ever sexy. It’s just math. . . .
Bayesian statistics hasn’t fallen, but the hype around Bayesian statistics has fallen. The utility of Bayesian statistics has improved as the theory and its software tools have matured. In fact, it has matured to the point that people don’t emphasize that it’s Bayesian.
So, despite the title of his post, John also doesn’t think Bayesian statistics has fallen! I will say, though, that it’s fallen as a relative share of statistics. In the past twenty years, the statistical and machine learning “pie” has become much bigger, and the Bayesian slice has become much bigger in size, but it’s become smaller relative to the pie as a whole. Which is fair enough. As John says, Bayesian statistics has matured–I’d like to take some credit for that, actually!–and it’s less new and exciting. There’s still lots of active research and many open problems (see here, for example), but I’d say that Peak Bayes occurred around 2010, before Bayesian methods became part of the background.
Beyond that, I have a few other reactions to John’s post:
Regarding his remark that Bayesian statistics used to be controversial, Christian Robert and I wrote an article back in 2013, “Not only defended but also applied”: The perceived absurdity of Bayesian inference, along with a followup, The anti-Bayesian moment and its passing. Howard Wainer proposed the theory that some of the more ridiculous anti-Bayesianism was a Cold War feint, an attempt to steer the Soviets away from a powerful statistical method that had been used to crack the German codes in WW2. I’m skeptical but it’s a good story.
Even now, there remains the Bayesian cringe: The attitude that we need to apologize for using prior information. For example, in Bayesian Data Analysis and Regression and Other Stories we use flat priors all the time, which yields results that I wouldn’t really believe. See further discussion in the long comment thread to my post from 2019, What’s wrong with Bayes (followed up with What’s wrong with null hypothesis significance testing, but that’s another story).
I’m getting better about using informative priors–there’s this paper with Erik van Zwet and lots more in our forthcoming book on Bayesian workflow–but I’m still not all the way there. So let me say in response to John that, yes, Bayesian methods have matured but they have a bit more maturing to do. As is the case with all statistical methods.
Beyond this, Bayesian computation is huge, and computational methods used in Bayesian inference are also used for other problems in statistics and machine learning. In many settings, Bayesian inference is recognized as an ideal: it’s the uncertainty quantification that users are aiming for, even if the computation can only be approximate.
I’ll conclude this parade of links with a pointer to my 2021 paper with Aki Vehtari, What are the most important statistical ideas of the past 50 years?, which really should’ve been a discussion paper but something got screwed up in the communication with the journal and it ended up as a standalone article.
Then again, and there is always a “then again,” there is this famous put-down quotation from Maurice Kendall,
“If they [Bayesians] would only do as he [Bayes] did and publish posthumously we should all be saved a lot of trouble.”
Did Kendall ever change/alter/amend his view? If so, was it sudden or gradual?
Paul:
I dunno, but I wouldn’t trust Kendall to combine multiple sources of information in the ways that we need to do in pharmacology, election forecasting, and other applications.
Based on a look at Kendall’s wikipedia page, I’d say that he did a lot of important work. It’s just too bad when people can’t leave it at that but instead make ignorant criticisms of other approaches. Another such example is discussed here.
The best is yet to come :)
So that we can understand and trust them, generative models, predictions and inferences that use ML/AI require principled foundations.
Those foundations will be Bayeisan. The Bayesian & ML/AI synthesis will take all the pie.
Any sign in the existing literature to take a peek?
In short, the _acceleration_ of Bayesian statistics has fallen.
Twan:
The umpteenth derivative of our progress is negative.
Sorry for a digression here. I wonder if you (or another statistician) has posted at all about this paper on sex at birth: https://www.science.org/doi/10.1126/sciadv.adu7402
(In case the link fails, the paper is “Is sex at birth a biological coin toss? Insights from a longitudinal and GWAS analysis” in Science Advances)
I read about it in the popular press (e.g. https://www.sciencefocus.com/news/boys-girls-birth ), where some surprisingly (to me) large effects were reported: For instance, “they found that if a couple had already had three boys, they had a 61 per cent chance of having another boy. Similarly, there was a 58 per cent chance of having another girl after having three girls.”
I have nothing bad to say about the paper or the authors and I am not an expert on any aspect of the paper. It just seemed interesting and surprising to me, and I know from reading this blog over the years that sometimes the popular press overhypes things (and sometimes the papers overhype themselves).
Of course, no one has any obligation to post anything about it, it’s just my curiosity.
Anon:
Someone pointed me to that paper awhile ago and it doesn’t impress me. It looks like one more bit of noise mining which is just so characteristic of the sex-ratio literature. And, no, I don’t think that 61% would replicate. It seems like a classic example of grabbing something striking from a pile of essentially random numbers. All things are possible, but that’s my Bayesian take.
I think I did write a post on this paper. If so it should show up sometime during the next few months.
Thanks for responding!
People been publishing on this sex ratio stuff for a long time. The fact its not exactly a binomial distribution can be used to justify anything you want. Eg, the first one ever was “sex ratio is not exactly 50%. Therefore god exists, and polygamy is evil”:
https://www.jstor.org/stable/103111?seq=3
300 years of applying NHST to the phenomenon has cumulated in your paper I guess.
In the introductory proseminar in experimental psychology at the University of Michigan, in 1966-67, Ward Edwards told us to ignore everything we were learning in the statistics course because Bayesian statistics will soon change everything. Happily, we did not take his advice, but I at least remained curious about it over the years.
The term, “proseminar” was new to me so I found this:
https://www.merriam-webster.com/dictionary/proseminar
“a course of study like a graduate seminar but often open to advanced undergraduates”
To my additional surprise,
“The first known use of proseminar was in 1888” so I have some catching up to do.
20 years ago, it was thought that objective priors were just around the corner. Not long after, the leading objectivity Bayesians (e.g., Bernardo, Berger) declared there was no such thing as an objective prior. There were as many priors as parameters in a model, and one has to put them in order of importance, etc. At my 2010 conference in London Bernardo repeated that Bayesian testing was a work in progress, but here too the promise hasn’t materialized. The long awaited text on Objective Bayesian Inference by Berger, Bernardo and Sun (2024) does not include a chapter on testing, not even Bayes factors promoted by Berger. Granted, Bayes factor tests are, apparently, now widely used as replacements for statistical significance tests, but I don’t think they’re up to the job.
https://errorstatistics.com/2025/08/07/my-bjps-paper-severe-testing-error-statistics-versus-bayes-factor-tests/
Many will respond by denying science really needs to test the adequacy of models and claims with error statistical guarantees, but my view is that this is something scientists are not generally willing to give up. Of course, there’s much more to Bayesian analysis (and I don’t count merely using “Bayes Theorem” as Bayesian), but if one was looking to point to “a fall”, it might be here. I realize many on this blog will vehemently disagree…
Frequentists in exile, the blog
Deborah:
You write, “20 years ago, it was thought that objective priors were just around the corner.” That may well be, that some people thought this. But I never thought it! The big thing that I was working on 15-20 years ago was weakly informative priors.
As usual, I think it’s kinda funny when people agonize over the prior distribution, p(theta) while using conventional off-the-shelf models for the data given parameters, p(y|theta). There are some simple problems where this makes sense, some settings where p(y|theta) is very well understood but p(theta) is not, but in general both parts of the Bayesian model will require judgment. This is not to say that conventional models can’t be useful–for example, deep learning models are full of logistic curves, a form that is somewhat arbitrary but has worked well in the past, it’s justified based on some combination of mathematical logic and practical experience. For reasons discussed in my paper with Hennig, I don’t think it’s particularly useful to describe particular data models or prior distributions as “objective” or “subjective.”
Perhaps not coincidentally, there’s a series of conferences on “objective Bayes theory, methodology and applications,” which have been going on for almost 30 years, and they haven’t invited me even once!
As you say, judgment is required to choose both the parameter prior and the parametric data model, so if somebody doesn’t like judgment/subjectivity, it doesn’t make sense to only criticise the former, not the latter (and then there are some reasons to like judgment, see our paper).
However in practice the argument does make some sense to say that both of these choices are hard and problematic and may have an impact on results, so it is better to need only one of them (frequentist inference) than both (Bayes). At least it looks like that to a user who doesn’t have much of a clue how their prior should look like. Furthermore, in a straightforward problem with, say, 500 observations assumed to come from the same parametric distribution, the user has 500 observations worth of information regarding the parametric shape of the data model in the data, whereas the data don’t even offer a single observation worth of information regarding the prior (in standard Bayesian analysis, the whole dataset is modelled as generated by a single realisation of the prior, which isn’t observed precisely as otherwise no inference would be needed). Of course the prior is about prior information, so if this is there, fine. Still, as long as we’re talking about relating our models directly to the available observations, we’re in a better position regarding the data model than the prior.
My personal stand on this roughly is, use a prior when it helps, don’t use it when it doesn’t (taking into account everything including the difficulty to choose and justify the prior).
> use a prior when it helps, don’t use it when it doesn’t (taking into account everything including the difficulty to choose and justify the prior)
You’re still using a prior – one that may have been easy to choose but is not necessarily easy to justify (unless it just doesn’t make a difference).
Nope. The fact that one can mathematically connect a certain outcome of a frequentist analysis to a Bayesian analysis with a certain prior doesn’t mean that this prior is in fact used. The frequentists don’t make statements about the a posteriori distribution of parameters, so what they do does not amount to doing a Bayesian analysis based on that prior.
> the frequentists don’t make statements about the a posteriori distribution of parameters
To be clear, are you using “frequentists” here in the “classical frequentist inference” sense?
Then it’s true that they don’t need a prior. There are so many things in life that can be done without a prior!
(Otherwise it seems that von Mises, who is a frequentist by anyone’s standards, wouldn’t agree.)
Not very relevant to the point before, but OK, if you insist, I could’ve more precisely written “to the amount that frequentists don’t make statements about posterior distributions of parameters”.
Or rather “to the extent”. Juggling different languages can confuse me.
It sounds to me like there are two main issues that you bring up: (a) the explicit use of prior information in a Bayesian analysis; and (b) the scientific value of testing the adequacy of specific models.
With respect to (a), I think a lot of the controversy about using prior information comes from a belief that statistics should be agnostic about the meaning of the models/quantities under consideration. Certainly, this is how statistics is still commonly taught, as a collection of “default” methods that work pretty well in cases that resemble common scenarios. The desire for “objective” priors, “empirical” priors, and even “weakly informative” priors thus strike me as attempts to preserve the illusion that statistical methods can be applied without understanding what is being modeled. So from that perspective, I’m not surprised that such approaches haven’t amounted to much, because they are often trying to solve the wrong problem.
With respect to (b), it does seem like Bayesian methods have been developed largely around the idea of model comparison. Moreover, it’s my impression that, historically, model testing was considered “out of bounds” for Bayesian methods for philosophical reasons. In principle, model testing can be done with Bayes factors or other model comparison techniques as long as the alternative model can be specified appropriately. But I share your concern that most actual uses of Bayes factors don’t do this and are thereby not going beyond the typically misapplied significance testing procedure.
Ultimately, it’s my impression both as a scientist and as someone who struggles to teach both graduate and undergraduate students about statistics, that the precise inference framework used matters less than the amount of care and fidelity put into specifying the models that couple the theoretical constructs of interest with their empirical consequences. Applying Bayesian methods well requires building models with meaningful parameters that do a good job accounting for data. That way, model comparisons are tailored to the inferential goals of the researcher and any priors used can be justified within context. The same is true, I think, about model testing, to ensure that the model being tested is not a meaningless “straw man” (as in NHST) and that the test is well-designed (so that it doesn’t accidentally test an auxiliary assumption rather than something core to the meaning of the model).
Bringing it back to Bayes, it is my impression that the wider adoption of Bayesian methods over the last 20-25 years has led more practicing scientists to consider the relationship between their statistical models and the inferences they want to draw. While there is still considerable work to be done along those lines, I’d say that is a net positive regardless of whether the method used is ultimately Bayesian or not.
Hi, gec. I’m not sure I understand your comment, “The desire for “objective” priors, “empirical” priors, and even “weakly informative” priors thus strike me as attempts to preserve the illusion that statistical methods can be applied without understanding what is being modeled. So from that perspective, I’m not surprised that such approaches haven’t amounted to much, because they are often trying to solve the wrong problem.”
I would say these desires are not all coming from the same groups. My understanding is that the “objective” prior thing is for the cringing Bayesians (as Andrew would call them) who don’t believe they should be using prior information, but for some reason I can’t fathom, still want to do Bayesian statistical inference. But you can’t create an objective prior without knowing the likelihood, and you can’t design a likelihood without understanding the data, so I’m not seeing how you could create an objective prior for a problem without understanding what is being modeled.
The use of “empirical Bayes” (I agree with Andrew that it’s a bad name because it’s no more empirical than the usual form of Bayes) has at least two motivations. The first motivation is to use it as a regularization step in frequentist analyses. For example, you can see that in Efron and Morris’s 1975 paper, Data Analysis Using Stein’s Estimator and Its Generalization, and they wrote many other things on this topic. The second motivation is just for efficiency in an otherwise intractable hierarchical model—Andrew argues for this approach all the time.
“Weakly informative” priors arise from thinking about what the scale of the parameters will be in your model and designing a prior around that. Andrew’s described his own trajectory as moving from favoring flat priors (not the “objective” prior, but a truly improper flat prior) to favoring weakly informative priors. Usually this is enough in that the data’s going to dominate the prior in most cases and the weakly informative prior can go on the hierarchical components which are not very sensitive (see, e.g., Andrew’s and my paper on Covid PCR testing sensitivity and specificity estimation across sites). It was only later realized that a bunch of independent weakly informative priors can be very uninformative jointly and has led to more serious prior predictive checking, following the Gabry et al. paper on visualization (it’s more about prior predictive checks in my opinion).
gec also says, “Applying Bayesian methods well requires building models with meaningful parameters that do a good job accounting for data.” But isn’t that true of frequentist modeling, too? Don’t you need a decent likelihood that’s close to the true generating process if you want to get good inferences?
> I would say these desires are not all coming from the same groups.
I guess I was thinking more about the desires of the potential *users* of these methods more than the desires of the folks developing them. I agree that there has been considerable progress around weakly informative priors and I’ve learned a lot from the efforts of people like yourself and Andrew and your colleagues on that front! However, from the perspective of a researcher “shopping around” for different methods to use, often they are concerned about justifying their approach to reviewers/colleagues. This can be done more easily if the approach is viewed as “standard” or “default”.
> But isn’t that true of frequentist modeling, too? Don’t you need a decent likelihood that’s close to the true generating process if you want to get good inferences?
My answer to that is an emphatic “yes!”, as I hoped would be clear from the context of the rest of my remarks (e.g., “the precise inference framework used matters less than the amount of care and fidelity put into specifying the models that couple the theoretical constructs of interest with their empirical consequences”).
> moving from favoring flat priors (not the “objective” prior, but a truly improper flat prior)
What do you mean by “objective” prior? It seems it may be a truly improper flat prior – depending on the problem.
I mean the priors that the so-called “Objective Bayes” folks come up with and call “objective”. Here’s a web intro:
https://www.objectivebayesian.com/p/intro
There’s a whole book, Objective Bayesian Inference, and a sequence of Objective Bayes conferences run by ISBA.
Thanks. An “objective” prior may be a improper and may be flat. (It’s literally the first example in your first link.)
“improper” priors are an artifact of using infinities to get an approximation. They aren’t a thing for actual (combinatorial -> rational, discrete) probabilities.
Perhaps of interest (particularly the rejoinder)-
“Defining a Credible Interval Is Not Always Possible with “Point-Null” Priors: A Lesser-Known Correlate of the Jeffreys-Lindley Paradox (with Discussion)” (https://projecteuclid.org/journals/bayesian-analysis/volume-19/issue-3/Defining-a-Credible-Interval-Is-Not-Always-Possible-with-Point/10.1214/23-BA1397.full)
Deborah,
Thanks for participating here! One thing I’m not clear on is, are you interested in ‘error statistical’ properties of models or of derived quantities from models (e.g. predictions) or of the specific parameters they contain? I guess my perspective here is that all the good stuff is happening in *workflow* from thinking generatively, to prior predictive modeling, iterative model fitting, to posterior predictive checking. We can very well check how well our models predict in various ways, how well calibrated, compare sets of candidate models in various ways. For one, I am often looking to break my models in various ways to learn where they fall short. If we have pre-specified hypotheses or outcomes, with clear mathematical definitions, we can report those statistics too. We can evaluate the frequency properties of our Bayesian models just fine as far as I can tell.
How is all this not testing the adequacy of models?
Chris:
I got very busy and neglected to check back here, sorry. I’m not sure what you mean by checking the frequency properties of Bayesian models. Thank you.
Deborah: no worries! We can study, for instance, the behavior of our models under repeated sampling easily via simulation, or with real data-sets. Do our credible intervals also have good coverage properties? This is readily checked. I have found numerous cases where a Bayesian interval has better coverage and calibration than an interval derived from a non-Bayesian method like least squares or even max likelihood.
Chris,
how can a Bayesian interval have better coverage than a Frequentist interval? By construction a Frequentist confidence interval with confidence level 1 – alpha is an interval that has a coverage probability of 1 – alpha. That means, if we repeat the exact same random process it will contain the true parameter with probability 1 – alpha.
How can you improve on that? If you add an informative prior, the coverage will go down. If you want a higher coverage probability you can make the interval wider but I assume that is not what you have in mind.
huan, you are expressing a common but incorrect view of statistical methods. The properties you talk about are desiderata for *estimators*, not all of which hold equally well all the time. Check out John Hughes ‘A Bayesian Tour of Binomial Inference’ to see how a variety of estimators perform in practice when evaluated for frequentist properties like coverage, and to see how common Bayesian models can have better coverage than non-Bayesian estimators.
Chris,
unfortunately I don’t have access to this paper.
Since you mentioned Binomial inference, I assume you are talking about confidence intervals for the success probability of N iid Bernoulli trials. I am sure some Bayesian intervals outperform some Frequentist intervals, in the sense that they have a certain coverage probability and are shorter than a Frequentist CI with said coverage probability (Just compare a Bayesian interval to [- infinity, + infinity].
However, I don’t believe that a Bayesian interval will be uniformly better than any Frequentist interval. There do exist exact CIs for this Binomial problem I think which should perform better than Bayesian intervals.
Mayo, I don’t think you ever understood the distinction between distributions that (1) objectively reflect frequencies, (2) objectively reflect a given state of information, and (3) objectively reflect and given state of information about future frequencies.
Bayes stalled not because that’s where Bayes in it’s natural development, but because the big cheeses of bayes themselves only partially grok’ed the full meaning and implications of non-frequency definitions probability.
I’m with Laplace and John Stuart Mill on this one. Here’s Laplace on why probability has to be considered epistemically rather than physically:
https://en.wikipedia.org/wiki/Laplace%27s_demon
Now we know this isn’t right from a quantum perspective, but all we really need is that we can predict coin tosses at more than 50% accuracy given more information to sink this idea that a perfectly repeatable 50% coin toss exists in reality independently of observers and the information they have.
Adoption hasn’t even happened yet. People are still oversimplifying by 10-100x, only fitting parameters of one model at a time.
This is implicitly placing priors of 0 on all other possible models, which considerably hobbles its application. Once it becomes computationally feasible to consider multiple models as a matter of course, then we will feel the true power.
Or, will we just find out how many irrelevant null hypothesis people can devise for use as strawmen?
Does anyone know who coined the term “improper prior”? I’d like to read the context of it.
My search lead me to Haldane 1932. While he is clear on working with approximations, the term “improper prior” is not used. https://www.cambridge.org/core/journals/mathematical-proceedings-of-the-cambridge-philosophical-society/article/abs/note-on-inverse-probability/1BC33DBEA96916D0A31998DCD5CA6D3B
Bayesians: once mysterious and feared…now quietly pulling the strings while frequentists take the credit.