Here’s what it says:
Bayesian statistics is a theory in the field of statistics based on the Bayesian interpretation of probability, where probability expresses a degree of belief in an event.
I disagree with this definition!
Why don’t I like it? Because I don’t think that probability, Bayesian or otherwise, “expresses a degree of belief.” I mean, sure, it can represent belief, but more generally it’s a mathematical model. Call it a model, call it an assumption if you’d like, but it doesn’t have to be a belief. To put it another way, all statistical methods use assumptions. The normal distribution is an assumption, the Poisson process is an assumption, logistic regression is an assumption, and so forth. You can fit a proportional hazard regression and say it’s your belief, or that it’s your model, or that it’s your method, or that it’s your assumption–and all of this is the case, whether or not you’re using Bayesian inference.
It could be worse, though. Back in 2011, Wikipedia’s article on Bayesian inference began as follows:
Bayesian inference is statistical inference in which evidence or observations are used to update or to newly infer the probability that a hypothesis may be true.
That really bothered me.
So what’s a better definition? Here’s something, not perfect, but better than Wikipedia’s, I think:
Bayesian statistics is a method for obtaining inferences and predictions by constructing a joint probability distribution over all parameters, observed data, latent quantities, and potential alternative or future data.
I don’t think this is so great–it’s too jargony and requires too much background–but it includes these key elements:
– It’s all about probability. The probability does not require a “Bayesian interpretation”; it just needs to be applied to everything in the model. And it’s probability distributions over data, parameters, and other unknown quantities in the system–not on “events.”
– It requires a joint distribution. From this can be derived Bayes’ theorem and all the rest.
– The model can be big. This points to the costs and also the benefits of Bayesian inference.
If you want a crisp definition, Merriam-Webster handles it well:
Bayesian: being, relating to, or involving statistical methods that assign probabilities or distributions to events (such as rain tomorrow) or parameters (such as a population mean) based on experience or best guesses before experimentation and data collection and that apply Bayes’ theorem to revise the probabilities and distributions after obtaining experimental data
That’s much better than Wikipedia. Professional lexicographers FTW!
P.S. Some related posts:
2025: Bayesian inference is not what you think it is!
2024: Bayesian statistics: the three cultures (from Bob Carpenter)
2016: Bayesian statistics: What’s it all about?
2016: What is Bayesian data analysis? Some examples
2012: What is a Bayesian?
2010: Bayesian statistics then and now
2009: Clearing up some misconceptions about Bayesian statistics
2009: Different meanings of Bayesian statistics
2008: Bayes, Bayesians
2008: Bayesian inference from the outside, vs. Bayesian inference as it is practiced
2008: “Bayesian reasoning” and “Bayesian data analysis”
2005: Are we not Bayesians?
Unfortunately, the definition of “Bayesian” in terms of degree of belief is still the dominant one in most intro stats books and the one regurgitated by many professors, so I cannot necessarily blame Wikipedia for representing what I think is still the “majority” position. I find that I have to waste quite a bit of time in my Bayesian stats class disabusing students of that erroneous definition.
To your point, defining frequentist methods in terms of a philosophy that probability is long-run relative frequency is just as useless.
An analogy to these kinds of definitions might be, “a lathe is used by people who adhere to the philosophy that rotational symmetry is inherently beautiful.” The lathe, like any statistical method, is just a tool that performs a function which may or may not be appropriate to any given situation. A user’s philosophy may guide their choice of tool, but the tool itself exists and should be defined independently of the beliefs of its users.
> The probability does not require a “Bayesian interpretation”; it just needs to be applied to everything in the model.
What alternative interpretation can you give to the probability of some meaningful parameter, say the location of Malaysia Airlines Flight MH370, or in general to the prior predictive distribution for whatever may be a quantity of interest when it’s not a parameter?
> it’s probability distributions over data, parameters, and other unknown quantities in the system–not on “events.”
> Merriam-Webster handles it well: […] statistical methods that assign probabilities or distributions to events [or parameters]
@Andrew :
Maybe there is, for most lay users, a confusion between the interpretation of probability and what is Bayesian statistic that rely partially on the subjective interpretation of probability. Of course, Bayesian statistics are much more than just a definition of probability. It also involves all modeling asumptions, etc.
Belief is too … religious to me. (probably a cultural context is at stake here, at least for me). Would the modified definition be better ? “…where probability expresses a degree of CREDIBILTY of an event/parameter…” ? or a “certain amount of knowledge about theta” ? The parameter being unique in the population, the probability can’t be a frequentist one. We nevertheless increase our knowledge about its value by accumulating data.
The best formulation I found (and suggest to my students) is as follow : the a posteriori point estimate and 95% credibility interval are x (y,z), i.e., 95% of my knowledge about the (unique) parameter in the population tells me that its values lies in the (x,y) interval. Would that be correct ?
An extreme solution would be to avoid using any definition (except the one base on Kolmogorow axiomatic) but that would lead us to a sliperry slope where the usual frequentist definition would fill the vacuum.
Or not ??
Nicholas:
My preferred approach is to think of probability as a mathematical system which can be applied in many ways. Depending on the context, the laws of probability (e.g., Bayes’ theorem) can be applied to:
– Subjective beliefs
– Decision analysis
– Long-run frequencies
– Physical probabilities
– Bets
and, more generally, all sorts of models constructed to yield useful inferences in various applied problem.
Probability is a branch of mathematics, just as algebra and geometry are branches of mathematics.
To say that “Bayesian probabilities are subjective” or “probability is relative frequency” or “probabilities come from bets” makes as much sense to me as saying that addition is inherently all about apples (one apple plus one apple equals two apples) or that a straight line in geometry inherently represents the edge of a folded sheet of paper.
When people take one particular application of probability and use it to define the whole, it makes me want to scream. Hence all these posts.
The posterior mode may fall outside a central 95% posterior interval. A simple example is exponential(1) with mode at zero and central 95% bounded away. Point estimates are summaries minimizing an objective (e.g., means minimize expected square error and medians absolute error).
Andrew’s point is that we should be saying, “given these modeling assumptions, and this data, there is a 95% probability some unknown falls in its central 95% posterior interval.” This is just a calculation relative to a mathematical model.
For comparison, we might say, “given a newtonian model and this data of initial t=0 object masses and observed positions and velocities, at time t=100 years, the bodies are predicted to be in these positions with these momenta.”
The probability predicted for an event depends on the data and the model. The positions predicted for planets depends on data and model. Neither are something that needs to exist in a metaphysical sense. Both are something that can be compared to reality if we think of events as corresponding to real events.
We could make the newtonian predictions probabilistic by assuming a measurement error model. Then we can do things like use our bayesian astronomy model to estimate the gravitational constant or planetary masses given planetary observations. Astrophysicists do just this!
What Bob said. Except for the non-capitalization of bayesian and newtonian; that seems kinda nonstandard to me!
Oops—I was writing from my iPad. Back on my computer now.
«The probability predicted for an event depends on the data and the model.»
As this discussion is about being precise about definitions, to me to talk about the probability of an event is somewhat meaningless without variance, and the words “event” and “data” also make me uneasy: I would rather use “future samples” instead of “event” and “past samples” instead of data.
«We could make the newtonian predictions probabilistic by assuming a measurement error model.»
Here I am uneasy with “measurement error” because when sampling from an ergodic source the samples are not necessarily involving “error” in the non-technical term sense, perhaps “sampling dispersion” might be more descriptive.
The term “event” was probably too technical here. I was referring to the technical notion which is a subset of the sample space that can be assigned a probability. See: https://en.wikipedia.org/wiki/Event_(probability_theory)
I literally meant a measurement error model in the sense that I might have a telescope and be measuring angles with some error (I’m only recording finitely many digits even if I can align it perfectly). Usually in something like a regression, the error term involves both molding error and measurement error, though you can often extend with a measurement error model. We then think of our observation as a sample from the measurement error model to tie it back together to thinking about convergence.
The “central” intervals you talk about are not particularly interesting. High density intervals (or regions, in general) are more interesting. They will contain the mode (but the mode is not particularly interesting either).
> Andrew’s point is that we should be saying, “given these modeling assumptions, and this data, there is a 95% probability some unknown falls in its central 95% posterior interval.”
The question is what does “probability” mean in that sentence. In frequentist inference that is meaningless or based on a particular interpretation of the term. One cannot escape the need to differentiate between different interpretations if we want to interpret that statement.
I don’t like highest density intervals because they are not invariant to parameterization. Either way, I think we can both agree that what we really care about it is event probability.
By probability I literally mean a number. You can then think about how probability corresponds to the world the same way you can think about how Newtonian mechanics applies to the world.
To be a bit more concrete, as a student of Gelman’s, I’d say I’m particularly concerned about making calibrated posterior predictive inferences. This is a very frequentist take on Bayes, but I’m happy to use all of probability theory in that modeling. Personally, I like to think of probability as uncertainty characterized by a certain state of knowledge and certain ability to model the world rather than “subjective belief”.
> By probability I literally mean a number. You can then think about how probability corresponds to the world the same way you can think about how Newtonian mechanics applies to the world.
I don’t understand what that means. The mass of a planet is number (with a unit) and if I say it’s 10^24 kg I can think about how that number corresponds to the world. That’s the mass of the planet. If I say that the mass has a probability distribution uniformly distributed between 10^23 kg and 2*10^23 kg I can calculate other numbers (probabilities) for “the mass being between X kg and Y kg” but the correspondence of tjose numbers with the world is not so evident. Many people will say that this is just meaningless because the mass of the planet is what it is.
> I like to think of probability as uncertainty characterized by a certain state of knowledge and certain ability to model the world rather than “subjective belief”.
We could debate what’s the difference between “subjective belief” and “someone’s characterization of uncertainty characterized by their knowledge and ability to model the world” but I think we may agree that a non-Bayesian (or frequentist or whatever word we want to use to refer to that position) will find both equally unacceptable.
I think part of the confusion stems from the fact that under Frequentist inference the researcher is allowed to condition on an assumption about the value of a parameter — such as the treatment effect is zero — that the researcher does not personally believe is true, but the researcher is not allowed to condition on a probability distribution of values that the parameter could take. So, conditioning on theta = 0 is fine from a Frequentist perspective but conditioning on theta being distributed normal (before you see the new data) with an expectation of zero and a standard deviation of $s$ is forbidden, even in the limit as $s$ approaches zero and even if all researchers in a subfield were to agree that $s$ is small.
What I don’t like about framing the prior distribution over the parameters as “just another assumption” is that Frequentist assumptions cannot change during the process of executing the estimator while the point of Bayesian estimation is to update the distribution of the parameters when conditioning on the observed data. You could say the posterior distribution is also “just another assumption” about the data-generating process but that does not convey the distinction that Frequentist assumptions are pre-data and posterior distributions are not.
Agreed. I think of Bayesian probability as quantifying “credibility under the assumptions and available information”.
So if I assume just that the mean of some value like the age of dogs in my town is a number that is between 0 and 15 and more likely around 6 than other values this constitutes some assumptions about dog ages and some relative credibility assigned to different values of the average.
If I KNEW the average, I could also make some assumptions about what the distribution of individual ages might look like. Conditional on a given average, and my knowledge of dog longevity I might say something like exponential(mu) truncated to [0,18] years assigns reasonable credibility to the number of dogs in each interval of age. I don’t have to *believe* that this is a frequency distribution of actual dogs, only that given the limited amount I know, it’s credible to think that the number of dogs in each interval of age could be those amounts.
Combining this credibility with an additional survey of randomly chosen dogs, I might find that the assumptions I made implied that if I considered those earlier descriptions credible, then I should also after seeing this data consider the new distribution for the mean parameter credible as well. In other words, a kind of imperfect implication: If A then B except with the imperfect knowledge being represented by probability rather than strictly boolean 0 and 1.
The point of Bayes for me is to make assumptions then have a mechanical way to calculate the implications of those assumptions. If I don’t like those conclusions I can also see which of the assumptions I should either relax or make stronger… because the assumptions don’t represent actual beliefs they represent simplified models of relative credibility.
To me, both the Wiki and the Merriam Webster versions are still quite technical.
The Bayesian framework is very useful and pervasive. In fact, our minds may be using it without us knowing! So why not give it a layman’s definition?
Don’t forget that Bayesians are frequentists. Also here
I like to think of this question by thinking about how one might be able to distinguish a frequentist from a Bayesian simply by observing their respective decision making choices:
“In practice, there is nothing preventing a Bayesian from using a frequentist test calibrated in such a way so as to minimize the Bayes’ risk, and nothing preventing a frequentist from using a Bayes factor calibrated in such a way so as to control the type 1 error (alpha). There is, however, a fundamental difference between the Bayesian and the frequentist when it comes to how they consider sample size (N). The frequentist will maintain the same value for alpha regardless of N, whereas the Bayesian will adjust alpha depending on N.”
Is this fair?
(this is from https://arxiv.org/abs/2104.07834)
I don’t think so. The Bayesian is distinguishable from the Frequentist by being willing to use probability for purposes other than what the Frequentist is willing to do. Specifically, to represent credibility under some knowledge and assumptions, rather than frequency in repeated data collection.
If you assign probability to a parameter, and there is no sense in which under repeated trials the value of this parameter changes in each trial and comes from a frequency distribution… then you are necessarily Bayesian. While I don’t think Bayes models “belief” to the normal Bayesian statistician, it does model some kind of “information about” or “credibility of” a quantity, including quantities that are fixed but unknown constants!
Andrew’s right that these are all just assumptions. You can assume normally distributed frequency errors and get some Frequentist tests, you can assume normally distributed mean parameter and get some Bayesian credibility calculations, but the content of the assumptions is different. A frequentists assumptions imply “if the frequency assumptions are sufficiently good, then the future data will come sufficiently close to this particular frequency curve when calculating the histogram” whereas the Bayesian assumption is more like “each future data point will individually be sufficiently typical of draws from this distribution” without demanding that long run frequencies converge to the given curve.
I personally think of frequentism as a specific interpretation of probability, whereas Bayes is more about actually handling probabilities, which may have various meanings (this is Andrew’s point I guess). Just using a frequentist interpretation of probability, I’m in no way committed to any “alpha” or sticking to it “regardless of N”; this is rather about a specific method that is usually used by frequentists, but there is nothing in frequentism (interpretation of probability) that forces any frequentist to do so. My kind of frequentist can well apply Bayesian methods (as in “Bayesians are frequentists” as referred to earlier in the discussion).
Re Daniel by the way, the frequentist concept of probability is an idealisation and will be applied as a thought construct to situations in which we do not necessarily think that “we can repeat this infinitely often” or “we know that the relative frequencies will stabilise”. Frequentist probability is used treating situations *as if* they were like this, and this can be useful even if they are not; this would need to be discussed on a case by case basis.
«Bayesian: being, relating to, or involving statistical methods that assign probabilities or distributions to events (such as rain tomorrow) or parameters (such as a population mean) based on experience or best guesses before experimentation and data collection and that apply Bayes’ theorem to revise the probabilities and distributions after obtaining experimental data»
That is not too dissimilar to how I would define it, because to me “bayesian statistics” if it is anything must involve Bayes’ theorem:
Given a sample from an ergodic source from a population the bayesian approach uses Bayes’s theorem in inferring some measure of the population from some measure of that sample and from some “priors”; that it is presumed that there is valid information about the population that is not provided by that source and can be reliably used in that inference.
Andrew:
Given the apparent importance of the these distinctions why not set the record straight by editing the Wikipedia page accordingly, perhaps by adding a new subsection about dissenting terminological interpretations of Bayesian statistics?
I sometimes am of the opinion that trying to define Bayesian statistics by a single belief is like trying to define a political party by a specific belief. It is a social-constructed grouping of many similar and usually co-occurring ideas, but that don’t necessarily need to all be shared by an individual “Bayesian.” To name two:
1) It is valid to consider beliefs as probability distributions, at least insofar as they can be priors in Bayes’ Theorem.
2) It is favorable to apply Bayes’ Theorem to make inferences and predictions over using traditional frequentist methods.
But also, when you talk about Bayesian statistics, it feels a bit lacking to not mention the belief in joint probability distributions as causal models, the use of probabilistic programming languages, or the common motivating interest of conducting “better” science.
Bayesian inference compares how well various hypotheses fit the data, then weights each (the prior) according to a rough guess of how likely each is based on each individual’s past experience/information. It is an approximation of the ratio “favorable” outcomes divided by total possible outcomes, something intractable without knowing the current and all possible states of the universe.
Frequentest inference limits itself to using observed frequencies to estimate that favorable/total ratio.
NHST, which is by far the most common application of statistical inference, ignores the research hypotheses altogether for the sake of simplicity.
How does this relate to probabilities in quantum mechanics? There is a lot of literature, with no agreement. They could be (1) physical entities, like energy; (2) subjective beliefs; or (3) predictions that pop out of reliable models.
According to Andrew “the usual rules of conditional probability fail in the quantum realm” but definitely there is no agreement on that. There have been several discussion on the subject here, see also https://arxiv.org/a/0000-0002-6070-0784.html
What words would you use for the practice of expressing a degree of belief using probability theory (i.e., epistemic probability)? The fact is that most people call it Bayesian statistics.
If you want a word for *any* use of probability theory, why don’t you just say probability theory? Why do you want to muddy the waters and call any use of probability theory anything other than probability theory, and in this case use a word for it that is already taken. You are sowing confusion by overloading the word ‘Bayesian’ for no reason that I can see.
Other:
“Expressing a degree of belief using probability theory” is one application of Bayesian statistics. Obtaining consistent betting odds is another application of Bayesian statistics. What we do in Bayesian Data Analysis–inference from data given an assumed model–is another application of Bayesian statistics.
This is not “overloading the word Bayesian”; it’s just that Bayesian inference can be used in different scenarios, and I think it’s a mistake to identify it with “degree of belief,” as this is only one such scenario, and not the most important one.
Other Andrew’s point is, I think, that your statement above could also be written as:
“Expressing a degree of belief using probability theory” is one application of probability. Obtaining consistent betting odds is another application of probability. What we do in Bayesian Data Analysis–inference from data given an assumed model–is another application of probability.
It’s not clear what is gained, if anything, by saying “Bayesian statistics” instead of “probability” given that “Bayesian statistics” is already used evoking some ideas that you don’t quite like.
Yes, exactly. Carlo has made the point very well.
Carlos:
“Bayesian” is a powerful brand name. Back in 1991 when I started putting together what became the book Bayesian Data Analysis, I considered various alternative titles. My preference was “Statistical inference using conditional probability” or “Data analysis using full probability models” or something like that. But I settled on Bayesian because, although lots of people associate Bayesianism with a philosophy that I think is mistaken (see here, for example), there was lots of other stuff called “Bayesian” that was close to what I liked. No term is perfect. The word “Bayesian” is out there and it has some power so I use it.
To put it in your terms, yes, “Bayesian statistics” is already used evoking some ideas that I dislike. Also, “Bayesian statistics” is already used evoking some ideas that I do like. That’s the point of the above post!
It’s not Bayesian if you don’t use Bayesian conditioning (as Bayes originally did). This of course isn’t necessarily done when doing statistics, so Neyman and Pearson and Fisher are rightly not called Bayesian, but the (in my view) correct reason is not that they used a frequentist concept of probability. That can be used well with Bayesian reasoning.
The “Bayesian” label appears first in the literature in 1950, to refer to inference using inverse probability methods (finding the probability of causes from the outcomes, as Laplace put it). Fisher wanted to distinguish those “old” methods from the “improved” methods that didn’t involve an invalid assignment of probabilities to causes. In 1951 Savage wrote “the problem of dealing with uncertainty when probability doesn’t apply to the unknown states of the world in unBayesian, statistical theory.”
Bayesian has come to mean “statistical methods where unknowns are represented by probabilities” and, by opposition, frequentist refers to “statistical methods where unknowns can not be represented by probabilities”. Reading some things you wrote it seems that your definition of Bayesian is maybe even more restrictive if every prior distribution has to have a frequentist interpretation (or maybe not, if any probability can be given a frequentist interpretation simply by putting it in the reference set of things with that probability).
Maybe you would agree with a definition like the original but without mention of “belief”: “Bayesian statistics is a theory in the field of statistics based on the Bayesian interpretation of probability, where probability expresses uncertainty about an event.” Maybe you would leave out the “about an event”, or prefer “about something”. In any case, being based on a Bayesian interpretation of probability is the main defining point, it is what makes constructing a joint probability distribution possible in inference problems.
The distinction between Bayesian and non-Bayesian statistical methods remains relevant. “Bayesians are frequentists” is not particularly shocking if it means that someone may accept the interpretation of some probability as uncertainty and the interpretation of some probability as frequency (de Finetti’s book has a hundred of pages on random processes) or that the same person can apply Bayesian methods and check their frequentist properties (that practice is as old as the term “bayesian”) or apply frequentist methods. However, there are still many statisticians that are frequentist in the non-Bayesian sense and “Bayesians are not non-Bayesians”.
“Bayesians are frequentists” is a memorable slogan anyway, I give you that. It looks a bit like 1984’s “doublethink”: War is peace. Freedom is slavery. Ignorance is strength. Bayesianism is frequentism.
Perhaps a useful framing is the idea that ALL probabilities are conditional upon a model. Different model, different probabilities, and there is no sense in which anyone has to believe the model. As Ed Jaynes put it in his articles and posthumous book, we should always include a “background information” in any probability we write down. The prior is P(θ|I) for some specific background information I encoding the model; change I and you of course get a different number.
(As far as quantum mechanics is concerned, both the Many Worlds and the QBists take Bayesian probability very seriously — with models that have completely different ontologies, but exactly the same mathematical repercussions. That is, they both reproduce the formalism of textbook quantum mechanics.)
Agreed.
Consider the contortions and confusion and ‘fake’ subtleties that this use of the word Bayesian brings about. You have that Fisher isn’t a Bayesian, but not that because he was a frequentist. That frequentism can be Bayesian. That Bayesian can be frequentist. That encyclopedia entries should be rewritten.
Now consider saying Bayesian when you mean Bayesian and saying probability when you mean probability. What’s wrong with that?
No terminology is perfect, but it’s hard to understand why you insist on terminology that deliberately overloads the word Bayesian and quite evidently leads to misunderstanding, that is mostly cleared up by calling different things different names.
In this simpler terminology, Fisher wasn’t a Bayesian because he violently objected to epistemic probabilities. Frequentism isn’t Bayesian because it doesn’t quantify belief using probability theory. We don’t need essays to clear up the confusion that we wreaked.
Other:
You can get a sense of what I, and many others, think of “Bayesian” from the contents of Bayesian Data Analysis. In chapter 1 of that book we discuss the foundations of probability and give several empirical examples.