From my 2011 article. Bayesian Statistical Pragmatism:
Claims of the subjectivity of Bayesian inference have been much debated, and I am under no illusion that I can resolve them here. But I will repeat my point made at the outset of this discussion that Bayesian probability, like frequentist probability, is except in the simplest of examples a model-based activity that is mathematically anchored by physical randomization at one end and calibration to a reference set at the other. I will also repeat the familiar, but true, argument[1] that most of the power of a Bayesian inference typically comes from the likelihood, not the prior, and a person who is really worried about subjective model-building might profitably spend more effort thinking about assumptions inherent in additive models, logistic regressions, proportional hazards models, and the like. Even the Wilcoxon test is based on assumptions! To put it another way, I will accept the idea of subjective Bayesianism when this same subjectivity is acknowledged for other methods of inference. Until that point, I prefer to speak not of “subjectivity” but of “assumptions” and “scientific judgment.”
The above passage is not intended as a provocation or as “red meat” for the Bayesians in the audience. I think the statements above are commonplace, and it is only because of some combination of idealism and disillusionment that they could be viewed in any way as controversial.
Along the same lines, I recommend Rob Kass’s article, Statistical Inference: The Big Picture, of which my above-linked article is one of several discussions, and also chapter 1 of Bayesian Data Analysis, which has more on the foundations of probability.
[1] As a friend remarked to me in tenth-grade English class, “I don’t know why they don’t want us to use clichés. These sayings are clichés because they’re true!”
I am a newbie at this, but wouldn’t a frequentist be able to claim objectivity (in the math world of the model) at the limit, where inference is no longer needed and even the bayesian result could be said to be objective in said world?
> the math world of the model
The world of mathematical models is, to paraphrase Andrew, the world of assumptions and judgment. I think you could claim to be “objective” as long as you were just keeping track of frequencies of events without ever talking about probabilities (though even then there is the issue of measurement, i.e., how are “events” defined). Andrew’s point is that talking about probabilities requires making a model, which requires making assumptions (i.e., some degree of “subjectivity”). You need a model to take a limit or to specify the latent processes that generate observed events.
Same as any scientific model, you make assumptions then derive the consequences. The likelihood *is* the scientific model that actually gets tested.
Anon:
With enough assumptions you can make objective statements. In your example, the assumption is some sort of stationarity, so that as N increases there is some underlying constant parameter being studied.
I think the other respondents (thus far) are assuming quite a bit of familiarity with the lingo here. Let me try it at a more basic level.
There are some very simple cases in which one could be said to be performing “the same experiment” many times. Flipping a coin, rolling dice, pulling black or white or red balls out of an urn, etc., these are probably close enough to “doing the same thing many times” that one can create a mathematical model of them and then make frequentist statements based on the model. Or, yes, Bayesian statements that are “objective”.
But in almost all real-world situations that we care about, we aren’t really doing “the same thing” over and over. Presidential elections are all different, football games are all different, etc. Every day and month and year is different from any day or month or year that has ever come before, so the idea of a long-run frequency of identical events doesn’t really make sense. There is no long run of identical presidential elections, or identical monarch butterfly migrations, or identical Super Bowls. Any model one might care to make about these real-world situations is going to require a lot of “subjective” assumptions. Is the distribution of margins of victory around the point spread in Super Bowls normal? Perhaps it’s t-4? Perhaps it’s t-6? (In reality it is none of these things, and anyway there will never be enough Super Bowls to tell for sure even if it is one of these).
Andrew often points out that some people object to Bayesian models that put a prior distribution on a parameter (“look, you’re doing something subjective!”) but these same people often don’t object to other kinds of “subjective” assumptions such as assuming a relationship is linear, or assuming model residuals have a Normal distribution.
What is the probability a game ends with a score of 11-5? I had a dream it would happen due to my random football squares pool numbers: https://www.pro-football-reference.com/boxscores/missing-scores.htm
It did not happen.
I know the question was a little facetious, but the odds of 11-5 probably isn’t too bad. I’d expect we’ll see it someday. I always like the display at https://nflscorigami.com/.
Obviously for losing team to get 5 they’d need to only get a safety and a field goal. It happens once every few years, it seems. Meanwhile the other team needs a field goal and a touchdown, with the 2-point conversion. So the biggest question is whether a team would strategically go for the 2-point conversation over the extra point. It seems to me they would, as long as the other team had already gotten the safety and it was late enough in the game. Immediately after the touchdown, the score would be 9-5, 6-5 , 9-2 or 6-2. In all those cases, I’d think a coach would opt to go for 2 points (except maybe 9-2). So I’d think that behaviorally we could easily end up with 11-5. The team getting 11 also only scores twice, so it’s not like their offense is running wild, so a safety seems reasonable enough possibility.
There are a few scenarios that plausibly lead to 11-5. Eg, if the score was 5-3 (requiring only one rare safety) then the behind team scored a TD, they’d almost surely go for two. At least in the 4th quarter.
But the frequency of a game ending on 11-5 is very low, indicating there are simply that many more scenarios that end in other scores.
Point is the frequency f is an empirical estimate of p = “# 11-5 scenarios”/total. If you imagine infinite games played then f ~ p.
Ok, but how does a numerical correspondence in a special case (and not a case ever observed in reality) turn a frequency into a probability? Why is the estimate being given the same name as the estimand?
The whole thing is extremely shady. Like a slight of hand trick. Then once people fall for the trick they refuse to admit it.
A frequentist can well admit that some subjectivity is needed also in frequentism, but could say that the Bayesian needs the same subjectivity as the frequentist and then some more on top of it. (I’d agree with this, although I wouldn’t agree that this is necessarily bad.)
Christian:
As the man said, it is perhaps merely an accident of history that skeptics and subjectivists alike strain on the gnat of the prior distribution while swallowing the camel that is the likelihood.
For frequentists, the notion of repeatable experiment is conditional on covariates. For example, consider a simple linear regression,
y[n] = beta * x[n] + epsilon[n]
epsilon[n] ~ normal(0, sigma).
What is repeatable in the frequentist sense the residual noise epsilon[n], not the observations y[n]. So you can add i.i.d. errors (i.e., homoscedasticity) to your list of subjective assumptions made by frequentists in some cases.
This way of modeling is popular, but by no means necessary. Nothing in frequentist philosophy forces you to use i.i.d. errors. And of course Bayesians may also well model the errors as exchangeable, i.e., i.i.d. conditionally on the parameters. Or they may not. As frequentists.
The question is how much “being able to claim objectivity (in the math world of the model)” actually helps. (It’s a major issue with the whole concept of objectivity that usually the stronger and clearer an objectivity claim can be “technically” justified, the less it points to something really valuable for understanding the world.)
I’d take this to mean that one shouldn’t refer to Bayesian and frequentist probability at all, but rather reserve “Bayesian” and “frequentist” to describe different ways of bringing models into contact with data (i.e. different ways to use models in statistics).
Andrew, you are a child of your time. Your approach to statistical philosophy is reminiscent of Clinton’s political strategy of triangulation in the 1990s.
Clinton wanted people to think that he was neither left nor right, but an elevated, balanced synthesis. One that understood and acknowledged all sides, and that most parties could find in some way palatable.
What (if this can be pinned down) is the axis to which the “end”s refer?
Unlike Andrew, I don’t think the basis of Bayesian probability theory is “mathematically anchored by physical randomization.” Like Laplace, I’m on team epistemological when it comes to probability. To yet again quote the English translation of Laplace’s book, Essai philosophique sur les probabilités (“A Philosophical Essay on Probabilities” en anglais).
The probability one assigns to an event is going to depend on what information you have to condition it on. If I tell you it’s August in Edinburgh, you can give me a probability that it will rain today. Is that a physical probability? I don’t think so, because if I tell you it’s August 3rd, you can make a better prediction. And if I tell you the weather on August 2nd, you can make an even better prediction. If I give you up to date satellite information during August 3rd, I can make still better predictions. Where’s the physical probability in all of this?
Well put. I would go further and say it’s subjective degrees of belief (and that ‘science’ itself is remarkably subjective). Since I know little about the weather in August in Edinburgh, my probability likely differs from yours. If we both had data from the past 100 Augusts in Edinburgh, our probabilities might come closer to converging toward consensus, but even given that we are both of the Bayesian persuasion and have both conducted numerous various statistical modeling analysis in Stan for several years, we would likely still code different observational and prior models leading to slightly different probabilities for rain. People find the word ‘belief’ quite unpalatable, so choose a different one if you like, but my preference is for not sweeping things under the carpet.
Bob:
What I wrote is that Bayesian probability theory is “mathematically anchored by physical randomization on one end calibration to a reference set at the other.” Ideal coin flips, radioactive decay, etc.: these are processes with known probabilities that anchor numerical probability statements. On the other end, random sampling from a fixed population is another ideal process that anchors numerical probability. In the middle is everything between, including pharmacology, weather forecasting, sampling in the real world, etc. The anchors at the two ends provide unambiguous definitions of numerical probabilities such as 1/4.
> Bayesian probability, like frequentist probability, is except in the simplest of examples a model-based activity that is mathematically anchored by physical randomization at one end and calibration to a reference set at the other
Bayesian probability, unlike frequentist probability, doesn’t require those anchors. One can have unambiguous definitions of numerical probabilities without referring to physical frequencies or reference sets. You may anchor yourself if you that makes you feel safer – with more or less effort – but there’s no need to do so.
Carlos:
The anchors are needed because we want to apply probability to the real world, not just to make consistent mathematical statements. There’s a reason they carefully build roulette wheels so that all the outcomes are equally likely.
Rain in Edinburgh falls (or not) in the real world. However, we don’t need to think of its probability as the frequency of a hypothetical sequence of weather events or as a sample from an imaginary reference sets. If they are imaginary or hypothetical they don’t exist and we don’t need them!
*for all practical purposes “equally likely”
My probability that the roulette wheels produce precisely equally likely outcomes is rather close to zero, but I’ll grant we could all be dead before some experiment could provide enough data to be pleased with:p These are only anchors in the sense of some tangible example. The actual precise probability of some outcome from any given roulette wheel is not known and varies from wheel to wheel.
Jd:
Of course I meant, “close enough to equally likely for betting purposes.” Nobody thinks roulette wheels are infinitely precise. The point is that they try their best to make the probabilities equal using physical principles.
Carlos,
> Rain in Edinburgh falls (or not) in the real world. However, we don’t need to think of its probability as the frequency of a hypothetical sequence of weather events or as a sample from an imaginary reference sets. If they are imaginary or hypothetical they don’t exist and we don’t need them!
If you agree that all models are only approximations to the world, then your statement is equaivalent to the claim that all models are hypothetical, hence they don’t exist and hence we don’t need them. If I want to learn something about the world, I would disagree with this statement.
I agree with the view that inferences about the real world require predictions about frequencies of events because this is the only way to produce data with which to estimate and test models. That doesn’t mean that all probability estimates have to be frequencies — there is no difference between “we don’t need to think of … probability as the frequency of a hypothetical sequence of … events” and “it is sometimes (under specific conditions) useful to think of probability as the frequency of a hypothetical sequence of events”, apart that the latter statement is more specific and thus potentially more useful, if your aim is to learn something about the world.
> The point is that they try their best to make the probabilities equal using physical principles.
They try to make the long-run frequency of outcomes equal. Still, the “probability” of an outcome in one play is not an objective property of the physical device (for example, it can be controlled to some extent by the croupier).
The probability that Trump had of winning the 2024 election is also not a physical property of Trump (nor of the 2024 election) but in that case one cannot even imagine a meaningful long-run frequency of anything.
“One” can imagine many things. Apparently more than you can imagine. ;-) To what extent such an imagination is convincing when applied to a real world situation is a different matter of course.
But then epistemic probabilities are just as much an idealisation as frequentist ones. Both are better described as constructed than as “really existing”, and whether or not such a construction is helpful is a case-by-case discussion.
> But then epistemic probabilities are just as much an idealisation […] better described as constructed than as “really existing”
I agree, epistemic probabilities are ideations of the mind. Does anyone disagree?
Thank you all for your replies. Fascinating! If I only knew probabiy theory and statistical inference were so fundamental and interesting and fun when I was taught them in the totally wrong way 30 years ago :(
An inference connects physically real inputs to a physically real outputs. There’s no requirement that the inputs or outputs be frequencies, and there’s no inherent reason that intermediaries, such as probabilities be a prediction of a frequency.
Some people get it, some people don’t. I’ve yet to see a single example of someone in the later category move to the former. That’s fine. We can’t all be geniuses. The world need statistical ditch diggers too.
You ain’t a genius, pal.