Palko writes:
I’m just an occasional Bayesian (and never an exo-biologist) so maybe I’m missing some subtleties here, but I believe educated estimates for the first probability vary widely with some close to 0 and some close to 1 with no good sense of the distribution. Is there any point in applying the theorem at that point? From this Wired article:
If or when scientists detect a putative biosignature gas on a distant planet, they can use a formula called Bayes’ theorem to calculate the chance of life existing there based on three probabilities. Two have to do with biology. The first is the probability of life emerging on that planet given everything else that’s known about it. The second is the probability that, if there is life, it would create the biosignature we observe. Both factors carry significant uncertainties, according to the astrobiologists Cole Mathis of Arizona State University and Harrison Smith of the Earth-Life Science Institute of the Tokyo Institute of Technology, who explored this kind of reasoning in a paper last fall.
My reply: I guess it’s fine to do the calculation, if only to make it clear how dependent it is on assumps. Bayesian inference isn’t just about getting the answer; it’s also about clarifying the mapping from assumptions to inference to decision.
Come to think about it, that last paragraph remains true if you replace “Bayesian inference” with “Mathematics.”
I think this post highlights a function of modeling that is often missed by the modern fixation with models as predictive tools: A good scientific model shouldn’t just produce data that looks like the data you’ve already seen, it should also help you understand why the data might look that way. Prediction is related to understanding—if your model doesn’t fit the data, that tells you that one or more assumptions in the model are wrong—but it is not sufficient. Often, as in the post, the primary value of a model is less in its predictions than in how it lays bare the hypothesized logical relations between observables and latent (assumed) constructs.
gec wrote: “A good scientific model shouldn’t just produce data that looks like the data you’ve already seen, it should also help you understand why the data might look that way.” I think that gec’s term “model” could be replaced by the term, “theory”.
As I recall, there is a similar sort of statement relating to the term “Scientific Theory.” The honorific term, “Theory,” as in, for example, “Theory of Evolution,” “Quantum Theory”, “Theory of Relativity”, is an honorific term bestowed when it explains data which has occurred AND successfully predicts and explains data that will occur.
Unfortunately, outside of science, the term “theory” often means mere idle speculation and is deemed inherently inferior to “data.”
While I realize it is a controversial position, I consider many models to be realizations of theory. A model can translate the concepts and relations posited by theory into a formal framework that couples those abstract entities with observables.
To be clear, I don’t think all models are realizations of theories—it is possible to have purely descriptive/predictive models and even just aesthetically or intellectually appealing models. But a model, like the one in the post, that is part of a scientific argument deserves to be judged not just by its predictive qualities (and maybe not at all) but by how much insight it grants.
” The purpose of computing is insight, not numbers.”
Richard Hamming: Numerical Methods for Scientists and Engineers (1962) Preface
I was hoping some fervent Bayesians would chime in to denounce this usage. Maybe they still will.
I like Palko’s example of what I call Bad Bayes. The uncertainty we are managing here is the possibility that something is happening that no one has ever thought of,* an abiotic process that produces a gas that we only associate with biotic processes. On a different planet. Uncertainty can take different forms; this one has a touch of perfection to it. We have many ways of knowing the world around us. None of them help.
Palko is being generous when he describes the Bayesian model input as “educated estimates for the first probability [which can] vary widely with some close to 0 and some close to 1 with no good sense of the distribution” (although I do admire the phrasing!). It might be more accurate to call them wild guesses by educated persons. Bad Bayes is the homunculus on your shoulder telling you that all you REALLY need to build a Bayesian model is a few numbers to punch in that need not be anything more than guesses in the form of probabilities.
*In the Wired article Peter Vickers call this uncertainty “unconceived alternatives,” I prefer “poverty of the imagination.” Donald Rumsfeld called it “the unknown unknowns.”
With regard to Rumsfeld’s “the unknown unknowns,” I recall that he was roundly criticized for that flight of linguistic fantasy regarding that unfortunate period of American history. Rumsfeld is no longer with us and “the unknown unknowns” is a phrase to be avoided because of all it evokes.
On the other hand, just a tiny bit ago there was ridicule regarding the thought of buying Greenland and, by golly, that concept has now been resurrected. So to speak, we are once again reliving the (immortal) concept of always shooting a dead horse because we can never be sure it is dead.
Paul, aren’t you the guy that came up with name “Seward’s folly” for Alaska?
I see no problem. There is no proof or disproof in science, it is all relative. Ie, the posterior probability depends on how well the various explanations fit the data.
Where D is data and H[0:n] are the various hypotheses/theories/explanations put forward, the denominator of Bayes rule is: sum(p(H[0:n])*p(D|H[0:n])
If someone comes up with a new theory, then that changes the posterior for all the existing theories. Some, like god/aliens did it, have very wide likelihoods (they are vague theories that explain anything). Those can often be dropped from the denominator due to being negligible relative to the others.
If the posterior for a given theory seems “too high” to someone, that usually means the denominator is ignoring some other explanation(s) they find plausible (ie, implicitly setting them to zero prior and/or likelihood).
“There is no proof or disproof in science, it is all relative”
I think there is proof drinking a class of arsenic will kill you.
That specific statement has a lot of problems (eg, dosage and there are people with arsenic tolerance), but it doesn’t really matter.
There are always other explanations. Eg, maybe in all those previous instances people drank water contaminated with arsenic + deadly thing, or it depends on the exact isotope.
https://en.wikipedia.org/wiki/Affirming_the_consequent
I have enjoyed the stimulating comments but we haven’t differentiated the purposes of modelling and the disciplines in which they are put to work. Modelling in health/epidemiology for example can serve various aims that are distinct from other disciplines but also have some overlap. Causal inference is an overlap with physics for example but epidemiology can have descriptive and predictive models that are distinct from the aims of physics modelling. Also, talking about Bayesian or frequentist approaches, what exactly is probability?
Don:
For “What is probability?”, I recommend chapter 1 of Bayesian Data Analysis. Also this post from 2018 entitled, “What is probability?”. And a bunch more links here.
Defenders of the Drake Equation often use the “we’re just trying to clarify the problem” defense but no one actually uses it that way. Drake originally just plugged in almost one for all his estimates and the resulting silliness has has become common wisdom. Serious people and respectable publications treat the Fermi paradox as a given and flakes like Robin Hanson (yes, that Hanson) are treated like deep thinkers when they talk about the great filter and grabby aliens.
One day the Drake Equation is going to be used to determine funding for Starfleet Academy and you’re going to have to eat those words.
I’m already on Elon Musk’s shit list.
Mark:
There’s no way you’re on Elon Musk’s shit list, no? I assume he hasn’t ever heard of either of us. He spends his time on twitter, not reading blogs, right?