Hi, this one comes up from time to time so I thought I’d devote a whole post to it.
The question is: what is conditional probability? And here’s what I wrote:
Everyone agrees that P(A,B) = P(A|B)*P(B).
The question is, what comes first?
In traditional probability textbooks, P(A,B) is defined first, then P(A|B) is defined as P(A,B)/P(B), and the conditional probability is only defined if the joint probability exists.
But it is completely consistent with the mathematics to first define P(A|B) and then to define P(A,B) as P(A|B)*P(B). In that case, the conditional probability can exist even if P(B) is not defined and even when there is no joint probability.
The math works either way.
This question sometimes arises in a non-Bayesian perspective, how can you write p(y|theta) at all—isn’t it meaningless if theta is not a random variable? The answer in that it’s perfectly kosher to define the conditional probability on its own without being part of a joint distribution. Even though it’s not always written that way in probability texts, there’s no mathematical need for the joint distribution to come first.
P.S. We’ve had some interesting discussion in comments, and I’d like to echo some of the comments here that the math can be tricky. If you define probabilities in a nested way (first P(A|B) and then P(B)), it’s no problem at all, but if you start trying to define P(A|B) and P(B|A), or P(A|B,C) and P(A|C), etc., then things can get tangled fast. Indeed, this was the problem that resulted in my notorious false theorem (with correction here). Anything that can confuse Terry Speed, me, three referees, and a journal editor—that counts as legitimately confusing! And with Stan we just work with the joint distributions, not with conditionals at all. So I do think there’s more to be said on the matter; maybe my above post was too glib.
That said, I don’t think notation such as P(A;B) resolves anything at all. If you use P(A;B) based on the principle that you can’t include B in a conditional probability statement if there’s no joint distribution, then you’ve already postulated a separation of your variables into “random variables” A and “constants” B, in which case you can separate the problem by first specifying P(A|B) and then separately specifying P(B) if needed. By using the P(A;B) notation, you’ve already restricted your problem enough that all the mathematical difficulties with P(A|B) would disappear in any case.
Dear Prof Andrew,
I’m a research fellow in Rotterdam, The Netherlands. I’m studying on a project and really have some advices for statistical methods for analyzes. I saw your blog and your conversations. Can I ask you about my project? With your kind permissions, I want to tell in a very short paragraph about my project:
We implanted 2 different types of stent into coronary arteries of 8 pigs. We used one type of stent for one coronary artery in animals. And each animal had 3 stents in their 3 coronary arteries. But in 1 coronary there is only one stent. Totally we had 11 stent-A 6 stent_B. After implantation we did intravascular imaging and computational fluid dynamic study to get shear stress values for these stents. For each stent we divided the stent into cross-sections longitudinally, and in each cros-section we divided the cross-section into 72 equal subunits with 5-degree angles each other. For each subunit we got a shear stress value. For one stent nearly we had 100-110 crossection and 7000-7200 shear stress values from subunits. Totally we had 30000 shear stress values from stent-A and 60000 shear stress values from stent-B group. But we couldn’t decide which methodology is more suitable for comparison between the two types of stent group; GEE or Generalized multi level Lineer model? Which one will be more suitable for our analyzes, Prof? Could you please give advise to us? I really need help for that issue.
Some comments:
1. The notation p(y|theta) induces the practitioner/statistician to apply rules of inferences that are not allowed in the classical statistics. The statistician “infers” that the operation p(theta|y) = p(theta)*p(y|theta)/p(y) is a theorem in probability theory. However, for this to be a theorem you have to assume that (y,theta) is a random vector well defined in a common sigma-field endowed with a common probability measure, which is NOT the case in classical statistics. Therefore, this kind of transference rule is not allowed among quantities that are NOT of the same type (the same type means they lie in the same sigma-field).
2. In formal mathematics, a generic definition has definiendum and definiens. Every definition must be non-creative and eliminable. That is, the former means that from the definition we cannot derive new results that couldn’t be derived without it, if we can derive new results, then the definition produces contradictions; the latter means that at any time the definiendum can be replaced by the definiens. In our case, the statistical books usually defined: P(B|A)=P(A,B)/P(B) where P(B)>0. The problem with it is that it is not even eliminable, since when P(B)=0 you cannot replace the definiendum P(B|A)=P(A,B)/P(B) by any definiens.
3. “Even though it’s not always written that way in probability texts, there’s no mathematical need for the joint distribution to come first.” It depends on what you mean by ‘mathematical need’ and which level of formality you are willing to consider.
All of this applies in the fundamental level. It is evident that some rules of inference that are employed in the Bayesian statistics are not allowed in the classical statistics. That is, the classical statistics cannot be interpreted inside the Bayesian language without applying systematically prohibited rules (in a positive sentence: you must apply systematically prohibited rules in the classical statistics in order to interpret it inside the Bayesian language). In general, Bayesians tend to find this irrelevant, since they want to interpret everything through their language. But it is the same as using the law of the excluded middle in the intuitionist logic: it is NOT allowed! if you use it, then it is NOT the intuitionist logic!
I will presume to channel Box in response to Alexandre’s comments: “All models are wrong but some are useful” (and refer those who might not be familiar with this to https://en.wikipedia.org/wiki/All_models_are_wrong)
Hi Marta, I did not understand the connection with my comment.
Alexandre,
You were talking about formal definitions and rules — i.e, formal mathematical models. But when we are doing applied statistics, we expect models will always be “wrong”in the sense that they do not fit perfectly. In particular, a less formal model (such as Andrew proposed) may be more useful, even if one can point to formal technicalities that might make it “wrong” in a formal sense.
A corollary to Box: “Don’t let the perfect be the enemy of the good.”
Models will always be “wrong” in some sense when we are doing applied statistics. It is not a critique for what I said.
A statistical model must be built under consistent rules and methods. If it does not work well for the observed data, at least it does not generate contradictory predictions/descriptions.
If you have a statistical model built under inconsistent methods, then you have a big problem. Do you understand that they are different things?
I must add that you can detect that a statistical model is “wrong” in practice, mostly because the model was derived under consistent methods. Otherwise, your model could predict any thing and be useless.
As in classic Logic: “from a contradiction anything follows”:
1. If “a AND not-a” (assumption)
2. “a” can be derived (rule of inference: conjunction elimination)
3. “not-a” can be derived (rule of inference: conjunction elimination)
4. From 2. “a OR b” can be derive for any b (rule of inference: disjunction introductions)
5. From 3. and 4. b can be derive (rule of inference: disjunctive syllogism)
Therefore, if “a AND not-a”, then b.
That is why we impose for our statistical methods to be consistent in a strong way. If you model produces trivial contradiction, it is problematic, unless you use a paraconsistent logic. Probability theory depends on the classic logic to derive theorems.
No no no no: this isn’t abstract measure theory. We are not creating a conditional distribution on an arbitrary measure space (which would also require a distinguished function to define the conditioning, but that’s besides the point). We are doing it on the product space of the space of models, often parameterized with real numbers theta, and the space of measurements, D. Both have their own sigma-algebras and the product space assumes a sigma-algebra generated by the product topology, and then the likelihood or posterior can be thought of as disintegrations along the product space projections. Because of the product space topology we can always start with a disintegration and a marginal and reconstruct the joint, even when conditioning on measure-zero sets.
Don’t use pedantry for evil.
Michael, you simply did not respond to any of my points. There are rules of inference (logical rules) that are allowed in one language but are not allowed in another. As for conditional probabilities, I was writing specifically about the definition Gelman posted here. We can defined a conditional probability in a probability space via sub-sigma-fields and avoid the problem with sets with zero probability. You can of course start from the disintegration and marginal and reconstruct the joint probability model, but in this case you are in a Bayesian perspective.
Gelman wrote:
“This question sometimes arises in a non-Bayesian perspective, how can you write p(y|theta) at all—isn’t it meaningless if theta is not a random variable?”
If you are in a non-Bayesian perspective, you would not impose a probability model for Theta. This is justifiable in a Bayesian perspective but not in a non-Bayesian perspective. In a non-Bayesian perspective, we can deal with all subsets of Theta, even the non-measurable ones. The reason is simple: some inferential rules are NOT allowed.
If you interprete classical statistics with Bayesian language, you must use prohibited rules of inference in the domain of classical statistics. I must quote myself: “it is the same as using the law of the excluded middle in the intuitionist logic: it is NOT allowed! if you use it, then it is NOT the intuitionist logic!”
I hope you are not gonna be mad at me with the following. At the end of the day, a Bayesian model is a simple probability model, since for the Bayesian rule to be a Theorem you must have an underlying probability space where both X and theta are measurable functions. A classical statistical model is a meta-probability model. If you impose some specific rules, you project the meta-probability model in a probability model. You cannot claim that the latter procedure is more general, it is actually more restricted.
If you want to discuss it personally, I will be happy to show you in a blackboard exactly what I mean. They are not just words to confuse you. As you say: “Don’t use pedantry for evil”, it seems that you did not understand some passages of my text, it can also be a side of effect of my terrible English (it is not my native language). Anyway, I see no reason for you to not be respectful with me.
From my own perspective, I just throw out measure theory. I don’t think it corresponds to how math is used in modeling. To replace measure theory, if I have to justify something mathematically (almost never) I work in Internal Set Theory of Edward Nelson. It just fits better with my notions of what people doing modeling mean.
In Internal Set Theory we can easily see that there exists a finite set that contains every model that will ever be considered by any actual person in the past or future of the human race (there will always be a finite number of humans and each one will live a finite number of seconds, and will consider at most a small number of models per second). Since this is a finite set, we can easily put discrete probabilities over each model in a given situation (ie. conditional on what we’re modeling). In IST it is possible to prove that there is a finite set that contains all the “standard” numbers (and some non-standard ones too!) more or less this is the same thing.
As for conditioning on a set of measure zero, it never occurs, we condition on a set of infinitesimal but non-zero size. All the “apparent paradoxes” I’m aware of simply evaporate and we can get down to actually doing models pretty quickly. Most of the time people explore one or two or a small set of models anyway. They do this basically as a shortcut, truncating models that seem to have small enough prior probability out of consideration…at least until the data makes the models under consideration very unlikely and then “back to the drawing board” which means “reach into the bag of models and pull out something we had ignored as too improbable previously”
I take you to mean that classical statistics using classical definitions etc has problems with starting at the conditional probability level… I suppose I just don’t care. Happy to be a Bayesian, and happy to acknowledge that classical statistics is basically wrong-headed most of the time for a variety of reasons.
Daniel,
I am not familiar with the Edward Nelson’s Internal Set Theory. I now that he claimed to find an inconsistency in the Peano’s arithmetic, which was a result of a mistake found by Terence tao (https://golem.ph.utexas.edu/category/2011/09/the_inconsistency_of_arithmeti.html#c039553) and recognized by Edward Nelson (https://golem.ph.utexas.edu/category/2011/09/the_inconsistency_of_arithmeti.html#c039590). The most accepted set theory is the ZFC (Zermelo and Fraenkel plus the axiom of choice) and all formal apparatus to defined statistical models can be defined in a model of ZFC set-theory.
“I suppose I just don’t care. Happy to be a Bayesian, and happy to acknowledge that classical statistics is basically wrong-headed most of the time for a variety of reasons.”
I’ve read so many arguments against classical statistics. The majority of these arguments try to interpret the classical statistics inside the Bayesian language, which is a basic error of applying systematically prohibited rules of inference. Let us meet personally, this kind of subject is better to discuss face to face to a avoid misunderstandings. I am in Ottawa, Canada.
Yes, IST is a proven-conservative extension of ZFC. It offers us nothing new, but it gives us a different language for talking about things. I think that language happens to match the requirements of people building mathematical models of natural phenomena more closely than does measure theory. A little like how C and R are both computer programming languages which can be reduced to some assembly code and therefore have exactly the same expressive power as the instructions of whatever processor we run our code on… but I choose to use R over either C or writing in assembly when working on statistical models.
I am in Los Angeles, so I am afraid we will not so easily meet. The internet is a useful medium of communication for such cases, though I agree limited.
As far as interpreting the classical statistics inside Bayesian language… yes, I have basically drunk the Cox/Jaynes Kool-Aid and am happy to interpret the Classical Statistics as flawed from this perspective, but I don’t think it has to do with measure theory or whatnot. The biggest problem in classical statistics is that it uses the concept of “IID trials” to replace the concept of “physics”.
Daniel,
Nice to read you. Classical statistics is much more than measure theory. A classical model is a meta-probability model, it is beyond measure theory. I never said that only measure theory is sufficient to explains classical statistics (I really do not know what people repeat this here all the times). Many theorems of measure theory can be used in classical statistics, though. Classical statistics can be formalized in a model of the ZFC set-theory as the majority (or should I say all?) of mathematics. It is at least foolhardy to say that writing statistics in terms of well-defined sets must be avoided.
“I have basically drunk the Cox/Jaynes Kool-Aid and am happy to interpret the Classical Statistics as flawed from this perspective”
I have too. Notice that I have explained here many times why the Cox/Jaynes/Paris/DeFinetti/Lindley arguments do not say that “probabilities SHOULD be use for modelling all types of uncertainties.” There is a domain of applicability of their results, I already gave a long list of papers proving the in some domains probabilities must be avoided. Did you read some of them? let us discuss them if you’ve read.
“The biggest problem in classical statistics is that it uses the concept of “IID trials” to replace the concept of “physics”.”
classical statistics have many models that do not use iid trials. You can just make assumptions and use classical statistics. You can model many kinds of dependence structures for the errors and random quantities. What is you experience with classical models?
The real problem is that people are already convinced about these issues, they do not want to read other things. It an intellectual confirmation bias, which is too bad since there are many richness beyond probability models.
I have not read these papers. Here is a question for you though. Is the reason why “in some domains probabilities must be avoided” because it leads to some problems with formal definitions? Or is it because in some applications it’s found that it gives wrong answers to predictions or inferences about a real scientific model?
When it comes to statistics, I think people who find flaws at the level you are concerned with tend to start from too few assumptions. People doing applied statistics start from very strong assumptions, many times not very explicitly (ie. they may not realize they are making such assumptions), even when they think they are making “weak” ones.
As a very basic example, most people would make the assumption that it is wrong to interpolate a time series when it leads to a function that oscillates wildly. You can formalize this as them putting high probability on “smooth” functions and low probability on “wildly oscillating functions that go through all the data points” but I just don’t think we get anything of practical value out of trying to quantify that probability, or of trying to formalize some alternative to probability… people just choose smoothing over interpolating because it suits their needs.
Of course, sometimes people actually DO put probability measures over functions: gaussian processes, etc. But even when they do that, they choose covariance kernels from among a small set because that kind of kernel “meets their needs” well enough. Formalizing probability measures or alternatives to probability measures over choice of covariance kernels… it doesn’t float my boat.
Alexandre: I think my summary of your concerns is that within the domain of “model choice” we don’t know which model to use, so there is uncertainty, and in this domain we need not put probabilities over models… and I honestly think that we probably kind of agree here. Although I have no problem with saying that in a particular case there is actually a finite set containing all the models I will ever consider, and there is in principle some approximation to my own state of knowledge which could be formalized via a probability model on the finite set… I don’t DO that, in practice I choose a model, or maybe I choose a few models, and I work with it unless and until I don’t like how well it’s working… and then I try something else. I’m never getting anything out of a formal structure over these models… I don’t do inference on the full set of models, I don’t run computer programs using the prior probability over the model choices… in fact, to make my life easier, I don’t even specify anything about the alternatives to the one or two or a few models I do actually use.
But, if what you’re interested in doing is building formal structures out of this model choice aspect of statistics. I guess, more power to you, but I’ll await the computer program that exploits this structure to make my life better before I drink that particular flavor of Kool Aid.
Daniel,
I should say that it is very pleasant to discuss with you and very tiresome to discuss with many others who do not want to think about it. I would like to ask you if I am rude or closed minded or whatever? Because I really do not understand why some guys are so rude and disrespectful with me here. It is rare to find someone to discuss like adults.
I already explained the results of those paper here in some place. They are basically:
1. if you use probabilities you will lose for sure. It all depends the way you define the game. I really do not know why people do not realize that in the game proposed by de Finetti the wage is linear Sp, if you replace it by f(S,p) you can have may type of rules, not just the sum.
2. the cox assumptions do not necessarily reflect coherence. You can instead consider that the function F(.,.) is only non-decreasing. Then, possibility measures are also justified.
3. In general knowledge is not sharp, you have problems to elicit sharp prior probabilities. It justifies family of probabilities models.
and the listing goes on… If you read the Cox’s paper and de Finetti’s, you now that de Finetti already derived the Cox’s results in 1937.
I am really concerned that people are convinced without reading critically the main sources and they refuse to read other literature. It is intellectual confirmation bias. Times goes by and they seem to be more dogmatic. Don’t you think?
Daniel,
Here I commented for you, in our previous discussion, on the results of the papers:
https://statmodeling.stat.columbia.edu/2015/07/03/why-should-anyone-believe-that-why-does-it-make-sense-to-model-a-series-of-astronomical-events-as-though-they-were-spins-of-a-roulette-wheel-in-vegas/#comment-224141
To give you some flavor of IST. When someone specifies that X is normal(0,1) in standard mathematics, the definition relies on Lebesgue measure dx and the normal density. The probability of an outcome between a and b is integral(normal(x,0,1),a,b) using lebesgue integration etc…
The typical thing to do in IST would be to interpret the statement X is normal(0,1) in the following way:
The actual value of X is one of the following values 0, 1/N, -1/N, 2/N, -2/N…. N/N, -N/N … N^2/N, -N^2/N
the probability of X being say Q/N for Q N^2
in this system there are a finite (but nonstandard) number of actual outcomes. That agrees with reality where even the most precise measurement devices are typically something like 30 or 40 bit A/D converters… In reality, the number of possible outcomes is not only finite, but well defined (2^32 for example) but by choosing to use a continuous probability model, we are acting really “as if” it were vastly larger.
There is a lot more to this, but in most cases, I think this type of interpretation matches my intuition about what I am doing when I build models. The vast majority of the time I am secretly modeling some discrete situation using a continuum as an approximation. When I put a gaussian process on some “function” I’m not doing probability on function spaces, I’m doing probability on a finite but potentially large set of samples. IST lets me connect that to function spaces in a straightforward way.
All this is to say that I enjoy having a foundation that “makes sense” in the context of models as approximations, and that foundation isn’t measure theory.
argh, blog ate everything after a less than sign, so you will have to fill in some blanks. DANG YOU WORDPRESS!!
But a version of measure theory can be derive in a model of IST, can’t it?
So far, the mathematical foundation for statistics is not measure theory, is ZFC set-theory.
IST contains all of ZFC, plus 3 extra axioms, so measure theory is automatically contained in IST in the usual way, it is what is called part of “standard” mathematics (since it doesn’t use the “nonstandard” axioms). However, the usual way of working with functions etc within nonstandard analysis is not necessarily the SAME as the measure theory way. For example, here are two integrals:
sum(i=1..N, f(x+i*dx)*dx)
(the nonstandard integral, where N is a nonstandard number and dx is an appropriately sized infinitesimal, the function f is evaluated on an infinitesimal grid)
sum(i=1..N, f(st(x+i*dx)) * dx)
(this is the lebesgue integral. In IST thanks to the infinitesimals, you can define it in a simple way, but it *is* different from the above, since at each integration point, the standard value of the argument is used to calculate the function value f, it is not evaluated on a grid, it’s evaluated at only the unique standard number nearest to each grid point)
so, the nonstandard approach gives you new ways of working with things. In particular, it directly gives you “separation of scales” which corresponds in an algebraic way to the kind of thinking typified by asymptotic analysis. Since almost all modeling is really asymptotic analysis (ignoring things when they are small enough). I think it maps well into the thought pattern of modeling. For example see Andrew’s post on first, second, and third order corrections:
https://statmodeling.stat.columbia.edu/2015/11/18/first-second-and-third-order-bias-corrections-also-my-ugly-r-code-for-the-mortality-rate-graphs/
I guess the point of all of this is that if the formalism doesn’t suit the real world need, it’s the formalism that goes out the window, not the real world need.
Daniel,
thanks!
Could you pin-point a statistical analysis that ZFC set-theory cannot be used to formalize while the IST can?
I think the formalism does not serve to suit real world problems, it serves to give you consistent basis for you to build a model that possibly explain the real world problems or even abstract worlds.
If the formalization is not enough, you extend it. For instance, the usual membership “in” is a binary function: x in A (x in A or is not in A). It is possible to extend the ZFC set-theory for the case where the membership function is not binary (which is what fuzzy set theory does). The equality can also be defined in term of functionals. The set-theory can be viewed as a sub-product of function-theory and vice-versa.
Actually, what matter is that your theory is built inside a consistent system. Of course that if the system is consistent and it can generate at least the natural numbers, then using its own language (axioms and rules of inferences) it is not possible to proof its consistency (Gödel, Incompleteness theorem).
People here seem to think that the (statistical) probability theory does not depend on any extra formal systems. Of course it depends! How do you proof any theorems inside probability theory? the rules of inference to derive theorems in (statistics) probability are not part of the probability theory. Foundational issues depend on the allowed rules of inferences. Probability theory cannot handle with its foundational issues by itself, you need a meta-language inside the same formal system to analyze the foundational issues of probability/statistics.
In order to understand the differences between Bayesian and classical statistics, or to judge if former is better than latter, we must understand their domain of applicability and which rules they use to make inferences. For me, it is evident that you can simply impose Bayesian language to interpret every thing, if you do that all of your results have a label “BAYESIAN LANGUAGE BEING APPLIED”.
The explosion principle that I derive in a previous post (https://statmodeling.stat.columbia.edu/2015/12/12/defining-conditional-probability/#comment-254588) is not derivable if we block one logical rule of inference. The same occurs with classical statistics, some conclusions make by Bayesians simply does not follow in the domain of classical statistics.
Alexandre: IST proves nothing new about objects in ZFC. That is what is meant by “conservative” extension. So, IST gives us nothing new and we can ignore it? Just like R gives us nothing new and we can program in intel machine language!
Except, R does give us something useful, a way to do things that makes things clearer to the user, even if the result could have been gotten by programming in machine language.
So, it’s not a matter of IST giving us something different, as much as IST’s approach matches the approach that intuitively people use when building models: they ignore “small” things, they work with finite sets of data, finite precision of measurements, they do first or second order thinking, they safely ignore the fact that infinite series don’t converge because they only use a few terms and the approximation is already good! etc etc.
I see from your comments in various places that you seem to agree with me: possibility theory is a way to formalize the process of choosing a model from among many options where no obvious probability measure is available, and maybe some other related issues. It doesn’t add anything to an analysis that is already a Bayesian Hierarchical model for example, it is “meta” to that model in some sense.
I think the resistance you get here is that people already know that they have to choose a model, that they might have chosen the wrong family, that they should consider other options… and they have ways of doing this which are NON-FORMAL and they don’t think investing time in FORMALIZING that process is going to add anything to their applied analysis.
Also, Bayesian statisticians are aware that the claims they make don’t make sense to classicists! This does not bother the Bayesian, though it certainly seems to bother the classicists!
Whether you describe what a Bayesian does in choosing a model in terms of some kind of possibility theory, or in terms of some kind of asymptotic approximation of a probability distribution over a finite but very large set of models… is really not helping the applied statistician get anywhere I think.
But, the computer program Maxima lets you do things like say: “take the total derivative of this complex expression and give me back all the terms involving the variable q or its derivatives” and it will grind out the answer in a formal way. The rules of algebra and calculus are mechanical to this program. This is something people want! That is what formalisms give you: mechanical rules to connect one description to another. So, I think for your possibility theory, you need some kind of application for the formalization before you will convince people that it’s worth the work of being formal.
Physicists used the Dirac delta function happily for years before a formalism was invented that could explain it! This is because it was useful in their analysis. Your possibility theory “feels” like an explanation in search of a use… but most things are adopted because they have a use and need an explanation. In fact, hunting through the possibilities for “good” models is already something that is DONE! But physicists didn’t rejoice when “distribution theory” was invented… They already had a Delta function etc. It was the mathematicians who rejoiced, they could figure out what those wacky physicists were “really” doing! They could also give those physicists some “rules” for things that didn’t really make sense… so that is another area you might consider. But keep in mind, if you tell applied statisticians “oh, gotcha! you are now formally no-longer allowed to work on this problem because you’ve exhausted some formal options” you will not get very far! And if instead your “possibility” theory is so broad that it doesn’t provide *any* restrictions of practical use to the statisticians…. it will not be very interesting either.
It seems that it must do something in the middle path: provide some helpful rules that can prevent statisticians from going wrong in ways that they hadn’t thought about, but DO care about, while allowing them to do anything that they currently do when what they currently do makes good sense.
So our choice is between keeping conditional probabilities or p-values?
In that case I’ll dump p-values faster the real statisticians dump measure theory.
What do P-values have to do with this conversation?
It’s a stand-in for classical statistics
We have other measures of evidence. P-values, S-values, C-values, Likelihood measures and so on.
Moreover, probability conditionals are part of the classical statistics. However, they are applied in some specific statements (or events).
One example:
A: “the bread’s weight is lesser than 30g” and B: “the bread’s weight is greater than 40g” and C: “the probabilities of A and B are computed from N(35g,2g²)”
A and B are of the same type and marginal, conditional and joint probabilities can be well defined in classical statistics given the information C, i.e., P_C(A), P_C(B), P_C(A&B), P_C(A|B) and P_C(B|A), provided that some requirements hold. The statement C is not of the same type of A and B, it is a statement about A and B, therefore it is not mixed with them.
Bayesian theory mixes all types of statements, i.e., P(C), P(A),… However, there is a problem here, since what does mean P(not-C) ??
not-C means “It is not the case that the probabilities of A and B are computed from N(35g,2g²)”, but how do you define a probability in a set of all probability densities except N(35g,2g²)?
The way Bayesian deal with it is postulating that all distributions must be normal (or another family), then relative to this restriction not-C is “well” defined. The real problem for me is that not-C is taken for granted in the Bayesian formulation, the real complication is simply not considered. That is one reason why Bayesian theory is an analysis of certainty, it is 100% sure (in probability terms) that the model is normal in this case. In Classical statistics we do not have any certainties of the statistical model, it is just a possibility.
That is one of the reasons Fisher argued against acceptation of a hypothesis, to accept an hypothesis, you must postulate a closed universe with is a fiction.
If we know that all models are wrong, is it reasonable to specify probability 1 for a model that we know is wrong?
It is, so long as we understand that it is an asymptotic approximation.
If you have a random variable that is modeled as normal(0,1) and someone tells you that it generated 30 data points and one of them was 10.301 and wants to know what is the probability that it will generate something bigger than 10 under the model assumption… you answer “zero” but you know that it is really “epsilon” which is very very close to zero, so close that no one really cares. The fact that you observed a 10 suggests that your model is wrong, not that you observed an event whose probability of exceedance is truly 7.6×10^-24
That is the different between probability and possibility. Possibility can be positive when probability must be zero. The likelihood ratio statistics is a type of possibility measure. The s-value is a type of posterior possibility measure.
Possibilities do not have the problem of non-measurability.
basically Classical statistics can be viewed as a theory of possible models
It starts saying that M is a possible family of probability models for the random variables.
It finishes saying that M0 is a family of probability models most plausible according to the maximum likelihood. You can also consider the family Ms which contains all probability measures that generate likelihood functions greater than “s x the maximum likelihood in M”.
We can invent many other quantities, since we have freedom of creation because the classical statistical model is rich and powerful. We can inclusive use Bayesian modeling, you just say that M = {P} is a family of one element that define the joint distribution of all random variables involved.
We have different types of rules. Internal rules (which are probabilisitic) and external rules (which are possibilistic). Probabilistic rules are defined just for events that can be defined in a closed universe. Possibilistic rules are defined for statements/events that cannot be well defined in a closed universe.
This discussion is like fish arguing over whether it’s possible to define water.
+1
It is important at least know what means a definition to avoid ambiguity and contradictory results.
One trivial example: Let A, B and C be natural numbers. Let # be a binary operation such that A#B = C if and only if A<C and B<C.
You can derive many Theorems from this definitions. Notice these two cases:
1) For A = 3, B = 4 and C = 5, we have by definition that 3#4=5.
2) For A = 3, B = 4 and C = 50, we have by definition that 3#4=50.
Then, we have 3#4 = 3#4 and by transitivity of equality we also have 5=50. That is, our definition is creative, since it generates new results not allowed in the system it was defined.
We define many objects in statistics and probability without verifying if they are eliminable and non-creative. How do you guarantee that all definitions in statistics and probability are free of such problems? I am 100% sure that there are many published papers the considers problematic definitions.
Maybe you are right "This discussion is like fish arguing over whether it’s possible to define water."
Alexandre,
I love measure theory. Unlike most stat people I was already highly proficient at measure theory and real analysis before ever taking a stat course. But it has nothing to do with statistics.
Measure theory is second only to frequentism for having wasted the most man-hours on irrelevant research questions. Measure theory is completely irrelevant to either applied statistics or foundational questions.
You think measure theory will give you some special insight into statistics. You think it will give you an edge or tool for studying the foundations of statistics. You are absolutely 100% wrong. The only thing it will do is waste so much of your time you never get around to anything important.
Anonymous,
I am very curious to see why measure theory has nothing to do with statistics. I am presenting examples here, I am not just claiming that something is wrong or right.
I am not talking about measure theory here, though. I am talking about rules of inferences and logic and definitions. If for you the ZFC set-theory is not appropriate to formalize statistics, then show why. I would enjoy to see such examples or at least a good argument.
I think there are aspects of statistics which simply do not NEED to be formalized. The process of building a model is a creative one, involving using your brain to think about how the world works. It doesn’t need to be placed into a formal cage and made into something that could in theory be carried out by cranking a handle….
Now, I use computer algebra systems, and I love the fact that I can crank a handle and carry out a transformation and get an answer which is more or less implied by my inputs. So if you offer me a computer language in which I can put in very simple description about the structure of a natural system and you will allow me to push a button and get a fairly complex Stan program out of it that I agree with… I will start to pay attention.
But, I suspect that your system will require more input than if I just wrote the Stan program myself. That is in some sense what I mean by “we don’t need to formalize some things”.
Daniel,
The process to build a model must be creative, but it must be also consistent. It is not only creativeness that is needed.
To program you MUST follow strictly rules, you know and I know. I already programmed a lot to publish my papers.
Try to program a Bayesian model with improper priors. You will end up with inconsistent results generated by improper posteriors. If you do not know the underlying theory, you will be using a statistics as a blackbox.
I can give you a number of examples where you can end up with real problems if you do not know theory or implicit rules of inferences that are behind a statistical theory.
Not knowing theory is a big problem. No question. However, it does depend what the theory describes. If the theory describes the formalism of quantum mechanics, and you are working with ocean circulation currents, you really do not need that particular formalism.
So, what is it that possibility theory offers a Bayesian? As near as I can tell, it is a way of describing formally something that is already being done. Namely, trying out some models, and then sometimes rejecting them and trying some other ones, without actually putting probabilities on the models.
But, when a Bayesian statistician is explicitly interested in working with multiple models at once, they will typically assign them each a probability and do Bayesian analysis on them. This is fine, since ultimately they’re working with a small finite set of models. If in the end they reject the whole lot and go to yet another model… my suspicion is that you would say “aha, this model had probability zero before. probability failed you. possibility theory is needed”
but it is only needed if you want to formalize this process… but statisticians aren’t interested in describing the process that statisticans go through.. they’re interested in describing the real-world science process that generated the data (medical, or biology, or chemistry, or ecology etc…)
But, the Bayesian can do more than that if they like. they can respond to your criticism something like this:
“My previous selection of 6 models with assigned discrete probabilities was an asymptotic approximation to my real state of knowledge. There are of course millions of other models to consider, and I chose to assign zero probability to those which had sufficiently small probability… but now that I know that none of my 6 models work well, I can always go back and look at the next largest probability models which I had truncated to zero probability previously. In fact, they simply had ‘much smaller’ probability than the original 6.”
IST and its infinitesimals lets us formalize this notion of “so small I considered it as if it were zero”.
As for the Cox issue and its reliance on dividing by the derivative. I suspect that this may be a purely formal “problem” which goes away when you are working on a finite but nonstandard probability space. I’m not sure though.
I agree with you when it comes to applying statistics. But we are discussing foundations of statistics. In foundational discussion much more is needed than running programs and seeing the final result.
We are discussing about logical systems that make inferences. And, based on some issues of the reality, you built models based on assumptions and rules of inferences. Some assumptions may be dropped and some rules of inferences as well to derive more appropriate inferences in real world. For instance, you may have a concern with confirmatory biases, therefore you can block some rules of inferences to avoid this issue…
I expect that the vast majority of people who read this blog would be happy to work within the following reduced formalism, particularly as Stan improves its ODE solvers etc… though we could add a few additional Bayesian inference programs to help with PDEs and other specialized whatnot:
“The set of models is the set of all Stan (or a small set of other languages) programs less than 1 terabit (10^9 bits) long that compile and run in less than 1 month of wall time on a modern desktop computer”
Clearly as the models are less than 1 terabit long, there are fewer than 2^(10^9) of these models. Since compiling is a big restriction, probably there are MANY fewer. But, this is a finite number. We can add the following rule:
The prior probability over a model is proportional to 1/N if there are N models being considered, unless otherwise explicitly specified, and proportional to 1/2^(10^9) for any models not formally considered.
For a person willing to work with this HEAVY theoretical restriction, what does your possibility theory provide? This heavy theoretical restriction is “very light” for the vast majority of statisticians. Most of them would balk at writing even 1 megabit of Stan code.
ack, proofreading problems… terabit is 10^12 not 10^9…
Agree with Andrew that starting from conditional probability as basic is the way to go. Agree with Alexandre that it may not be trivial to axiomatize.
Here’s an interesting philosophy paper on the issue: ‘What conditional probability could not be’ by Alan Hájek (2003, Synthese)
pdf: https://philrsss.anu.edu.au/people-defaults/alanh/papers/what_cp_couldnt_be.pdf
BTW I think (without claiming to have worked it all out) this may be one of the reasons behind Judea and Andrew (and me) disagreeing over do calculus vs hierarchical modelling etc. I would argue a hierarchical model takes conditional probability as basic and can represent any model expressible in do calculus.
Judea’s arguments against being able to express causal assumptions in probability language appear on a brief look to me to rely on taking unconditional probability as basic. I doubt they hold up if you take the ‘conditionalist’ view.
My impression is that “do calculus” lets you do things that distinguish different situations “mechanistically”. The same hierarchical statistical model can be used in two different situations: one in which there is merely predictive association between one thing and another, and one in which we actually assume there is a causal connection between one thing and another. The “do calculus” would let you create some formal description of the difference and therefore distinguish between the two cases mechanistically… which is just not a very high priority for me. For example, I rarely use theorem proving programs either, though I use computer algebra systems… so I agree that formal methods and automatic symbolic calculations CAN be useful
For me, the dissatisfaction is that I am perfectly happy explicitly claiming in words: “I believe that taking aspirin causes reduced pain… and so therefore I specify the following statistical model…”
I also have no problem distinguishing in words “we assigned people to take aspirin or not via a random number generator” vs “we observed a variety of people some of whom reported taking aspirin recently” and the whole point of “do” as far as I can tell, is that it distinguishes between “we made X happen” vs “we observed that X had happened”
If you’re already thinking in terms of what is causal and what isn’t, and you’re making fairly mechanistic models, for which you need estimates of unknown parameters… you probably sort of skip over the stuff that “do calculus” does for you. On the other hand, if you have not much mechanistic information in your models (ie. like you’re doing some kind of macro-econ or psych or something) then you might actually be more interested in all that jazz.
RE: The same hierarchical statistical model can be used in two different situations: one in which there is merely predictive association between one thing and another, and one in which we actually assume there is a causal connection between one thing and another.
My instinct is to actually disagree with this. Or at least claim it is not obviously true. The issue to me comes down to implicit ‘closure’ or boundary conditions, which may differ in the two cases.
Jaynes’ position, as I understand it, is that there is no such thing as unconditional probability, all probability is conditional on something. He points out that we frequently leave out (as understood) the conditions, but says that they are still there even if not written down. He also gives examples of errors of inference that arise as a result of failure to condition properly on all relative assumptions.
From this point of view, Andrew’s approach is the right one. Start with conditional probability as basic and derive the product rule from that. But I suppose Jaynes would like to write it this way:
P(A,B|C)=P(A|B,C)P(B|C),
where C is the conjunction of all those background conditions that we usually leave out.
I confess that I haven’t looked at Jaynes’ book recently to verify that he actually does it this way, but it would seem likely given his basic point of view.
[As Andrew correctly points out, doing it this way avoids the awkward “except when P(B|C)=0” provision.]
…should be “relevant assumptions.” in first paragraph.
That’s basically right Bill. Conditional probabilities predate any axiomozations of probability theory the way counting predates the axioms of arithmatic.
If a proposed axiomatization of finite arithmatic doesn’t comport with our more basic notion of counting then you need to find a different axiomatization. You don’t run around claiming counting doesn’t exist.
Bill,
I agree with you that “all probability is conditional on something”. However the conditional concept is wider and it doesn’t mean exclusively that it should be a conditional probability. Consider the cases:
1) A: “the coin lands on heads” and B: “the coin lands on tails”
2) A: “the coin lands on heads” and C: “the probability of A is 1/2”
In the first case, A and B are of the same type: they talk about coins turning up heads or tails (observable events). In the second case, A and C are intrinsically different, since C refers to A.
In classical statistics: in the first case, P(A|B) can be defined as a conditional probability, while in the second case, P_C(A) is a probability attained through a conditional knowledge, but it is not a conditional probability. We could define D as follows:
D: “The probability measure is the unique coherent way of modeling ALL uncertain events”
Would you write P(A|D) or P_D(A)?
And how about:
E = “The probability measure is NOT the unique coherent way of modeling ALL uncertain events”
Would you write P(A|E) or P_E(A)?
You might eventually find good justifications to say that A and D (or E) must be treated in the same level of the language. But it is also possible to find justifications to say that they are not in the same level and you must use different rules in each level.
Alexandre: good motivation for adopting a hierarchical approach, no?
ojm,
It depends on what you mean by hierarchical approach. Such hierarchical is embed in a unique probability model? then, I do not think so.
The hierarchical approach can be defined in a family of probability measures. The hierarchies might be probabilistic or possibilistic or deterministic or… We can have hierarchies among observable statements, among non-observable statements, among probabilistic statements, among mathematical statements and so on.
I prefer not mixing statements that are of different types. This implies in some sense that a unique probability measure is NOT sufficient to model all kinds of uncertainties.
‘Such hierarchical is embed in a unique probability model?’
I’d say no.
ojm,
Are you not considering the Bayesian model for modeling uncertainty?
There is no set of all sets. Presumably (?) there is no hierarchical/conditional Bayesian model of all hierarchical/conditional Bayesian models.
ojm,
yes, there is no set of all sets, at least in the ZFC set-theory. What do you mean by hierarchical approach? I can invent many hierarchical approaches, Bayesian and non-Bayesian. I’d like to know what you mean by it.
We can certainly model random variables (X,Y1,…,Yk) hierarchically:
X | Y1=y1 ~ P1
Y1| Y2=y2 ~ P2
… …
Yk ~ P{k+1}
you will have at the end a joint probability measure P induced by the vector (X,Y1, …, Yk). This can be modeled either in classical or in Bayesian models. If Y1, …, Yn are non-observable random variables, then they are latent variables in the classical model (such as in mixed models, structural model, item response item, errors-in-variables models and so on).
In the classical model, we have by definition a family of joint probabilities measure G = {P: P is of a certain type}, this means that we have many possible joint distributions to model the above hierarchical structure. You can impose a hierarchical structure on the elements of G with or without an external probability measure (a prior probability). If you impose this external probability measure over the subsets of G, you are projecting the statistical model in a probability model.
question: What if G is so big that it is not a proper set in the ZFC set theory? can you define a prior probability over its subsets?
‘I can invent many hierarchical approaches, Bayesian and non-Bayesian.’
Sure. I’m referring to the Bay si an flavour, in particular the one where all probabilities are conditional and hence require provisional closure assumptions for applications to real problems.
‘you will have at the end’
What do you mean by ‘at the end’?
‘You can impose a hierarchical structure on the elements of G with or without an external probability measure (a prior probability). If you impose this external probability measure over the subsets of G, you are projecting the statistical model in a probability model.
question: What if G is so big that it is not a proper set in the ZFC set theory? can you define a prior probability over its subsets?’
Exactly. I prefer to work with provisional model closures. Expanding the hierarchy is always an option.
ojm,
OK, you are referring to Bayesian models. It is difficult to define exactly a Bayesian model, since there are so many types of it.
1. Some Bayesians say that you must have a single prior distribution. In this case you have one single underlying probability model that governs all random quantities of your problem.
2. Some Bayesians say that you can have more than one prior distribution. In this case you have a family of prior distributions and a family of joint probabilities that governs all random quantities of your problem.
You can model either cases 1. and 2. with a classical statistical model (the mathematical definition of a classical model). The first case is a probability model, then you can simple reduce your family of probability measures to one probability. The second case we have the classical model by definition.
That is to say that the use of a prior over the family of probabilities just reduces the potentiality of a statistical model.
“At the end” was of rhetorical use.
‘use of a prior over the family of probabilities just reduces the potentiality of a statistical model.’
Agreed. To me this is a good thing – models need a closure. A prior is the last conditional probability you write down in a sequence of conditional probabilities. It is conditional on a non-random (wrt the closure) quantity.
This is fine (?) if you take conditional probability as basic rather than the ‘ratio’ definition and unconditional probability (for example). See the paper I linked for more discussion.
ojm,
I am reading the paper, but it will take some time, because I usually follow all steps and proofs.
You are saying about closure. Then, the closure makes your statistical model a probability model. Let me give you an example:
it is possible to model classically:
1. X| theta ~ bin(n,theta)
2. theta ~ Beta(a,b)
3. for a,b>0.
Then, you have a family of probability measures for each marginals, conditionals and joint distributions. You can estimate a and b and so on and so forth.
In you ‘closure’, do you have a prior distribution for the restriction in 3.?
In this particular model a and b appear to be non-random, so it seems the answer is no here (for me).
I can also imagine embedding that model in a further model in which a and b are random and c and d (eg – or should I say efg?) are introduced as non-random quantities.
OK, then what is the closure in this model?
Depends what you want to do with it.
As given, it appears to be a family of models index by a,b. Do you want to draw inferences about a and b or about X and theta?
In the latter case I would say that reasonable closure assumptions are something like
1. X | theta,a,b,c,d,e,f,… = X | theta ~ bin(n,theta)
2. theta | a,b,c,d,e,f… = theta | a,b ~ Beta(a,b)
3. a,b are known and >= 0
I could implement this on a computer.
If a and b are unknown or you want to compare models for different values, I would prefer to take a,b as random functions of c (for example) and take c as fixed an known. In general I think Andrew’s advice to distinguish inferences ‘within’ and ‘without’ a given model is sound. I would be nervous about drawing inferences about a or b without embedding them ‘within’ a model.
BTW I don’t actually disagree too much with a number of your comments here.
ojm,
Here a and b are unknown. As I understand, your position is then to build a unique (underlying) joint probability model, since if you will use a prior distribution for them you will have a single joint probability model.
I would use a prior for a and b without problems. But I could also consider a possibility measure over them and compute a possibility posterior by using the s-value.
Possibilities can model uncertainty and certainty while probability models only certainty. If you are completely unsure about a and b, the only thing you really know is the they are positives, then any probabilities will give a kind of information that you did not have before, while possibilities do not impose extra information.
I have no problem with the concept of a ‘possibility measure’, though I haven’t used it before.
I find the hierarchical bayes approach sufficiently flexible for most applied work but am open to other approaches, of course.
BTW I also find that many people don’t seem to understand what the hierarchical bayes approach means. In particular that it requires ‘closure’ assumptions, which are also testable through (for example) ‘significance test’ style reasoning (or embedding in a further model).
ojm,
it is flexible, but it seems to assume only probability measures for every uncertain statemets and events.
thanks for this nice discussion. Hope to meet you some day.
best regards
ojm,
thank you for the paper. I am reading slowly, pg 22.
RE: ‘it seems to assume only probability measures for every uncertain statemets and events.’
And your comment above about ‘internal’ probability vs ‘external’ possibility rules of inference.
I believe this is very compatible with the hierarchical approach Andrew has described many times. See eg his paper with Shalizi.
https://www.stat.columbia.edu/~gelman/research/published/philosophy.pdf
Basically this approach means
– be Bayesian (probabilistic) within a model (with a temporary ‘closure’)
– be possibilistic externally to the boundaries of the model
I’m sure you could formalise this somewhat informal description better than many (definitely me!)
Konrad,
You can mix these statements, but you can also not mixing them and treat them in different levels of modeling.
E = “The probability measure is NOT the unique coherent way of modeling ALL uncertain events, given Z”
Would you write: P(A|E)? and how about the knowledge that influenced your choice but you did not remember to explicit?
Remember that you cannot explicit all conditional terms you are using to build you measure. I see much more value in saying just that my mathematical/philosophical knowledge to build a measure P is in P itself. We cannot explicit everything in P(A|Z,…). Saying that everything must be conditional is redundant, moreover you will never operate with many conditional statements. Why do they must be written in P(.| ” your knowledge that even you don’t know exactly what it is” ) at all?
Alexandre,
There is a problem in your example 2. The context was you agreed that all probability is conditional on something. But in “the probability of A is 1/2” you did not specify what this is conditioned on. The conditioning is critical in the context of this discussion – the variable containing the information on which you are conditioning _must_ be included in your notation.
So if we define some other information state Z and change C to “the probability of A given Z is 1/2” (or: C: P(A|Z)=1/2), then C is just a statement about what can be inferred about A from Z (i.e. C is a mathematical rather than an empirical claim, and in principle we can use mathematics to decide with certainty whether it is true or false), and there is no problem with conditioning on this statement or with writing down the probability of it being true. Both P(A|C) and P(C|A) are meaningful (though we should probably make things more explicit by defining I as the set of background information and writing P(A|CI) and P(C|AI) instead of leaving that information implicit, and clarify whether the probability used to define C was P(A|Z) or P(A|ZI), since these are different).
Konrad, my response is up here: https://statmodeling.stat.columbia.edu/2015/12/12/defining-conditional-probability/#comment-254730
Sorry I replied to another post.
Ok, I’ll continue to reply here. You raise several points.
“You can mix these statements” – my point was that your example, once made complete, does not pose a problem of the sort you were trying to demonstrate. Since you agree we can mix the statements, does that mean you are agreeing with me?
“but you can also not mixing them” – the fact that you can choose not to mix them doesn’t impact on the fact that mixing them is possible. Did you mean something different?
Conditioning on E: that’s a strange discussion point, why would we want to assume that probability theory is wrong and then still use probability theory to calculate the implications of that assumption? As you know I am working from a propositional logic framework as proposed by Jaynes rather than a measure theoretic framework; in that framework, conditioning on a contradiction is undefined (since you could derive any probability when starting from a contradiction). And E seems to contradict the framework (one could quibble on whether this is really true, but it seems better to come up with a simpler statement of what arises when what you had in mind for E is combined with the axiomatic development of probability as logic).
“Remember that you cannot explicit all conditional terms you are using”: I’m not sure why you say this – we routinely make all of our assumptions explicit enough to be implemented on a computer.
“Saying that everything must be conditional is redundant”: Not when you are presenting an example which appears problematic when you leave out the conditioning, but becomes completely ok when the conditioning is added.
Information theory models communication channels by conditional probabilities. This perspective suggests that the conditional probability can be the primary object, as for a fixed channel, we can obtain many joint distributions of input and output by varying the input distribution.
However, there is an interesting counterpoint. If we deal with *quantum* information then the concept of a communication channel can be readily generalized. However, the concept of a “conditional probability” does not generalize in a satisfying way. In particular, there is no useful definition of a “joint distribution of input and output”. Instead, one has to resort to more indirect means, wherein one sends in one part of a correlated into the channel and studies the resulting joint distribution of output and reference.
I wonder what the takeaway message from this is with regards to the “primacy of conditional probabilities”.
The primacy of conditional probabilities is not a statistical question but an epistemological question,
and declaring yourself “a Bayesian” does not license you to ignore the epistemological question:
Is the ratio P(A|B)=P(A,B)/P(B) a legitimate representation the phrase “given that we know B”?
Here is an excerpt from our UG book “Causal Inference in Statistics – A Primer” (2016)
******************
Bayes Rule is useful, because often we will find ourselves in a situation where we cannot assess ????(????) directly, but we can through a weighted sum of conditional probabilities
????(????) = ????(????|????1)????(????1) + ????(????|????2)????(????2) + · · · + ????(????|????????)????(????????)
It is generally easier to assess conditional probabilities such as ????(????|????????), which are tied to specific contexts, rather than ????(????), which is not attached to a context
There is some controversy attached to Bayes’ rule.The controversy stems from the subjective nature of P(Bk)—how are we to know whether the assigned ????(Bk) accurately summarizes the information we have about the hypothesis? Should we insist on distilling all of our pro and con arguments down to a single number? And even if we do, why should we update our subjective beliefs about hypotheses the same way that we update objective frequencies?
***************
Apropos the legitimacy of conditioning, here is a related paper where it surfaces through Savage’s sure-thing principle
https://web.cs.ucla.edu/~kaoru/r466.pdf
Judea:
Sorry, but if the probability P(Bk) is subjective, so is the probability P(A|Bk). They’re all part of the model. How do we ever know that an aspect of a model accurately summarizes the information we have about that process? The answer is that almost never do we know such a thing.
Andrew,
You say: “if the probability P(Bk) is subjective, so is the probability P(A|Bk). They’re all part of the model.”
I must disagree, here is another excerpt from our 2016 Primer:
******************************************
“Suppose you are in a casino, and you hear a dealer shout “11!” You happen to know that the only two games played at the casino that would occasion that event are craps and roulette .. What is the probability that the dealer is working at a game of craps, given that he shouted “11?”
In this case, “craps” is our hypothesis, and “11” is our evidence. It’s difficult to figure out this probability off-hand. But the reverse—the probability that an 11 will result in a given round of craps—is easy to calculate; it is specified by the game.
Moreover, what if we did not know the proportion of roulette tables to craps tables in the casino, how on Earth could we determine the prior probability ????(“craps”)? We might be tempted to postulate ????(B) = 1/ 2 as a way of expressing our ignorance. But what if we have a hunch that roulette tables are less common in this casino, or the tone of the voice of the caller reminds us of a craps dealer we heard yesterday? In cases such as this, in order to use Bayes’ rule, we substitute, in place of ????(B), our subjective belief in the relative truth of the hypothesis compared to other possibilities. The controversy stems from the subjective nature of that belief—how are we to know whether the assigned ????(B) accurately summarizes the information we have about the hypothesis? Should we insist on distilling all of our pro and con arguments down to a single number? And even if we do, why should we update our subjective beliefs about hypotheses the same way that we update objective frequencies? Some behavioral experiments suggest that people do not update their beliefs in accordance with Bayes’ rule—but many believe that they should, and that deviations from the rule represent compromises, if not deficiencies in reasoning, and lead to suboptimal decisions. Debate over the proper use of Bayes’ theorem continues to this day.
***************** end of excerpt ************************
In short, I do not understand what you mean by “They’re all part of the model”. We break down P(A) into a product involving P(A|Bk) because
the latter invokes more reliable judgments of the uncertainties involved; often because the latter matches the organization of our scientific knowledge.
Notice also that, when we find such decomposition useful, we rarely break P(B) into a product involving P(B|Ak) , although mathematically the two
are symmetrical. Why? Why is it that we usually find one decomposition useful and the other useless. If “they’re all part of the model” then symmetry
should prevail; after all, P(B|Ak) is also “part of the model”.
The answer lies in epistemology, namely in the way we organize knowledge. One way of organizing knowledge would make some aspects of the process
more reliably estimable than others. They’re all part of the model, true, but some parts are more natural than others.
Judea:
A casino is one of the rare settings in which certain probabilities are exactly known. I don’t work with any casinos. I work in problems such as political science, pharmacometrics, environmental science, education research, etc etc etc., where probabilities are all part of the model and the data model must itself be built out of assumptions. I agree that for the study of casinos it can make sense to consider certain probabilities to be known.
When speaking of the sort of applied problems that I work on, your statement, “the latter invokes more reliable judgments of the uncertainties involved; often because the latter matches the organization of our scientific knowledge,” is itself a subjective statement that I do not think applies particularly well to the problems that I’ve studied.
Andrew,
This must be the 100th time that I hear this argument from you:
“I dont work on this problem, I work on other problems”.
I do not understand what it proves, or what it teaches us.
To explain,
I was not speaking about Andrew Gelman and the problems
he worked on. I was speaking about the philosophy of Bayes’ inference,
and the reason we decompose problems one way and not the other.
To that end, I brought up the simplest example
I could think of, the casino; it saved readers going to
my books and articles and made them understand the essence of
the argument in just 5-lines of blog text.
As everyone understand, we do not need “settings in which certain probabilities are exactly known.”
It is enough that some probabilities are more reliably estimable than others.
If you think it is possible to substantiate your objections through
such transparent examples, I am very very curious. If not,
I would be very very worried: Why not? Why not?
What is it about Gelman’s interpretation of Bayesianism
that can only be unveiled in problems having more than 48
variables and lots and lots of messy data?
As a teacher, how would I defend your argument before
students. As a scientist, how would I convince myself that
you have a valid point?
I am eager to see an example that refutes my statement
(slightly rephrased) “the benefit of Bayes’ analysis lies
in matching the organization of human knowledge”
I refuse to believe that I am the only one to stand behind this obvious statement,
it must be somewhere in Savage’s or Lindley’s writings. Can anyone help me dig?
Judea
Judea:
It is common in science that theories are elaborated and extended in the context of more complicated, realistic examples.
If people really want to be pedantic, then there’s no argument you can make that the casino odds are known so exactly. They are emergent from laws of physics and chaotic dynamics. When assuming known odds are you making a strong epistemological assumption of what emerges from those laws.
As Andrew says, it’s assumptions all the way down and none are any more valid than others a priori.
Andrew,
It is common in science that theories are elaborated when complex applications call for elaboration,
not when foundational questions are being discussed and are begging for simplification, not complication.
I am gratified to learn that you could not find a transparent example to refute
my claim about the intimate connection between Bayes analysis and the organization
of scientific knowledge
@Michael Betancourt: in fact, I really enjoyed reading “The Eudaemonic Pie” back years ago… and also, I remember some European incident where the modern equivalent occurred using cell-phones to both measure the roulette wheel and communicate between partners… They had actually won millions of dollars before being caught if I remember correctly.
Judea:
Sounds like a plan. You can work on the simplification, I can work on the complication. There’s room in science for many different sorts of contributions!
Andrew:
Sounds like a great plan.
It has one bug though: the conclusions of your
complications seem to contradict those of
my simplifications — not too healthy for science.
For example, Michael Betancourt (above) quotes you as saying:
“As Andrew says,it’s assumptions all the way down and none are
any more valid that others a priori”
Did you really mean to say that?
Isn’t it the case that some assumptions (eg P(A|Bk))
are systematically more reliable than others (eg P(Bk|A))
given the experience and training of the probability
assessor? The literature on behavioral decision making is
full of studies supporting it.
What do we do now? Complication or simplification?
Simplification says that Bayesian analysts, who
believe so strongly in “it’s assumptions all the way”
should also study the source of those assumptions,
namely, human judgment and the way humans store information.
What says complication?
Judea: certainly some assumptions are more valid than others, I don’t believe that Michael really thinks the opposite (he added the rider “a priori”, which presumably he interprets in some way that makes his statement true).
Your claim that Bayes is useful in organizing scientific knowledge seems self-evident (and I don’t see anyone disagreeing with it), but is it _the_ central benefit? I would have said the point of Bayes is that it allows us to calculate the implications of any set of assumptions. These assumptions are _not_ all equally valid; in fact, we often calculate the implications of assumptions that we actually think are false (in order to refute those assumptions). Sometimes we use assumptions that we hope are good approximations of the truth, but even then it is generally the case that some parts of our models are based on more reliable assumptions than others. But they’re still all part of the model – what is it about this phrase that bothers you?
Are you perhaps trying to say that priors are fundamentally different from and inherently less reliable than likelihoods? If so, I can easily come up with a counterexample.
Konrad
You say that it seems self-evident that Bayes is useful in organizing scientific knowledge.
I did not mean “organizing”. I said that the benefit of Bayes’s conditioning (ie, P(A,B)=P(A|B) P(B))
lies in allowing us to compose a query of interest (say P(A,B)) from judgments that are more
compatible with the organization of scientific knowledge.
This may be an obvious truth among practitioners, but I have not seen any attention given to the question of
compatibility in the literature of Bayes Data Analysis (Can an owner look up the index in Gelman’s books?)
not have I heard compatibility discussed on this blog.
Now, can we tell “apriori” which judgments are more compatible than other? In may cases we can.
For example, I would bet you 100:1 that, among practicing physicians, the assessment of
P(fever| flue) would be much more reliable than P(flue or malaria | fever or headache).
The latter needs to be mentally derived from stored relationships like the former.
It is not that “priors are fundamentally less reliable than likelihoods”, but some likelihoods match
the way knowledge is organized and some do not.
You doubt whether this is the central benefit of Bayes methods and you propose, instead, that the
central benefit lies elsewhere, in allowing us to calculate the implications of any set of assumptions,
however capricious. I am not sure. There is certainly an advantage to this capability, but I am not sure
it can stand alone without attending to the compatibility issue.
To take a far out example, assume that you assign priors to 100 probabilities of the type
P( Xi or Xj | Xk or Xm) with i,j,k,m varying all over the place.
You update them with data, and out comes a posterior probability on your query of interest, say
P(Joe has cancer) = 0.34 (or some range of probabilities).
What does it tell us about Joe? Almost nothing.
If we do not find the prior premises meaningful, we do not know how to interpret the posterior.
The guarantees provided by the posterior are only as good as the conviction we have in the veracity of
the premises.
Things get only worse when instead of observables, (Xi, Xj , Xk , Xm) we deal with a hierarchy of
abstract parameters (theta1, theta2…) each specifying a prior on the distribution of another.
Interesting question: Is there any benefit beyond knowledge compatibility?
Judea,
I think we are more or less in agreement. In the cancer example, I agree that the numbers tell us nothing directly about Joe, and how much they tell us depends on how much we trust the model, which in some cases may not be much at all. But I contend that this sort of analysis is nonetheless useful in many cases, particularly when we work with complex but well-established models that closely mirror known mechanisms in nature.
Perhaps the difference between our descriptions is that I consider all knowledge to be assumptions, but not all assumptions to be knowledge, so I am claiming a broader set of benefits.
OK, let me try.
Suppose you are building some physical devise, say, charged lepton counter that reacts differently on muons and electrons. You are not quite sure how exactly it works and want to calibrate it and make it better and whatnot. The thing you are pretty sure about is that a charged pion will enter the device and decay in it either by electron or by muon channel. The ratios of the two channels are known very well (muonic is about 10^-4). Naturally, you’ll calibrate the device by (among other things) observing its output. I don’t think you have any choice, but apply the Bayes rule with fixed prior probabilities and somewhat uncertain conditional probabilities P(signal|muon) and P(signal|electron).
Ooops. It’s electronic channel that is suppressed by 10^-4. But it has no effect on the overall argument.
And of course probability text books are another place where “certain probabilities are exactly known.”
“Sorry, but if the probability P(Bk) is subjective, so is the probability P(A|Bk).”
I agree with the point of needing to consider the model as a part of the subjective aspect of an analysis, but I think this is not quite right; a conditional probability can be objective but the corresponding marginal probability subjective. The simplest example is trivial: P(A|A)=1 is a tautology and objectively true, even if P(A) is subjective.
To make the point less trivially, the Bk’s only have meaning assuming the model itself. In Pearl’s casino example, this is exactly like assuming rules of the game. Thus, even in modelling, we have the same sort of objective conditional probabilities as in the casino.
Priors are also conditional probabilities, right? In a typical hierarchical model they just vary with factors that the ‘internal’ conditional probabilities are invariant (conditionally independent) wrt.
Judea:
I thought I would respond the this comment of yours here given I can also point to Daniel’s comment here.
> I for one see nothing wrong with this summary:
> “A hierarchical model would give a complete joint probability distribution for the data and parameters,
> and transportability analysis leave this completely unspecified, trusting that the hierarchical modeling experts would do a good job at
> that”
Nothing formally wrong but it seems to delegate the distinction between predictive association and causal connection to others.
From Daniel- “The same hierarchical statistical model can be used in two different situations: one in which there is merely predictive association between one thing and another, and one in which we actually assume there is a causal connection between one thing and another. The “do calculus” would let you create some formal description of the difference and therefore distinguish between the two cases mechanistically…”
For instance, with transporting a parameter randomly drawn from a common distribution (rather than just a common parameter) the (often implicit) assumption that the differences between trials were not systematic (or even more strangely systematic but symmetric) is very questionable. Not so much a problem for association but more so for causality.
Keith etal,
I knew I heard about the primacy of conditional probabilities someplace.
I found it, of all places, in my 1988 book Probabilistic Reasoning.
Page 39 reads:
Contrary to the traditional practice of defining conditional probabilities in terms of joint events, P(A I B) = P(A, B) / P(B) Bayesian philosophers see the conditional relationship as more basic than that of joint events, i.e., more compatible with the organization of human knowledge. In this view, B serves as a pointer to a context or frame of knowledge, and A I B stands for an event A in the context specified by B (e.g., a symptom A in the context of a disease B). Consequently, empirical knowledge invariably will be encoded in conditional probability statements, while belief in joint events, if it is ever needed, will be computed from those statements via the product P(A, B) = P(A IB) P(B)
The main reason to read this paragraph again, after 27 years , is to reaffirm the obvious connection between Bayes reasoning and the organization of knowledge which,
apparently, practicing Bayesians have allowed to be forgotten. It is knowledge organization that dictates the primacy of conditioning.
I have no objection to anything you wrote there. I have also found much of value in your work.
I’m just not convinced that you understand Andrew’s approach as given in BDA3 or that the issues do calculus has with things like the ideal gas are not symptomatic of issues with the formalism.
Why do calculus and not sets and mappings, differential geometry, topos theory etc etc?
ojm,
I did not think anyone would object to
what I wrote, nor was I asking for
an overall approval of my work, but thanks anyhow.
I am writing on this blog because I thought some
readers would be curious to find out what researchers in
related disciplines are doing, how they think about
similar problems, what new tools they have developed.
and whether those tools can be applied to specific problems
that readers on this blog are facing.
Regarding Andrew’s approach, I was indeed curious to find out how it relates to causation, after Andrew presented
hierarchical modeling as a technique that should be part of every attempt to handle extrapolation.
I have now satisfied my curiosity, as summarized in the consensus we reached a few days ago.
I do not understand the problems you are having with the ideal
gas. If you have a specific problem, I will try to help.
But I will need to ask you to be specific and specify the problem
in an “input-output” format. For example:
Needed: An estimate of the pressure if we were to raise
raise the temperature 10 degrees.
Given: current measurements, first law of thermodynamics.
Assumptions: iid, no measurement errors, …
The do calculus is not an abstract theory of mathematics.
It is a tool for solving problems of causal inference,
especially concrete problems (see format above) concerning:
model testing, identification, mediation, extrapolation,
explanation, attribution, etc.,
in short, causal inference problems that the potential
outcome literature is avoiding (See discussion of Imbens and Rubin’s book).
Hi, Judea.
I don’t actually think you understand why I was recommending hierarchical modeling for generalizing from scenario A to scenario B. The idea is simple from my perspective (it’s a compromise between no pooling (that is, not using the scenario A information at all) and complete pooling (that is, assuming that the model from scenario A applied exactly to scenario B), but for whatever reason it doesn’t seem to fit into your experiences or views of the world.
That’s fine, it’s not necessary for everyone to understand every method. There are lots of statistical methods that are evidently useful to many people, that I don’t understand myself. But, given all the above, I’d prefer you to say that you don’t understand what I’m talking about, or that it baffles you, or that you don’t see the point of it, or whatever—but not to imply that your earlier comment on the topic represents a “consensus” of anyone but you.
That is: say what you think, feel free to admit confusion or disagreement, but please don’t try to imply a consensus. I don’t have the energy to reply to every blog comment; that doesn’t mean that when I don’t reply, that I’m agreeing with what you wrote.
Hi Andrew:
First, the consensus I believe we reached was that
hierarchical modeling is a powerful estimator of
the joint probability distributions involved and, as
with any other powerful estimator, the aim is to
minimize sampling variability, not systematic bias.
Put in other words, at the asymptotic level,
powerful estimators can be emulated by the assumption that we already have
the distribution itself, not samples from the distribution.
Second, while I do not exactly understand the details
of your approach, all I wished to verify was that it works
at the estimation level, and not at the identification
level. The consensus I believed we reached was that this
indeed is the case, hence, to solve external validity
problem like those presented in recent papers you brought
to our attention (by Dehejia etal), additional methods (eg do-calculus) are needed;
hierarchical modeling can only deal with the estimation part.
Third, I hereby glad to replace the phrase
“consensus we reached last week” with the phrase
“consensus I believed we reached last week”
Judea:
Yes, I’m saying that if you’re extrapolating from scenario A to scenario B, that I prefer hierarchical modeling to a choice of no pooling or complete pooling. But the way to do the extrapolation must be determined by some combination of subject-matter knowledge and data. When I say “use a hierarchical model,” that’s within the context of whatever assumptions are required for causal identification etc. So I don’t think of hierarchical modeling as a competitor to your causal framework; rather, I think that it can fit within your framework when you move from the identification phase to the estimation phase of your analysis.
Andrew,
I am glad we reached a consensus, OOPS, sorry,
we reached a perceived meeting of the mind.
Judea
From Judea: “hierarchical modeling is a powerful estimator …, the aim is to minimize sampling variability, not systematic bias.”
But in many cases partial pooling is trading in bias for less variability (though often pragmatically) given the assumption of exchangeability is overly hopeful.
On the other matter raised by Judea, I would agree with this “It is not that “priors are fundamentally less reliable than likelihoods”, but some likelihoods match the way knowledge is organized and some do not.”
Keith:
I’d say that the dividing line between variance and bias is arbitrary, it depends what is being conditioned on and what is being averaged over. Variance can be thought of as unmodeled bias.
Andrew:
And association can be thought of as un-(do)modeled causality ;-)
Andrew:
Please elaborate. If I observe an iid sample X1,…,Xn and want to estimate E[X], then the variance of the sample mean has nothing to do with “unmodeled bias”.
Anonymous:
Let me first elaborate from my perspective (of disagreement/agreement).
Think of Simpson’s effect in no pooling or complete pooling of two studies with binary outcomes.
No pooling – both studies provide negative estimates of effect.
Complete pooling provides positive estimate of effect.
If extreme enough – partial pooling will also provide a positive estimate of effect.
If no pooling happened to provide unbiased causal estimates – partial and complete pooling would not.
Now from considerations of trying to pool or partial pool only parameters that are common from the two studies one might specify that the unknown proportions of success in groups are different (not common) but the relative risk is common and only pool/partial pool that (meta-analysis 101). It is this common parameter that one wants to transport to a new study/setting (transportability 101) and its estimation by partial pooling would be of reduced variance with no increase in bias.
But most situations are much more involved than the above and my concern was who will be responsible to make sure partial pooling and transporting are jointly being worked out together.
Randomization makes variance a good representation of underlying unknown variation (biases), the harder questions is in what other situations is variance a a good representation of underlying unknown variation (biases)?
For “underlying unknown variation (biases)” read “unmodeled bias”
Try:Systems without a graphical causal representation
https://link.springer.com/article/10.1007/s11229-013-0380-3
And: Caveats For Causal Reasoning With Equilibrium Models
https://link.springer.com/chapter/10.1007/3-540-44652-4_18
I would like to add something: that those equilibrium examples are “difficult” because we are only looking at the marginal distributions of a stochastic process. If we represent the *dynamics* of a stochastic process, then the DAG is back. The difficulty there is representing continuous-time structures, which is work in progress. But there is plenty of work on do-calculus-like methods for discrete time systems. See for instance the many contributions of James Robins in epidemiology, where dynamic treatment regimes are considering using tools analogous to the do-calculus. If you think the development of longitudinal AIDS treatments should be halted until we have a full language for continuous-time causal models, then I’d better just stop the discussion here…
The ideal gas example is dull because in this special case the “marginal distribution” can be written as an equation in which the lack of dynamics won’t preclude prediction of what will happen when equilibrium returns. But what happens in between an intervention to pressure/temperature/etc. and the next equilibrium point is not captured by it. Fortunately, the system will quickly go back to another equilibrium, where the ideal-gas equation can then be used again, and we can ignore what we missed in between.
Would you say time-series models are “incomplete” if they fail to produce short-term forecasts when given only the marginal distributions at each time t of a system, without ever specifying what happens between t and t + 1? The information is just not there in such equations, and yes, equilibrium equations do fail to produce causal predictions when looking at small enough time scales. Would we care about market equilibrium under interventions to the economy without knowing how long and what would happen in between equilibrium points? See Kevin Hoover’s work (as in his book, Causality in Macroeconomics) for examples on how do-calculus, causal graphs and equilibrium can be tied together in messy applications.
Most sciences are not lucky enough to have such clean problems as in classical mechanics. Repeating that reasoning I’ve discussed in the other thread, just because more sophisticated causal models don’t add much to exceptionally clean situations, it doesn’t mean we have to pull a Rutherford and lazily ignore them.
@Ricardo
Do you have any comments about why DAGs haven’t really caught on in most sciences? At least that has been my observation, correct me if I’m wrong. I totally agree with your observation that “Most sciences are not lucky enough to have such clean problems as in classical mechanics” but somehow they don’t seem to be turning to DAGs to address their messiness.
It sounds like one of those ideas that’s elegant but somehow doesn’t catch the interest of applied practitioners? You could say they haven’t been around for that long but then again I’m hearing of them for the last 10 years at least.
Is it destined to be one of those revolutionary ideas that’s perennially just around the corner?
That’s an excellent question, Rahul. My own theory is the difficulty of doing empirical validation. Causal models are mostly useful when we have observational data only or a mixture of different interventional and observational regimes.
The former is much more common, but if we have observational data only, how can we validate what we got makes sense? In many sciences, there are standard adjustments where there is no “need” for causal graphs: just condition on everything measured that might be a confounder, hope for the best. Does this work? Hard to tell if we have no (easy) way of doing an experiment. One could use explicit causal assumptions on how the covariates might or might not be adequate for adjustment, and do sensitivity analysis on how the results might vary by varying the assumptions on causal structure. This requires a lot of work, and if for decades people got away without it (less work, more publications? Win-win), it is hard to break this tradition. (Surprise, surprise, not everybody is willing to add to the uncertainty by adding alternative causal structures to the mix.) You would be totally right to say that is people who work on causal graphs that should try to change this culture, and I’m guilty as anyone of not having worked on bigger applied projects pointing to these issues. But then again, showing sensitivity is still not the same as showing successful causal effect estimation if an experiment cannot be easily executed.
I would also say that, coming from the machine learning community myself, I’m mostly interested in out-of-sample prediction. There is a large literature in statistics about in-sample estimation, and that can be largely detached from those interested in causal graphs. Even good textbooks such as Imbens and Rubin start right at Chapter 1 by making clear their main interest is in-sample estimation, which at that point most machine learning folks would just shut the book and never open it again. I myself hardly care about what “would have happened had such and such taken place”, as opposed to “what will happen if such and such takes place.”
Concerning the use of causal graphs in problems which combine different experimental setups, I think this is still in their infancy as good data of that type seems to be not that widespread. Sachs et al. (Science, 2005: Vol. 308 no. 5721 pp. 523-529) is one of the most common examples and it is well-cited, but similar studies are rare. One needs the appropriate facilities to run these experiments (the Microsoft paper I’ve mentioned elsewhere also required a major infrastructure), but I’m optimistic opportunities like that might be more common in the future.
Meanwhile, another reason to be optimistic is the emergence of some very good textbooks such as Morgan and Winship might slowly change the culture in those communities who do purely observational studies but currently don’t dare to challenge the usual “let’s adjust for everything / and hope for the best / and ignore the all the rest” approach. And there is already a number of people in social sciences and epidemiology who are not shy of doing so.
But, to summarize, I agree that much more could be done, and people like me are much to be blamed for not trying to do more on large applied projects. I’m trying to get there.
@Ricardo
Fine if you think that the ideal gas is ‘dull’. I just think that thermodynamics is one of the nicest examples of an abstract stucture capturing an incredibly wide range of applied problems.
I would rather abstract from things that work well on real problems. For example, classical mechanics can be understood in terms of symplectic geometry. Thermodynamics can be understood in terms of contact geometry.
General dynamical systems can also be understood in terms of differential geometry. For example I often talk to people who work in the field of geometric singular perturbation theory applied to models of signalling in biological cells.
Very messy, complicated problems where geometric structure expressed in terms of modern mathematics can give a lot of insight into both the dynamics and the quasi-steady state (or slow manifold) behaviour.
My impression is that do-calculus lacks the expressive power and ability to analyse a wide range of systems that existing mathematics – e.g. the field of dynamical systems – already possesses.
RE: Rutherford. I have some sympathy for his (admittedly provocative) statement that ‘all science is either physics of stamp collecting’.
Firstly because I see that people in areas like mathematical biology are constantly making progress by borrowing tools from statistical mechanics, differential geometry etc – i.e. by slowly adopting the mathematical formalisms of theoretical physics and adapting them to their particular problems – and secondly because I am a NZer too!
possible typo: physics *or* stamp collecting, though physics of stamp collecting also sort of works…
Absolutely, by all means I think it’s great to borrow and adapt good ideas developed elsewhere. Let’s be inclusive, I couldn’t support this more. Just let’s not exclude other good ideas based on no reason but the belief that “that’s not the way I think about it, and the way I think about it can do everything”. In general case, we can’t fit models to (equilibrium) observational data and magically predict the outcome of an intervention without making assumptions linking observations to interventions. We will need all we can get from our toolboxes.
Rahul, that’s the correct quote. I think I was too obscure while referring to a previous post of mine. I never learned the exact quote, but the spirit is the same, “that’s not the way I think about it, and the way I think about it can do everything”.
But by now I think I’ve abused Andrew’s hospitality with these off-topic posts…
@Ricardo
Sure!
I was pushing back against Judea’s attitude not yours :-)
(My point re mathematics is wondering what expressive power we are giving up in adopting do-calculus. But I’ve used up daily comments on Gelman’s blog again!)
Ha, which Rutherford quote did Ricardo have in mind? I assumed it was “If you need statistics to do science, then it’s not science”
Things are a bit messy above, so I’ll start a separate reply here.
OJM wrote:
“Agree with Andrew that starting from conditional probability as basic is the way to go. Agree with Alexandre that it may not be trivial to axiomatize.”
As also pointed out above by Bill Jefferys, this is exactly how it was axiomatized by Jaynes in the first two chapters of his book (2003), expanding on the earlier, less complete, attempt by Cox (1947). I’ve previously had an email discussion with Alexandre to try and see if he would point out any issues in Jaynes’s approach, but the only issue he raised was a reluctance to work with two-valued statements. (I.e. if we want to work with statements that can be partially true instead of just either true or false, then we have to throw out principles like the law of the excluded middle and are clearly not dealing with probability theory anymore.)
So instead of saying “it may not be trivial to axiomatize”, why don’t we discuss the _existing_ axiomatization that does exactly this? I don’t see any issues with it, but perhaps others do?
The exposition by Jaynes is perhaps not as complete and explicit as a mathematician would like, but from reading his text it’s pretty easy to fill in details oneself. Available as pdf from this site:
https://bayes.wustl.edu/
For a shorter version that states the axioms more explicitly (but without all the written justification), see Chapter 2 of this book by Baldi and Brunak:
https://www.amazon.com/Bioinformatics-Learning-Approach-Adaptive-Computation/dp/026202506X
Konrad,
I enjoyed our discussion by email. I gave you some examples and many explanations and many papers. But you barely comment them in our discussion by email, you seemed to be already convinced that the Jaynes’ approach is better and you seemed to not hearing any of my explanations.
Hope you have read (or eventually read on day) the papers I sent you that discuss the problems of probability in non-monotonic reasoning. If you are an applied statistician, maybe they are not interest for you, you seem to be not interested in looking for other explanations and other ways of reasoning.
Well, thanks for all, I will avoid to comment here. I got the message, but take care with the intellectual bias confirmation. I will not read the response, If you want to write something, send me an email.
best regards.
Hi Alexandre,
I’m sorry if I caused any offense, and I also enjoyed our discussion. I agree with you that the use of two-valued logic is not forced on us, but the focus of our discussion (and any discussion about probability theory, since this is what probability theory deals with) was on systems of reasoning about two-valued statements. But as it turned out, none of your examples pointed to any issues with probability theory (whether the Jaynes axiomatization or any other) that arise after accepting that statements are two-valued. So I’ll just repeat my request to point out issues in probability theory as the correct extension of two-valued logic to conditions of uncertainty. If no such issues are raised, I will continue to assume the derivation is correct.
You’re right, it’s messy above, so I’m reposting with an edit:
I expect that the vast majority of people who read this blog would be happy to work within the following reduced formalism, particularly as Stan improves its ODE solvers etc… though we could add a few additional Bayesian inference programs to help with PDEs and other specialized whatnot:
“The set of models is the set of all Stan (or a small set of other languages) programs less than 1 terabit (10^12 bits) long that compile and run in less than 1 month of wall time on a modern desktop computer”
Clearly as the models are less than 1 terabit long, there are fewer than 2^(10^12) of these models. Since compiling is a big restriction, probably there are MANY fewer. But, this is a finite number. We can add the following rule:
The prior probability over a model is proportional to 1/N if there are N models being considered, unless otherwise explicitly specified, and proportional to 1/2^(10^12) for any models not formally considered.
For a person willing to work with this HEAVY theoretical restriction, what does your possibility theory provide? This heavy theoretical restriction is “very light” for the vast majority of statisticians. Most of them would balk at writing even 1 megabit of Stan code.
And the point of this is that when considering infinite sets all kinds of weirdness occurs. But, when working with the kinds of situations applied statisticians deal with, we work with finite everything, sometimes we pretend that we have infinite sets because the set is so big, but if a problem occurs theoretically, we can always retreat to the more accurate view of reality, which is that we’re dealing with finite everything, and the infinite was an approximation.
This is opposite to the usual mathematician’s view, where the ODE or the continuous function is the “reality” and the finite differences or finite set of samples are the approximation,
Some people might object, perhaps they like 1 year instead of 1 month, or they want 100 terabits, or whatnot, but there’s always some finite upper bound that has to be “good enough” because we can’t wait until after the sun runs out of hydrogen to use our model, and we can’t compute with a computer that can only be implemented using more mass than the sun…
Hi Daniel,
I’m not sure exactly what you’re responding to here – assuming it’s Alexandre’s claims about possibility theory. At the risk of misrepresenting him, I’d say that possibility theory is what applies when you want to reason about statements that do not have to be two-valued. Statements like “John is tall”, in a context where we agree that we can reject “John is tall” without accepting “John is not tall” (perhaps we can say “John is somewhat tall”, while rejecting both of the other statements). So the law of the excluded middle does not apply, which means ordinary logic goes out the window.
Probability theory is simply not a framework for dealing with statements of this type.
Yes I was trying to nail down what Alexandre’s goal was. He seems to point out that probability is not the only way to deal with uncertainty, and seems to want to apply this to cases where we’re dealing with decisions to be made about model choice.
My understanding is that it’s not just applicable in the absence of two-valued logic, but also when you can’t really well-define a probability space. For example, you can say that any positive number is a possibility for a parameter, but there is no true probability measure on “all positive real numbers” or you could say that there are vast families of priors and likelihoods that you could choose to use, and no probability space over all possible priors or likelihoods…. but I just don’t care. I only care about problems that fit WELL inside the 1 terabit of Stan code and limited time resources restriction… which is just so well behaved that I don’t think I need possibility theory to help me.
“you can say that … there is no true probability measure on “all positive real numbers” ” – why would one say that?
“or you could say that there are vast families of priors and likelihoods … and no probability space over all possible priors or likelihoods” – why would this matter? It is true that probability theory only allows us to deduce the implications of assumptions we can actually state, but if anyone thinks there is a defensible way of deducing the implications of an in-principle-unstatable assumption, they’ve got their work cut out backing up that claim.
Sorry, I meant “no true *uniform* probability measure on all positive real numbers”
As for your second point, yes, I am generally happy to work with probability within a model, and then, when I don’t like my model, move on to another model. I don’t feel a need to formalize this process, except maybe when I am explicitly working on a small set of models where I can in fact do model averaging with priors over the models… so if I feel the need to formalize model selection, it seems that a discrete set of models and a prior are sufficient. And when I find out that they are not, it doesn’t bother me to go back to the drawing board and start working with a model that wasn’t being considered before. I don’t see this as a “failure of probability”. Yet, it seems like one of the major issues that Alexandre seems to be bringing up.
With my discrete 1 terabit of code example, I’m kind of pointing out that “back to the drawing board” can be seen as simply elaborating a higher-precision probability model that had been approximated asymptotically as having zero probability over the new model… if it makes you feel better.
I think it’s perfectly plausible to have well-stateable assumptions that are not covered by probability:
“The measurement error should be modeled by a unimodal distribution with a continuous density and mode at 0”
I don’t think I can put a probability distribution over all these possible probability distributions… but, nor do I think I really need to.
This is the kind of thing I think Alexandre’s possibility measures can provide “possibility” for… but I’m not convinced that it will help me in any way. For example, I could just state “I think the measurement errors are well modeled by some continuous density with mode at 0”. I don’t feel the need to formalize that. I’d like to see an application where formalizing this kind of thing helps me in some way before spending more time on this topic.
You have specified a constraint that is matched by multiple specific assumptions. We could either say it is an ill-posed problem (the stated assumption is incomplete/ambiguous), or we can add principles like maxent into the mix to make the problem well-posed. Either way I don’t see that there is a (fundamental rather than technical) problem here. If someone wants to propose resolving the ambiguity using a principle distinct from maxent, they should start by addressing in which way maxent is incorrect/unsatisfactory.
As for the infinite-domain uniform probability measure – sometimes we can find ourselves stating assumptions that are mathematically impossible. We already know from ordinary logic that a contradiction implies all statements; there is nothing further to be learned there.
konrad: working inside IST, I actually can easily specify a nonstandard uniform prior, uniform(0,N) where N is a nonstandard integer. Since it’s a nonstandard integer, it’s larger than all the standard integers. This means it is a uniform probability on a set that contains all the standard positive real numbers (plus some additional nonstandard ones less than N).
This is clearly a nonstandard probability model. Yet, I can then perform my Bayesian analysis with an appropriate likelihood, and find out that I have sometimes (often or even usually!) a standard posterior (this is well known, the likelihood is often normalizable by itself).
The complaint “ah but you’re using an improper prior” actually leads to no problem at all for the posterior. But what is the logical basis for even claiming that an improper prior can be used? Well in IST it’s straightforward.
One of the things I love about IST is that it eliminates a lot of nonsense because it vastly opens up the space of functions to include all kinds of things that aren’t standard functions (like the Dirac delta!). These don’t suddenly become a totally different type of object (like say “a functional that maps a function to its value at a particular argument”) They become straightforward pointwise but non-standard functions or even families of functions.
Similarly, the issue with maxent arises… the differential entropy is not the same as the entropy over a discrete set. But we can define the probability distribution over the discrete grid of a fixed chosen infinitesimal size, and we can define the nonstandard maximum entropy problem, and then we can say that within this formalization we can maximize the entropy and arise at a standard distribution (such as normal) which nevertheless has a nonstandard entropy calculated within this model. So what if I can’t calculate the entropy, I can prove that it’s maximal and it occurs with a *standard* normal distribution, so off I go.
I’m not saying it eliminates all problems, but I am saying it gives you a way to quickly determine when you can avoid spending your time on things that aren’t really problems.
I agree you don’t need to, but I don’t think it’s too difficult. I haven’t done any math to verify this, but if I had to, I’d start by checking:
X ~ F
where:
log (g(x)) ~ some stationary GP()
F”(x) = -x * exp(-x^2) * g(x)
Corey: I’m not quite sure I understand what you’re doing there. Is F the CDF or pdf ? In any case, even if we are able to specify a probability measure over PDFs with mode at 0 it’s not at all clear that such a measure corresponds to any real-world state of knowledge.
In any case, I think the maxent solution handles the problem nicely, though I confess to liking the mathiness of your general idea even if I don’t quite understand what you’re up to.
F is the CDF. The idea is make the derivative of the pdf the product of a random positive function and a function that is that has one zero-crossing at 0. But now I perceive that this is a distribution over unimodal distributions with differentiable densities, which is not what was called for.
For it to have a well defined mode, it seems like it’s going to have to be a differentiable density, so that is maybe ok.
However, I don’t think your system results in necessarily properly normalized distributions. You’ll probably want to use an ODE for F which prevents it from having a positive derivative when F approaches 1, so
F’ = something_nonnegative(x) * (1-F)
But anyway, suppose we get something that defines some sort of a distribution over distributions… it’s not entirely clear how to match the distribution to a real-world state of knowledge. We could perhaps specify a small number of quantiles as random variables, etc. I’m sure there are interesting things to be done here when in fact our real GOAL is to figure out an unknown distribution. But if your goal is just to figure out what some data implies about some parameters in a model, choosing a maxent distribution is most often going to give reasonable results, and unless your goal is really to model the process of choosing a model… yeah it’s sort of a turtles all the way down issue.
Silly of me to forget about the normalizing constant!
Also, you’re probably going to need to have two ODEs, one for F going from x=0 towards infinity, and one for F going from x=0 towards negative infinity, since you can’t specify a boundary condition “at negative infinity”, then you’ll need a couple of matching conditions at x=0 and a distribution over the value F(0)…
Incidentally, I actually DID do something like this for a model, I had an observed dataset of positive time-to-events, and some hypothesized ODEs that described a chemical, and a hypothesized parametric relationship between that chemical concentration and the probability per unit time of an event, and Maxima, which actually solved the ODE in closed form for me, so I could get a closed form expression for the distribution over distributions of time-to-events… it was pretty cool, but worked in part because I had a definite boundary condition at t=0.
Last time I read Jaynes I remember he mentioned ‘interface’ conditions like:
“The robot always takes into account all of the evidence it has relevant to a question. It does not arbitrarily ignore some of the information, basing its conclusions only on what remains”
My impression (it’s been a while) is that he spends more time on working ‘internally’ to the system and less on the implications of requiring ‘interface’ conditions. If he runs into a contradiction within the system, he expands the system.
Good and interesting, sure, but I don’t see why people shouldn’t be encouraged to think more carefully about the implications of the ‘within’/’without’ the model/interface distinctions. What if the interface condition is only ‘approximately’ satisfied? Is the resulting inference ‘stable’? Etc etc.
I find the subtleties of ‘interface’ conditions (or ‘closure’ conditions as I like to call them) interesting. These seem to relate to Gelman’s (and Box’s) model checking requirements, Alexandre’s ‘possibility’ measures, others’ ‘adequacy’ or ‘approximate model’ approaches.
Fine if you want to deal with them informally, but there’s plently of wiggle room for someone to think
‘why bother with perfect inference within an imperfect model, maybe I could try imperfect inference within an imperfect model and possibly do better wrt to external reality?’.
You don’t have to do it yourself, but I myself wouldn’t be so vehemently opposed to people doing this. Of course you might say ‘well I just embed this in a bigger Bayesian model etc etc’ but then we get a bit of a chicken an egg situation.
@ojm you might (or might not) find john boyd’s writings on deconstructing and reconstructing modeling frames fun to read. it’s more in the context of competition rather than science since he was in the military.
Smart and interesting guy but unfortunately his slides are almost incomprehensible out of context and, unlike his contributions to energy–maneuverability theory, he never really mathematized this body of work.
https://www.goalsys.com/books/documents/DESTRUCTION_AND_CREATION.pdf
I enjoyed that, thanks!
I agree that those are important and interesting questions. But if you view probability theory as a framework for calculating the implications of crisply stated assumptions, then the interface problem is outside of its scope. This means you can have a cleanly axiomatized system (which performs formal and correct reasoning under uncertainty) at the core of statistics, without needing _everything_ you do in practical applications to be part of that system.
I don’t consider ‘the interface problem [to be] outside of its scope’ though.
It’s like saying a chaotic dynamical system is sufficient to calculate the implications of crisply stated initial conditions.
Why do people who claim to love ‘formal and correct reasoning’ based on ‘clean axiomatizations’ start talking about ‘oh but practical applications’ whenever any of the axioms is called into question?
To me this is sloppy thinking, and makes it hard to pin down whether we’re arguing about axiomatic or informal issues at any one point in time.
I think the Jaynes/Gelman Bayesian approach of work inside, check, expand is nice (and I use it myself!), but to me the need for interface conditions leaves plenty of room for people like Alexandre who want to think about different approaches.
tldr: I try to keep my interface conditions at least a little open!
Many here don’t like defining probability crisply, and many don’t consider the interface problem to be outside of its scope. My impression is that these two sets of people largely overlap. Be that as it may, my claim is just that a crisp axiomatization exists and that it so happens that the interface problem is outside the scope of that particular axiomatization. Personally I prefer a crisp axiomatization with limited scope to a fuzzy description with broad scope, but each to their own.
But I really don’t get this: “Why do people who claim to love ‘formal and correct reasoning’ based on ‘clean axiomatizations’ start talking about ‘oh but practical applications’ whenever any of the axioms is called into question?”
I don’t see where you called any axioms into question. If you did, which ones? And I didn’t say ‘oh but practical applications’. My point was that a formal system does formal things, which are then usually interpreted broadly (beyond the scope of the system) in an application context.
I probably shouldn’t keep answering at this point…but…internet discussions…
I implied that Desideratum IIIb of Jaynes (which *is* part of his formal assumptions) may be (note – *may*) somewhat naive, especially in light of other formal mathematical frameworks such as the theory of dynamical systems.
So of course one could treat it purely as a formal system, but that formal system may or may not be a good axiomatization of what you’re trying to represent, or may be a special case of something more general.
Euclid’s parallel postulate etc.
Just ran across this 10-year-old post. Andrew wrote:
> But it is completely consistent with the mathematics to first define P(A|B) and then to define P(A,B) as P(A|B)*P(B).
Indeed. Many years ago I went looking for such a treatment, and found one in M.M. Rao’s book Conditional Measures and Applications (2nd edition appeared in 2005).
I’d always wondered why this wasn’t the standard approach, since it avoids problems with conditioning on a set of measure 0 (e.g., the Borel-Kolmogorov paradox). You can choose the definition you want for p(A|B) when p(B)=0 … provided that it is coherent with the other conditional probabilities you define, i.e., no axioms are violated.