3 Jul 2024, 11am, at the International Society of Bayesian Inference meeting at Ca’ Foscari University of Venice:
Every philosophy has holes, and it is the responsibility of proponents of a philosophy to point out these problems. Here are a few holes in Bayesian data analysis:
1. The usual rules of conditional probability fail in the quantum realm;
2. Flat or weak priors lead to terrible inferences about things we care about;
3. Subjective priors are incoherent;
4. Bayesian decision picks the wrong model;
5. Bayes factors fail in the presence of flat or weak priors;
6. For Cantorian reasons we need to check our models, but this destroys the coherence of Bayesian inference.
Some of the problems of Bayesian statistics arise from people trying to do things they should not be trying to do, but other holes are not so easily patched. In particular, it may be a good idea to avoid flat, weak, or conventional priors, but such advice, if followed, would go against the vast majority of Bayesian practice and requires us to confront the fundamental incoherence of Bayesian inference. This does not mean that we think Bayesian inference is a bad idea, but it does mean that there is a tension between Bayesian logic and Bayesian workflow which we believe can only be resolved by considering Bayesian logic as a tool, a way of revealing inevitable misfits and incoherences in our model assumptions, rather than as an end in itself.
This work is joint with Yuling Yao.
This is not an “anti-Bayesian” talk (for my thoughts on anti-Bayesianism, see here and here). Incoherence is a necessary part of any generally useful statistics or data-analytic workflow. One cool feature of Bayesian inference is that we can leverage its local coherence as part of a larger workflow—but to do this most effectively we should understand where the paradigm breaks down. Hence this talk.
Perhaps I’ll also have a chance to make the point that Bayesians are frequentists.
(FYI, The “see here and here” links are missing)
Links fixed; thanks.
I wouldn’t say that an advice such as “avoid priors which are inadequate for the problem” requires us to “confront the fundamental incoherence of Bayesian inference”. Not more that “avoid models which are inadequate for the problem” anyway. If anything, it’s the fundamental coherence that needs to be confronted.
> Perhaps I’ll also have a chance to make the point that Bayesians are frequentists.
To make the point clear to them you may want to start by defining “frequentist” – that word means different things in different contexts.
From our paper:
Your paper says many things – some make more sense than others. Even in that short fragment the last sentence contradicts the first.
> The usual rules of conditional probability fail in the quantum realm
I doubt that is true.
David:
Click through the link and read section 2 of the paper. This is basic quantum mechanics: uncertainties are combined using superposition of complex amplitudes, not linear combination of probabilities.
To put it another way, what we in statistics call “probability” is, in physics jargon, called “Boltzmann statistics.” At the quantum level, particles follow “Fermi-Dirac” or “Bose-Einstein” statistics. It’s kinda weird, and you’re not the first person to doubt it, but it’s how the world works.
I’m gonna guess that David Marcus is well aware of the way you calculate the frequency with which certain outcomes of certain QM experiments will be certain quantities… I’m guessing he just denies there’s anything wrong with using probability properly.
Let’s give the simplest example I know of… We have a double slit experiment and individual photons are sent into the system, and then detected at different locations on the “screen”.
We can cover one hole, and get a spot of light, cover the other hole and get a spot of light at a slightly different place, or uncover both holes and get a well known “interference pattern” with multiple light spots.
You might argue that “conditional on going through hole A” you get the first spot of light as the distribution of outcomes, and “conditional on going through hole B” you get the shifted spot of light, but “conditional on going through either A or B” you don’t get the p(location | A) + p(location | B) two spots of light that probability theory would predict.
But that’s not correct, because you aren’t “conditioning on going through either A or B” you are “conditioning on both holes being open and all the wave function propagation that implies”
You will find that if you condition on the wave function, everything obeys standard probability.
Someone even took the time to write a 15-page reply:
The rule of conditional probability is valid in quantum theory [Comment on Gelman & Yao’s “Holes in Bayesian statistics”]
PierGianLuca Porta Mana
https://osf.io/preprints/osf/bsnh7
> I’m gonna guess that David Marcus is well aware of the way you calculate
> the frequency with which certain outcomes of certain QM experiments will
> be certain quantities… I’m guessing he just denies there’s anything
> wrong with using probability properly.
Good guess.
Andrew: If you don’t want to believe me that probability works just fine for quantum physics, maybe you will believe Sheldon Goldstein:
https://www.scholarpedia.org/article/Bell%27s_theorem#Classical_versus_quantum_probability_.28and_logic.29
“However, as long as the usual meanings of words are kept, there is no need to get rid of classical probability theory (or classical logic).”
I expect that if you email Professor Goldstein, he’d be happy to explain this to you.
Andrew,
I looked at Section 2 of your paper. Your error is to assume that experiment 4 simply provides more information about experiment 3. You can’t assume the “detectors” don’t affect the particles. The theory tells you whether they do or don’t. If they do (and the theory says that they do in this case), then experiment 4 is not just more information about experiment 3. It is a different experiment.
You could have the same confusion classically if you used water waves and assumed your “detector” wasn’t affecting the water when it really was.
When you posted about this paper on February 23, 2020, (https://statmodeling.stat.columbia.edu/2020/02/23/holes-in-bayesian-statistics/) we pointed out the error. I don’t know why you haven’t corrected your paper.
> The photon does not go through one slit or the other; it goes through
> both slits,
This is (of course) nonsense. The particle (being a particle) goes through one slit. But, the wave (that guides the particle) sees both slits.
David:
I get that you disagree with what I wrote. That doesn’t make what I wrote an error, or nonsense, or (of course) nonsense, or whatever. It’s a disagreement. There are different ways to translate quantum mechanics into a description in natural language.
Andrew, you can argue about the goes through one or both slits terminology, or even argue that particles don’t exist, as some QM people seem to. But I think you have to acknowledge that experiment 4 isn’t more information about experiment 3 it’s a different experiment.
I’m not an expert in QM, far from it, but I understand that double-slit type experiments involve some material making up the slit, that material is made out of atoms, and the atoms have electrons and things which produce potential energy in the space around them, making it so that the QM wave propagates through the space differently. Detectors are also made out of atoms, and may emit particles or light or whatever ultimately altering the physical reality in the vicinity of the detector. The wave propagates differently in configuration space when the detector is there and when the slit is open vs is closed, etc.
We can create a perfectly fine classical experimental setup that has the same logical content:
You can imagine putting a pingpong ball in a river, and halfway downstream are two separate weirs which can be set to different heights and fully downstream is a series of baskets that catch the pingpong ball aligned next to each other across the river. If we set the weirs so that theyre both flowing, the ball floating down the river will, due to the flow of water, pass either through one weir or through the other. We can imagine putting a “detector” which is like a pipe that has a camera in it on the outflow of each weir. We can also imagine that the presence of the pipe alters the surface waves of the water flow so that there’s no longer the same kind of interference caused by the surface waves as compared to if it’s just the weirs without the detectors.
We wouldn’t say that putting the detectors in place gives us more information about what happened when we did an experiment that had no detector, because clearly the detector perturbs the surface waves in the water dramatically and eliminates the interference.
With either weir blocked we get a single spot that the pingpong ball always goes to, but it’s different spots for blocking different weirs. When both weirs are open, balls are directed by complicated waves on the surface to an interference pattern at the downstream detector. However with a “detector pipe” next to each weir, the interference pattern on the surface of the water disappears because of the pipe, and we get single spots again.
Each experiment is a physically different set-up where the apparatus itself matters. The probability we need to assign to being in a certain place is **conditional on the physical affects of the apparatus** and this is not in any way contradictory for the laws of probability.
In fact the whole reason why Physicists like Bayes is because Bayes is about **our knowledge** and not about **the way the world works**. So the truth or accuracy of Bayesian probability itself can’t possibly be dependent on the outcome of experiments. Only the accuracy of our model at explaining the physics is called into question. The Cox theorems are mathematical tautologies that don’t depend on physics.
Daniel:
I agree that experiments c and d in our paper are different experiments. We label them as such!
P.S. Physicists may like Bayes, but they know better than to apply Boltzmann statistics to electrons, or to combine probabilities using linear averaging rather than superposing complex amplitudes.
Andrew, I still disagree with your characterization of combining probabilities by squared amplitudes.
Suppose we have a double slit apparatus and a high quality unpredictable random number generator. The random number generator controls whether one or the other or neither slit is covered. It produces numbers 0,1,2 uniformly and they correspond to the states open-open, open-closed, closed-open.
The apparatus is in a dark room we are standing outside of, we can press a button outside the room and then a few seconds later a detection will appear on a computer screen corresponding to a photon that went through the apparatus and hit a detector at a certain point along the screen.
Our information is just that after pressing the button, the RNG runs, converts the apparatus to one of the three states with equal probability, and then sends a particle and the detection was at a certain point.
we’ll still say
p(certain_point) = p(certain_point | 1) p(1) + p(certain_point|2)p(2) + p(certain_point|3)p(3)
each of the p(certain_point | 1) etc will be a calculation in which the probability of a detection at a certain point is calculated conditional on the experimental set-up, just like we’d calculate the pingpong ball with the weirs.
In the process of doing the physical simulation of the pingpong balls we are free to do some calculations related to fluid mechanics equations. In the process of calculating where the photon hits we are free to do some calculations related to Bohmian mechanics. In both processes we have some uncertainty over the initial conditions of the particles as they enter the apparatus, and those small uncertainties grow in time to produce a variety of possible outcomes, giving us p(certain_point | 1) as integral(p(certain_point | 1, initial_condition)p(initial_condition))
Despite the mechanics being different (fluids vs DeBroglie), the structure of the mathematics is the same, you run a particle forward in time from its assumed initial condition until it hits the screen, and you do this for a weighted average over the prior for possible initial conditions
The DeBroglie equation assumed by Bohmian mechanics doesn’t even involve complex values per-se. Given an initial condition, the dynamics are weird, but **deterministic** like a pingpong ball. And it builds up an interference pattern as the sum of individual outcomes weighted by priors over initial conditions using standard probability calculations we do every day in problems like pharmacodynamics.
Andrew:
> That doesn’t make what I wrote an error
The fact that we disagree doesn’t automatically imply you made an error. But, I pointed out what your error is. Conversely, just because you don’t admit it is an error doesn’t make it correct.
> There are different ways to translate quantum mechanics into a
> description in natural language.
Some make sense. Some do not make sense (i.e., are nonsense). If you pick one that doesn’t make sense, it is not too surprising that you conclude that probability is broken.
As I said, you should contact one of the experts. Or, read some of the many good books on the subject.
Hmm to clarify, whether DeBroglie-Bohm involves complex numbers is not quite the right way to think about the question and is a red herring.
https://en.wikipedia.org/wiki/De_Broglie%E2%80%93Bohm_theory#Derivations
shows that the original Bohm derivation used 2 coupled real equation, but that they’re equivalent to a single complex equation. The point is more or less irrelevant. In the pingpong ball example we calculate p_NS(outcome | apparatus) by integrate(p_NS(outcome | apparatus, initial_cond) p(initial_cond)) using Navier-Stokes equations to propagate position forward in time. In quantum 2 slit experiments we can do it by integrate(p_DBB(outcome | apparatus, initial_cond) p(initial_cond)) using DeBroglie-Bohmian mechanics to propagate position forward in time.
In both situations we get a probability by the sum rule and integration over a prior combined with “some physics”. There’s no “weird probability” involved everything is standard Bayes.
Daniel:
> But I think you have to acknowledge that experiment 4 isn’t more
> information about experiment 3 it’s a different experiment.
Exactly!
Good example with the river.
Andrew,
If you read any of Sheldon Goldstein’s papers
https://sites.math.rutgers.edu/~oldstein/index.html
it is obvious that he understands probability. So, if he writes an article with a section “Controversy and common misunderstandings”
https://www.scholarpedia.org/article/Bell%27s_theorem#Controversy_and_common_misunderstandings
and one of those common misunderstandings is something that you believe is true
https://www.scholarpedia.org/article/Bell%27s_theorem#Classical_versus_quantum_probability_.28and_logic.29
you might want to discuss it with him before assuming that you are right and he is wrong.
Apparently de Broglie and Bohm pilot wave theories are *not* the same. And this is a major source of confusion when talking about the topic.
https://www.mdpi.com/2073-8994/16/2/149
Anoneuoid,
> Apparently de Broglie and Bohm pilot wave theories are *not* the same.
I believe de Broglie showed how a particle could be guided by a wave, then Bohm later turned it into a complete theory. Hence, some call the complete theory Bohmian mechanics and some call it the de Broglie–Bohm theory. Unfortunately, other physicists convinced de Broglie to abandon his theory.
Afaict the Bohm theory is Markovian, while the de Broglie theory is not. Ie, according to de Broglie, the behavior of each particle is determined by the past motion of all particles, including itself. For Bohm, the next state can be treated as purely a function of the current state (plus some “quantum randomness”).
So complete or not, seems to be a different theory. But that is why I shared the quote/link.
But in both the mathematics and quantum foundations literature probability is just called “probability”; the simplification of the jargon reflecting the knowledge that “classical” (Kolmogorovian) probability isn’t all of probability (see e.g.).
> But in both the mathematics and quantum foundations literature probability is just called “probability”
Also in physics in general.
“[Maxwell-]Boltzmann statistics describes the distribution of classical material particles over various energy states in thermal equilibrium.”
Usually when people “in statistics” call something “probability” they are not thinking of that.
“The usual rules of conditional probability fail in the quantum realm.”
Yukalov and Sornette have solved this problem…or at least proposed a reasonable way to formulate conditional probability in the quantum realm.
See:
V.I. Yukalov and D. Sornette, Positive operator-valued measures in quantum decision theory, Lecture Notes Comput. Sci. 8951, 146-161 (2015)
external page(https://ssrn.com/abstract=2579278)
and especially
V.I. Yukalov and D. Sornette
Quantum probabilities of composite events in quantum measurements with multimode states,
Laser Physics 23, 105502 (14pp) (2013) doi:10.1088/1054-660X/23/10/105502
(arXiv:1308.5604 and https://ssrn.com/abstract=2316701)
> the usual rules of conditional probability fail in the quantum realm
Sounds bad.
> The standard paradigm of Bayesian statistics is to define a joint distribution of all observed and unobserved quantities […]
Ok, let’s keep that in mind.
> Consider four possible experiments: […] The problem is that p4, […] is not the same as p3 […]
Is it a problem that different experiments have different outcomes?
> This violates the rule of Bayesian statistics, by which a probability distribution is updated by conditioning on new information.
Does it? Having a probability distribution for the outcomes conditional on the experiment being performed doesn’t seem problematic.
> At this point, we can rescue probability theory, and Bayesian inference, by including the measurement step in the conditioning. Define a random variable z carrying the information of which slots are open and closed, and the positions of any detectors, and then we can assign a joint distribution on (z, y) and perform Bayesian inference
Ah, we can rescue probability theory by following the standard paradigm of Bayesian statistics: taking into account all the relevant information about the experiment to create a joint distribution.
> In essence, the problem with applying probability theory to the slit x and screen position y arises because it is physically inappropriate to consider x as a latent variable.
So that was it, the model was physically inappropriate.
> Bayesian inference cannot resurrect a misspecified model, but it works fine to incorporate quantum mechanics within the model.
Amen.
> There are two difficulties here. The first is that Bayesian statistics is always presented in terms of a joint distribution of all parameters, latent variables, and observables; but in quantum mechanics there is no joint distribution or hidden-variable interpretation.
We did just rescue probability theory in your QM experiment(s) by defining the right joint distribution. Where’s the difficulty?
> The second challenge […] If classical probability theory needs to be generalized to apply to quantum mechanics,
Why would it need to be generalized if “Bayesian inference can be rescued by careful modeling”?
Carlos wrote:
> So that was it, the model was physically inappropriate.
Indeed.
There can’t be anything wrong with conditional probability because it is just math. It is possible to produce a mathematical model that is not a good description of reality. If one does that, then it is not surprising that the model will make statements that do not match reality. This just means the model is poor. In fact, it is the disagreement with reality that tells us that the model is poor.
In this case, Andrew and Yuling assumed the “detector” does not affect the particle. With this assumption, the model predicted things that did not agree with experiment. The solution is to remove the bad assumption from the model. Once you do that, everything is fine.
I don’t understand why Andrew thinks this is just a “disagreement” rather than an error. What Andrew is saying and what we are saying are contradictory. They can’t both be true.
David:
You write, “Andrew and Yuling assumed the ‘detector’ does not affect the particle.” That is not correct.
What we actually wrote is, “p4, which is a simple mixture distribution, is not the same as p3, which has nodes and which cannot be expressed as any linear combination of p1 and p2. This is a manifestation of Heisenberg’s uncertainty principle: putting detectors at the slits changes the distribution of the hits on the screen.” Which is pretty much the opposite of what you said we wrote!
We also wrote, “At this point, we can rescue probability theory, and Bayesian inference, by including the measurement step in the conditioning.” And then we continued, “Bayesian statistics is always presented in terms of a joint distribution of all parameters, latent variables, and observables; but in quantum mechanics there is no joint distribution or hidden-variable interpretation. Quantum superposi- tion is not merely probabilistic (Bayesian) uncertainty—or, if it is, we require a more complex probability theory that allows for quantum entanglement.”
Anyway, it’s ok to disagree; I’d just prefer you not say that we assume things that are the opposite of what we write!
Andrew,
Let’s take it one step at a time. I am discussing your claim “The usual rules of conditional probability fail in the quantum realm”. Your justification for this claim is the part of your paper from “Consider the canonical experiment” through the paragraph that starts “This violates” (i.e., the paragraph before “At this point”). Is that correct?
David:
I very much appreciate you engaging here. The applicability of conditional probability to quantum mechanics is a complicated and controversial topic, and I get that your take and others’ is different than the view held by Yuling, me, and the reviewers of this article. I don’t think a discussion in blog comments here will resolve these disagreements.
To step back from the details for a moment: I’m no longer a physicist, and my goal in writing about that example was not to make any contributions to physics or to advance understanding of quantum mechanics in general or the two-slit experiment in particular. Rather, I’m interested in the implications for applied statistics of macroscopic phenomena in areas such as pharmacology, political science, psychology, etc. As Yuling and I discuss in our article, it is an open question if there are any useful applications in macroscopic statistical modeling for non-Boltzmannian or non-Kolmogorovian probability ideas from physics such as non-latent variables, superposition of probability amplitudes, and Bose-Einstein and Fermi-Dirac statistics. I don’t know; I just thought it was interesting and perhaps important to raise the question.
Andrew, I appreciate that you think maybe this can’t be hashed out in a blog post. But as I’ve said above, you can get the quantum outcomes from straightforward application of Bayesian probability (sum and product rule) and Bohmian mechanics.
You put a prior over the initial conditions, then you use the physics to define the wave function which describes how the particles move, and then you get a probability distribution over the location where the particle will be found.
The idea that you “need” something other than probability is just wrong. It’s a product of physicists being unwilling to accept that Bohm’s mechanics + straightforward Bayesian uncertainty on initial conditions solves the problem in exactly the way that Newton’s mechanics plus straightforward uncertainty on initial conditions solves the pingpong ball in the river problem.
Some group of physicists **vehemently deny** the possibility that the world could be non-local in the way that Bohm’s mechanics requires. This seems to be, if anything, a psychological and or political problem, not a scientific issue. Yes, Bohm was once a communist in a time when that was enough to get you ostracized from society. And yes non-locality is weird and it’s not clear how to reconcile it with Einstein, but simply crossing it off as a possibility has done a lot of harm. I mean, we can’t reconcile gravity with QM either but we don’t throw away gravity! (or QM)
Bell tried to argue that his inequalities imply Bohm’s mechanics makes the most sense. Instead his inequalities have been used to further the anti-Bohm sentiment.
Sometimes I think the biggest reason people don’t want to deal with Bohm’s mechanics is that it implies that the whole world is one enormous wave function and that factorization of that wave function into a little “system” that is mostly independent of the rest of the world is hard. Of course, they also will spend billions of dollars a year making complex ultra-low-temperature vacuum laser tabletop devices to isolate a few particles from the rest of the world, and not consider that in the slightest bit contradictory.
Daniel:
I agree that this stuff is important, and it’s not a topic I know much about at all! So, again, I’d rather pivot to the question of possible applications in macroscopic statistical modeling for non-Boltzmannian or non-Kolmogorovian probability ideas from physics such as non-latent variables, superposition of probability amplitudes, and Bose-Einstein and Fermi-Dirac statistics.
I’ve never seen these used in serious applied work outside of quantum physics. For example, I’ve never seen someone analyze a time series of prices using probability amplitudes, or analyze a psychology experiment using Fermi-Dirac statistics, or analyze survey responses using a two-slit-like model.
Daniel,
> And yes non-locality is weird
The 2022 Nobel Prize in Physics was awarded to experimenters who demonstrated that the world is nonlocal. So, we are stuck with it.
> and it’s not clear how to reconcile it with
> Einstein
There are relativistic versions of Bohmian mechanics that are Lorentz invariant. Whether they are really “relativistic” is more of a philosophical question. Goldstein has some interesting papers on his website that discuss this.
> Instead his [Bell’s] inequalities have been used to further the anti-Bohm
> sentiment.
The press release for the 2022 Nobel Prize in Physics says, “John Clauser developed John Bell’s ideas, leading to a practical experiment. When he took the measurements, they supported quantum mechanics by clearly violating a Bell inequality. This means that quantum mechanics cannot be replaced by a theory that uses hidden variables.”
If that last sentence were true, it would have been quite a blunder for Bell to promote Bohmian mechanics (since Bohmian mechanics is a “hidden variables” theory). Of course, Bohmian mechanics does predict the Bell inequality will be violated. So, the press release is in error.
Andrew,
> I don’t think a discussion in blog comments here will resolve these
> disagreements.
> it’s not a topic I know much about at all!
You claimed “The usual rules of conditional probability fail in the quantum realm”. That’s an amazing claim. When we question it, you say that you don’t want to discuss it in this blog and you don’t know much about it.
If you don’t think you know much about the topic, maybe you should just retract the claim. Then we could all move on.
Since you don’t wish us to discuss it here, this will be my last comment on the topic. Of course, anyone is welcome to email me directly if they wish to discuss this topic.
David Marcus, Paul Hayes, Carlos Ungil, and anyone else interested in QM and Bayes as a topic
I’m happy to discuss more publicly on mastodon if you like just mention me: @[email protected]
David:
The statement that “The usual rules of conditional probability fail in the quantum realm” is not amazing at all. It’s basic quantum physics. What physicists call “Boltzmann statistics” is what statisticians call probability theory. And it’s well known that Boltzmann statistics fails in the quantum realm. Again, though, my interest here is not in using insights from probability theory to do better physics—I’m unqualified to do that!—but in using insights from physics in order to do better applied statistics. I’m still not sure that there’s anything useful to be done in that area, but I’m open to the possibility.
Andrew. I would say that “Boltzmann Statistics” is a particular probability model, in which we assign the probability of being in a state at energy E as proportional to something like exp(-E/kT) and it is related to maximum entropy principles and relies primarily on the lack of physical constraints to prevent the system from being in certain energy levels and therefore we simply maximize the entropy of the distribution obeying the known constraint of average energy (which is expressed as kT). When there are constraints other than an “average energy” (ie. temperature) then we use different maximum entropy distributions (Fermi-Dirac or Bose-Einstein). All of these are applications of “normal probability theory” using the principle of maximum entropy, and some constraints to come up with a prior on achievable states.
When we have something like QM double slit experiments and in your article you say:
“There are two difficulties here. The first is that that Bayesian statistics is always presented in
terms of a joint distribution of all parameters, latent variables, and observables; but in quantum
mechanics there is no joint distribution or hidden-variable interpretation. Quantum superposition is
not merely probabilistic (Bayesian) uncertainty—or, if it is, we require a more complex probability
theory that allows for quantum entanglement.”
David and I disagree with this statement, particularly “in quantum mechanics there is no joint distribution or hidden-variable interpretation” and also “…we require a more complex probability theory…”
In the experiment you reference, the simple double-slit experiment, we can calculate the entire result as:
integrate(p(y | A,x) p(x) , dx)
where y is the detection location, x is the unknown initial condition from the photon emitter, p(x) is a prior over the initial condition based on our physics knowledge (perhaps uniform on an aperture or approximately so), and A is the description of the state of the apparatus which tells us which trajectories the particle takes if we know its initial location.
In fact, Bohmian mechanics says p(y|A,x) is a deterministic delta function distribution which you must calculate using some coupled partial differential equations. Similar to how you might calculate the final temperature distribution after say 30 seconds of shining a bright light in a certain physical pattern on a dark black metal plate, or calculate the position of a bead embedded in a thick syrup 10 seconds after opening a hole in a tank containing the syrup and allowing it to drain out… There’s nothing mysterious about needing a physical model to project particles forward in time.
In other words, there’s no unusual probability required at all. you just start with uncertainty over the initial conditions, and a rule for moving the particles through space given the initial conditions.
The part that’s weird, is that the particle transport mechanism in QM is *Bell nonlocal* that is, stuff far away from one particle can “instantly” affect the experimental outcome. That’s the point of the EPR experiment, basically you couple two particles, then you measure one of them and instantly know what will happen when someone else measures the other one even though that other one is “far away” (space-like separated) and it’s not possible to send light-signals from the first measurement to the second particle to tell it what to do.
David and I just accept that to be the case because it’s apparently reasonably well established experimentally. That’s really the *only* weirdness in QM, that it’s bell-nonlocal. Even Paul agrees that it’s Bell-nonlocal.
Einstein argued in the EPR experiment that “of course” everything is Bell-local, and yet we see these correlations, and so QM is incomplete. Bell later came along and said “Hold up, here’s a proof that QM **is Bell nonlocal**” and so we no longer have to agree with Einstein that it’s incomplete. Basically post-Bell we’re in the state that we are free to believe that QM is Bell-nonlocal, and a complete description (modulo relativity), and that complete description is equivalent to Bohm + Bayesian uncertainty on initial conditions.
This is the content of Goldstein’s papers such as the scholarpedia article that David has pointed you to. It’s also the content of the excellent comprehensive description by Tim Maudlin here: https://arxiv.org/abs/1408.1826
David and I, and I believe Carlos though I’m not sure, all agree that you should publish a correction to your paper in which you clarify that there’s nothing *actually wrong* about Bayesian probability and that it does in fact describe the Physics of the double slit experiment, provided that you use a model fit-for-purpose. There *is* a hidden variable description of QM, namely Bohmian mechanics, and its weirdness is just that it’s Bell-nonlocal.
Maybe there is some such “group” but as far as the group of numerate and quantum foundations literate physicists is concerned you have that backwards. Math. phys. and quantum foundations literate physicists (and philosophers) are well aware that BM or some other classical model of QM – aka “hidden variable theory” – with its non-locality is possible but reject it because they know that it’s wholly unnecessary (as well as problematic). As Werner pointed out in this article it’s the advocates of BM who seem plagued by psychological or political problems:
The (C=) Classicality prejudice is sort of understandable, and even some respectable and prominent quantum foundations researchers – e.g. Matt Leifer – do share it and work on ways (e.g. retrocausality) to restore classicality somehow to physics, but there’s a huge difference between their fully informed prejudice and the blind ideological commitment of many Bohmians (and advocates of other schemes such as superdeterminism).
David Marcus,
They didn’t and we’re not. Far from it. See my [possibly still waiting moderation] previous comment. (Bohmian) claims to the contrary are simply false.
Andrew,
“Quantum cognition” is
an application of “quantum probability” outside of quantum physics.
Paul:
When I look up quantum cognition I see lots of theory but no applied work. I’m not saying the applied work isn’t out there; I just haven’t seen it.
Daniel,
Thanks for the kind invitation but if you’ve read the Leifer and Spekkens article that I tried to draw attention to in the earlier “Holes” article here then you already know more than I do about the topic of QM and Bayes. (It’s not a particular interest of mine and I’ve only ever skimmed through it).
Paul,
I have a hard time following your posts, because for the most part they seem to involve some assertions and a web of links back to sources. This may be enough for a specialist with a lot of time, but I’m neither a specialist nor do I have a lot of time or skill to construct my own arguments to support your views. I’d much prefer your argument, then with links to sources.
For example, some time ago, perhaps a year or three, David Marcus gave a summary of Jean Bricmont’s argument from Quantum Sense and Nonsense. That gave me an understanding of why Marcus (and Bricmont) thinks the world is inevitably nonlocal. Apparently all four of us had the same conversation, and got to the same place… in other words, Marcus has given a logical argument from some foundational assumptions, and you have given something… else. I don’t want to be uncharitable or controversial, I just found that what you wrote didn’t address Marcus’ points and always seemed to point to some other people and some “lack of foundation” on our part.
https://statmodeling.stat.columbia.edu/2022/02/03/epistemic-and-aleatoric-uncertainty/#comment-2044173
In any case, perhaps you can address the following question which I consider to be a question of Bayes + QM.
In this comment above: https://statmodeling.stat.columbia.edu/2024/07/02/holes-in-bayesian-statistics-my-talk-tomorrow-at-the-bayesian-conference-based-on-work-with-yuling/#comment-2375013
I argued that there is a 1-1 analogy between dropping a pingpong ball in a river, and sending a photon/particle through a double slit experiment. That both of those can be described, probabilistically using standard Bayesian probability subject to Kolmogorov type axioms, as uncertainty about initial conditions, perhaps some uncertainty about the transport process (caused by for example external interference, a child splashing in the river, uncertainty about the precise conditions of the river channel, or noise from lab apparatus or the AM radio station a mile away), and a procedure for propagation of position forward in time.
In essence the probability distribution over the final location is the integral over the prior for the initial condition and/or prior for the transport process, pushed-forward through the transport process to the final detector location.
There is **nothing** in this description requiring any “weird probability” it’s literally just completely standard application of Bayesian probability.
From a logical perspective it looks like this: “if I know a particle started at A and the apparatus is in the state B then the particle will wind up at place C according to mechanical rules” symbolically p(C | A,B) is given by mechanics, and then we admit that we never know exactly where the particle starts, so we have a probability distribution over its initial conditions p(A), and then … we will have a probability distribution over the final conditions C. Let’s for the moment assume complete knowledge of the apparatus B and no external interference, then:
p(C | B) = integral(p(C | A,B) p(A), dA)
Which is just the product and sum rules for normal probability.
Can you tell me, in your view, are there any experimental or theoretical results which **rule out** my taking this as descriptive of the real world? I’d just like to follow up on those references which in your view make my view untenable.
Daniel,
Your pingpong ball in a river is potentially descriptive of part of the real world – the double slit phenomenon – but not the (quantum) real world beyond that. Contra Feynman, interference phenomena aren’t any longer regarded as the “essence of the quantum” anyway.
In that earlier discussion you linked to I see I’ve already pointed out there (and included references) that Bricmont’s and other Bohmians’ error is their use of “local causality” aka “Bell locality” as the definition of locality in their ‘logical’ argument. I really can’t imagine what more I could possibly do for anyone who still can’t see the problem; the breathtakingly glaring circularity in it.
Paul, I don’t know if it’s my fault or not, but you have apparently completely misunderstood the point of my river example.
I’m not saying “fluid mechanics works just like Bohmian mechanics a-la the bouncing droplet experiment, and therefore there’s nothing special about QM”. I am merely saying that **probability** plays the same role in both experiments even though the **mechanics** are different.
That bouncing oil droplets don’t work like Bohm’s mechanics is of ancillary interest to me, but irrelevant to my argument about probability. I’m not saying ping pong balls flow in rivers similar to electrons in Bohmian mechanics, I’m just saying that in both cases:
p(final_location | apparatus) = integrate(p(final_location | initial_location, apparatus) p(initial_location), d_initial_location)
This has the structure p(A|B) = integral(p(A|B,C) * p(C), dC)
where p(A | B,C) is the “mechanics” for an apparatus B, A is observation of a location at the end of the experiment, and C is initial location.
In particular, there’s no “complex probability amplitude” here, and there’s no violation of the sum or product rule. This is 100% usual probability.
The mechanics can of course be different in the river and the Bohmian pilot wave, in fact I assumed they were quite different myself but maybe you thought I was referring to the bouncing droplet pilot wave idea. I wasn’t. All that’s necessary is that there be some way to compute the physical transport from initial condition C to final location A.
Next we go to “local causality” vs “Bell locality” vs “locality”. I honestly don’t know what to say here.
Experiments prove that if Alice measures an EPR experiment as UP then Bob will measure it as Down 100% of the time. Bell’s inequalities prove that this isn’t because of a quantity carried along by the particles in the way that Bertlemann’s socks are mismatched because at the beginning of the day he put them on mismatched. That is, the electron sent to Alice wasn’t “always Up” while the one sent to Bob wasn’t “always Down”. Bell basically says that Alice making the measurement ensures that Bob will make an opposite measurement even when they are space-like separated and the choice of measurement was made at the last possible moment.
You claim there’s a “breathtaking circularity” in the logic “if local causality then not Bell’s inequalities, yet Bell’s inequalities are observed, therefore not local causality”
Please provide a single professor of logic who will dispute that argument. David Marcus has a PhD in Mathematics and specifically studied Probability https://www.davidmarcus.com/Personal.php he does not dispute this argument. I have an undergraduate degree in mathematics and a PhD where I use applications of mathematical methods to Engineering problems. I don’t dispute this argument. Do you dispute this argument?
Now, there’s another possibility, you say “local causality” is an irrelevance, you don’t care about it, because you care about something called “locality”. Then it is *up to you* to define what *you* mean by “locality” and to tell us why David and I should *not care* about “local causality / bell locality”?
I would ask, if you choose to do so, that you provide a brief, perhaps 100 words or less paraphrase or summary of the definitions you wish to use, the importance of the distinctions you want to make, and the argument itself, and THEN point us to some backing references, because so far I am generally finding you reference previous posts by you which are also just as unsatisfying and there’s some recursion eventually leading to some article by someone, often someone who is flat out 100 mistaken.
For example at one point you refered to Werner. Tim Maudlin gave the reply here https://arxiv.org/abs/1408.1828 in which he explicitly quotes Werner. I won’t reproduce the entire quote here but will paraphrase. In essence Werner says that the *epistemic* state is the state that matters, and that this state is not changed by measurement. Unfortunately he isn’t completely clear on whose epistemic state we’re discussing but there are some epistemic states that ARE changed by measurement.
It’s TRUE that when Alice measures spin UP, Bob’s epistemic state doesn’t change. Only when Alice has sent a data packet on a laser beam to Bob telling Bob the results of her measurement does Bob’s epistemic state change to “100% chance spin down”. However, Alice’s epistemic state changes *immediately* and the Bell correlation implies that no matter how far away Bob is, eventually when he makes the measurement it’ll be DOWN.
In this sense, the change in epistemic state is local (it’s entirely at Alice’s location and places reachable by her laser).
Nevertheless, we expect that if we send an EPR pair to Alice and Bob, and Alice has an apparatus capable of rapidly changing between x and y axis, and she has a complicated apparatus which she presses a button and it changes the apparatus to measure y axis at the last possible moment, if Bob is doing y axis measurements, and he is space-like separated from Alice’s random-axis device when Alice measures UP he will still get 100% correlated DOWN measurements. That is, pressing the button and forcing Alice’s apparatus to measure y axis imposes a 100% correlation on the Bob result in a way that can’t be determined by the Bertelemann’s socks explanation.
Do you dispute that?
Do you dispute that one way in which the last possible moment change of Alice’s machine can affect Bob’s measurement at space-like separation is that both Alice and Bob are making measurements of some mechanical properties that are determined by a **global, universe wide wave function** which determined both the outcome of the random apparatus change, as well as the measurement at A and the measurement at B?
Unfortunately I’m putting in quite some time to explain all this, mostly in hope that some other people are reading it. So, If you provide me a link and a couple phrases and claim that this was all explained long ago, I will be forced to consider you a troll.
Daniel,
Indeed I see I don’t understand the point of your river example. That one could in principle use “ordinary” (classical, Kolmogorov) probability to describe any phenomenon in the physical world, even the quantum ones? Well of course that’s true: Many phenomena can be so modeled with clever artifices a la Spekkens and of course general (nonlocal) classical models of QM and other such schemes haven’t been ruled out. But everyone knows that already…
Anyway…
Yes. Experiments confirm QM. Bells theorems and inequalities prove that QM is incompatible with “local realism” (+ certain other reasonable assumptions). No local HV theory is possible.
No. How many more times?! What I’ve pointed out rather than claiomed is that there’s a breathtaking circularity in the ‘logic’ of the Bohmian argument which uses Bell’s “local causality” as the definition of locality thereby reaching the conclusion that QM itself (and “the world”) is “non-local”. Of course QM is incompatiblle with “local causality” – that “very specific locality condition invented as a constraint on hidden variable theories” of Bell’s (1976) theorem. QM (and “the world”) is “Bell non-local”. But there are no HVs in QM itself. QM is not a local hidden variable theory. It is an “Einstein local” theory.
I’m sorry but I’m not willing to put up with any more of this crankery and nonsense. I’ve tried my best to explain this stuff and represent the “mainstream QF” knowledge and position on these issues here – over and over again already. I’ve backed it all up with high quality refs (and quotes from those refs) but theyve clearly not been even half understood, and have been repeatedly variously misconstrued, ignored and treated as incomprehensible or nonsense (and the ilikes of Werner, Landsman etc. portrayed as the cranks rather than the Bohmians whose ‘arguments’ trigger remedial responses in respected journals). Well maybe it is all my fault – my best just isn’t good enough – but this really is just too ridiculous, futile and draining.
Paul, then what we’ve been doing is arguing over “the one true meaning of nonlocality”. Which is a totally futile argument I agree.
I was easily willing to agree there’s no faster than light signaling. I think you call this Einstein locality. So we agree there! There’s no FTL signalling because Alice can’t *control* the outcome of her experiment.
You are willing to agree that QM violates what we can call Bell Locality so that it is a Bell Nonlocal theory… I agree with this!
And you are willing to agree that I can use standard Kolmogorov probability to do all the predictions needed for probability in QM by applying standard priors + the mechanics of Bohm. No probability amplitudes or redefinition of probability required. We agree here!
Apparently the only thing we disagreed about was whether it was ok to call Bell Nonlocality “nonlocality”.
So I will concede this point, I’ll add Bell in front of nonlocal. QM is a Bell Nonlocal theory that if we do Bohemian calculations we get a 100% accurate theory (as accurate as any non relativistic QM of any other form) that uses standard prior and posterior calculations to predict probabilistic results for particle experiments. Nothing woo woo about it, it works, and it uses probability in the normal way, and it has particles in the normal way and it has no faster than light signalling, and it is Bell Nonlocal. That’s all David Marcus and I have ever claimed.
When I looked into it this was not the case. The first Aspect 1982 experiment did come up with a post-hoc QM prediction (rather than only an upper bound that amounts to a vaguer prediction than the classical model). This equation was never used for a prediction about future data or seen again. Ie, eq 5 here: https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.49.91
Rather, the experiments reject a certain classical model then assume QM is the only alternative. In particular, one assumption made in that classical model is that the universe is Markovian.
If you know of a paper where they predict the result using QM beforehand I would love to see it. A couple experiments like that would indeed “confirm” QM in my mind.
There is no need to understand all the details to see that they are rejecting theory A, then accepting theory B. This is a logical fallacy, not a confirmation of theory B.
Its summed up in this paper: https://arxiv.org/abs/2208.08940v2
The upper bounds are:
Markovian classical model
– S = 2
QM
-S = 2*sqrt(2) ~ 1.83
Non-markovian classical model
– S = 2.49
QM
-S = 2*sqrt(2) ~ 1.83
Anoneuoid,
> Ie, eq 5 here: https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.49.91
That paper says Figure 3 is the comparison of experiment with QM. The text on the same page says the agreement is better than 1%. (I can’t judge the paper. I’m just pointing out that the authors say their results agree with QM.)
Yes, but said equation needs to predict future results. Not just fit the results already observed.
Later papers never use this equation, or even bother to come up with their own post-hoc fit. Correct me if I am wrong.
The upper bound of S ~ 2.83 (sorry about the failed typo correction above!), is a true QM prediction. But the exact result predicted by QM depends on the details of the experiment.
It doesn’t mean much when a model making imprecise predictions is consistent with observation while one making more precise predictions (the classical model) fails to fit. Eg, the ultimate vague explanation is “God/aliens did it”, which is consistent with anything.
As shown in the last paper I referenced (Papatryfonos 2022), classical *non-markovian* models can be consistent with observation up to S ~ 2.49.
Thus, observing S greater than 2 is not sufficient to confirm QM.
Anoneuoid,
> As shown in the last paper I referenced (Papatryfonos 2022), classical
> *non-markovian* models can be consistent with observation up to S ~ 2.49.
That is just an analogy. It isn’t a model for the actual Quantum Mechanical experiment.
After Aspect 1982, no quantum mechanical models have been published for these experiments either. Once again, you should be able to easily show I am wrong about that.
So what we have is:
1) A non-markovian classical model that yields a bell violation, but no application of that model for an actual bell test.
2) QM that yields a bell violation, but (with one unreplicated exception) no application of that model for an actual bell test.
This seems to be pretty much equal footing to me. I really want to know why the later papers don’t try to do the same as in Aspect 1982? This must be known amongst the insiders on this topic.
Anoneuoid,
Looks like this list of experiments is as of 2011:
https://www.scholarpedia.org/article/Bell%27s_theorem#Experiments
You’ll have to ask a physicist if you want to know what has been done since then. I expect it would make the news if an experiment disgreed with Quantum Mechanics.
From your source:
This makes sense to me. The “non-locality” is in time rather than space.
I already looked into this before and most observations of that S-value are between 2 and 2.5. Most often ~2.5. Possibly there is a meta-analysis somewhere.
The issue isn’t that these results disagree with QM, it is that they agree with other explanations just as well… or even better. Ie, for the value of S allowed three theories:
A) Between 0-2
B) Between 0-3
C) Between 0-2.5
Most observations of S are between 2-2.5. Which theory is most likely correct? I would say A is rejected, C is consistent with the results, but B fits it best. Then it comes down to priors.