I’ve made lots of mistakes, and failure has been good for my intellectual development. But this only works if I learn from my errors, which in turn requires me to be aware that I got things wrong in the first place. Four times I’ve had errors that were big enough that I issued formal corrections. And then there was the time that I messed up on a statistical analysis that I’d posted on the blog and someone pointed out my error, causing me to spend a couple months figuring out how to do better.
There’s also this talk I gave on learning from mistakes and this one too. And also this.
And I’ve made various smaller slips over the years that I’ve had to correct (for example here and here). It happens!
And there was the embarrassing error in the second paragraph of this article. I think that particular mistake was introduced by the copy editor, but I’ll still have to take responsibility for not carefully checking and letting it slip through.
An opportunity for all of you
Given all this, it only stands to reason that there must be a bunch of things I’ve done wrong that I haven’t yet realized were in error. Or mistakes I’ve made that I’ve corrected, but the corrections are not prominent enough and people continue to miss the point. Or simply some places where we disagree: things I’ve written where you think I’m wrong, even past my insistence otherwise. Fair enough.
This is your chance! I’m speaking to my friends as well as the haters out there. Let me know what I’ve done wrong. This is your chance to help me learn from my mistakes, to force me to confront disagreements, also a way for you to share your gripes with others. Point out my mistakes right here in the comments, and tens of thousands of readers will be made aware of them.
It’s your opportunity to help me out, to embarrass me, and to get your perspective out there. So go at it. Complain loudly. The squeaky wheel gets the grease.
Funny, I just happened to think about this following example very recently for some reason. Here is an example that may point to something you might become more aware of and/or more optimally handle (or whatever word is appropriate here).
I think what I sometimes notice in your comments and/or behavior might relate to certain topics and/or certain discussions where it seems to me you might not read precisely what’s exactly been written and/or you might misinterpret things and/or you might act overly defensive (or whatever word is appropriate here).
Perhaps this following example makes clear what I mean, and what I have experienced in discussions with you on this blog from time to time (also see my comment to your comment concerning that). I am the same Anonymous in the following exchange with you, and I will link to a comment you made which I think might point to what I mean.
https://statmodeling.stat.columbia.edu/2025/08/15/whats-on-your-universitys-home-page/#comment-2402255
I don’t want to discuss the details of the topic in that particular exchange here, but strictly the interaction and reaction to my comment. If you read my Anonymous comment above your linked to here comment, and my Anonymous comment below I think all three comments together might make something clear.
Or not.
Or it could be that I am the one making a possible mistake, or possibly misinterpreting something.
Anon:
I appreciate your effort here, but I can’t quite see what is the mistake you think I’m making. Can you be more direct, as in, Gelman is saying X but the truth is actually Y? Or, Gelman is saying X but providing no evidence in support of that claim? Or, Gelman is avoiding engagement with argument X that was offered by person Y? Or something else along those lines?
I’d also advice following the advice of the much-maligned Strunk and White and replacing all uses of “and/or” by “or.” Try it–it really works!
You wrote the following in the comment I linked to: “You say, “I generally think it’s best not to use such words or sentences like ‘please do better.’” Fair enough; that’s your preference. But the only person in this discussion who used the phrase “please do better” was you, in an earlier comment! So you appear to be engaged in a spirited debate with . . . yourself.”
Can you tell me why you said “But the only person in this discussion who used the phrase “please do better” was you, in an earlier comment! So you appear to be engaged in a spirited debate with . . . yourself.”? Why am I appearing to be engaged in a spirited debate with…myself?
I don’t understand that sentence and your comment/reaction at all, except for the reason I stated in my comment or the possiblity that you were being overly defensive (or what’s the appropriate word here).
Just for the record, I understand very little of what you say (Anonymous – though admittedly I may often confuse you with the other Anonymi).
Quote from above: “Just for the record, I understand very little of what you say (Anonymous – though admittedly I may often confuse you with the other Anonymi).”
I think that I have certain general ways to comment here, some of which involve certain perhaps unusual ways to attempt to make a point or make something clear. Some of this is present in the comments referred to above which may complicate manners. However, just like concerning the specific topic, I don’t want to discuss that, I just want to mention the reaction and comment structure (or whatever term is most appropriate).
In my view, in the comment I simply say:
“Maybe some people at certain (kinds of?) universities might say things like “please do better” (..)”
and later on I say:
“Regardless, that’s not me. I generally think it’s best not to use such words or sentences like “please do better”
I don’t understand Mr. Gelman’s reaction to that (except for possible reasons I mentioned). I can understand his reaction IF he thought I was saying he was the one that said “please do better” first, or to me. I can also understand his reaction IF he thought I was saying he “should do better”. But I don’t think I said or implied these things.
Anon:
When I wrote that you appeared to be engaged in a spirited debate with yourself, what I meant was that you argued against the use of words or sentences like “please do better,” but you did not give any examples of anyone who was using such phrases.
Rather than saying you are debating with yourself, it would have been more accurate to say that you were debating with an unseen opponent, not anyone in that particular thread.
I don’t really have any complaints about Andrew, but I guess I am complaining about you. So, perhaps I should be more specific. I feel like your comments contain lengthy descriptions of your thought process. I personally don’t find these interesting and I feel like they detract from whatever point you are making. It could just be me so feel free to disregard. But since Andrew has opened this post to “Complain loudly” I’ll take the liberty of voicing my reactions to your comments rather than his. It would be fair game to now complain about my hijacking threads (which I know I do at times).
Dale:
I have a horrible feeling that most of the people who disagree with me aren’t commenting here at all. Which is kind of too bad. Maybe I need another post where people collect criticisms of me from others and post them here. I welcome criticism and can learn much from it. Even when the criticism seems completely off base to me, it still can show me where I’m failing to communicate.
Quote from above: “(…) what I meant was that you argued against the use of words or sentences like “please do better,” but you did not give any examples of anyone who was using such phrases.”
I said: “Maybe some people at certain (kinds of?) universities might say things like “please do better” (…)”
Quote from above: “Rather than saying you are debating with yourself, it would have been more accurate to say that you were debating with an unseen opponent, not anyone in that particular thread.”
See the above comment concerning “some people at certain (kinds of?) universities” which might be the “opponent” in light of your comment here just now? (I was also mostly just expressing some views and not having a “debate” in my view).
In light of these new comments, I’ll replicate part of what I stated earlier:
“I think what I sometimes notice in your comments and/or behavior might relate to certain topics and/or certain discussions where it seems to me you might not read precisely what’s exactly been written (…)”
“Or it could be that I am the one making a possible mistake, or possibly misinterpreting something.”
It’s interesting to me to have these little conversations about having conversations. I have had trouble before understanding what someone wrote, so perhaps this is merely another example of that. In that instane, someone else commented and made something clear which may happen here as well. If not, that’s okay too. Thank you for the replies!
I think what the Anonymous here is trying to claim, in an oddly indirect way, is that the Anonymous who said “please do better” in the linked thread was a *different* Anonymous.
Andrew, regarding “most of the people who disagree with me aren’t commenting here at all”, it’s generally not a good idea to comment on a blog if one is in great disagreement with the blog-owner. The people who do that tend to be hardcore culture-warriors of some sort. Even if someone is generally in agreement overall, the risk/reward proposition is still rather dicey. There’s not a whole lot of upside for the commentator, but there’s a chance of getting banned or slammed from “on high” (which can be very unpleasant). Sure, you’re of course a nice person who wouldn’t knee-jerk react that way, but one never knows. Even nice people can have a bad day and take it out on someone who left an irritating comment.
Do you think people really worry about making negative comments because they might get banned or slammed? My casual evidence suggests that there is little such worry, even to the point that civility hardly exists on most blogs (this one is an exception). I think the opposite is more the problem: people might disagree just to be jerks because there is so little consequence – and they can always hide behind being anonymous.
(to Dale: can’t reply at that depth?): I agree there are trolls/people who are jerks just for the joy of it, but there’s also an unsampled group who disagree/have complaints but *don’t* feel like dealing with the resulting drama – if probably not from “on high” in this case, from others on the list.
Reminds me of this cartoon: https://www.smbc-comics.com/comic/2013-04-07
Dale: Note the part of my comment about “… tend to be hardcore culture-warriors of some sort”. That is, it’s a sorting process. The ones who don’t worry tend to be those who are jerks, are anonymous, etc. The people who do worry tend to think it’s not a good idea. Nothing is absolute, of course. But there’s unfortunate incentives very much in play.
Seth:
I guess one reason for people who disagree with me not to comment on the blog is that they don’t like me and they don’t want to give me free material.
Another reason is that, in the immortal words of Nate Silver, they “don’t really want a back-and-forth.” I think that attitude is crazy–I love a good back-and-forth–but maybe that’s just my academic temperament.
Also I fear that twitter has gotten people in the habit of just making assertions without backing them up.
So, there are various reasons why people might not want to comment here. On the other hand, this is kind of the perfect place to argue with me, first because I’ll notice and take a look at what people say, second because you’ll reach the readers of this blog.
Mostly a joke:
You are a liberal. This is incorrect. You should be a socialist.
Then I guess you really hated this post!
I love the idea of this post. If anyone wants to correct my mistakes they should feel free to do that too.
Quote from above: “If anyone wants to correct my mistakes they should feel free to do that too.”
There appear to only be 12 songs on the “Implicit Christmas” spotify playlist?!
(Thanks for the list b.t.w.! I am listening to “”Til I get it right” by Tammy Wynette as we speak. I was a new song to me, and I checked out a few more of her songs as well. Also, I prefer Chris Isaak’s “Wicked Game”, but I appreciated the version by Andrew Sa (which was new to me as well) on the list too. Talking about covers, “Nothing Compares 2 U” by Sinead O’Connor might fit with the Implict Christmas list perhaps).
I’ve looked in my music folder for songs that might fit the list of Implicit Christmas songs. I’ve tried to look for songs with a certain mood or vibe, and the jangly sounds of bells or guitar or a certain rhytmn that might be in line with some songs on the list of 12 you made. At the time of writing this comment that list has 4 saves, 12 songs, and a running time of 48:48 so it would only make sense for me to list 4 x 8 songs (if only because I needed a number to stop, and it makes for a nice way to list them all here).
I had fun trying to come up with songs that might fit the Implicit Christmas theme! (although I am not sure whether I captured the essence regarding your list of 12).
Sinead O’Connor – Nothing Compares 2 U, Riddy Arman – Help Me Make It Through The Night, Coldplay – Amsterdam, Sheryl Crow – The Difficult Kind, The Housemartins – Caravan Of Love, Tom Odell – Sense, Lou Reed & John Cale – Nobody But You, The Righteous Brothers – You’ve Lost That Lovin’ Feelin’
The Supremes – Reflections, The Hollies – The Air That I Breathe, The Mamas And The Papas – Monday, Monday, Marc Almond & Gene Pitney – Something’s Gotten Hold Of My Heart, Ray LaMontagne – New York City’s Killing Me, Del Shannon – Runaway, Charley Crockett – Hard Luck & Circumstances, Charlie Marie – Shot In The Dark
Etta James – I’d Rather Go Blind, Fred Eaglesmith – Cemetery Road, Gloria Jones – Tainted Love, Jackie Wilson – I Get The Sweetest Feeling, Randy Crawford – One Day I’ll Fly Away, Roy Orbison – In Dreams, Norman Greenbaum – Spirit In The Sky, Marvin Gaye & Tammi Terell – You’re All I Need
Fleetwood Mac – Need Your Love So Bad, Dion – Runaround Sue, Bobby Darin – Dream Lover, Ben E. King – Stand By Me, Arthur Baker And The Backbeat Disciples – Love Is The Message, Edwyn Collins – A Girl Like You, The Ronettes – Be My Baby, Tommy James & The Shondells – Crimson And Clover
Wow, I’m glad you were inspired! I know some of these and will enjoy trying out the others.
One mistake is not allowing multiple links to sites like pubmed without sending it to spam review. Rather than a hard rule for one link per comment it should allow multiple for whitelisted domains.
The blog already uses the email address as a kind of pre-approval, so should result in less work for you with little extra spam. This started a few years ago, not last year though.
Your other mistake is not making enough mistakes, of course.
Also: not having the ability to type equations in comments is a mistake. Add latex support (pretty please).
Anons:
Sorry, that’s on WordPress, not me! Or maybe it’s on me because I’m not expert enough in setting it up. Our sysadmin told me that WordPress has lots of plugins, but for security reasons he only trusts the ones that have a lot of users, so there are various potential features we haven’t implemented.
Regarding the spam review, all I can say is that, given the volume of spam we receive, I think WordPress does a pretty good job filtering it.
I brought it up before, and know you will not implement this. But I’ve seen in crypto communities they set up a spam channel for the spammers. It works because the day-to-day spammers get paid by clients paying at scale without checking where it really ends up, and no one needs to read the spam. You also need to be ruthless about the spammers who post elsewhere of course. The idea is that type of scheme was devised by spammers who understand the economics of that endeavor.
Quote from above: “I don’t really have any complaints about Andrew, but I guess I am complaining about you.”
(I am replying here for overview reasons, it’s easier to start a new thread in my view.)
I am not sure whether you correctly identify my posts in all the blog posts I have commented on over the past 12+ years or so (?) you have read, but if you do, it would be helpful for me to have another example that the one discussed here. I view the discussion here about the interaction with Mr. Gelman on the older post as pretty particular because it’s communicating about communicating and may therefore involve relatively more mentioning of my thought processes (and it’s very difficult, in my view at least, to keep track of things in such kinds of discussions).
I can try to sort of attempt to reply in general to your comment. I view this forum as a place where one can comment in different ways. I have recently posted a lot of poetry again for example, which I often try to make relevant to the specific post at hand. Now, I can imagine some, perhaps many, people may not necessarily like reading those posts, but I feel some may. Or perhaps noone likes those poetry posts, but I like to make them. I think that’s all okay, because it’s related to the topic at hand (most of the time), and it’s just one or two posts I make on a forum that is there for people to comment. But yeah, just like with poetry, I can imagine you may not like reading about my thought processes.
I can’t think of real mistakes, so here are suggestions! These all involve the comments, which are a wonderful part of the blog. (Thanks, everyone!)
1. Require that all comments have a username that isn’t “Anonymous” and that is consistently used, to distinguish different anonymouses (or whatever the plural is), and to allow the rest of us to more easily ignore particular commenters. (Sorry, Anonymous.) Enforce this strictly.
2. Your greater willingness to call out meandering, off-topic “garbage time” in comments is appreciated! It can be ramped up.
3. (Less important.) Pester whoever made the hosting software to allow an ‘edit’ button for comments, so our embarrassing typos aren’t staring at us for eternity.
I guess this is the right place to comment on the “correction” of Shira’s notation which I find “mildly horrifying” as ChatGPT puts it (even though I don’t necessarily agree with its technical arguments): https://chatgpt.com/share/694c3074-eac0-8001-a471-d30a35d1666c
Carlos:
I see your point, but that’s kind of why I did want to change from brackets to parentheses, because I think this is an artificial and confusing distinction. Fair enough to disagree on it, though.
We discussed a related notational dispute a few years ago. I think it can be helpful to argue about notation. These disputes are not empty; they represent real disagreements in how we use models, and I think about notation a lot.
To be clear what I find mildly horrifying is the act of changing someone else’s post because you don’t like their choice of notation (even though you say that brackets are fine too!). How would you feel if the editor of one of your books changed your equations to use brackets?
Carlos:
I did talk with Shira about it earlier and she’d said the brackets didn’t really matter to her, she was just doing it out of habit. Although I didn’t check with her on this particular post.
I agree with you in principle. It’s just that in this case I think it’s ok, and Shira is a friend.
The story that is told on this blog about plagiarism by Ed Wegman is part fabrication and part confabulation. It was based on a hit piece by an anonymous blogger who went by Deep Climate and further bastardized by John Mashey. Mashey was later ordered by a court to cease and desist with this exact story. The story as told by Andrew actually conforms to neither one of those, and Mashey has even stopped by since to attempt to correct the narrative to no avail. The fact is that Wegman offered a withering defense of his attribution decisions leading to full exoneration, and no reinterpretation of that by dishonest climate scientists can change that. If you’re going to be so strongly opinionated about a sequence of events, you should go back and find out what really happened because it is nothing like your version of the story. There are no straightforward examples of plagiarism by Wegman. As I said before, I am no fan of Wegman, but his opponents behaved far worse than he did and it is infuriating to see people simply parrot that. And this on top of completely ignoring the statistical arguments he made as if statistics somehow don’t matter because the subject is climate science.
Merry Christmas, everyone!
Matt —
The ad homs don’t do you any favors.
As someone with just a passing familiarity with the facts of this case, it’s striking how completely your comment relies on argument by assertion. You make a series of sweeping claims, but you don’t address any of the publicly documented material — the side‑by‑side text overlaps, the journal retraction, or GMU’s own findings. I don’t get why you would think that simply hand‑waving away material facts would persuade anyone. To convince people you’d need to construct an argument with actual supporting evidence.
Perhaps you gain some measure of satisfaction merely by posting a non‑persuasive argument by assertion.
@MattSkaggs, I’ve read (most of) the long thread here: https://statmodeling.stat.columbia.edu/2023/08/18/weggy-time-a-conceptual-blast-from-the-past/ Are there links that document the cease and desist order you mention? The only stuff I can find online states that W&S voluntarily withdrew a suit against Mashey in 2015. Similarly, links to Mashey “attempting] to correct the narrative to no avail” ? At least in the thread linked above, I read his comments as supporting rather than correcting the narrative?
* “attempt[ing]” …” (sigh)
“Mashey was later ordered by a court to cease and desist with this exact story.” … this does not appear to be true, or at any rate I can’t find any evidence that it is true.
“The fact is that Wegman offered a withering defense of his attribution decisions leading to full exoneration”, this is not true according to the Wikipedia page about the Wegman Report, which says “Reviews by outside experts contacted by USA Today found plagiarism from textbooks which was obvious and inappropriate; the social network analysis section had also been partly copied from Wikipedia.” The Wikipedia article also “the 2008 social network analysis paper was investigated by a separate committee which unanimously found “that plagiarism occurred in contextual sections of the (CSDA) article, as a result of poor judgment for which Professor Wegman, as team leader, must bear responsibility”, and Wegman was to receive an “official letter of reprimand”.
The claim that “there are no straightforward examples of plagiarism by Wegman” is amusing in its phrasing and its narrow scope. Are there examples, but no straightforward examples? Are there straightforward examples of plagiarism in the Wegman report, but not _by_ Wegman? The latter, at least, is Wegman’s claim: he admits there are examples of plagiarism, but he blames them on a student: “When allegations of plagiarism in this section were examined, Wegman said that material in it had been “basically copied and pasted” by a student who was the “most knowledgeable” person about such analyses on his team.” Hey, could be, maybe a student wrote a section of the report, and Wegman and his co-authors included the section without checking it and without crediting the student as an author.
That said, I agree that in the big picture all of this stuff is a distraction from what should properly be the main topic of discussion…but that topic is not the errors Mann et al. made fifteen years ago or whenever it is, it’s what is _really_ happening with the climate. The climate skeptics have won the day as far as policy, but the mainstream climate scientists have been proven correct in their warnings about rapid anthropogenic warming. If only it had been the other way around!
Matt:
I can’t speak for “Deep Climate” or for Mashey. I’ll just say that it seems beyond dispute that Wegman plagiarized multiple times, most notoriously when a passage from Wikipedia appeared with only tiny changes (most notably, the inclusion of a mathematical error!) in an article with Wegman listed as author.
Here’s the story, here’s the original Wikipedia version:
and here’s the version in the Said and Wegman paper:
I guess nowadays Wegman would cover his traces by using a chatbot.
1. It would be nice to have links to outside sources open in a new browser tab, so the original post is still open.
2. Edit button for comments
3. Threaded replies are sometimes hard to follow. Not clear what is a reply to a reply vs. a reply to the original post.
4. Some way of displaying post titles only, all on one page. Sometimes I remember the post a few days old, but it may take scrolling through multiple pages to get to it.
Perhaps some more colors for various levels of posting, etc.
Re #1: it is so ingrained in me to right-click and open a link in a new tab when reading any article online that I barely think about it. I agree, though, that this should be the default; for my WordPress-hosted site, it is not, and I have to manually make each link open in a new tab for the reader when creating a post.
Navigator:
1,2. I don’t know how to do these but I can ask the sysadmin.
3. We set a limit to thread depth because otherwise it can be very difficult to read the threads on the phone.
4. Maybe that’s possible using another panel on the right side of the page?
It wasn’t a mistake but it was kinda funny. There was one of your blog posts that I started reading and as I was reading I was getting more and more disquieted, getting the feeling of “oh, no, I really don’t understand anything at all about statistics, this just doesn’t make sense with everything I’ve come to learn”. And then I started having feelings of acute imposter syndrome, accentuated by starting a new job. The next day I went back to read it again to see if it a second reading made it make more sense and then I noticed the blog post was dated April 1rst. Such a huge sense of relief.
This post, praising Allison Mattek (https://statmodeling.stat.columbia.edu/2019/07/22/alison-mattek-on-physics-and-psychology-philosophy-models-explanations-and-formalization) was a mistake. Mattek’s paper is not good. See Anoneuoid’s and Daniel Lakeland’s comments on that post.
Experiment comes first. Definitions and equations come later, all for the purpose of describing the data and making predictions about what new data on different variables looks like. Example: Galileo did many experiments on motion, and then using the data and his own creativity, defined velocity in a way that made it useful for predicting the position of an object. Mattek does not understand this. Describing her work as being like Meehl’s implies that you have some misunderstandings here as well (though, given your post about how utility theory is unfalsifiable, less misunderstandings).
Anon:
Thanks for the pointer. I’d forgotten about that entire discussion! This kinda makes me feel I should write a book or article to put all my philosophy-of-science ideas in the same place. I guess that such a book would sell approximately zero comments, but writing it could be helpful to me.
I also had forgotten about the Mattek paper so I reread it. It’s hard for me to evaluate it–I think its intended audience is research psychologists, which I’m not, and I don’t really know where they’re coming from. I thought it was interesting but I can’t really judge its utility. I wouldn’t agree with you that the paper is “not good”; I think its goodness depends on how it works with its audience.
I was curious what was up with Alison Mattek now so I googled but there was nothing recent–I guess she’s working in private industry? I did turn up a webpage for her family . . . they seem to be a pretty accomplished group!
For what it’s worth Andrew, if you wrote a book collecting your philosophy of science ideas in one place I would click that preorder button before I’d finished reading the subtitle!
OK, I’ll do it!
You use the term ‘awhile’ in many of your posts but you often are using it incorrectly. It’s pedantic, but to me it’s jarring every time I read it, and I don’t want to comment each time it happens because that would be annoying for you!
Here are examples of proper usage/when to use awhile vs awhile vs for a while – a general common error is saying ‘for awhile’ but this usage should be ‘for a while’:
https://www.merriam-webster.com/grammar/awhile-usage
I wish I had something more substantive to offer but the actual content is enjoyable! Merry Christmas
Hey–I’ll try to do better!
Great thank you! Also apologies I assume happy holidays/seasons greetings is more appropriate for you than merry Christmas!
yup
The claim “The usual rules of conditional probability fail in the quantum realm” appears in multiple posts and in a slightly different form in the “Holes in Bayesian statistics” paper.
Blog post “Holes in Bayesian Statistics”,
Posted on February 23, 2020 1:00 PM by Andrew
Commenters explained in detail why the claim is false.
Blog post “Holes in Bayesian statistics (my talk tomorrow at the Bayesian conference, based on work with Yuling)”
Posted on July 2, 2024 2:07 AM by Andrew
Commenters explained in detail why the claim is false.
In a comment on the February 23, 2020, post, Andrew on February 24, 2020 6:06 PM at 6:06 pm said:
“And I gave my talk to a room of physicists, and they didn’t think what I was saying was nonsense.”
Appealing to authority is only as good as the authorities. Even with good authorities, one should check the reasoning.
Blog post “Carroll/Langer: Credulous, scientist-as-hero reporting from a podcaster who should know better”
Posted on October 19, 2024 9:19 AM by Andrew
This post points out that physicist Sean Carroll is not always a reliable authority. Carroll is supposedly an expert on quantum mechanics.
“Physicists disagree wildly on what quantum mechanics says about reality, Nature survey shows”
Nature News Feature, 30 July 2025
This article surveyed physicists as to the correct version of quantum mechanics. No version got more than 36% of the votes.
Conclusion: A room full of physicists agreeing with a talk provides no evidence of correctness. If anything, it may indicate error.
David:
Ahh, yes, I recall that you disagree with me on that point. My view on this point does not come from an argument from authority; it comes from the well-known fact that Boltzmann statistics (what we in statistics would usually just call “probability”) does not apply in the quantum realm, where reality is described by Fermi-Dirac statistics, Bose-Einstein statistics, and interference (superposition of quantum states rather than of probabilities).
I never thought of my take on this as controversial, but given your sources, I accept that it is actually controversial. I don’t think it’s controversial that electrons follow Fermi-Dirac statistics, not Boltzmann statistics, so I guess the question is whether Fermi-Dirac statistics and other quantum phenomena can be expressed using the usual rules of conditional probability. I think the difference between my position and yours is not about physics but about what are the rules of conditional probability.
Based on previous discussion threads, I think there’s approximately zero chance we’ll resolve this one here, but I appreciate you bringing it up, as this is the sort of thing I was asking for in this post!
Since you were asking for it, and since I think it’s worth explaining the main area of disagreement, I’ll try to give the overview here, and then I agree we’re unlikely to resolve it so I wont’ press the point.
The main area where David and I disagree with you is in the kind of set up that the 2 slit experiment represents. This is where you have a source of particles, halfway along the path you have a screen with two slits with a shutter on each slit, and at the end a detector screen. You can shutter one slit, and see a pattern of detections that looks lumpy, and you can close that shutter and open the other, and get detections that look lumpy but shifted over slightly. Then, you can open both shutters and instead of seeing two lumps next to each other, you’ll see an “interference pattern” in which the detections are spread out and have bright and dark regions.
The set-up and the results are described here https://en.wikipedia.org/wiki/Double-slit_experiment with the basic visual result shown in this image with a red laser: https://en.wikipedia.org/wiki/Double-slit_experiment#/media/File:Single_slit_and_double_slit2.jpg
Your previous assertions were that because of the interference pattern with the double slit, the probability of seeing a photon hit the detector at a particular place was not the sum of the probability for it to go through the first slit (which you took as the density for the one slit pattern) plus the probability to go through the second slit (the same pattern shifted slightly to align with the other slit).
Where David and I think your error is is that you take the probability in the two slit open experiment to be the sum of the probabilities coming from the single slit experiment. But the double slit experiment is a different experiment.
According to Bohmian mechanics, the only version of QM where particles have positions and trajectories that are explicitly described by the theory (AFAIK), the photon DOES go either through the first slit, or through the second slit in the double slit experiment, but the **trajectories** that it could go along are different trajectories than if only one slit is open. This is similar to how if you had a wind tunnel and it had a screen in the middle with two holes through which the air would squeeze, that the turbulence and things it would induce would cause particles of smoke to move to different places than if just one hole was open. No one would say that the concept of probability fails because particles of smoke going through two holes in a wind tunnel screen fail to go to the same places that they would go to if one or the other but not both holes were open. What we would say is that conditional on knowing where the particle of smoke started, and the wind patterns caused by the apparatus, we could estimate probabilities for the particle to finish at a certain spot at the end of the tunnel… and with different patterns of open holes, we would use different wind patterns to get different predictions for the final location.
Here are the Bohmian trajectories
https://upload.wikimedia.org/wikipedia/commons/thumb/0/02/Doppelspalt.svg/960px-Doppelspalt.svg.png
As you can see, conditional on the particle going through the bottom slit, it will be detected on the bottom half of the screen. And conditional on it going through the upper slit, it will be detected on the upper half of the screen. The probability of it being detected anywhere at all on the screen is the sum of the probabilities to go through the lower slit, plus the probability to go through the upper slit. Showing that conditional probability works in the usual way and there is no incompatibility with the predictions of QM.
I think we’re reaching garbage time and will never agree so I will not comment further.
The Veritasium video “There Is Something Faster Than Light” posted on YouTube Dec 19, 2025, is pretty good except for the end.
Andrew I wrote a detailed post related to this, but it seems to be held in moderation because it links to a few wikipedia places. Anyway I hope someone can approve it. Thanks
Daniel, after going through the posts that David mentioned, I have a question for you: do you think any empirical evidence can justify changing the laws of probability and logic? Andrew’s answer (and many other statisticians’ answer) is yes. Yours seems to be no.
The laws of probability are a consequence of Cox’s theorem. They aren’t an empirical fact they are a formal consequence of certain axioms about how numbers representing plausibility should work. Could any empirical evidence convince you that its wrong that for all real x the formula 0*x = 0 ? Thats just how real numbers are created to work.
In any case as my longer post says, the empirical evidence question doesn’t arise because in the only version of QM in which it makes sense to say that a particle goes through a given slit in the 2 slit experiment, the probability of a detection at a given location p(x) = p(x|slit1)p(slit1) + p(x|slit2)p(slit2) just as conditional probability requires. The evidence doesn’t disagree.
Bohmian mechanics is the version of QM in which particles exist and have positions and continuous trajectories described by the theory. All other versions of QM are silent on the question of whether particles exist or have continuously defined positions AFAIK. Its not possible to formulate the question of p(slit1) in any other form of QM.
Daniel:
You can talk about Cox’s theorem all you want, but electrons follow Fermi-Dirac statistics, not Boltzmann statistics.
I like the analogy to non-Euclidean geometry. From Euclid’s axioms, the angles of a triangle add to 180 degrees, but on a sphere the angles of a triangle add to more than 180 degrees. Euclidean geometry works fine on Earth for small distances but not for large distances.
Andrew, the thing is, Probabilities dont describe physical anything. Physics describes fphysical stuff. Probability describes what your available information implies about the plausible physical things that might happen. Geometry describes physical things like angles, probability describes what logic tells you about your conclusions within a formal mathematical system when your information is less than perfect. If the probabilities lead you astray its a sign your physics is wrong, not that probability logic fails in the world.
I’m not sure what the Fermi-Dirac and Boltzmann statistics stuff has to do with the question of the double slit experiment which is where your original claim about conditional probability failure came from.
F-D statistics are a result in the thermal physics of a large collection of Fermions. It tells us the distribution of energies of a randomly chosen particle in this thermally equilibriated group. Fermions have restrictions on what is physically possible for them to do, this knowledge of their restrictions implies a different distribution of energies, just like our knowledge that there are only 2 seats in a certain sports car implies something about how many people there are in a given mile in a traffic jam full of those sports cars. We dont know exactly because the cars could have 1 or 2 people in them but we know if they’re driving they dont have 0 and we know that they can’t fit 3 or more. Also the cars have a certain length, so they can’t be more densely packed than bumper to bumper. if we know the average density of cars over a 10 mile stretch of road, and the total number of people over that stretch, we can from that determine plausible numbers of people in a 1 mile sub-stretch. That all follows from standard probability calculations incorporating the physical restrictions of the cars and doesn’t involve anything that violates probability laws like conditional probability.
The original context for David and I to object to your statements was in the context of double slit type experiments where you argued that the distribution of outcomes with both slits open was not the sum of the distributions of slit 1 alone and slit 2 alone.Thats true, but is not a failure of probability, its a failure to use physics to determine what the outcome distribution should be for both slits open. when you do use physics you find that the outcome distribution is the sum of the two conditional distributions, “goes through slit 1” and “goes through slit 2” at least for the QM interpretation that admits particles exist and have paths so that they can “go through” a slit. Probability isn’t violated. There are two possible mutually exclusive sets of paths, and the total set of possible outcomes comes from adding together the possibilities from each exclusive subset just as required by probability.
Daniel:
From my perspective as a former physics student, the statement that the laws of probability do not match physical reality is just a commonplace. Fermi-Dirac statistics and quantum superposition are two examples of this.
Anonymous wrote:
> do you think any empirical evidence can justify changing the laws of
> probability and logic?
Probability is a part of mathematics. We make definitions and prove theorems. We then apply probability in models and theories about the real world. If you use the wrong model or theory, then you may find the model or theory does not apply to the real world. So, we may stop teaching a model if it has no useful applications. This cannot invalidate the proofs of theorems.
> From my perspective as a former physics student, the statement that the
> laws of probability do not match physical reality is just a commonplace.
Is “former physics student” an argument from authority? I confess to not having majored in physics.
By commonplace, do you mean others said it? Or that it is obviously true? I can believe others said it. But, it isn’t obviously true.
As an example of what other authorities have written on this topic, I submit Section 10.4, “Classical versus quantum probability (and logic)”, in
Sheldon Goldstein et al. (2011) Bell’s theorem. Scholarpedia, 6(10):8378.
The authors are
Sheldon Goldstein, Mathematics Department, Rutgers University, NJ
Travis Norsen, Smith College, Northampton, MA, USA
Daniel Victor Tausk, Mathematics Department, University of São Paulo, Brazil
Dr. Nino Zanghi, Physics Department, University of Genoa, Italy
Andrew, as I said, I dont think we will resolve this here. in my view you were taught some statements about probability in QM in an undergraduate QM class which were wrong, and you internalized them in a way that makes it difficult for you to see what David and I are arguing. Part of why I argue here is because I think these common but wrong statements in undergrad QM are holding back understanding.
Note that neither of us argues against the ideas that F-D statistics are different from Boltzmann, nor that F-D statistics better describe Fermions in large collections. we both accept that. Also we dont argue that the distribution of detections in the double slit experiment is the same as the sum of distributions of each single slit expt. Instead we agree there is an interference pattern.
what we disagree with is statements like “the laws of conditional probability do not hold in QM”
we both think that the laws of probability are formal laws about certain calculations which can’t be invalidated by observation any more than you could convince us that there is a real number x for which 0*x is something other than 0
What could be invalidated by experiment is the physical description of the experiment, ie. the physics. Indeed a Newtonian description of the two slit experiment is invalidated by observation.
However we dont think the physics of QM is wrong in any important way. Therefore we expect the laws of probability, which are a calculation tool on top of physics rather than a co sequence of physics… they should hold.
In the one form of QM that admits the existence of particles with trajectories, the probability of a detection at location x is p(x) = p(x|slit1)p(slit1) + p(x|slit2)p(slit2) which incorporates both the sum and product rules of probability in the normal way and the idea that slit1 is mutually exclusive from slit2. Ive shown you a graph of the trajectories calculated from QM, and you can verify that particles never cross over the centerline of the experiment and so slit1 detections and slit2 detections are mutually exclusive. The ultimate result of all this could be summarized as:
Bohmian QM predicts that particles either go through slit1 or slit2, and that these are mutually exclusive and therefore the probability calculations assuming mutual exclusivity hold in the usual way.
Other forms of QM are incomplete theories in that they dont have anything to say about whether particles have paths or they even exist. Quantum stuff is full of confusion
I wrote a longer post on my phone, then something happened weird with touchscreen baloney and I think it was lost. Or maybe it shows up later. I don’t know.
I’ll try to summarize the points here. Basically David and I don’t have any issue with saying something like “Newtonian mechanics fails in the Quantum realm” or “Quantum particles obey different energy statistics in thermal equilibrium than predicted by classical theories” or any of that. What we object to is any notion that probability somehow fails in the quantum realm.
First off, probability isn’t an empirical theory. Probability is a logical structure in mathematics that tells you how to calculate with certain kinds of quantities. Similarly, you could no more convince me that probability fails by doing particle experiments than you could convince me that there exists a real number x such that 0*x doesn’t equal 0 by doing particle experiments.
I can prove that 0*x = 0 in the reals, it’s not contingent on the outcome of experiments.
If probabilistic calculations using some physical theories fail to predict experiment, this is evidence that **the physics is wrong** not that probability doesn’t work.
However, in QM, the only theory that has explicit notion of particles with positions and trajectories, Bohmian mechanics, predicts as I showed in links above, that detections on the screen in the double slit experiment come from particles going through slit 1 or through slit 2 and these are mutually exclusive, and the conditional probability statement p(x) = p(x|slit1)p(slit1) + p(x|slit2) p(slit2) which uses both the sum and product rule of probability, works just fine to predict the empirical results. In this physical theory, slit1 and slit2 are separate mutually exclusive sets of events, and because of that, the probability math works correctly.
What doesn’t work is to assume that the two slits open experiment has observation probabilities that are somehow the sum of the slit1 open experiment and the slit2 open experiment. That’s wrong physics. That’s not the distribution predicted by QM.
I would accept a statement “if you replace QM by a naive sum of the two distributions of individual slit observations, it will give the wrong result” but I don’t think this means “probability fails” I think it means “using the wrong physical predictions fails”. It would also fail to assume that for example the distribution of particle detections would be uniform, or have a shape like an elephant, or be triangular. Those are not predicted by a physical theory, and neither is the “sum of the individual slits” distribution.
My example of the wind tunnel, shows that you get the wrong results even in very very classical experiments if you assume the wrong physics.
Daniel, pretty simple: there are empirical situations where the standard laws of logic and probability do not work.
This comes up when you have a database with incomplete information, for example, or situations in electrical engineering. If part of a circuit has a defect and you don’t know whether a certain switch in that part is on or off, you can represent it as a third logical value, “unknown”. Multivalued logic was invented to deal with these situations. Two-valued logic fails. Why can’t quantum be the same way?
https://en.wikipedia.org/wiki/Many-valued_logic#Applications
The laws of logic are just models. They can be rejected as irrelevant when they disagree with physical reality or are not useful in making predictions, which is what I really mean by “wrong.”
Probability is the same way. Bell’s inequalities can be restated as necessary and sufficient conditions for a probability distribution to exist that describes all the possible measurements you can get from a system (this fact that you can so restate them is called Fine’s theorem). Bell’s inequalities are of course violated, so there is no classical probability distribution that can reproduce the results of quantum mechanics. Hence we need to change the laws of probability.
See e.g. https://arxiv.org/abs/1403.7136v2
About specifically Bohmian mechanics, there are two problems that serious Bohmians know about and try to overcome:
1. Bohmian mechanics disagrees with relativity.
2. Bohmian mechanics is far more complicated than standard quantum (it has a whole extra guiding wave and an extra equation to describe it). This complication does not pay off in the form of extra correct predictions.
See e.g. here: https://philsci-archive.pitt.edu/20537/1/qm%20underdetermination.pdf
So there is no reason to follow it. The fact that it has an ontology is meaningless, if I said everything was controlled by fairies that’s also an ontology.
I don’t think you’re going to be convinced but there are serious arguments against your positions that you ignore.
I recommend that people read the “Bell’s Theorem” article on Scholarpedia. The mistakes Andrew makes in his paper are discussed and corrected in Section 10.4 of that article.
It is important to understand that the experiments with one slit open are incompatible with the experiment with two slits open. You can’t model incompatible experiments using a common probability space. This is just as wrong for classical experiments as for experiments in quantum mechanics.
If your physical theory says that opening the second slit does not affect the particles, then it is your physical theory that is wrong. It would be just as wrong if you were discussing water waves.
Bohmian mechanics tries to fit a classical peg in a quantum hole. Why should I accept that different experiments done on the same system are “incompatible”? I want one model to describe them all (one per system). Bohmians want there to be multiple models, which is another way it is more complicated than standard quantum, with no payoff.
Anonymous,
> Bell’s inequalities are of course violated, so there is no classical
> probability distribution that can reproduce the results of quantum
> mechanics.
Bell’s inequality being violated implies that the assumption used to derive Bell’s inequality (i.e,, locality) is false.
> 1. Bohmian mechanics disagrees with relativity.
There are versions of Bohmian mechanics that agree with relativity, i.e., are Lorentz invariant.
> 2. Bohmian mechanics is far more complicated than standard quantum
Textbook quantum mechanics has all the measurement postulates and the need for observers.
Textbook quantum mechanics does not have tables and chairs.
David:
To reply to your comment higher up in this thread, when I say, “From my perspective as a former physics student,” of course it’s not an “argument from authority”! Having taken some physics courses 40 years ago is not much of a sign of authority at all. I was letting everyone know where my perspective was coming from.
David,
> Bell’s inequality being violated implies that the assumption used to derive Bell’s inequality (i.e,, locality) is false.
This is equivalent to what I said, read it again and look at the link. Classical probability assumes these things (locality and realism). It is wrong as applied to reality. Fine’s theorem.
> There are versions of Bohmian mechanics that agree with relativity, i.e., are Lorentz invariant.
Not ones that work. You need to assume a preferred reference frame. Even Goldstein has tried to escape from that and failed.
> Textbook quantum mechanics has all the measurement postulates and the need for observers.
Many-worlds is another interpretation that does not need measurement postulates. They fall out of the formalism. Same with informational interpretations.
Textbook quantum mechanics has all the measurement postulates and the need for observers.
> Textbook quantum mechanics does not have tables and chairs.
Neither does Bohm. These are emergent properties.
Note even for Bohr style quantum mechanics the conditional probability calculations you might do are exactly the same. For example, you can condition on seeing the particle hit the lower half of the screen, and then if you want to know the probability that it hit between position A and B, you would calculate the probability of hitting between A and B unconditionally and divide by the probability of being seen on the lower half of the screen. That’s just a standard conditional probability calculation. It works just fine.
The only issue with bohr-style quantum mechanics is that you can’t condition on the particle going through a particular slit since it has no notion of trajectories.
it simply isn’t needed to get into the bomien versus other interpretations of quantum to see that when you do a probability calculation using quantum mechanics, all the usual rules of probability apply. When you take a probability of one event and you take a probability of another event and the events are mutually exclusive, then you can add them together to get the probability of the unified event, etc. None of that breaks.
Anonymous,
> Classical probability assumes these things (locality and realism).
Like Andrew, you appear to be changing the meaning of words, or making up your own. I don’t know what “classical” means when applied to probability.
> Not ones that work. You need to assume a preferred reference frame.
The world is nonlocal. So, it is not surprising that there is a preferred frame: You need a frame in which the nonlocality happens.
The only thing experiment confirms is Lorentz invariance. Lorentz invariance means we can’t detect a preferred frame, So Einstein said it doesn’t exist. It the theory is Lorentz invariant, then it will agree with experiment.
> Many-worlds is another interpretation that does not need measurement postulates.
Many-worlds is just nonsense: There are no atoms, no tables, no chairs, no cats. Plus, it gives the wrong probabilities for experiments. Many-worlds is what you get from staring at the equations for too long.
> Neither does Bohm. These are emergent properties.
In Bohmian mechanics, tables and chairs are made out of atoms, which are made out of electrons, protons, and neutrons. Just like people are taught in high school. This time, high school got it right.
This discussion is not really relevant to Andrew’s mistake.
A Lorentz invariant world that is somehow actually nonlocal. Bohmians think this isn’t tortured? Don’t make me laugh.
Dude, I feel like Paul Hayes from previous blog comments. There’s only so much arrogant ignorance I can take.
Meese,
We are left with just what experiment shows us is true. EPRB shows us the world is either nonlocal, or there is a local hidden variable missing from the QM description. Bells inequalities and the Aspect and other experiments confirm there are no local hidden variables. of the two options that could solve the EPRB question the only one left is the world is nonlocal.
Many experiments also prove Lorentz invariance.
We are left logically with the requirement for a Lorentz invariant nonlocal theory. Anything else fails to be scientific in the sense of following the logical implications of experimental results.
“Standard QM” is nonlocal. Einstein showed this with his boxes experiment. Take a box, place a particle in it. Partition the box into two boxes. Carry one of them far away. From a position halfway between the boxes, send a radio signal, at each box when the Signal is received open the box and photograph the particle. It will always be in exactly one box, but no light signals could arrange that perfect correlation.
Bohm says it was always in one box but other forms of QM say it has no position until measured so the world somehow arranges for the perfect correlation between boxes in a nonlocal way.
The world is Lorentz invariant, and the world is nonlocal. 🤷♂️
Daniel, not at all. Bell’s original paper clearly is just a sharpening and completion of EPR’s argument. The experimental violation of Bell’s inequalities shows that the world cannot be “local realist”. You can reject locality, or you can reject realism (the particle has no position until measured), or you can reject both. Different choices give different interpretations of QM. Lorentz invariance is ironclad.
Meese wrote:
> You can reject locality, or you can reject realism (the particle has no
> position until measured), or you can reject both
If the particle has no position until measured, then you are back to a theory that requires observers and needs the measurement postulates. And, you have cats that are both alive and dead at the same time.
You can’t save locality by discarding “realism”. See Section 10.2 of the Scholarpedia article. As Daniel explained, Einstein’s boxes show that standard QM is nonlocal.
If you don’t think particles exist until “measured”, do you think the moon is there if nobody looks?
> Bell’s original paper clearly is just a sharpening and completion of
> EPR’s argument.
I’ve read Bell’s paper (several times). It uses the EPR argument at the beginning to show locality implies determinism for the particular experiment. But, Bell’s inequality is not in the EPR paper, and EPR were undoubtedly unaware of it.
Anonymous wrote:
> I feel like Paul Hayes from previous blog comments.
For the benefit of those who don’t know, Paul Hayes was a commenter involved in a similar discussion on this blog. He eventually gave up when he couldn’t justify his claims. He did not admit that he was wrong.
David,
> If the particle has no position until measured, then you are back to a theory that requires observers and needs the measurement postulates.
Nope! Any interaction counts as a measurement. No observer required. This is very standard.
> As Daniel explained, Einstein’s boxes show that standard QM is nonlocal.
Einstein’s boxes only show that standard QM is not “locally causal,” which is a two-part assumption. Some people call this “locality” but it includes “realism” as well. All arguments claiming QM to be nonlocal smuggle in realism too.
> I’ve read Bell’s paper (several times). It uses the EPR argument at the beginning to show locality implies determinism for the particular experiment. But, Bell’s inequality is not in the EPR paper, and EPR were undoubtedly unaware of it.
Read it again, then. You don’t seem to have understood the main point. I never said EPR were aware of Bell’s inequalities. Their argument, when taken to its logical conclusion, gives a set of inequalities which are experimentally violated. This is Bell’s point.
Perhaps you should read some of the vast literature about all types of interpretations, and engage with it instead of strawmanning it and claiming that other interpretations than Bohm are obvious nonsense. You may learn something. Try e.g. Zurek’s “Decoherence and the emergence of a classical world from quantum theory,” or this paper: https://arxiv.org/abs/1402.0351
Sorry, I forgot to write my name by accident. The latest anonymous comment refuting David’s claims here is mine.
> For the benefit of those who don’t know, Paul Hayes was a commenter involved in a similar discussion on this blog. He eventually gave up when he couldn’t justify his claims. He did not admit that he was wrong.
That’s really rather funny. You are yourself confidently wrong and you refuse to admit this.
Meese/Anonymous wrote:
> or this paper: https://arxiv.org/abs/1402.0351
Wiseman’s paper says this about Bell’s theorems (page 2): “The first is that proven in 1964 [1], that there are quantum predictions incompatible with any theory satisfying locality and determinism (L&D).”
Indeed. This combined with the EPR argument gives us that QM is not local.
Wiseman’s claims that Bell didn’t understand his own papers are not convincing.
> Some people call this “locality” but it includes “realism” as well.
OK, I’ll bite: Please define “realism”.
> Wiseman’s claims that Bell didn’t understand his own papers are not convincing.
Well yes, those are the two camps. Bell forgot the details of his own argument since he gave it decades prior, and only remembered the intuitive idea. (He did use “local causality” many times after he introduced the term in his 1976 paper to describe the assumption, though, with occasional lapses.) This is quite plausible, it happens to most people, and it explains his statements quite well, and better than the alternatives. (Why would he state four times in the 1964 paper that his result requires two assumptions otherwise? etc.)
> Please define “realism”
“Realism” is probably better stated as “counterfactual definiteness” (CD), or the idea that unperformed measurements have no results. That is, under standard meanings that nearly all people who talk about interpretations agree with (measurement = interaction), the rejection of realism/CD means that if it doesn’t interact with anything, it doesn’t matter.
As an aside, I just learned about this paper, you may be interested: https://arxiv.org/abs/1807.02092
Supposedly you can derive the Born rule and measurement postulates from the others, in an interpretation-independent way. No nonsense many-worlds or complicated Bohmian guiding wave required. I will have to think about this.
Meese,
Can you try again in your definition of “realism” or Counterfactual Definiteness?
i didnt get a definite sense of what you meant or how it applies.
Perhaps even better you could give a simple logically sound argument for the meaning of the EPRB + Bell results incorporating “realism” into the argument explicitly.
I gave such an argument previously and did not incorporate this concept perhaps you can define realism and incorporate it into my argument in a way you believe completes the argument correctly.
The basic argument I gave is EPR implies either nonlocal, or local + incomplete description. Bell implies no hidden variables can complete a local description. the two together imply nonlocal.
you believe some premises are wrong and should involve “realism” or “counterfactual definiteness” (CD)
defining CD and restating my argument using it according to your view would help me immensely to understand what you’re saying.
Meese,
I agree with Daniel’s comments..
Section 10.3 of
Sheldon Goldstein et al. (2011) Bell’s theorem. Scholarpedia, 6(10):8378.
discusses and disposes of the claim that Bell’s reasoning is suspect due to the use of counterfactuals. I won’t quote their entire discussion, since people can read it for themselves. However, I think this is the key:
“in order to explain (without violation of locality) the fact that the outcomes will be perfectly anti-correlated if the experimenters both measure spin along the z-axis, one has to assume that these outcomes are pre-determined. The same goes for measurements of spin along the x-axis.”
Do you disagree with this? If so, what is your alternative explanation for how this can be?
Daniel, David:
The classical prediction for the correlation function agrees with the quantum prediction when correlations are perfect. [Citation below.] If the spin states of e.g. entangled spin 1/2 particles are perfectly correlated, there is no need for any weirdness, they act classically. [This applies to your classical Einstein’s box argument too. In real experiments, you always have to use repeated measurements, hence the appearance of probability in the quantum version.] The particles act like left and right-handed gloves (looking at one allows you to perfectly predict the state of the other) and they came from a common source. It is when correlations are imperfect that you get issues, because the particles are more correlated than you expect classically.
For a highly cited and very physical proof see Peres, “unperformed experiments have no results”. It’s very short and you can easily get a pdf on sci-hub.
https://doi.org/10.1119/1.11393
The key points are 1) the one I stated above, and 2) also that in deriving the Bell inequalities you need to consider unperformed measurements [supposed proofs of the inequalities with just locality smuggle the assumption in]. This can probably be improved into a general theorem applying regardless of the way you derive the inequalities.
>what is your alternative explanation?
The explanation for imperfect correlation varies. I wouldn’t venture to give just one, because nobody knows which one is correct (I am biased towards superdeterminism, though). And ALL of them have strange features. Pick your poison.
1. In many-worlds, within each world, everything is local. But, over the universe of worlds, there is not just one outcome for a measurement, all of them happen, with the right fraction of worlds to violate Bell. Some people say the Bell inequalities are *uninteresting* in MW.
2. In [most?] informational interpretations, there is no nonlocality. Your state of information, represented by the wavefunction, changes upon measurement.
3. You know what the Bohmian explanation is.
4. Retrocausality. The result of your measurement travels backwards in time and influences the other (?!).
5. Superdeterminism. Everything in the universe is determined in advance and it conspires to give an apparent violation of the inequalities, even though there is no real violation of locality or CD. (If we live in a deterministic simulation, this is what happens.)
Etc. These interpretations and others I have not mentioned do not exhaust the space of possibilities. Maybe Zurek (I mentioned him and his work earlier) is making progress in this regard, he at least has new experimental predictions. We need to take more drugs and think harder to solve this problem.
Its the same thing as the three polarizer paradox:
https://sciencedemonstrations.fas.harvard.edu/presentations/malus-law
See eq 8-9, Bell tests are the equivalent scenario, so we should apply Malus’ law:
https://cds.cern.ch/record/142461/files/198009299.pdf
So, we don’t need any weirdness to explain the results. There’s a long mastodon thread with daniel, david, and I where I discovered many people with opinions on quantum mechanics stuff are completely unfamiliar with the experimental details of these bell tests.
That’s a great Bell essay.
1) He uses the phrase “local causality” quite often. Unfortunately he neglects to mention it’s a two-part assumption.
2) He shows clearly that perfect correlations can be explained with socks and you need more to explain imperfect ones.
3) He mentions superdeterminism, but not under that name.
Many quantum optics books (e.g. Mandel and Wolf) show that you can explain the quantum mechanics of a single photon with the classical Maxwell equations. Even if you didn’t know this, there are many examples where classical wave intuition is extremely useful for explaining QM, so perhaps the concept of “particle” needs to die and be replaced entirely by a “wave” concept. (I need to think more about that Malus’s law thing.)
I also found the proof I was looking for.
Proposition 2 of Fine (1982) (https://faculty.washington.edu/afine/PRL1982.pdf) shows that if you have a Bell inequality, you can use it to construct a local hidden variable model (the converse of the usual, where a local hidden variable model implies a Bell inequality). Therefore, the experimental violation of the Bell inequalities implies a violation of the compound assumption of local hidden variables (locality + CD).
Meese,
Consider the experiment where QM says there are perfect anti-correlations if both experimenters choose the same direction. Bell derives an inequality for this. QM says this inequality is violated. Bell’s logic is
1) Locality and the perfect anti-correlations imply the values for all directions are determined at the source.
2) The values being determined at the source give the inequality.
Which of these steps to you think is wrong?
David, Anon and I have now both given citations (one by Bell himself) that show that QM and classical predict the same thing for perfect correlation (the particles are like gloves or socks). Bell’s inequality is violated for imperfect correlations only.
But for those imperfect correlations, the wrong step varies by interpretation. My bias is towards saying that “the experimenters freely choose the direction” is wrong. That explanation is called superdeterminism (though it doesn’t need a whole “cosmic conspiracy”. Regular inefficiences can be modeled like this, because if it is inefficient enough, you won’t choose it: https://plato.stanford.edu/entries/qt-epr/).
Step 1, locality implying determinism, is also wrong (it secretly assumes CD too). See the five explanations I gave in my previous comment. There are other choices you can make than Bohmianism.
@meese wrote:
Please do and give your opinions here.
Let’s look at step 1. If we see one of Bertlmann’s socks, we predict the other sock is a different color. Every time we do this, we are correct. Our conclusion is that Bertlmann always puts on different-color socks when he gets dressed. Can you give an alternative explanation?
David, I have made my point about this repeatedly and given four alternatives (numbered in one of my previous comments). I think we have reached garbage time.
Anon: Your idea is very interesting. Tragically, it predicts correlations that are a factor of 2 too small, see e.g. https://www.nature.com/articles/s41598-021-84438-9
The functional dependence is correct, however. Only the factor of 2 is wrong. It can be used to gain physical intuition.
You may also be interested to know that there are classical states of light (“non-separable states”) that violate an analog of Bell’s inequality and can be used to do a limited amount of quantum computing:
https://iopscience.iop.org/article/10.1088/1367-2630/16/7/073019
https://www.sciencedirect.com/science/article/abs/pii/S037596011500376X
Meese,
In your previous comment, you wrote that your numbered alternatives were for “imperfect correlations”. Bertlmann’s socks are always different, i.e., the (anti-)correlations are perfect. So, you haven’t given any relevant alternatives.
Daniel asked you to give the derivation of Bell’s inequality and explicitly include counterfactual definiteness or realism in the argument so that we could see where you believe it is needed. You didn’t do this.
The only possibilities anyone has ever given for Bertlmann’s socks is Bertlmann put the socks on when he got dressed or nonlocality.
Bell stated clearly that the only assumption for his inequality is locality. I’ve read his papers and the papers by people claiming Bell made a mistake. Bell did not make a mistake. (There are also the no-conspiracy assumption and the experiments-have-outcomes assumption, but denying these is like clutching at straws.)
David, perfect correlations can be explained perfectly classically, so if that is the only thing you consider, you don’t need alternatives (because you want the simplest explanation). The alternatives I gave, however, work if you are considering both imperfect and perfect correlations, as they should. Perhaps that was not clear in my statement.
I can’t give equations in a blog comment. I linked to a very short derivation that has CD. All derivations of the inequalities implicitly or explicitly assume that there are real properties (in the philosophy encyclopedia I linked to this assumption is called “separability” for some reason). If you are only convinced by arguments that do not have equations, I cannot help, and neither can anybody else.
I can say that I consider Bohmianism to be clutching at straws too. This differing philosophical intuition is why nobody can agree on QM interpretations, and why I gave four alternatives, because I recognize that at this time, I can’t be truly sure which is right, if any. I won’t comment further.
Meese,
> perfect correlations can be explained perfectly classically,
Yes. And, that explanation can be applied to EVERY AXIS, since the choice of axis can be made after the source emits the particles. Therefore, the values for all axes are determined by the source. That is all Bell needs to derive his inequality. There is no additonal assumption of “CD”.
> The alternatives I gave
Let’s look at them:
1. Many worlds: Travis Norsen says this theory has a for-all-practical-purposes solipsistic character. See “Quantum Solipsism and Non-Locality” by Travis Norsen.
2. Informational interpretations: Information about what? Travis Norsen disposes of this one, too.
3. Bohmian mechanics: Nothing wrong with this. It is the least nonlocal of all the theories that agree with experiment. The theory is just a pair of differential equations and initial conditions for the equations. All the usual textbook QM stuff including the measurement postulates are derived from the equations. The theory includes tables and chairs. Cats are never both alive and dead. Tables and chairs are composed of atoms that are composed of electrons, protons, and neutrons.
4. Retrocausality: The universe looks to see what experiments we do, then goes back in time to fix things so we think the world is nonlocal when it really just has time all messed up.
5. Superdeterminism: The universe knows in advance what experiment we will do and makes us think the world is nonlocal.
Andrew wrote:
> Boltzmann statistics (what we in statistics would usually just call
> “probability”)
That’s wrong right there at the beginning. These “statistics” (really probability models) describe the distribution of particles over energy states. In the double-slit experiment, there need only be one particle at a time going through the apparatus, and all the particles are prepared the same way. Distributions over energy states are not relevant to this experiment.
> I guess the question is whether Fermi-Dirac statistics and other quantum
> phenomena can be expressed using the usual rules of conditional
> probability.
I don’t know what that sentence means. You seem to have defined “conditional probability” as “Boltzmann statistics”, then noticed that the “Boltzmann statistics” model does not apply to all situations. I agree that if you use the wrong model, it will disagree with experiment. But, I don’t agree that you get to define “conditional probability” as “Boltzmann statistics”.
> Based on previous discussion threads, I think there’s approximately zero
> chance we’ll resolve this one here
A recurring theme of this blog is people who don’t correct their errors when they are pointed out to them. This is a good example of that.
David:
This is a difference of opinion. It’s not an “error” I can correct, but it is a difference of opinion that I can and should acknowledge.
Just by analogy, I get frustrated when the Nudge authors do not acknowledge criticism. I’d love for them to flat-out admit their errors, but I’d be ok if they would acknowledge that there are other perspectives out there.
> I guess the question is whether Fermi-Dirac statistics […] can be expressed using the usual rules of conditional probability.
The answer is yes – unless you want it to be no, of course, you can always decide that the appropriate conditioning “doesn’t count”!
https://chatgpt.com/share/6951910a-70d8-8001-a4e1-252fbdcc1e90
Andrew wrote:
> This is a difference of opinion. It’s not an “error” I can correct, but
> it is a difference of opinion that I can and should acknowledge.
I don’t know why you put “error” in quotes.
We (I include Daniel and Carlos) are using the standard meanings of words. You are changing the meaning of the phrase “conditional probability”, and you don’t seem to understand the physics (although it is hard to tell).
You aren’t really engaging in the discussion at all: You don’t respond to the content of what we write, and you bring in irrelevancies like Boltzmann statistics and Fermi-Dirac statistics.
If you don’t understand what we write, you should ask questions, not assume that we are wrong.
> Having taken some physics courses 40 years ago is not much of a sign of
> authority at all. I was letting everyone know where my perspective was
> coming from.
What do you mean by “perspective”? Are you saying that the only evidence you have for your claim is what you heard in a physics course 40 years ago?
David:
You write to me, “I don’t know why you put ‘error’ in quotes.” The answer is that I put “error” in quotes because I was quoting you! I’m glad to have cleared up that mystery.
You also ask, “What do you mean by ‘perspective’?” I just mean that ever since I took those courses in quantum mechanics, I understood that the laws of probability that we apply to rolls of the dice, etc., don’t apply at the quantum level. You say that Boltzmann statistics and Fermi-Dirac statistics are “irrelevancies,” but they’re not irrelevant to me!
In any case, again, thanks for the discussion. I guess I should change the title of this post from “Correct me! Point out all my mistakes” to “Inform me! Point out all my mistakes and places where we disagree.”
> You say that Boltzmann statistics and Fermi-Dirac statistics are “irrelevancies,” but they’re not irrelevant to me!
However, we still don’t know why do you think they are relevant – or what do you think they are to start with, Boltzmann statistics is not what we in statistics call “probability”.
As I mentioned in another comment one can derive Fermi-Dirac statistics instead of Boltzmann statistics using exactly the same probabilistic calculation taking into account what we in physics call the exclusion principle and the indistinguishability of particles.
Looking forward to disagree for years to come!
Carlos:
OK, how’s this? If you roll 2 dice, there are 36 possibilities. You’re twice as likely to get an 11 as a 12, because you can get an 11 from 6,5 or 5,6, whereas you can get a 12 from 6,6. But that’s because the dice are distinguishable. If they were undistinguishable, as with electrons in Fermi-Dirac statistics, 6,5 and 5,6 would be the same thing, and 11 would be as probable as 12. This is just a silly example; the point is that if you want to figure out the probability of a set of fermions being in different states, you need to use Fermi-Dirac statistics; you can’t just do the intuitive probability calculations corresponding to Boltzmann statistics.
But, yeah, I recognize there are differences of opinions here, which I think are less about the physics and more about the application of probability. My claim is that when we use probability models outside of quantum mechanics, we use Boltzmann statistics or what I call standard conditional probability without even thinking about it.
> My claim is that when we use probability models outside of quantum mechanics, we use Boltzmann statistics or what I call standard conditional probability without even thinking about it.
So either
A) your claim is that when you use probability models outside of QM you use a classical description of how large numbers of distinguishable, non-interacting particles distribute among energy levels in thermal equilibrium, typically at high temperatures and low densities where quantum effects are negligible – and that seems a silly mistake that you wouldn’t make
or B) you have decided for some unknown reason to ignore the definition of Boltzmann statistics and make the term synonymous with “probability” – which seems an even worse mistakes as it makes any communication extremely hard.
While it may be just a difference of opinion I expect that you will continue to misuse the term and we will continue to point it out.
Carlos:
The Maxwell-Boltzmann distribution of classical thermodynamics is derived from probability theory.
> the point is that if you want to figure out the probability of a set of fermions being in different states, you need to use Fermi-Dirac statistics; you can’t just do the intuitive probability calculations corresponding to Boltzmann statistics.
The Fermi-Dirac distribution of quantum thermodynamics is also derived from probability theory. All of statistical mechanics is derived from probability theory!
Is it surprising that applying probability theory to solve problems which are related but different we may obtain different answers?
Sampling with replacement is also different from sampling without replacement. Even if I were to label the former Platonic statistics and the latter Socratic statistics we would be using “probability” just the same.
Perhaps a review/summary is in order.
Andrew repeatedly claimed that “The usual rules of conditional probability fail in the quantum realm”.
The prior on this claim is pretty small since mathematicians prove their theorems; they don’t verify them by experiment.
The evidence Andrew provided for his claim was an analysis of the double-slit experiment. In Andrew’s analysis, he assumed that the probability of detecting the particle in a particular region only depended on the state of the slit the particle went through, not the other slit. Andrew then noted that using this assumption and the rules of conditional probability, you get a result that differs from experiment.
The natural conclusion is that Andrew’s assumption that the state of the other slit is irrelevant is incorrect. Instead, Andrew concluded that the rules of conditional probability are incorrect.
It is hardly surprising that if you make a false assumption, your analysis will disagree with reality. In fact, we can get the same disagreement by replacing the electrons with macroscopic particles and using wind or water. More generally, in any situation, if you are interested in the probability of X, and you assume this probability only depends on A when in fact it also depends on B, then you will get a result that disagrees with reality. It is by using the usual rules of conditional probability that you determine that your assumption is wrong.
Andrew’s response to this was to say that by “probability” he meant Boltzmann statistics.
It is generally an error to make up your own meaning for standard terminology. In Through the Looking-Glass, Humpty Dumpty said, “When I use a word, it means just what I choose it to mean—neither more nor less”. Humpty Dumpty can get away with it.
But, using Andrew’s definition of “probability”, i.e., Boltzmann statistics, doesn’t save his claim. The evidence for his claim is an analysis of the double-slit experiment. Boltzmann statistics does not appear in his analysis. Neither does Fermi-Dirac statistics. So, with this definition of “probability”, there is no relevant evidence at all.
Finally, Andrew claims that all these errors are actually differences of opinion due to his perspective formed forty years ago when he took physics courses in college and learned then that the “laws of probability … don’t apply at the quantum level”.
The mistakes Andrew made in his analysis of the double-slit experiment are well-known to experts in the field. The mistake he made by thinking that Boltzmann statistics or Fermi-Dirac statistics is relevant to the double-slit experiment is perhaps a new mistake.
> Boltzmann statistics (what we in statistics would usually just call “probability”)
Like other commenters, I don’t understand what that may mean.
Boltzmann statistics describes the thermal equilibrium distribution of distinguishable particles over energy levels/states. The probability that a given state of energy E is occupied is inversely proportional to exp(E/(kT)).
Is there anyone else in statistics who would just call that “probability”? Maybe you have a source, I’m curious about where did you get that idea.
It’s true that it’s about a “probability” in a specific context but the same can be said for Fermi-Dirac statistics which describes the thermal equilibrium distribution of indistinguishable fermions (particles obeying the Pauli exclusion principle) over single-particle energy levels/states: The probability that a given state of energy E is occupied is 1/(exp((E-μ)/(kT))+1). [The chemical potential cannot be factored out here, on the other hand we get actual probabilities that don’t require further normalization.]
It’s also true that “probability” is used to derive the Boltzmann distribution – but it’s just as true for the Fermi-Dirac distribution. We just need to include in the calculation Pauli’s exclusion principle when counting micro states (each single-particle state can be occupied by one particle at most). Then we maximize entropy under the energy and number of particles constraints just like for the Boltzmann distribution.
I think there was a response to the Holes paper here https://arxiv.org/abs/2007.08160
I don’t know if Andrew commented on this response already somewhere in the blog.
The paper “The rule of conditional probability is valid in quantum theory” by P.G.L. Porta Mana is good in its discussion of the probability calculations. He gives a very simple non-quantum example. He clearly explains Andrew’s errors.
I’m less enthused about his discussion as to whether photons are particles.
I have to keep beating a dead horse here on Reinhart & Rogoff, because you’re right about such horses not really being dead but instead lingering on as zombies. Will I have to keep beating it in the years to come? That’s up to you.
I think it was a mistake to ask for mistakes on December 24.
Messe/anon:
To be clear, Bell’s inequality is violated by Malus’ law. The reason is that polarizers rotate the light rather than block/pass it like assumed by Bell when deriving his inequality.
I’d also note that the photons are affecting the polarizer itself, so we should not assume complete independence between sequential events. Ie, there should technically be some kind of relaxation time (after which the polarizer is “rerandomized”)
involved in the derivation. But this is a minor issue relative to the Malus’ law problem (that the (non-strawman) classical model from 1810 predicts the same thing as QM).
Its not my idea, its just what Bell described. They are exactly the same. This paper introduces a new model that contains a self-contradiction (obey Malus’ law and only 50% “pass”). This additional 50% assumption is contrary to the original Bell model, Malus’ law, and nature. Just apply Malus’ law and you get the exact same prediction as Bell described for QM.
It really is quite simple, the observations in the Bell tests follow Malus’ law.
I know I’m late to the game here, but I’ll add my 2 cents’ worth. I think your criticisms of the late Seth Roberts were unfair and verged on being mean-spirited, particularly because Seth is no longer around to defend himself. I learned a lot from Seth, and his Shangri-La diet helped me to lose weight. His love of self-experimentation inspired me to pursue numerous n=1 studies on myself.
Alex:
Is there a specific thing in one of my posts on Seth that you disagree with?
For one thing, I think you’re too dismissive of the effectiveness of the Shangri-La diet that he developed. I also think you overstate how much Seth deceived himself, though self-deception is something we’re all susceptible to. But mostly, I object to the tone of your posts. You said he was a friend of yours, but your criticisms seem surprisingly harsh. On the other hand, you knew him much better than I did, so perhaps you have your reasons.