Newcomb’s paradox is considered to be a big deal, but it’s actually straightforward from a statistical perspective. The paradox goes as follows: you are shown two boxes, A and B. Box A contains either $1 million or $0, and Box B contains $1000. You are given the following options: (1) take the money (if any) that’s in Box A, or (2) take all the money (if any) that’s in Box A, plus the $1000 in Box B. Nothing can happen to the boxes between the time that you make the decision and when you open them and take the money, so it’s pretty clear that the right choice is to take both boxes. (Well, assuming that an extra $1000 will always make you happier…)
The hitch is that, ahead of time, somebody decided whether to put $1 million or $0 into Box A, and that Somebody did so in a crafty way, putting in $1 million if he or she thought you would pick Box A only, and $0 if he or she thought you would pick Box A and B. Let’s suppose that this Somebody is an accurate forecaster of which option you would choose. In that case, it’s easy to calculate that the expected gain of people who pick only Box A is greater than the expected gain of people who would pick both A and B. (For example, if Somebody gets it right 70% of the time, for either category of person, then the expected monetary value for the “believers” who pick only box A is 0.7*($1,000,000) + 0.3*0 = $700,000, and the expected monetary value for the “greedy people” who pick both A and B is 0.7*$1000 + 0.3*$1,001,000 = $301,000.) So the A-pickers do better, on average, than the A-and-B-pickers.
The paradox, as has been stated, is that from the perspective of the particular decision, it’s better to pick A and B, but from the perspective of expected monetary value, it appears better to pick just A.
Resolution of the paradox
It’s better to pick A and B. The people who pick A do better than the people who pick A and B, but that doesn’t mean it’s better for you to pick A. This can be explained in a number of statistical frameworks:
– Ecological correlation: the above expected monetary value calculation compares the population of A-pickers with the population of A-and-B-pickers. It does not compare what would happen to an individual. Here’s an analogy: one year, I looked at the correlation between students’ midterm exam scores and the number of pages in their exam solutions. There was a negative correlation: the students who wrote the exams in 2 pages did the best, the students who needed 3 pages did a little worse, and so forth. But for any given student, writing more pages could only help. Writing fewer pages would give them an attribute of the good students, but it wouldn’t actually help their grades.
– Random variables: label X as the variable for whether the Somebody would predict you are an A-picker, and label Y as the decision you actually take. In the population, there is a positive correlation between X and Y. But X occurs before Y. Changing Y won’t change X, any more than painting dots on your face will give you chicken pox. Yes, it would be great to be identified as an A-picker, but picking A won’t change your status on this.
One more thing
Some people have claimed to “resolve” Newcomb’s paradox by saying that this accurate-forecasting Somebody can’t exist; the Somebody is identified with God, time travel, reverse causation, or whatever. But from a statistical point of view, it shouldn’t be hard at all to come up with an accurate forecast. Just do a little survey, ask people some background questions (age, sex, education, occupation, etc.), then ask them if they’d pick A or A-and-B in this setting. Even a small survey should allow you to fit a regression model that would predict the choice pretty well. Of course, you don’t really know what people would do when presented with the actual million dollars, but I think you’d be able to forecast to an accuracy of quite a bit better than 50%, just based on some readily-available predictors.