I wrote this post awhile ago but it just appeared . . .
I liked this line so much I’m posting it on its own:
We should be open-minded, but not selectively open-minded.
This is related to the research incumbency effect and all sorts of other things we’ve talked about over the years.
There’s a Bayesian argument, or an implicitly Bayesian argument for believing everything you read in the tabloids, and the argument goes as follows: It’s hard to get a paper published, papers in peer-reviewed journals typically really do go through the peer review process, so the smart money is to trust the experts.
This believe-what-you-read heuristic is Bayesian, but not fully Bayesian: it does not condition on new information. The argument against Brian Wansink’s work is not that it was published in the journal Environment and Behavior. The argument against it is that the work has lots of mistakes, and then you can do some partial pooling, looking at other papers by this same author that had lots of mistakes.
Asymmetric open-mindedness—being open to claims published in scientific journals and publicized on NPR, Ted, etc., while not at all being open to their opposites—is, arguably, a reasonable position to take. But this position is only reasonable before you look carefully at the work in question. Conditional on that careful look, the fact of publication provides much less information.
To put it another way, defenders of junk science, and even people who might think of themselves as agnostic on the issue, are making the fallacy of the one-sided bet.
Here’s an example.
Several years ago, the sociologist Satoshi Kanazawa claimed that beautiful parents were more likely to have girl babies. This claim was reproduced by the Freakonomics team. It turns out that underlying statistical analysis was flawed, and was was reported was essentially patterns in random numbers (the kangaroo problem).
So, fine. At this point you might say: Some people believe that beautiful parents are more likely to have girl babies, while other people are skeptical of that claim. As an outsider, you might take an intermediate position (beautiful parents might be more likely to have girl babies), and you could argue that Kanazawa’s work, while flawed, might still be valuable by introducing this hypothesis.
But that would be a mistake; you’d be making the fallacy of the one-sided bet. If you want to consider the hypothesis that beautiful parents are more likely to have girl babies, you should also consider the hypothesis that beautiful parents are more likely to have boy babies. If you don’t consider both possibilities, you’re biasing yourself—and you’re also giving an incentive for future Wansinks to influence policy through junk science.
P.S. I also liked this line that I gave in response to someone who defended Brian Wansink’s junk science on the grounds that “science has progressed”:
To use general scientific progress as a way of justifying scientific dead-end work . . . that’s kinda like saying that the Bills made a good choice to keep starting Nathan Peterman, because Patrick Mahomes has been doing so well.
A problem I see is that the defenders of junk science are putting themselves in the position where they’re defending Science as an entity.
We really need more Nathan Peterman analogies. I contend that he is the same object-class as Paxton Lynch and Brock Osweiler. In other words, people who got their jobs because they looked like a quarterback (or at least they matched the conventional thinking about what a quarterback should look like).
Brian Wansink may be an object of the same class. He didn’t do his job very well (that is research), but he certainly appeared like he could do his job well.
Dalton:
Is it true that Peterman looked like a quarterback? According to his Wikipedia page, his performance in the NFL combine was “mediocre.” But I don’t know enough to have a sense of why he kept getting chances to play in pro games.
He’s 6’2″ and 225 lbs with an $85 haircut. Falls pretty much on the greenline in the bottom panel of this (surprisingly good) USA Today graphic: https://usatftw.files.wordpress.com/2013/10/fbfksdb.png
Dalton:
Whoever made that graph should’ve put the labels next to the points rather than at the bottom of each graph in no particular order. It took me a moment to figure out that the label for centers is next to the quarterbacks and the label for quarterbacks is next to the centers.
Wow, I’m really off those charts! :~)
His combine stats seemed slightly better and he was drafted higher than that of one Tom Brady. I think it is a bit difficult to judge who is going to make it in the NFL:)
Jd:
Yes, I guess the right question to ask is not, Why did Peterman get to play in the NFL, but rather, Why did he keep getting more chances after he’d performed so poorly.
I would call this selective open mindedness a variant of the Gell-Mann Amnesia effect. It is extremely common in scholarly discourse where the scholar rejects a theory because of inadequacy of method while proposing an alternative based on data obtained by the same method. Or even bases the critique of that method itself on positive outcomes of the method in question. For example, critiquing a ‘culture of measurement’ by relying on measures of alternative data using the same method.