PPPPPPPPPPNAS!

Jochen Weber writes:

As I follow your blog (albeit loosely), I figured I’d point out an “early release” paper from PNAS I consider to be “garbage” (at least by title, and probably by content).

The short version is, the authors claim to have found the neural correlate of a person being “cognizant of” the outcome of an action (either doing something on purpose or by accident), but frame it as a correlate of “knowledge vs. recklessness” in the context of crime and assessing culpability in the legal domain.

Without having read the whole thing yet, it would seem important that once you tell someone that some negative consequence occurred (or could occur), the neural correlate would likely change.

This may not be so much bad based on the methods, but rather by its completely over-hyped framing (and the fact that it spent less than 3 months in review; received 11/23/2016, accepted 2/9/2017, which is about 10 weeks). Presumably the reviewers were so enthralled by the work, they couldn’t think of anything to improve…

And just reading the methods now, I would say they tested something different altogether; the amount of “knowledge” that was manipulated between groups had nothing to do with to what extent what they were doing was criminal, but rather to what extent they were likely to be caught… It is maddening to think that popular news media will pick this up as “finally the answer to judging culpability based on fMRI”–what utter nonsense…

Weber continues:

Finally, just a couple issues I would point out…

– showing people a bowl of poo in the scanner and asking them whether they would eat it may elicit a lot of brain activation; to claim that that is the neural basis of how the brain looks like when eating poo is a fallacy

– besides the point that this was a game (and neither risk nor knowledge had any real-world consequences), this doesn’t seem to be about crime so much as about representing fear of being caught—a criminal who is certain they won’t be caught may not show this pattern

– the first author didn’t design the research (see contributions section), and the senior author never first-authored an fMRI paper (so he doesn’t know the modality), a toxic combination it seems…

A bowl of poo, huh? I’m liking this experiment already! Seriously, I do find this paper a bit creepy as well as hyped.

14 thoughts on “PPPPPPPPPPNAS!

  1. The study will likely be overhyped and it’s certainly a stretch to generalize these results immediately to criminal courts, but on the last point, the last author (Read Montague) has been doing fMRI for a while, despite coming initially from biophysics/computational biology. He has a reputation as a bit of a showman (sure, he has a TED talk), but he doesn’t seem over the top in his quotes from the VA Tech press release (although later news articles might be more dramatic)

    http://research.vtc.vt.edu/news/2017/mar/22/researchers-find-brain-imagery-shows-difference-be/

    …the researchers cautioned that the assessment of the mental state of a defendant should not be reduced to the classification of brain data.

    “In principle, we are showing these brain states can be detected when the activity is taking place,” Montague said. “Given that, we can start asking questions like, which neural circuits are engaged by this? What does the distribution look like across 4,000 people instead of 40 people? Are there conditions of either development, states of mind, use of pharmacological substances, or incurred injuries that impinge on these networks in ways that would inform the punishment?”

    • Anon:

      There’s something about the published paper that bothers me in the way they move back and forth between legal terms and scientific measurement. But I could believe that there are some valuable ideas and data there, underneath the hype.

      • I agree with your conclusion, I just don’t think it’s the “toxic combination” of a clueless senior author and a late-coming first author. My guess is the senior author and friends came up with the idea, gave an incoming postdoc the reins, and the final writing style was a deliberate strategy to land the work in a high impact factor journal.

  2. I don’t know anything about fMCI and so only skimmed the paper but Weber’s comment “this doesn’t seem to be about crime so much as about representing fear of being caught” seems a valid point.

    Would the differing probabilities of having a footbridge collapse as you crossed it give the same results?

    I am left wondering if we are seeing the development of a new high-tech version of a polygraph machine. And, as far as I can see, a polygraph works if the victem (err subject, suspect, or whatever) believes in them. (And they are not a psychopath?)

  3. >>>showing people a bowl of poo in the scanner and asking them whether they would eat it may elicit a lot of brain activation; to claim that that is the neural basis of how the brain looks like when eating poo is a fallacy<<<

    ….how about measuring player's performance in artificial lab settings and using it claim that the phenomenon of hot hands exists in real game play (although no evidence can be found by anyone when analyzing actual play data)?

    • Rahul:

      1. It was Gilovich et al. who analyzed basketball shooting in a non-game situation and made strong claims about the non-existence of the hot hand. Unfortunately their analysis had two different biases; put this together and a correct analysis of their own data shows evidence in favor of the hot hand. So you’re kinda getting it backward about who’s making broad conclusions from non-game shooting data.

      2. As Miller and Sanjurjo have discussed, analysis of in-game shooting also shows evidence for the hot hand. So your statement, “no evidence can be found by anyone when analyzing actual play data,” is incorrect.

      3. The “artificial lab setting” of shooting baskets in a gym is not so different from free throws taken in real game play.

      • #1 Oh, I’m not at all saying Gilovich is right and others wrong. All I’m saying is that it’s far fetched to extrapolate especially when the effect seen in actual game data is small to non-existant.

        #2. Here’s what Miller wrote in a recent comment: “In any event, I agree that if you tie your hands and restrict yourself to game data, current studies either fail to find a signal (Gilovich, Vallone & Tversky (1985, Study 2) Huizinga & Weil (2009), Rao (2009)), or it’s appears small (Bocskocsky, Ezekowitz & Carolyn Stein (2014)).”

  4. I vaguely remember hearing an interview about this on the radio and the authors were careful to say this is not about directly assessing intent for the courtroom. It was about attempting to show that intent as a concept has a neural basis to assist with legislation. Some in the legal field were arguing that it’s not a real distinction, recklessness and intent.

    • Very good comment. Anyone interested should read the paper’s own disclaimers.

      I can’t really get behind the general tone of Weber here. What the paper claims to show, is that state-of-the-art fMRI can (sorta-maybe, given some crazy advantages) tell the difference between two mental states that every normal person has always considered obvious.

      So, hurrah for fMRI, maybe it’s not 100% noise, some of us had some doubts.

      So, hurrah, maybe there is a difference between driving a car like a maniac and trying to hit a passing car. Some people were having doubts on that one (they probably had someone relation to someone who was run over by a driver who claimed mania) and they can maybe have very-slightly less doubt.

      On the other hand, while not much, its much better than garbage. It’s state of the art fMRI! Which puts the lie to most fMRI hype.

  5. On page 2 they write ‘To assess the “significance” of the results, correcting for finite sample sizes (23), we ran a permutation test (Supporting Information).’. Why those quotation marks around “significance”? By the way, gotta love those 100 +/- 0.00%

  6. I’m pretty sure this is D.O.A. The box on the first page entitled “Significance” reveals the kind of lack of conceptual clarity that is common when scientists approach a topic that’s heavy on the theoretical/philosophical side. Some analysis:

    »Because criminal statutes demand it, juries often must assess criminal intent by determining which of two legally defined mental states a defendant was in when committing a crime. For instance, did the defendant know he was carrying drugs, or was he merely aware of a risk that he was? Legal scholars have debated whether that conceptual distinction, drawn by law, mapped meaningfully onto any psychological reality.«

    Fine so far. Does the legal distinction map onto a psychological distinction?

    »This study uses neuroimaging and machine-learning techniques to reveal different brain activities correlated with these two mental states.«

    Wait a moment. Wasn’t it just the question whether there are actual mental states corresponding to the legally defined states? Now we’re already sure they exist and can spend a few million bucks looking for their neural correlates in a brain scanner??

    »Moreover, the study provides a proof of principle that brain imaging can determine, with high accuracy, on which side of a legally defined boundary a person’s mental state lies.«

    Same as above, but also, what does “proof of principle” mean in such a context? What could have prevented this from being possible, *in principle*? And how can the results of a single study (with a group size of 20) ever be *proof* of something? IMO, these are just big words to justify a sexy study whose approach is so far removed from any reality as to be completely useless for any practical purposes.

    • »This study uses neuroimaging and machine-learning techniques to reveal different brain activities correlated with these two mental states.«

      Wait a moment. Wasn’t it just the question whether there are actual mental states corresponding to the legally defined states? Now we’re already sure they exist and can spend a few million bucks looking for their neural correlates in a brain scanner??

      GS: One question: Did you have to sell your alleged soul to the alleged devil to become nearly as smart as me? Unfortunately, what you said could be said about a very large proportion of neuro”science” and a very large proportion of topics. It is an example of what I used to always call “homunculism” but now sometimes call the “mereological fallacy” after Bennett & Hacker. Here is a good review of their first book:
      https://ndpr.nd.edu/news/23573-philosophical-foundations-of-neuroscience/

      I don’t agree with everything B&H say, but the sort of category error that characterizes much of neuro”science” has got to go before real progress can be made. The source of all this is, of course, mainstream psychology (i.e., cognitive psychology). Why should mainstream psychology be the source of the conceptual cesspool?

      http://www.behavior.org/resources/88.pdf

      I have to chuckle a little bit at the attacks on psychology that occur on this blog – fixing psychology will require far more than methodological reform. Most here, if given a hand in attempting to reform psychology, would perpetuate the conceptual nonsense that characterizes it. Neuroscience, as influenced by mainstream psychology, would continue the search for the ghosts of mentalistic philosophy – they just wouldn’t use p-values.

Leave a Reply

Your email address will not be published. Required fields are marked *