Skip to content
 

Documented forking paths in the Competitive Reaction Time Task

Screen Shot 2016-04-08 at 6.45.15 PM

Baruch Eitan writes:

This is some luscious garden of forking paths.

Indeed. Here’s what Malte Elson writes at the linked website:

The Competitive Reaction Time Task, sometimes also called the Taylor Aggression Paradigm (TAP), is one of the most commonly used tests to purportedly measure aggressive behavior in a laboratory environment. . . .

While the CRTT ostensibly measures how much unpleasant, or even harmful, noise a participant is willing to administer to a nonexistent confederate, that amount of noise can be extracted as a measure in myriad different ways using various combinations of volume and duration over one or more trials. There are currently 120 publications in which results are based on the CRTT, and they reported 147 different quantification strategies in total!

Elson continues:

This archive does not contain all variations of the CRTT, as some procedural differences are so substantial that their quantification strategies would be impossible to compare. . . . Given the number of different versions of the CRTT measure that can be extracted from its use in a study, it is very easy for a researcher to analyze several (or several dozen) versions of the CRTT outcome measures in a study, running hypothesis tests with one version of the measure after another until a version is found that produces the desired pattern of results. . . .

3 Comments

  1. jrkrideau says:

    That’s not even forking paths, it’s spaghetti junction!

    I cannot, immediately, see how you would establish a reliability measure let alone any kind of real validity measure though I see there are some attempts at construct validity. I am not sure they are all for the Competitive Reaction Time Task.

    I am reminded of the fMRI study on the dead salmon http://blogs.scientificamerican.com/scicurious-brain/ignobel-prize-in-neuroscience-the-dead-salmon-study/.

  2. “running hypothesis tests with one version of the measure after another until a version is found that produces the desired pattern of results. . . “

    Yikes! If only good science were so easy. Hopefully, this blog post might raise awareness that this type of analysis isn’t a selling point t of the CRTT.

    The fMRI link made me think of some of the articles of Arina K. Bones: http://www.projectimplicit.net/arina/

    Enjoy.

  3. Anon— says:

    I wish the font on that site were easier to read.

    On the other hand, maybe it made my reasoning about these issues better… http://statmodeling.stat.columbia.edu/2016/06/26/29449/

Leave a Reply