Skip to content
 

Olivia Goldhill and Jesse Singal report on the Implicit Association Test

A psychology researcher whom I don’t know writes:

In case you aren’t already aware of it, here is a rather lengthy article pointing out challenges to the Implicit Association Test.

What I found disturbing was this paragraph:

Greenwald explicitly discouraged me from writing this article. ‘Debates about scientific interpretation belong in scientific journals, not popular press,’ he wrote. Banaji, Greenwald, and Nosek all declined to talk on the phone about their work, but answered most of my questions by email.

This attitude seems similar to the one you have pointed out in the past, wherein certain professors (and sometimes editors) seem utterly unwilling to even countenance challenges or be open to debate about their work. This attitude strikes me as very unscientific. Oftentimes “outsiders” can recognize deficiencies where “insiders” cannot, because they come at things with a very different point of view.

My reply: I thought the linked news article, by Olivia Goldhill, was excellent.

I’ve been skeptical of the implicit association test for a long time; see for example this from 2008, long before I’d heard about any replication crisis in psychology or elsewhere.

And I agree that it’s disturbing when people say, “Debates about scientific interpretation belong in scientific journals, not popular press.” Scientists have no problem their work being uncritically discussed in the popular press, and they have no problem with the popular press speculating on the real-world implications of their important work. So why is the press suddenly shut out when the explorations turn critical? Especially given that Goldhill’s “popular press” article is much more thoughtful and sophisticated than most journal articles I’ve seen on these topics.

My correspondent replied:

Agreed! Imagine if this attitude had prevailed in the 1920s and Walter Lippmann had not been able to criticize the interpretation and use of IQ tests.

And then this, which really says it all:

In the off chance you mention this in your blog, please don’t mention who sent it to you—I don’t want to accidentally embroil myself in any controversy.

P.S. More here in this hard-hitting piece by Jesse Singal, including these bits:

The problem, as I [Singal] showed in a lengthy rundown of the many, many problems with the test published this past January, is that there’s very little evidence to support that claim that the IAT meaningfully predicts anything. In fact, the test is riddled with statistical problems . . .

One striking thing about the process of reporting that article was the extent to which Banaji tried to smear her critics . . . She also accused the test’s critics of having a “pathological focus” on black-white race relations and the black-white IAT for reasons that “will need to be dealt with by them in the presence of their psychotherapists or church leaders.”

This is the definition of a derailing tactic — shift the focus from critiques of the IAT itself, some of which in this case appeared in a flagship social-psych journal, to the ostensible moral and psychological failings of the critiquers.

Yes, I hate that tactic.

Singal continues:

The idea that journalists shouldn’t write about scientific controversies would have been highly questionable even before the replication crisis exploded onto the scene, but it’s hard to fathom why anyone would take this argument seriously in 2017. . . . Greenwald, of course, doesn’t appear to have any problems with positive coverage of the IAT.

And he concludes:

Society desperately needs more open scrutiny of scientific claims, not less, whether in scientific journals, the media, or anywhere else. Especially when it comes to claims that seem to change every two years.

I agree.

68 Comments

  1. Anonymous says:

    Quote from above: “Society desperately needs more open scrutiny of scientific claims, not less, whether in scientific journals, the media, or anywhere else. Especially when it comes to claims that seem to change every two years.”

    I agree as well.

    I would like to point out a possible principle that, i feel, gets overlooked a lot in (social) science and might be related to the quote above and perhaps even the topic: “Primum non nocere” which translates as “first, to do no harm” (https://en.wikipedia.org/wiki/Primum_non_nocere).

    I sometimes get the idea that (social) scientists see the world as one giant experiment, and the people in it as participants. I have often thought that (social) scientists, to me at least, seem to perhaps not even realize, or entertain the thought, that 1) their research may have negative consequences, and 2) that science, and scientific results, are not the only thing to take into consideration when it comes to policy, advise, etc.

    Having good intentions, or thinking you have good intentions, is not enough. Scientific principles, values, and responsibilites should come first. And i think they should come with the wisdom of how, why, and what to do with the science that results from these principles, values, and responsibilites as well.

    I feel, and fear, this has been, and is currently, all not the case. And in the case this makes any sense, i feel, and fear, that this is all not even realized and/or acknowledged.

    • yyw says:

      This applies to anyone trying to effect social changes. Communism that led to much suffering was brought by people with noble intention and willingness to sacrifice themselves for what they perceived to be greater good.

      • Anonymous says:

        “This applies to anyone trying to effect social changes”

        I have thought a lot about this comment.

        If i am not mistaken, science is (at least primarily) NOT about effecting social changes. It’s about trying to explain, predict, understand, etc. things.

        Perhaps science, and scientific results, can be used to effect social change in the form of being a source of information or point of view, but i reason it’s very important for scientists to separate these issues.

        I also think this might be a possible confusion, and conflation, that may explain a lot of (social) science’s problematic issues.

    • Terry says:

      1) … research may have negative consequences, and 2) that science, and scientific results, are not the only thing to take into consideration when it comes to policy, advise, etc.

      Having good intentions, or thinking you have good intentions, is not enough. Scientific principles, values, and responsibilites should come first. And i think they should come with the wisdom of how, why, and what to do with the science that results from these principles, values, and responsibilites as well.

      I’m a bit confused.

      You say that scientific principles, values, and responsibilities should come first. This sounds like you think truth should be our guiding principle and we shouldn’t distort or shape our results for ulterior reasons.

      But you also talk about the wisdom of how, why, and what to do with the science that results from these principles. This sounds like you are saying truth is not our only guiding principle and we should wisely mold our research in light of the effects of our results.

      Could you be a bit clearer? Could you give an example of how wisdom would shape our scientific principles?

      Or are you saying that we should react wisely to the truth we find from rigorous scientific principles? If so, could you give an example?

      Genuinely confused.

      • Anonymous says:

        Quote from above: “Could you be a bit clearer?”

        I am going to try.

        1) You wrote: “You say that scientific principles, values, and responsibilities should come first. This sounds like you think truth should be our guiding principle and we shouldn’t distort or shape our results for ulterior reasons.”

        I (think i) am saying that scientific principles, values, and responsibilities are more important, and should come first, compared to having good intentions, or thinking you have good intentions (I wrote that stuff after i wrote about intentions). To use your words, i reason “we shouldn’t distort or shape our results for ulterior reasons”.

        2) You wrote: “But you also talk about the wisdom of how, why, and what to do with the science that results from these principles. This sounds like you are saying truth is not our only guiding principle and we should wisely mold our research in light of the effects of our results. “

        To use your words, i am NOT saying we should “wisely mold our research in light of the effects of our results”.

        Your use of the word “truth” makes it hard for me to say anything about the fist part of both sentences you wrote that i quoted. I reason that what is considered the “truth” may differ from time to time, and/or that it’s perhaps very hard to know what the “truth” is.

        1) I reason that the primary goal of social science is to try and understand behavior/phenomena/etc.
        2) I reason that logical reasoning, experimentation, and other scientific things, can possibly help with that goal.
        3) I reason that during the process of trying to understand things, a very critical and careful approach is welcomed and needed (and in line with scientific values, principles, and responsibilities)
        4) I reason that the wisdom of how, why, and what to do with the science that results from these principles is NOT an antithesis (if i understood this word correctly) of science.
        5) I reason that the wisdom of how, why, and what to do with the science that results from these principles might be an un-acknowledged, and perhaps even un-realized aspect of science.

  2. Jonathan (another one) says:

    Put in its most charitable light, public *acceptance* of some article is a sign you understood the conclusion, no matter how ignorant you are of of the details, while public *criticism* often superficially focuses on the conclusion without any consideration of the details of the methodology. Thus, one is tempted to say “Thanks” in the first case and “I don’t have the time or the patience to teach you everything I know” in the second case. But the solution to ignorant public criticism is not to shut off all public criticism. Even the dumbest critique can usually be answered fairly simply and without condescension, and the answer may help you explain the work to others. There is still a risk that you’ll be ensnared in a series of follow up critiques that waste everyone’s time, but you are always free to say: “Clearly, I’m not going to convince you. So I’m going to stop trying.”

    But really, critiques can’t be judged from the source in any case. They should be judged by the content.

  3. Anonymous says:

    Quote from above: “One striking thing about the process of reporting that article was the extent to which Banaji tried to smear her critics . . . She also accused the test’s critics of having a “pathological focus” on black-white race relations and the black-white IAT for reasons that “will need to be dealt with by them in the presence of their psychotherapists or church leaders.”

    This is the definition of a derailing tactic — shift the focus from critiques of the IAT itself, some of which in this case appeared in a flagship social-psych journal, to the ostensible moral and psychological failings of the critiquers.”

    Some of the people in this IAT story, and perhaps (social) science in general, always remind me of some lyrics of the song “Jesus he knows me” by Genesis:

    “Won’t find me practicing what I’m preaching
    Won’t find me making no sacrifice
    But I can get you a pocketful of miracles
    If you promise to be good, try to be nice
    God will take good care of you
    Just do as I say, don’t do as I do”

    https://www.youtube.com/watch?v=35K6vQRt67g

    • Andrew says:

      Jake:

      Your link points to controversy about Singal’s writing about transgender kids; the above is about his writing on the implicit association test. I don’t see the connection.

      • AllanC says:

        I think Jake is trying to say something like…Singal was definitely wrong (in a big way) on something pretty basic in the past, so why should we pay attention to what he is saying now.

        Clearly misguided as Jonathan said above and yyw quotes below.

    • yyw says:

      Quoting Jonathan above, “But really, critiques can’t be judged from the source in any case. They should be judged by the content.”

    • Jesse Singal says:

      Could you highlight a factual claim from that Jezebel article about my work that you’d like me to respond to? I’ll respond right here if you do. The article gets a *very* basic thing wrong — Walker says she believes I am against informed consent, when my article rather explicitly presents informed consent as an important advance in trans healthcare (at the end of a lengthy section in which I lay out the many obstacles to care trans people have faced over the decades). I do not think Walker read the article before writing her piece; that’s not the sort of thing she would have missed if she had.

      Anyway, I’m asking you to point to something *specific* I got wrong — not someone else’s complaint, not another link — because my frustration with a lot of online conversations about this subject is that they tend to go like this:

      A: Wow, I can’t believe you wrote this! It’s so bad! Holy crap how was this published.
      B: Can you explain what you disagree with me about and I’ll respond?
      A: LOL right no way.

      If there’s an actual thing I got so wrong in my coverage of gender dysphoria that my work in *other* areas shouldn’t be trusted either, that strikes me as important enough for me to respond to. So you definitely have my attention and I’d be happy to respond to your specific critique. I don’t really know what to do with the question of “What My Fucking Deal” is, though — that’s not a critique.

      Thanks,
      Jesse

      • Corey says:

        ’m asking you to point to something *specific* I got wrong

        This is a deflection. The criticism of your reporting on transgender issues isn’t that you’re inaccurate — it’s that you’re selective and the choices you make promote a narrative damaging to trans individuals. The people who comment here are concerned with statistical best practice, so we know well what selection bias does to information — it vitiates it.

        • A. Tasso says:

          @Corey, this is not a deflection and you know it. Singal is inviting you to make a specific critique. If you think he’s selectively omitting something important, then raise it. If you think he reported on something and made a mistake, then raise it. Otherwise, the “WTF is wrong with you” critique doesn’t hold water.

          • Corey says:

            If people tell me I’m doing bad thing X and my response is, “Show me one instance of me doing bad thing Y! Oh you can’t? Checkmate!” that is indeed a deflection.

            As to raising specific issues, well, I’m reluctant to further derail the comments section, so my last action on this particular subject will be to link Julia Serano’s account of her interactions with Jesse Singal.

            • J. J. Ramsey says:

              “If people tell me I’m doing bad thing X and my response is, ‘Show me one instance of me doing bad thing Y! Oh you can’t? Checkmate!’ that is indeed a deflection.”

              On the other hand, if someone tells you that you are doing bad thing X, and your response is “Yeah, a lot of people have repeatedly tried to argue that, and they’ve repeatedly botched their arguments. Show me a specific instance of me doing bad thing X instead of pointing to one of those botch jobs,” that’s not a deflection. I’d say that’s also far closer to what his comment was saying, especially, since he started his comment by claiming that the article criticizing him got a basic thing wrong.

              I remember reading the post by Serano to which you linked, and what stood out was her attempt to argue that when Singal wrote, “as an aside, you should read her Daily Beast article about navigating the dating scene as a trans woman in San Francisco,” he was trying to insinuate that she had loose morals. Even if one were to grant that he should have picked a different article of hers to cite, the aside is so blatantly ineffective as a shaming tool — especially given the likely audience of liberals who would read it — that it’s hard to believe that a professional writer like Singal would have even tried to use the aside for such a purpose. That’s a bad enough botch that it’s hard to take her other criticisms at face value.

              This also isn’t the only time she’s made dodgy claims about Singal. In a part of her Twitter feed trying to criticize his article, “When Children Say They’re Trans,” she writes: “THERE IS NO DISCUSSION OF HOW AFFIRMING ≠ TRANSITION!!!” (https://twitter.com/JuliaSerano/status/1008830500083920897). Yet the article itself an expert named Aron Janssen saying, “Many people misinterpret affirming care as proceeding to social and medical transition in all cases without delay, but the reality is much more complex.”

              The pattern that I repeatedly see from Singal’s critics is that they make exaggerated or distorted claims that don’t follow from what he’s actually said. It’s a pattern that I’d expect to see if in certain circles it had become conventional wisdom that he was somehow bad, but the “wisdom” was poorly supported. I’ve seen a similar pattern when, for example, those on the far right talk or write about Hillary Clinton or Barack Obama. It’s taken for granted that they are bad people, but when they try to justify it, they end up getting things grossly wrong (e.g. uranium deal, death panels, Pizzagate).

              • Re: The pattern that I repeatedly see from Singal’s critics is that they make exaggerated or distorted claims that don’t follow from what he’s actually said. It’s a pattern that I’d expect to see if in certain circles it had become conventional wisdom that he was somehow bad, but the “wisdom” was poorly supported.

                I hate to say or admit this. But this is a pattern that nearly always of us has been guilty of. If not in a public arena; then in a private space. I’ve known public intellectuals who talk a great game in terms of being open to and kind to others. But subsets can reflect some petty and bullying behaviors behind the scenes.

                At the expense of repeating myself [heaven forbid], I refer to such habits as emotional blackmail. It has seeped into many cultures in what seems like a big way. It is a consequence of so many factors. Very eminent thinkers can engage it too. It is constituted from addictions to certain patterns of behavior. I even saw it in my family. I was determined to unlearn it as much as possible. I saw also a graciousness in my family too. And began to discern what triggers each.

                I think that we need more gracious temperaments in so many contexts.

              • Jesse Singal says:

                Thanks for pointing that out, J.J. I’ll refrain from commenting more in this thread but I had no idea Serano did that and it’s ridiculous. Not only does the *article* of course make that point about affirmation being more complex than instant transition, but it’s right there in the section she’s referring to(!): https://twitter.com/jessesingal/status/1126885819611009025

                This has happened over and over and over and over and over. There have been corrections in ThinkProgress, the NY Times (a letter to the editor they printed), and Slate, with a major one on the way in The Advocate. I’m not sure what other evidence I can point to to convince people that there has been an attempt to spread misinformation about this article. All I can do is ask people to read it and respond to what I actually wrote, not to what commenters who either lied or who didn’t read the article all the way through (Serano is doing one or the other here) have said.

            • JK says:

              The treatment Corey and others are giving Singal is Kafkaesque. They refuse to specify anything that he has actually done wrong yet still, judging by the amount of vitriol, seem to consider him one of the great villains of modern times. It’s also comical how Corey does exactly what Singal says his critics always do: he makes highly damaging accusations, then, when challenged, refuses to provide any evidence for the accusations and just posts some link. My hunch is that Corey has never read a word of what Singal has written, but somehow has learned that Singal is “a bad person” and thus any libel against him is A-okay. As another commenter suggested, this campaign against Singal is at the same intellectual level as “Obammy is a gay Kenyan Muslim communist.”

              • J. J. Ramsey says:

                “My hunch is that Corey has never read a word of what Singal has written, but somehow has learned that Singal is “a bad person” and thus any libel against him is A-okay.”

                Hold it. Stop right there. It’s one thing to indicate that someone is *mistaken* based on flaws in the sources that they cite. It is a whole other matter to suggest *without evidence* that someone is “A-okay” with *libel*, which is an intentional spreading of falsehood. Don’t do the latter.

        • Jesse Singal says:

          Sorry to have forgotten about this thread, but your response tells me everything I need to know.

    • Anonymous says:

      You are citing to a Jezebel article as an authoritative refutation of Singal? (Here is an example of the Jezebel author’s public debating style … from the author’s own article: https://i.kinja-img.com/gawker-media/image/upload/bg5xklsacmd7covvf5un.png

      Really?

  4. Dave says:

    I have 3 points:

    1) As far as cognitive psychology is concerned the IAT is simply an amalgamation of the Simon task and the semantic priming task and therefore, the effects are **not at all surprising** unless, you want to go well beyond the data and say IAT reflect something the Simon/Semantic priming tasks never claimed to measure namely, implicit (race/sex/health…) attitudes. See https://replicationindex.com/2019/02/15/iat21/ for a more detailed review of validity of the IAT. The initial acceptance of the IAT as a valid measure of attitudes with little scrutiny led to a mini-industry of research which leads to the next point…

    2) I wonder whether the authors of IAT considered it valid simply because it received many citations and did so quickly. I say this from reading this (generally very good article): Greenwald, A. G. (2012). There is nothing so theoretical as a good method. Perspectives on Psychological Science, 7, 99–108 in conjunction with this: Mitchell, G., & Tetlock, P. E. (2017). Popularity as a poor proxy for utility: The case of implicit prejudice. In S. O. Lilienfeld & I. D. Waldman (Eds.), Psychological science under scrutiny: Recent challenges and proposed solutions (pp. 164-195). : Wiley-Blackwell.

    3) I am not sure the IAT would be with us if the original author had not decided to submit to a more cognitive-psychology orientated journal rather than JPSP (where, incidentally, he had served as an editor between 1977-1979).

    • Anonymous says:

      “The initial acceptance of the IAT as a valid measure of attitudes with little scrutiny led to a mini-industry of research (…)”

      Perhaps that’s 1) a possible result of the poor state of (social) science, and/or 2) exactly the (implicit) goal.

  5. I’ve never quite understood the IAT. Nor could I understand Philip Tetlock’s critique of it. I must not have paid sufficient attention. One reason is that over decades we have been discussing prejudice.

    • Terry says:

      Could you elaborate? Sometimes the IAT strikes me as odd too. Other times it seems obvious.

      • I am not knowledgeable enough about psychometric tests to assess their merits and demerits. My outsider question would be just how predictive and useful such tests are more generally. To assess the IAT, I will have to take the test. Obviously, that will be my own opinion.

        From several of the critiques of the IAT, I would explore the topic of prejudice from several other angles. It came to me after reading this thread. So that is something I might consider exploring I’m glad this topic came up.

        What bothers me more is that much testing is in the service of market-driven goals and objectives. They tend to be fads for several years or a decade or more. Then some go on to developing some other product. I was thinking back, for example, to the use of the Myers-Briggs. I couldn’t believe that the test had as much gravitas as it did.

        In my observation of American academia, there has always, at least from the end of World War II, subsets that have sought to weed out prejudice/racism. Gunnar & Alva Myrdal and Gordon Allport spearheaded efforts to alleviate it. Sissela Bok is Gunnar & Alva Myrdal’s daughter. Sissela’s husband is Derek Bok, former President of Harvard Univ. During Derek Bok’s tenure, much was done to address it. Harvard itself housed several academics who, for example, led interfaith dialogue all over the world. Roger Fisher conducted negotiations in South Asia and the Middle East. So there has been ongoing momentum to address prejudice. It’s that quality of leadership that has made a difference. They included me too. I was enrolled in Harvard’s Program on Negotiation. So had an introduction to the longer term goals and objectives, one which was to address prejudices among minorities here in the US.

        It would have been great if there had been preregistration of the IAT. That way we could see what went into conceiving it. From a cursory look at the questions, gleaned from the article above, I have seen these types of questions even before the advent of IAT. In mid-eighties and early nineties. Maybe in a compendium of a conference held in Toronto back then. I have to reread it probably.

        Whether there is quality information gain from the administration of such tests has been in dispute for so long, raising even more fundamental questions about the theory. I’m not a great fan of administrating them. My impression of academics is that a good % are cloistered in their own specialties. And certainly, don’t get out in the streets all that much. I see that here in DC.

        I know I’m probably not on point. But I will re-read some of the critiques. And try to be clearer.

          • Thanks for pursuing the topic. It has given me a few new angles to explore. I have been ensconced in international relations academics [colleagues of my Dad] since age 8 or so. I didn’t meet Gordon Allport. But did Erik Erikson, childhood development professor, who perked my interest in prejudice. It was a foundational topic in the effort to forge interfaith dialogue. Unfortunately, much of interfaith dialogue wrestled with theological themes. I thought that the psychological literature on religion couldn’t do justice to the subject of prejudice. Then again I had a better handle on the geopolitical factors that influenced the expertise on prejudice b/c of Niebuhr and Rawls blessing. I got to go to many forums in my teens. And bring my cassette of funky music to listen to. LOL Seriously I took a lot in despite my seeming disinterest and rebellious profile.

  6. Steve says:

    I’ve taken many of these IATs and I can attest that it’s fairly easy to get whatever result you want to get one by simply manipulating how fast or slow you answer the questions. The test pretty transparently uses your response speed when it determines how biased you are: if you’re quicker to associate ‘good’ concepts with one race, or gender, or whatever, and slower to associate those concepts with the other, then if you want the test to say you’re not biased, simply slow down your responses to every stimulus across the board. One of the fundamental rules to any social science experiment like this is that the subject shouldn’t know what exactly the researcher is trying to study, because that can influence the responses; and yet, in the IATs, the subjects *always* know what is being studied (it’s right there in the name). For the life of me, I can’t see how anyone has ever taken this test seriously.

    • Anonymous says:

      Quote from above: “I’ve taken many of these IATs and I can attest that it’s fairly easy to get whatever result you want to get one by simply manipulating how fast or slow you answer the questions. “

      Also possibly see this paper by Fiedler & Bluemke (2010) “Faking the IAT: Aided and unaided response control on the Implicit Association Task” http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.372.878&rep=rep1&type=pdf

      The abstract reads as follows:

      “One pragmatic goal of implicit tools like the Implicit Association Tests (IAT) is to rule out self-presentation and controlled responding. Three experiments examined whether the IAT meets this goal, using Turkish and German groups along with positive and negative traits. Experiment 1 was an Internet study. After completing a naïve IAT pretest, participants were instructed to fake on a posttest in three graded conditions that differed in the explicitness of faking instructions. Experiment 2 replicated and extended the approach in the laboratory, including no-pretest condition. Results demonstrate that participants who intended to fake were successful, provided the experience of a pretest. Experiment 3 ruled out an alternative account of faking in terms of pretest experience. Faking was mostly due to slow-down oncompatible trials, but a notable speed-up on incompatible trials also occurred. Faking remained inconspicuous, especially with non-blatant instructions; experts failed to identify faked data sets.”

  7. zbicyclist says:

    I am astounded at the number of studies in these meta-analyses:

    Validity of the IAT: “it came as a major blow when four separate (pdf) meta–analyses (pdf), undertaken between 2009 and 2015—each examining between 46 and 167 individual studies—all showed the IAT to be a weak predictor of behavior. Two of the meta-analyses focus on the race IAT while two examine the IAT’s links with behavior more broadly, but all four show weak predictive abilities.”

    Ability of IAT workshops to reduce implicit bias: “A 2017 meta-analysis that looked at 494 previous studies (currently under peer review and not yet published in a journal) from several researchers, including Nosek, found that reducing implicit bias did not affect behavior. “Our findings suggest that changes in measured implicit bias are possible, but those changes do not necessarily translate into changes in explicit bias or behavior,” wrote the psychologists.”

    I’m particularly surprised since the net conclusion is that there’s not much here — which means the number of no-effect studies that didn’t get written up would likely be much larger still.

    Reminds me of the title of a post a couple of days ago, about beating dead horses.

    • Martha (Smith) says:

      Methinks there may be a lot of dead horses lying around getting (or awaiting) beatings.

    • Terry says:

      Can you tell how many of these studies were done by low-level academics who had to churn out studies to qualify as “researchers”?

      The IAT could be a “turn-the-crank” machine for turning out “research”.

    • Anonymous says:

      “I’m particularly surprised since the net conclusion is that there’s not much here — which means the number of no-effect studies that didn’t get written up would likely be much larger still.”

      I think so too. I think the amount of unpublished (non-significant?) studies involving IAT stuff might be overwhelmingly large.

      I think this might fit with what i think is one of the most weird things i concluded from thinking about the state of psychological science, and following the discussions in the past years. It seems to me that certain topics, and/or associated researchers, at one point in time “gain some traction” (or what’s the appropriate word) and then everyone “tries their hand at” the same thing or something similar. A possible result of this is also that i think the “original” researchers achieve some sort of “special status” in which they can much more easily publish, get grants, receive awards, hire PhD students who will work on the specific topic again, etc. It’s a whole self-sustaining and expanding process, like a snowball effect.

      IAT stuff and all the associated other “implicit” tests (e.g. “implicit self-esteem”), all the priming stuff, ego-depletion, etc. may all be examples of this possible process.

      (As a side note: i think that recently proposed “improvements” in the form of large-scale “collaborative” efforts will exactly replicate this detrimental process in a whole new manner. I think that a focus on large amounts of data on just a few topics will “gain traction” and will lead to many researchers “trying their hand at” it via data-dredging, selective reporting, etc. I think the net result is the same: lots of wasted resources, lots of folks working on just a few topics, there will still be no actual (optimal) theory-building and – testing, no comparing theories, etc.)

      As Meehl wrote back in 1978 (!!):

      “I consider it unnecessary to persuade you that most so-called “theories” in the soft areas of psychology (clinical, counseling, social,personality, community, and school psychology) are scientifically unimpressive and technologically worthless.
      (…)
      Perhaps the easiest way to convince yourself is by scanning the literature of soft psychology over the last 30 years and noticing what happens to theories. Most of them suffer the fate that General MacArthur ascribed to old generals — They never die, they just slowly fade away.”

      • Martha (Smith) says:

        ““gain some traction” (or what’s the appropriate word)”
        I’d say, “become popular” or “become the current fad”.

        “Most of them suffer the fate that General MacArthur ascribed to old generals — They never die, they just slowly fade away.””
        So maybe we should coin the phrase, “old general theories”.

        • Anonymous says:

          “So maybe we should coin the phrase, “old general theories”.”

          Ooh, interesting comment! Maybe you should stick even closer to the source with it, so the term even more clearly refers to Meehl’s example and paper. You could call a psychological “theory” that never really dies, but just slowly fades away, a “General MacArthur theory”.

          Perhaps a nice addition to professor Gelman’s list of idioms/coined phrases or terms (or what’s the name for the list). The term could be listed there, with a short explanation as is usually the case concerning that list if i am not mistaken.

          The explanation could even include mentioning Meehls’s 1978 paper titled “Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology”, which could possibly help make people more aware of it!

  8. Peter Dorman says:

    I know essentially nothing about psychometrics, so I have nothing to add to the technical discussion around the IAT. One thing that strikes me as obvious, however, is that its implicit model of discrimination fits well into the dominant view of such things that was incubated in academia in the last decades of the twentieth century and then released into the wild: the assumed subjective basis of social outcomes.

    Granted, the old leftism/reformism of earlier years, especially strands influenced by Marxism, were unbalanced in the direction of structure over agency. “Objective forces” supposedly determined discriminatory outcomes, and individual consciousness was just another byproduct. Well, no. It was healthy for a reaction to materialize that gave due consideration to the role of cultural biases, personality dynamics, etc.

    But then the needle just kept going. More and more speculative arguments were adduced for the vast power of subtle cultural tropes, like nuances of language use, and the ubiquity of biased consciousness. It became normal to read about ruling paradigms or the dominant culture “believing” or “assuming” as if discursive patterns were purposive, determining agents, all acting and not acted upon.

    So suppose someone comes along and says, “I have a way to *measure* the presence of the dominant paradigm in places so deeply buried that it is invisible to the naked eye—the missing link between what we already know about the ubiquity of racism, sexism, etc. and the empirical realm of measurement and potential remediation.” Of course there was a market, a vast one. And the methodological shortcomings didn’t even have to be denied or suppressed; they could be out in the open in the literature and none of the diversity trainers would even care, since they had just what they thought they needed.

    If it sounds like I’m a bit emotionally wrapped up in this myself, I am. This presumptive certainty that discrimination is all about our hidden thoughts, our deep, ugly secret, and we must make ourselves therapeutically available to those who can help us “fix” them, was catastrophic at the institution I work at. (That adjective is not an exaggeration.) But, just as one of tragedies of a flawed prosecution is that it lets the real perpetrator go free, a false diagnosis of our serious social ills gets in the way of doing what we really should be doing. My college fell apart over accusations of subtle or even invisible racism, while entirely feasible steps to redress bias and inequality—structural, about better procedures and a different allocation of resources—went untaken, and remain so.

    End of soapbox.

    • Terry says:

      This presumptive certainty that discrimination is all about our hidden thoughts, our deep, ugly secret, and we must make ourselves therapeutically available to those who can help us “fix” them, was catastrophic at the institution I work at. (That adjective is not an exaggeration.)

      You can’t just drop this here and not elaborate. What was the institution? Has the story of the catastrophe been told in detail somewhere?

      • Peter Dorman says:

        I thought I might have mentioned it in an earlier comment; I’m not at all shy about this. It’s the Evergreen State College. And unfortunately, there isn’t any remotely accurate or objective account out there, but the catastrophe part (massive enrollment losses, budget cuts) is well known.

        • Terry says:

          Evergreen State! Of course!

          From a great distance (from where I stood), it looked like a toxic mixture of low-intelligence, narcissism, and immaturity egged on by low-level apparatchiks in the Grievance Industry.

          I welcome any insights you have. Even links to your former comments would be appreciated.

          • Peter Dorman says:

            Two quick links:

            To a blog post on the “independent” review of college administration: http://econospeak.blogspot.com/2018/04/evergreen-looks-in-mirror-and-says-its.html

            To a more recent blog post on the it-makes-me-feel-bad argument for curtailing public speech: https://www.nakedcapitalism.com/2019/04/free-speech-safety-and-the-triumph-of-neoliberalism.html

            The latter includes a pair of possibly useful exchanges about the political dynamic at Evergreen in the comment thread.

            • Terry says:

              Thanks.

              Can you point to anything specific where the publicly-received story is particularly inaccurate? Or, to put it another way, can you tell us anything that went on behind the scenes that is not well-known and helps explain what went on?

              • Peter Dorman says:

                I had a ringside seat in the run up to the Evergreen meltdown, although not the only one to be sure. Afterward I conducted a number of semi-formal interviews to acquire more information. My knowledge is still quite incomplete but more than enough to dispel *all* of the popular publicly accessible narratives. My goal at the time was to write up an account/analysis and get it placed in an appropriate outlet. I spent a few weeks honing a very long essay that folded historical reconstruction together with the sort of conceptual critiques you’ve seen here and in the Naked Capitalism post, and which I’ve also parceled out in many other blog posts over the past year or two. In retrospect that was probably a mistake on my part. The upshot is that I submitted it to perhaps a dozen outlets that one would think of as being the longer-form, more intellectually ambitious segment of left journalism and didn’t get the slightest traction with anyone — not even any rejections, just silence. I can only guess what that means.

                The journalistic problem is that, once a story has established itself, it takes a lot explanation to disestablish it. There is no single missing fact that would explode either of the two popular narratives — a campus overrun by left wing zealots or a campus struggling honestly to face up to the ubiquity of racism and then attacked by the alt-right. They are both wrong from the ground up, so it’s hard for me to give you what you’re asking for in a blog comment.

                If I had to point to one behind-the-scenes aspect to the matter, it would probably be that the 2017 events were the culmination of a long-developing power struggle between factions of the faculty and staff over a series of different and only partially overlapping issues. There was definitely coordination between some of these people and some of the students at the center of the protests, but I don’t know how deep and extensive it was. I should emphasize that over the entire process, and not just at the final explosion, faculty associated with the political left were bystanders. This was not in any way a left-right thing. After the meltdown some left faculty chose to publicly identify with the demonstrators for various reasons (mostly hostility to the Fox media onslaught and ingrained sympathy for student demonstrations as such), but they had no prior influence over events.

                But, really, this leaves out so much. I just can’t give you the response you want.

              • Terry says:

                Is your essay available anywhere?

                So are you saying that the SJW attacks were a club with which to bludgeon some people for petty career reasons?

                If so, it sounds like what is going on now at the Southern Poverty Law Center (where the prize is a half billion dollars in cash).

              • Terry says:

                I found your (Peter Dorman’s) essay online. If it makes you (Peter) feel any better, I was aware of your telling of the story before, and my general understanding of the episode incorporates your comments, so your input has penetrated the narrative, at least for more informed viewers. (Non-informed viewers will always be with us and we can’t hope to reach all of them.)

            • RE: Free Speech on Campus

              If there is a campus speaker who may be deemed controversial, the university has the wherewithal to arrange a forum that can afford opportunities to those who object to the speaker. Simply scheduling one speaker has drawbacks when there might be a ruckus brewing.

              I wonder whether there has been a comprehensive study of which controversial and non-controversial speakers at universities; in particular fields. That would be very interesting and establish one base rate analysis.

    • Anonymous says:

      Quote from above: “So suppose someone comes along and says, “I have a way to *measure* the presence of the dominant paradigm in places so deeply buried that it is invisible to the naked eye—the missing link between what we already know about the ubiquity of racism, sexism, etc. and the empirical realm of measurement and potential remediation.” Of course there was a market, a vast one. And the methodological shortcomings didn’t even have to be denied or suppressed; they could be out in the open in the literature and none of the diversity trainers would even care, since they had just what they thought they needed.”

      I wonder if these IAT people ever think, or have thought, about:

      1) how much money, and other resources, have been spent on this stuff that could perhaps otherwise have been spent on research that might be more valid, useful, eplanatory, etc., and

      2) if this research, and all the “diversity” training and all that stuff, might actually make (and/or have made) things worse concerning discrimination, etc.

      • I have been around a lot of children in urban and suburban playgrounds. Interestingly, I’ve never seen them discriminate on the basis of skin pigment nor on nationality.

        Moreover, As I pointed out in an earlier post, I think there have been influential academic circles & business leaders that have boosted the public profiles of minorities. Generally speaking, some market imperative has been on the horizon. I base this on my attending some business conferences on the 90s. In fact, this hypothesis dovetails with Samuel Huntington’s quite intriguing analysis of western international liberal elites sensibilities whom Huntington contrasts with the sensibilities of conservatives nationalists. The liberal elites tend to include elites of other races and ethnicities as they are increasingly being financed by transnational corporate endeavors. Again market forces seem to play a role.

        In addition, we neglect the fact that geopolitical factors [wars, conflicts, & resource acquisition] are pivotal in how minorities are treated. For example, when war aims are being conducted, specific national security prerogatives kick in, which obviously bear on how some minority groups are viewed in such times. Explains in good part the conduct of the last 17 years, exacerbated by the tragedy of 9-11.

        • Anonymous says:

          Quotes from above:

          “In fact, this hypothesis dovetails with Samuel Huntington’s quite intriguing analysis of western international liberal elites sensibilities (…)”

          &

          “Again market forces seem to play a role.”

          I never heard of Huntington so i looked him up. From the wikipedia page on Huntington (at least i assume that’s the Huntington you are refering to):

          “Huntington is credited with inventing the phrase Davos Man, referring to global elites who “have little need for national loyalty, view national boundaries as obstacles that thankfully are vanishing, and see national governments as residues from the past whose only useful function is to facilitate the elite’s global operations”.

          Anyway, your quotes are interesting to think about in relation to all this IAT stuff, and how it may have actually emphasized a focus on race, and may have increased discrimination, etc.

          Or maybe it’s all a giant distraction, so the “global elites” can go about their business more effectively and easily whilst the general public are quibbling amongst themselves. Maybe the “New World Order” has been, and/or is, branching out to Psychology :P

        • Andrew says:

          Sameera, Anon:

          Huntington, huh? I remember back in a poli sci class in college reading an essay by Huntington, in a book from the 1970s published by the ultra-elite Trilateral Commission, complaining that we had too much mass political participation and that the solution was to restrict political participation in some way so that elites had more influence.

          It’s not enough for these dudes to be rich and to have more access than everyone else. It seems that they’re really only happy if they can be assured that everybody else has no real power at all.

          • Anonymous says:

            “It’s not enough for these dudes to be rich and to have more access than everyone else. It seems that they’re really only happy if they can be assured that everybody else has no real power at all.”

            Perhaps it’s all about how one frames things, and interprets them.

            Concerning psychology for instance, i have recently learned that one man’s idea of giving a lof of power to an increasingly smaller number of people might be another man’s idea of “crowdsourcing” or “collaboration”.

            Perhaps it’s all about perspective.

          • Andrew, Anon.

            Aha. Someone who has read Huntington. I had forgotten that Huntington coined ‘Davos Man’. Maybe he referred to it in his book Who Are We? It’s worth reading b/c I think Huntington’s insights have relevance to the history of psychometric testing. I believe Howard Gardner drew a link between the administration of IQ test and selection for service in the British colonies. However, I think there is something to the fact that Harvard, in particular, has consistently emphasized the need to address prejudice. Gordon Allport taught at Harvard. As did Jerome Bruner, Erik Erikson, and Derek Bok. It was an influential group that continued to place minorities in key public positions, from chit chat with academics.

            I’ll be back to comment on this topic. A tad busy.

        • Martha (Smith) says:

          Sameera said,
          “I have been around a lot of children in urban and suburban playgrounds. Interestingly, I’ve never seen them discriminate on the basis of skin pigment nor on nationality.”

          An example that sticks with me: Many years ago, when I volunteered for church nursery duty, two little kids, one black and one white (both having a tendency to get hot under the collar easily) got into an altercation. The white one said to the black one, “You, you …” He didn’t know any pejorative words; he just stopped for loss of words.

          • Thanks, Martha,

            I have yet to hear any child refer to skin color. Having said that I would venture that I have heard a fair amount of ethnocentric characterizations from adults regardless of whether they regard themselves as liberal, conservative, or libertarian. And that is a more interesting story because ethnocentrism is bred into some homes, from my own observation. There are all varieties of ethnocentrism too.

          • elin says:

            I’ve heard plenty of kids refer to skin color. There is nothing wrong or biased about mentioning skin color any more than mentioning hair color or eye color. Kids would be pretty unobservant if they didn’t notice that they look different. If you asked them to describe their friend and have friends who vary in skin color they will say things like Bobby has dark skin. How important they think it is or whether they treat others differently on the basis of it is quite a different question. There are good reasons that boxes of crayons don’t have a color called “flesh” any more.

  9. Over time, Project Implicit has qualified and revised some of the statements on its website, but one statement remains woefully unrevised (on the Ethical Considerations page): “Because the Implicit Association Test (IAT) sometimes reveals troubling aspects of human nature, it poses the possibility of causing discomfort. If you are considering using the IAT in your research, your research plan should take this possibility into account.” In other words, the reason people should use the test with caution is that it may reveal troubling things. This makes it all too easy to dismiss someone’s skepticism over the results, i.e., “Yes, you may find the results puzzling, but that is because they reveal things about ourselves that we would rather not see.” If skepticism can be written off as discomfort, then there’s no room for criticism at all.

    A little further down, on the same page, the text states that “we cannot be certain that any given IAT can diagnose an individual”; this very uncertainty should be applied to the notion that the IAT can reveal something about human nature.

    I discuss this briefly in my book (with reference to Singal’s and Goldhill’s articles).

    • Anonymous says:

      ” “Yes, you may find the results puzzling, but that is because they reveal things about ourselves that we would rather not see.” If skepticism can be written off as discomfort, then there’s no room for criticism at all. “

      Yes!

      I think i saw this possible faulty reasoning mentioned recently when i came across a blog post i can’t find at the moment. I think it used something with “Kafka” (refering to the author and/or book) in the name to refer to this style of argumentation/reasoning.

      • Anonymous says:

        Quote from above: ” I think it used something with “Kafka” (refering to the author and/or book) in the name to refer to this style of argumentation/reasoning.”

        I think it’s called “Kafkatrapping”. Wiktionary describes a “Kafkatrap” as “A sophistical rhetorical device in which any denial by the accused serves as evidence of guilt.”

        Also possibly see https://www.thedailybell.com/all-articles/editorials/wendy-mcelroy-beware-of-kafkatrapping/

        “The term “kafkatrapping” describes a logical fallacy that is popular within gender feminism, racial politics and other ideologies of victimhood. It occurs when you are accused of a thought crime such as sexism, racism or homophobia. You respond with an honest denial, which is then used as further confirmation of your guilt. You are now trapped in a circular and unfalsifiable argument; no one who is accused can be innocent because the structure of kafkatrapping precludes that possibility.

        (…)

        Kafkatrapping twists reason and truth into self-parodies that serve victimhood ideologues who wish to avoid the evidence and reasoned arguments upon which truth rests. The term appears to have originated in a 2010 article written by author and open source software advocate Eric S. Raymond. He opens by acknowledging the worth of equality before the law and of treating others with respect. But, he notes, “[g]ood causes sometimes have bad consequences.” One such consequence is that tactics used to raise consciousness can veer “into the creepy and pathological, borrowing the least sane features of religious evangelism.”

Leave a Reply