The tabloids strike again

Under the heading, “Unlearning implicit social biases during sleep,” Nick Brown writes:

What do you make of this?

At first sight I’m unimpressed; it looks like just another glamour journal fluff piece. For example, it seems to me that Figure 1F commits the error described here; and the authors seem to ignore the large increase (regression to the mean) in the second column (of 4) between Figures 1D and 1E. But maybe I’m being too instantly skeptical, in what I suppose may come to be known as “LaCour month”.

I replied: Wow—the tabloids strike again! What made you look at this article in the first place?

And Nick responded:

It was the #3 item in the “World News” section of the BBC app a couple of days ago. Not the Science section, or even the Health section under which they filed it, but apparently the third most important piece of news in the world. “FFS”, as the kids say (or maybe that’s the UK only, and “WTF” is the international English version).

All the tabloid-y discussion was about ethics, brainwashing, brave new world, etc. To me it looks like yet another study which is just “obviously wrong” (insufficient power, etc), even before I read it.

Nick then blogged it, under the heading, “Dream on: Playing pinball in your sleep does not make you a better person.”

But, hey, it was a net win for the journal Science: the BBC listed their article as the third-most-important piece of news in the world. And, unlike with LaCour and Green, the data were real. What more could you ask for??

P.S. I’m thinking that a better title for this post would be “Unlearning common sense during research.”

27 thoughts on “The tabloids strike again

  1. I find this frustrating for several reasons. The authors (Hu et al) state “The IAT allows for an assessment of the strength of implicit associations between social groups and attributes” but the Editor concludes “Changes in people’s stereotypical attitudes toward race and gender were maintained”. So, we go from “associations” to “attitudes” and, by the time it hits the BBC it’s “racism”. I agree with the authors – the IAT measures “implicit associations”. I am skeptical of the idea that the IAT measures racist attitudes because (for 1 reason) the predictive validity of the test is weak/modest

    • PT:

      An stereotypical association is the foundation of an implicit attitude containing content consistent with such an association. It is possible to engage in critical thinking that allows one to both recognize and make decisions that are not consistent with that attitude. However, this requires both effort and well developed critical thinking skills. Left to the typical automatic processing that the vast majority of people rely on for everyday reasoning, the implicit association will likely rule the day.

      Also, while the sample and methods may not allow for such a strong conclusion, the notion that sleep deprived brains might be more likely to engage in a lazier forms of reasoning is reasonable hypothesis.

      • PT might say:

        We can measure implicit associations, but what is the point of positing the existence of some “implicit attitude containing content consistent with such an association”? If we can’t measure that attitude directly, there’s no reason to assume it exists.

        Also I’m of the opinion that the word “attitude” should be banned from psychology. It means completely different things to social psychologists, cognitive psychologists, and laypeople. I doubt you could even get a random group of 5 social psychologists to agree on a definition.

        Replace “attitude” with “association” in your comment about controlled/automatic decision-making and it means exactly the same thing.

        • If the implicit association exists, then the attitude also exists. The only question is whether one is able to think critically and challenge the logic of the implicit cognition.

          Implicit attitudes cannot be assessed directly when there is motivation to conceal such an attitude both from others as well as from oneself.

          • Replace attitude with beliefs about something that directly impacts affective responses to that something. In this case they would be beliefs that can be outside of conscious awareness and immediate access.

            • I guess what trips me up about this (maybe because I don’t know much about social psychology) is that I can’t see how to operationalize some of these concepts. A belief doesn’t have to be something that a person explicitly endorses (or even be consciously accessible at all), and we can’t measure it through behavior because then people can “believe” two contradictory statements (Kahan’s example of the rural farmer who doesn’t believe in climate change but prepares for and worries about its effects). Affective responses too don’t have to be accessible; maybe you could use something like amygdala activity during some task but it responds to many things.

              A cognitive account of IAT goes:
              Activating some representation also activates other representations, which we infer from accuracy and reaction time in the IAT. The strength of the relationship predicts (or doesn’t predict, depending on who you ask) behaviors, like various forms of discrimination. It does (or doesn’t) display change/learning through experience, and we can (can’t) successfully use controlled processing to reduce or eliminate it. It seems more useful to focus on reaching consensus on these questions rather than considering the logical implications in terms of concepts that can’t be observed (or statistically modeled/estimated).

              • You bring up some important points. One is the distinction between research attempting to understand broad aspects of a psychological construct and the other drawing individual level inferences. The amount of data required to make broad and reasonably crude statements about the former is not nearly enough to make specific statements about the latter. For this reason, there is still a lot of research to be done in the area.

                I have both done research in the area of implicit biases (though it was into the reasoning itself) as well as taken IAT assessments. While it has been quite a while since I took one of the IAT assessments, I came away with the impression that there needed to be additional research conducted into potential confounds embedded in the features of the stimuli that could undermine inferences drawn from the research. Teasing these out requires a lot of data at the individual level and thus makes it more difficult to do as participants will often bail out if it is taking longer than they want to spend or attentional issues become another confound with which to deal.

                It has been posited that there are both implicit and explicit processing brain systems, which certainly makes sense and that the implicit operates at the level of something like breathing or perception while the explicit involves the frontal lobe and executive functioning and that the interaction of the two makes for some interesting research. If you do a search on implicit processing and brain system you will find the research literature for this area.

    • I was having a quick look at the original Greenwald, McGhee, & Jordan (1998) and I’d worry about reliability of the scales—there probably is data but my cursory search did not find any.

      N-sizes were in the 32 in each of two experiments. Experiment 1 had 32 participants with “Data for 8 additional subjects were not included in the analysis because of their relatively high error rates, which were associated with responding more rapidly than appropriate for the task” Eh? I know nothing about the conventions of response time measurement but dropping 1/5th of the original sample seems ???

      No one seems to have been dropped from the second experiment. Still n= 32 is not exactly large for psychological test development.

      Some of the assumptions in the original study seem dubious to me but I’d have to do a lot more reading than I am willing to do at the moment to really criticize.

      However I did see reference to a paper measuring West German and East German attitudes using the Implicit Association Test. Presumably this is a slightly adapted version?

      Another reports an adaptation of the Implicit Association Test to assess anxiety (Egloff & Schmukle, 2002). The structure of the test is that same, items are completely changed

      There may be some theoretical justification for doing this but at the moment it looks like the Implicit Association Test is not a test of anything, just a way to throw a bunch of vaguely related words together and churn out something with, perhaps, a bit of face validity.

      I am possibly being too critical but currently I am not impressed with the instrument and its many mutations.

  2. “It was the #3 item in the “World News” section of the BBC app a couple of days ago. Not the Science section, or even the Health section under which they filed it, but apparently the third most important piece of news in the world.”

    Only because no one had announced a Mars mission that day.

    • Um, their result is confined to white non-Hispanics in the US. Even though the baby boom was not confined to white non-Hispanics in the US. And they compiled data on mortality (all cause, and broken down by cause). They’re not looking at (and misinterpreting) changes in the age distribution, which would of course reflect people aging into and out of any given age cohort as well as dying out of it.

      • Jeremy:

        I have not looked at the data or the research article in question, but I think Anon’s point was that, as the average age of people in the 45-54 group increases (for example, maybe the average age in that group used to be 48 and now it is 51?), you’d expect a higher death rate. A quick look at Wikipedia shows Pr(death) going up from .0038 to .0048 over this three-year period, which is a pretty huge change. So if there has been a big change in average age within the cohort, it’s important to adjust for it in the analysis.

        • That would only explain the difference between the WNH group and the other groups if the other groups did not have a similar baby boom (meaning the peak ages were different).

          • Single-year age by race appears unavailable for the 1990s. Here’s what I get for 2000-2010, no baby boom effect apparent for the Hispanic population:

            Intercensal Estimates of the Resident Population by Single Year of Age, Sex, Race, and Hispanic Origin for the United States: April 1, 2000 to July 1, 2010

            I used these columns:
            NHWA_MALE Not Hispanic, White alone male population
            NHWA_FEMALE Not Hispanic, White alone female population
            H_MALE Hispanic male population
            H_FEMALE Hispanic female population

            • Great data! It’s really hard to tell because you’d want to disaggregate native born from foreign born (or maybe look at Puerto Rico). (I love how you can see that peak birth year in the 2002 data).

              • I didn’t understand this comment earlier, but now I think I do.

                Below you mention “The peak birth year of 1948”. First, the wikipedia graph shows births/population rather than number of births, which is misleading to us here. Second, either way the peak was 1947 so that cohort must have been at least 55 in 2002, outside the age range we are looking at. The peak due to that birth cohort is visible up to 2001. Check the sources:

                This agrees with the data from and CDC wonder (which probably gets it from the Census Bureau though). It is possible they are all using the same bad data, but I have not seen a contradictory source.

                Also, the paper in question does not mention distinguishing native from foreign born, etc with regards to Hispanics. Right or wrong, the same lack of baby boom effect is found in the data provided by CDC wonder, which was the data source for that paper.

      • You are correct, they do not consider age distribution at all. If they did, they wouldn’t have published this paper. Which would you think shows higher mortality rate? A population consisting of people who are mostly of age:
        A) 45
        B) 54

        Unless you think mortality rates do not increase from 45-54 years old, a large group of people entering then gradually leaving the age group will first artificially lower the mortality rate and then raise it (ceteris peribus). I have not looked at the distribution by race, but it is very apparent just from the total population. My argument is based purely on the age distribution, not mortality which I have not looked at.

        The CDC wonder database they used apparently doesn’t give single-age mortality. I would like to see them perform the same analysis using the less aggregated data:

        • Ok, now I understand your point.

          It would still surprise me if that explains the entire result. Would the average age in the 45-54 group (or more precisely, changes in the average age within the 45-54 group over time) be very different for non-Hispanic US whites than for any other group considered? I’d be surprised if so, but I don’t know, I’m not a demographer. I agree it would be best to check.

        • It’s not like they don’t explicitly state in the paper that they are not using age adjusted results for the 5 year data.

          Let’s say that the baby boom was 1946-1964 since that is what Wikipedia says (and it’s all a bit arbitrary), that means that means that their data starting in 1999 would have captured the the relatively small group born in 46 for 1 year and would have caught the tail end for 5 years. So the whole entire study is made up of baby boomers but it does not capture all of the 45-54 ages of all baby boomers in those periods. The peak birth year of 1948 would only have been in the data for a couple of years and so on that basis as time was going on the age mix in the risk pool was getting younger not older as a general rule though of course it all bounces around a bit year to year.

          According to data the mortality rate for people aged 45-54 has been flat or slightly increasing for about 12 years while in the rest of the developed world it has been declining. This has, according to the data, been largely do to increases in suicide and “poisoning” (which they tell us basically means drug or alcohol abuse) among the non college educated population (5% missing data on education on death certificates). That is the observed data, that suicide and poisoning deaths have increased in that age group. If the excess deaths were due to cancer or diabetes or heart disease or Alzheimer’s all of which are associated with age, then it might make sense to think that they should be doing age adjusted mortality because the differences might be due to age distribution within the 5 year groups. Age and suicide are certainly related.

          Now I do think that there are some other issues. For example, the current 45-54 cohort of lower education people may have already lost of a lot of people earlier due to HIV, which would have been big impact for this group, and it could be that the early removed of substantial numbers people from the risk pool could be changing the risk of suicide and addiction. The decline in homicides on the other hand, could also be leaving some people as suicide risks (and we saw this week how homicide and suicide can be hard to distinguish without serious police investigation and the same is true for accidental death). Also as their data show, the risks of suicide and poisoning are much higher in the south and west and so some kind of contextual effect could be in play where opportunity (e.g. availability of guns), culture, etc are really the explanation. On the other hand, Oxycontin.

          So overall while it is fine to speculate about what the real story is behind the observed data, starting with the assumption that the researchers don’t know what they are doing and don’t pay attention to issues that come up in a first semester demography course (age adjustment) is not really something serious. Strangely enough I did take my first semester demography course many years ago with David Weir who is one of the reviewers listed. I didn’t look at the current bios of the other reviewers but I can tell you for sure he is not an idiot and issues as obvious as that of the impact of age adjustment is a standard thing to any serious demographer.

          • It turns out CDC wonder does have single-year ages data for 1999-2013, you just have to choose detailed rather than compressed mortality. Here’s what I get:

            For ages 45-48 mortality rate increases then decreases back to 1999 levels. Ages 49-50 increase until 2003 then plateau. Age 51 gradually increases and is highest for 2013. Age 52 increases until 2004-2005 then plateaus. Ages 53-54 decrease until 2002 then increase again about halfway to 1999 levels.

            I don’t know what to make of it, but clearly the Hispanic mortality rates are decreasing and the non-Hispanic white is not. However, only 51 year old mortality really shows a steady trend in the 21st century, and at that point we’re cherry picking.

            Also, we can see the baby boom effect on age distribution even better when years 2011-2013 are included. For example, in 1999 11% were age 45 and 8% were 54. In 2013 8.75% were 45 and 10.5% were 54 years old. From the earlier plot we saw mortality rate is ~270/100K people at age 45 and 550/100K at age 54. So I remain confused as to why they do not discuss accounting for the baby boomers. As you say, this is obvious.

            Non-Hispanic White: Race=White, Hispanic Origin=”Not Hispanic or Latino”
            Hispanic: Race=NA, Hispanic Origin=”Hispanic or Latino”

          • Looking at this more ( I noticed something interesting. The range in mortality rate for the same age over the years is ~20-50 per 100k. This is approximately the difference in mortality due to being one year older, when measured the same year. Looking at the Non-Hispanic White curves from age 45-52 it also looks like a high mortality cohort moving through. I think we need monthly birth and mortality rates, yearly is not high enough resolution because mortality rates increase by 2-4 deaths/100k every month in this age range.

            Also, an interesting paper (unfortunately, they do not share the data) on variations in US birthrate seasonality 1931-2008:

  3. Thanks for covering this, I sincerely believe that this piece is the all-time low for the Science magazine (not counting the editorials and other non-research stuff, of course). If anyone thinks otherwise, prove it.

Leave a Reply

Your email address will not be published.