Response to a question about a reference in one of our papers

Tushar Sunkum writes:

I like this particular study that you did [with Jeff Fagan and Alex Kiss] on racial profiling.

However, I believe that you misrepresented one of the sources on the paper. You state, “For example, two surveys with nationwide probability samples, completed in 1999 and in 2002, showed that African-Americans were far more likely than others to report being stopped on the highways by police (Langan, Greenfeld, Smith, Durose, and Levin 2001; Durose, Schmitt, and Langan 2005).” Yet, in the paper by Durose, Schmitt, and Langan 2005, it shows that the difference in number of African Americans stopped by police relative to whites was statistically insignificant. Can you explain the reason for this discrepancy?

I took a quick look at the paper and the references and responded that it seems that it would’ve been more accurate for us to have said “more likely” or “a bit more likely” rather than “far more likely.” But the real expert on this is Jeff Fagan, who replied:

I [Jeff] have looked again at the 2005 publication. Table 7 reports that Blacks are stopped more often (9.1%) compared to Whites (8.7%) as a share of drivers in the sample/population. The authors don’t report a significance level for the comparison, they use the term “likelihood.” If this is a statistical claim, it would be helpful to know that test they used and if there were controls for such important parameters as time of day, location (urban/rural), and other demographic markers. They claim that the dataset is a representative sample of the US population and its drivers, so it’s not clear what they mean by “significance.” What they report seems to be unadjusted percentages, which is important if the driving patterns vary by race of driver. For example, it would be helpful to break down Table 7 by race and size of jurisdiction (Appendix Table, p. 29), where the populations are less likely to be random and more likely to be clustered by race. This matters because there is different availability of drivers to the police for stops. We also don’t know the base rates of the reasons for the stops, so selective enforcement may underlie the observed patterns. For example, table 8 shows that the reasons for stops vary by race: Blacks are more likely to be stopped for pretextual reasons (e.g., alleged vehicle defect, such as broken taillight; record check) and less for more concrete reasons such as speeding. Pretextual reasons of this sort are quite common in the US literature on highway stops and other traffic stops, and have been valorized in constitutional caselaw (U..S. v Whren).

There are other disparities by race that are notable and (most likely) significant:

– Fewer blacks report being stopped by police for a legitimate reason (73%) than Whites (86%) or Hispanics (82%)

– Relative to their share of drivers, Whites were ticketed more often, suggesting that the reasons for stops of Blacks and Hispanics were pretextual and not substantive. The same differential was reported for arrests of drivers pursuant to stops.

– Black drivers (78%) were less likely than Whites (90%) to report that police acted “properly during the stop.”

– Black (10.2%) and Hispanic (11.4%) motorists stopped by police were more likely than whites (3.5%) to be physically searched or to have their vehicle searched. The search disparity was true for searches of drivers, for physical searches, and for searches of the vehicle.

– Searches without consent were far more likely for Black (58.6% of drivers compared to Whites (39.1%) or Hispanic drivers (46.2%) (Table 11).

– Blacks (3.5%) and Hispanics (2.5%) were far more likely than Whites (1.1%) to be subject to force or threats of force (Table 16). Black and Hispanic drivers were also more likely to report that police used “excessive force” than Whites (Table 20).

In sum, then, perhaps we might have qualified our initial statement based on the absence of details in the 2005 report about the specifics of their analysis and claim. But the totality of the circumstances in these stops shows a pattern of disparate treatment of Black and Hispanic drivers that provides more context to explain the meaning of an otherwise modest difference in the simple rates of police contact.

It’s good to see people reading our papers in enough detail to check the references!

16 thoughts on “Response to a question about a reference in one of our papers

  1. Jim:

    It is what it is. If the article were still in the revision stage rather than having already been published, I would’ve changed “far more likely” to “a bit more likely.”

    I wanted to share the above story because it illustrates the challenges that we have in writing an article, where an entire literature of published work can be compressed into a single one-sentence summary.

    • A comparison between the 2015 PPCS survey data with this 2002 data reveals at least one dramatic change in the overall public perception of the police, with respect to street stops.
      The percentage of Blacks who thought the reasoning behind their being stopped was legitimate went from 73% in 2002 to 50% in 2015. The percentage among Hispanics plummeted from 82% to 44%, while even among Whites the fraction declined from 86% to 68%. Could there be a society-wide erosion in overall public trust levels for the police, with respect to honest street stops, one that may still be evolving and is reflected in the current protest movement that is seemingly stretching across these racial groups?

      • A measure like “The percentage of [group] who thought the reasoning behind their being stopped was legitimate” is interesting. The only problem is no one knows what it means.

        The number of possible causes is huge, and could easily range from (a) cops are more aggressive and need to be reigned in; to (b) NPR keeps running pieces about racial profiling in the police, true or not.

    • Oh, I’m fine with the explanation for the error or whatever it should be called. I’m just having a hard time with the explanation for how this data demonstrates discrimination. And it’s not that I doubt discrimination occurs. But sorting out what is and what isn’t from a scientific perspective is yowza very, very challenging. It’s extremely difficult to find robust comparisons of red apples and yellow apples, where a) the data are unequivocally objective; and b) everything but the color of the apple is constant.

  2. > Whites were ticketed more often, suggesting that the reasons for stops of Blacks and Hispanics were pretextual and not substantive

    I don’t say that’s not correct but I wonder if, had it been the other way and Blacks and Hispanics were ticketed more often, the statistic would also have made it to the list of notable disaparities suggesting discrimination against them.

    • Yeah, I didn’t quite understand the reasoning here. My first, doubtless very bad, model of this situation would be like this: we assume that actual substantive traffic violations occur at the same rate for whites and minorities. Presumably, police WILL stop people when they see a substantive traffic violation. If they wish to stop a certain group at a higher rate, they make up pretextual reasons for additional traffic stops. If this was the case, it would seem to me to suggest the opposite conclusion as the one mentioned here. I’m wondering if I’m misunderstanding what’s meant by pretextual or what.

  3. Another way of looking at this, is that the person who regularly mocks and highlights debatable statistical claims by other professions has been hoist with his own petard!

    • Megen:

      I don’t consider anybody “hoist with his own petard” if they make mistakes. We all make mistakes. What’s important is for us to be open and to learn from our mistakes. Science is not a game; it’s a way we try to learn about reality. And anyone should be free to openly criticize published work by anybody.

        • Rahul:

          The problem is with the framing of this as some sort of morality play, with the appearance of a mistake in one of my published papers representing some sort of plot twist. Again: we all make mistakes. Four of my published articles have had such serious mistakes that I needed later to publish corrections for them! To think that a discussion of the references of one of our articles is some sort of dramatic turning of the tables is to miss understand how science works. It’s not that mistakes are a risk of doing science; mistakes are a necessary part of the process.

    • How many people do we know who make questionable scientific claims, and then when it’s pointed out to them, they publish a blog post about how questionable their claims were and how they’d modify them to be better?

      Few and far between

Leave a Reply to jim Cancel reply

Your email address will not be published. Required fields are marked *