Questions about our old analysis of police stops

I received this anonymous email:

I read your seminal work on racial bias in stops with Professors Fagan and Kiss and just had a few questions.

1. Your paper analyzed stops at the precinct level. A critique I have heard regarding aggregating data at that level is that: “To say that the threshold test can feasibly discern whether racial bias is present in a given aggregate dataset would be to ignore its concerning limitations which make it unusable in its ability to perform this task. The Simpson’s paradox is a phenomenon in probability and statistics which refers to a pattern exhibited by aggregated data either disappearing or reversing once the data is disaggregated. When the police behave differently across the strata of some variable, but a researcher’s analysis uses data that ignores and aggregates across this distribution, Simpson’s paradox threatens to give statistics that are inconsistent with reality. The variable being place, police are treating different strata of place differently, and the races are distributed unequally across strata. The researchers who designed the threshold test do not properly control for place, as modeling for something as large as a county or precinct (which is what they do) does not properly account for place if the police structure their behavior along the lines of smaller hot spots.”

2. How would your paper account for changes in police deployment patterns?

3. What are your thoughts on this article? It addresses a paper from one of your colleagues but if the critiques were valid would they also applied to your paper in 2007?

My reply:

1. I’m not sure about most of this because it seems to be referring to some other work, not ours. It refers to a threshold test, which is not what we’re doing. As to the question of why we used precincts: this was to address concern that the citywide patterns could be explained by differences between neighborhoods; we discuss this at the beginning of section 4 of our paper. Ultimately, though, the data are what they are, and we’re not making any claims beyond what we found in the data.

2. The data show that the police stopped more blacks and Hispanics than whites in comparable neighborhoods, in comparison to their rate of arrests in the previous year. All these stops could be legitimate police decisions based on local information. We really can’t say; all we can do is give these aggregates.

3. I read the linked article, and it seems that a key point there is that most of the stops in question are legal, and that “Those lawful stops should have been excluded from his regression analysis, since they cannot form the basis for concluding that the officers making the stop substituted race for reasonable suspicion.” I don’t agree with this criticism. The point of our analysis is to show statistical patterns in total stops. Legality of the individual stops is a separate question. Another comment made in the linked article is that the analysis was “evaluating whether the police were making stops based on skin color rather than behavior.” This is not an issue with our analysis because we were not trying to make any such evaluation; we were just showing statistical patterns. There was also a criticism regarding the use of data from one month to predict the next month. I can’t say for sure but I don’t think that shifting things by a month would chance our analysis. Another criticism had a prediction that a census tract would experience 120 stops in a month but it only had an average of 19. I don’t know the details here, but all regression models have errors. It all depends on what predictors are in the model. Finally, there is a statement that “It is a lot more comfortable to talk about the allegedly racist police than about black-on-black crime.” I don’t think it has to be one or the other. Our paper was about patterns in police stops; there’s other research on patterns of crime.

17 thoughts on “Questions about our old analysis of police stops

  1. I’m impressed that you’d take the time to craft a serous response to something that starts out like this:

    >…the arcane statistical models of Columbia law professor Jeffrey Fagan, the anti-cop advocates’ favorite expert witness.

    Such a signal of bad faith, imo, pretty much locks in a biased analysis.

    You don’t have to agree with people who are opposed to stop-and-frisk to acknowledge that there could be reasons to oppose it other than being “anti-cop.” You could even acknowledge that while “anti-cop” might fairly characterize some portion the opposition to stop-and-frisk, that label couldn’t fairly be applied to everyone who’s opposed. You could even say that for that portion of the opposition who could fairly be called “anti-cop,” Fagan is their favorite witness – AND believe that his models could fairly be described as “arcane” – but even with all of that, that sort of rhetoric from that paragraph is a basic disqualification from good faith exchange.

    It’s interesting to me that you’d bother to treat it as if it merits a good faith response.

    • Joshua:

      I agree that the linked article is coming from a strong political position. But strongly political positions can still lead to valid critiques. So I think it makes sense to consider the critique on its merits rather than to simply dismiss it. To flip it around, I wouldn’t want other people to dismiss my own writing because they feel that I’m engaging in overstated political rhetoric. Yes, the ideal criticism would be clean of political bias and exaggeration—but we have to take criticism where we see it.

      • Andrew –

        I agree that strong political positions should not be dismissed out of hand just because they’re strong political positions – irrespective of whether or not I (dis)agree with them.

        But I think there’s a meaningful distinction between strong political positions per se, and arguments built around bad faith, straw man, ad hominems (i.e., “anti-cop advocates”).

        That’a not to say that I think it’s never worthwhile to engage with bad faith, straw man, ad hominems. It depends, and sometimes it is. Maybe reading a thorough response will be persuasive to people who are undecided. Maybe going through the process will deepen one’s own understanding.

        But when a putatively analytical take is built upon that kind of a foundation, it lowers the probability of a net positive return on the energy invested, imo.

        I’m not offering that as criticism. Quite the opposite. I think it’s commendable that you engaged with the material in good faith.

        • I guess it depends on who you’re talking to. It’s true that there’s no point in arguing with someone who is arguing in bad faith (straw man arguments are generally easy to deal with, and there is no answer to an ad hominem attack other than to point out it is not a logical argument) but your responses are generally not going to the critic, but to everyone else. In that case, they’re generally important, because you can’t depend on the fact that *you* see bad faith argument as being congruent with others’ judgment.

        • Jonathan –

          >… because you can’t depend on the fact that *you* see bad faith argument as being congruent with others’ judgment.

          Sure.

          I think the chances of someone not yet committed to a position on this issue somehow skipping over the bad faith/straw man/ad hom to be convinced by Andrew’s patient explication of his position is pretty slim. But it’s possible. In that sena

        • Arrggh..

          In that sense, someone who doesn’t agree with my characteristizarion of the “anti-cop advocates” label would be convinced by his argument. I think more likely is that people either agree with my characterization and thus don’t need to be persuaded by Andrew’s that the author’s piece was quite flawed. Or that people have no have no problem with “anti-cop advocates” and thus won’t be persuaded by Andrew no matter how compelling I might find it to be.

          But there’s no downside to Andrew spelling it out and there could be a benefit.

  2. Thanks to redlining and related phenomena, “black-on-black crime” is just “neighbor-on-neighbor crime,” which constitutes just about all crime. So discussing it is not so much uncomfortable as irrelevant. You might call it a straw herring.

    • +1

      It seems obvious in retrospect, but I never put 2 and 2 together to realize that the reason why “black on black crime” is a distraction is because naming it implies it’s a special category. Instead, it’s just “crime” and for historical reasons people of similar backgrounds tend to live near each other. “Straw herring” is a great description.

      • Moodey –

        In what ways do you think that “black on black crime” is meaningfully distinguished from “white on white crime?”

        > Just like cop-on-black stops are just “cop-on-criminal” stops, and associating them with race is just a straw herring?

        I think part of the point is that the “cop-on-criminal” stops disproportionately turn out to be “cop-on-non-criminal” stops with minorities.

        • Joshua,

          Andrew was refuting the claim that he is anti-cop because he didn’t focus on black on black crime by making the logical argument that there already exists research on that, and that police racial profiling is also part of the two sided problem that should be studied.

          Then Michael made the comment implying that black on black crime essentially has no racially important element, and that the focus should be on neighbor to neighbor crime. Which is obviously nonsensical… plenty of white people are neighbors with other white people, but somehow they don’t have as high crime rates as black people.

          My point was that black on black crime *is* a problem. So is racial profiling by police. But the whole argument Michael uses to dismiss race/culture as a driving factor behind race on race crime could also be used to dismiss race/culture having an effect on police stops. Both would be erroneous.

        • Moodey –

          > Which is obviously nonsensical… plenty of white people are neighbors with other white people, but somehow they don’t have as high crime rates as black people.

          The logic of that statement seems to imply that “black people” have crime rates, as if “crime rate” is an attribute of being black.

          By that logic, it would seem that all black communities would have a higher crime rate than any white community.

          Do you actually know that to be true?

          For example, do you know that all rural black communities have higher crime rates than any (impoverished) urban white communities?

        • For that matter, do you know that all mixed communities have higher crime rates than any white communities, by virtue of having a higher % of black residents?

          Wouldn’t that also be the logical implication of white people not having “as high crime rates as black people?”

        • “By that logic, it would seem that all black communities would have a higher crime rate than any white community.”

          This is a bad-faith (il)logical conclusion from my statement. If I told you that fruits have a higher rate of tasting good than veggies, would you then say that by that logic, it would seem all fruits taste better than veggies? No, you wouldn’t.

          BTW, I have no clue what the race on race crime rates look like when parsed down, and with other variables controlled for. My assumption is that race still plays a factor considering Andrew didn’t just throw the argument to the wolves, but instead he stated that studies on that already exist. If this assumption is correct, my point remains that black on black crime is not a straw herring, as Michael suggested.

          You mentioned that you “think part of the point is that the “cop-on-criminal” stops disproportionately turn out to be “cop-on-non-criminal” stops with minorities.” I agree with this point. As I stated earlier, it’s a two-pronged issue. Pretending either prong doesn’t exist is equally bad-faith in both directions.

        • Moodey –

          > I agree with this point…

          Yes. You stated that earlier.

          > If I told you that fruits have a higher rate of tasting good than veggies, would you then say that by that logic, it would seem all fruits taste better than veggies? No, you wouldn’t.

          If you told me that fruits taste better than veggies, then you would be telling me that meals comprising all fruits would taste better than meals comprising all veggies.

          That’s a more parallel construction of my question to you.

          But sure, that wouldn’t mean that all fruits taste better than all veggies. That wasn’t the question I was asking you, however..

          You’d also be saying that meals comprising a mix of fruits and veggies would (very likely) taste worse than than a meal comprising all fruit. That’s the parallel to me follow-on question.

          As for this:

          > This is a bad-faith (il)logical conclusion from my statement

          Perhaps my logic is off (certainly not that far-fetched that it would be) – in which case I’d appreciate being straightened out. But I rarely, if ever, engage in bad faith. Don’t attribute to malice that which a lack of intelligence can explain.

  3. Thanks so much for the reply. I just had a few questions. My first remark is related to the critique of the threshold test that was predicated on location. I probably should have clarified what I wanted to ask. Essentially I was asking whether this critique that accounting for crime at a higher level of aggregation such as precincts/counties/neighborhood than crime hot spots which occur at the street segment or address level prevent us from making the claim that most of the race and criminal justice literature is suggestive of stop and search racial bias. To summarize Simpson’s paradox would presumably apply to any method if if it didn’t account for hot spotting at small locations. Second, regarding deployment patterns, I have heard that police model their behavior after changes that occur on a weekly basis. By accounting for crime rates from the previous year does this improperly model changes in police behavior and deployment. Lastly, do you believe your paper is sufficient for making a claim on racial bias or not?

    • Pa29:

      Regarding your first question: I think it makes sense to look at data at different levels of aggregation. In our paper we look citywide and by precinct and in between. It would be find to perform analyses of individual stops too. Different levels of aggregation tell us different things.

      Regarding your second question: It could well be that police change their actions on a weekly basis. For example, I’ve seen evidence that police behavior is different at the end of the month, and people sometimes speak of arrest quotas. I don’t think this changes our findings about aggregate levels of stops, but it makes sense to look at changes during the month and rapid response as well.

      Finally, I don’t really have any comment on racial bias. Our paper is motivated by claims of “racial bias” (the phrase is even in the title!) but all we talk about in our paper is “racially disparate impacts.” Obviously the two things are related. For example, when we released our report, the city lawyers responded by attacking us. They could’ve said something like, “yeah, police stops have racially disparate impacts but there’s no racial bias”—but they didn’t say that. So I guess the city felt vulnerable on that point. But from my perspective, I don’t really think I can say anything about bias, as that would require more of an individual-level analysis than anything we did.

Leave a Reply to Pa29Pa04 Cancel reply

Your email address will not be published. Required fields are marked *