It’s all about the denominator: Rajiv Sethi and Sendhil Mullainathan in a statistical debate on racial bias in police killings

Rajiv Sethi points me to this column by Sendhil Mullainathan, who writes:

Tamir Rice. Eric Garner. Walter Scott. Michael Brown. Each killing raises a disturbing question: Would any of these people have been killed by police officers if they had been white? . . .

There is ample statistical evidence of large and persistent racial bias in other areas — from labor markets to online retail markets. So I [Mullainathan] expected that police prejudice would be a major factor in accounting for the killings of African-Americans. But when I looked at the numbers, that’s not exactly what I found. . . . what the data does suggest is that eliminating the biases of all police officers would do little to materially reduce the total number of African-American killings.

Then come the numbers:

According to the F.B.I.’s Supplementary Homicide Report, 31.8 percent of people shot by the police were African-American, a proportion more than two and a half times the 13.2 percent of African-Americans in the general population. . . .

But this data does not prove that biased police officers are more likely to shoot blacks in any given encounter.

Instead, there is another possibility: It is simply that — for reasons that may well include police bias — African-Americans have a very large number of encounters with police officers. . . . Arrest data lets us measure this possibility.

At this point I just have to interject that every time I see “data” used as a singular noun, it’s like fingernails on a blackboard to me. Yes, yes, I know that in modern English, “data” is acceptable as a singular or plural noun. So I’m not saying that Mullainathan (or the New York Times style guide) is wrong. It just bothers me. I’m not used to it.

Anyway, to continue from Mullainathan:

For the entire country, 28.9 percent of arrestees were African-American. This number is not very different from the 31.8 percent of police-shooting victims who were African-Americans. If police discrimination were a big factor in the actual killings, we would have expected a larger gap between the arrest rate and the police-killing rate.

This in turn suggests that removing police racial bias will have little effect on the killing rate. . . .

He continues with some sentences that explain the basic idea which should be unexceptional to the readers of this blog.

My Columbia University colleague Sethi was not happy with this reasoning. Sethi writes:

Sendhil Mullainathan is one of the most thoughtful people in the economics profession, but he has a recent piece in the New York Times with which I [Sethi] really must take issue. . . .

A key assumption underlying this argument is that encounters involving genuine (as opposed to perceived) threats to officer safety arise with equal frequency across groups. . . .

Sethi argues that it does seem that police officers often behave more violently toward black suspects. But then, he asks,

How, then, can one account for the rough parity between arrest rates and the rate of shooting deaths at the hands of law enforcement? If officers frequently behave differently in encounters with black civilians, shouldn’t one see a higher rate of killing per encounter?

Sethi answers his own question:

Not necessarily. . . . If the very high incidence of encounters between police and black men is due, in part, to encounters that ought not to have occurred at all, then a disproportionate share of these will be safe, and one ought to expect fewer killings per encounter in the absence of bias. Observing parity would then be suggestive of bias, and eliminating bias would surely result in fewer killings.

The discussion continues

Sethi updates:

This post by Jacob Dink is worth reading. Jacob shows that the likelihood of being shot by police conditional on being unarmed is twice as high for blacks relative to whites. The likelihood is also higher conditional on being armed, but the difference is smaller:


[Damn that’s an ugly y-axis. And boy is it ugly to label the two lines with a legend. — ed.]

Sethi summarizes:

This, together with the fact that rates of arrest and killing are roughly equal across groups, implies that blacks are less likely to be armed than whites, conditional on an encounter. In the absence of bias, therefore, the rate of killing per encounter should be lower for blacks, not equal across groups. So we can’t conclude that “removing police racial bias will have little effect on the killing rate.” That was the point I was trying to make in this post.

Some interesting discussion in comments to Sethi’s post. I have no idea where the error bars are coming from in Dink’s post—I assume the data are some total count, not from a sample.

What’s my take on all this? First off, data are good (or, as the kids today say, data is good). So I appreciate Mullainathan’s numbers and also Dink’s (even if I can’t be quite clear on what Dink is actually plotting). I’m also sympathetic to Sethi’s general argument. Suppose there’s a sliding scale of police aggression, starting with arresting, moving to violence, and culminating in killing a suspect. I could imagine #killed/#arrested not varying much by race, while at the same time the cops are disproportionately arresting, beating, and killing African Americans.

To put it another way, this is an argument over the denominator. Mullainathan is using arrests as the denominator, but it’s not clear this is appropriate. These are tough questions. In our stop-and-frisk paper we used previous year’s validated arrests as a baseline. But that’s not perfect.

It’s interesting that Mullainathan uses the example of Tamir Rice and then recommends using arrests as a reference point. I assume that had the officer gotten close enough to Rice to see what was happening, he wouldn’t have arrested Rice in any case, right?

Also it is notable that it is two economists having this discussion, given that the topic is not actually economics! I say this not because I think economists should be discouraged from studying such topics, it just seems surprising to me. I suppose what’s going on is that there are a lot more academic economists out there, than there are sociologists or criminologists or even political scientists. Also it’s my impression that quantitative political scientists are discouraged from working on this sort of policy research, but I might be wrong about that.

98 thoughts on “It’s all about the denominator: Rajiv Sethi and Sendhil Mullainathan in a statistical debate on racial bias in police killings

  1. Sethi’s reasoning makes a lot of sense and is consistent with my priors, but I’m often suspicious of arguments that explain a lack of association by positing a bias that almost perfectly offsets the effect.

  2. This seems like the sort of analysis that would’ve been better to try to publish as an academic article before writing it up as an op-ed. In an academic article, (ideally) the model underlying the analysis would be laid out clearly so that we could see what he was assuming in the background. We could then refer to data that bear on those assumptions to decide their plausibility. Anticipating such responses, the article would likely have pre-empted the discussion by arguing on behalf of those assumptions. And if no such arguments could be made with a straight face in view of the data, then he would have decided to scrap the idea altogether. Instead, it seems like he had a hunch and saw two numbers that, on the surface, appeared to contradict his priors, and so decided to write an op-ed to be the first to break the story. Academic articles are not perfect (as this blog routinely observes), but they do enforce a level of quality control that op-eds rarely meet. Even if the article hadn’t yet been published, he would at least have something to point people to (on arXiv for example) which fully fleshes out the argument more than the confines of an op-ed would allow.

  3. “I have no idea where the error bars are coming from in Dink’s post—I assume the data are some total count, not from a sample.”

    The error bars, along with the rest of my analysis, are fairly problematic. I’m glad I bracketed this as a “quick and dirty” calculation throughout (and I’m glad you’re calling me out).

    I basically took the counts from each state, and treated each state as they were a “subject” in an experiment. Which… doesn’t actually make sense, for several reasons. Am I trying to make an inference to a larger population of states? Should each state be weighted equally? No and no. In general, this approach wasn’t really sound: I shouldn’t be building a statical model or generating error bars based on this data.

    If I take the raw counts, I get the same pattern, we just don’t get the fictitious error bars.

    However, I also think splitting by armed and unarmed was problematic because I don’t think data exists on what the relative armed/unarmed *arrest* rates are by race. Without that data, I shouldn’t be splitting by armed and unarmed deaths.

    “[Damn that’s an ugly y-axis. And boy is it ugly to label the two lines with a legend. — ed.]”

    Just the ggplot defaults. Quick and dirty analysis, like I said!

    • Jacob,

      Breaking it down nationally or even by state sounds to coarse and even within states there is not a homogenous distribution of the ethnic diversity as might be implied by “State X is 13.2% black.” Crime rates are also not homogeneous across states either, and I would suspect large cities to have locally higher crime rates and different local concentrations of ethnic backgrounds.

      Do you think there might be some sort of geographic equivalent of Simpson’s Paradox?

      • This isn’t well known, but racial gaps in homicide offending rates are fairly homogeneous across the states. There are a few exceptions, such as Hawaii and Alaska, where most of the few black residents have ties to the military and are better behaved than the black national average, and a few other places where whites are better behaved than the white national average, such as Minnesota and New Jersey, but in general the racial ratio in committing homicides is more consistent than most people who haven’t studied the subject in depth would imagine.

  4. In addition to the question of the right denominator, there is also a question about the meaningfulness of the numerator. Unless something has changed recently that I’m unaware of, there is no uniform requirement for reporting of killings by police to the FBI. This numerator represents whatever incidents the local jurisdictions choose to report. There could be considerable bias (of many different kinds) in this figure. I don’t think it means much and I wouldn’t draw conclusions from any ratio using it, no matter what the denominator.

    • Why is this? The USA seems to be one of the few Western countries (maybe the only one) where these sorts of national police data are not collected. Sure you have state and local police authorities as well as federal police (marshals, FBI etc), but so does Australia – the difference is only quantitative, not qualitative.

      • We have poor criminal justice data in general. Reporting from states and localities is very inconsistent. I think it is hard for people from other places to imagine how many police/criminal justice agencies there are and also hard for people to understand how little the federal government can mandate that agencies do. I would not really trust NCIC data very far; traditionally we’ve always said that it is only the supplemental homicide report that is pretty good though even there it is not 100%.

        • I think this difficulties are highly exaggerated. The heavy-handed approach would be to pass a federal law requiring reporting of each shooting death if the gun involved has “traveled in interstate commerce”, which would make it almost universal. But even without any federal law, shooting death by LEO is certain to make it to at least the local news that can be easily aggregated. Yes, that would not provide complete information, but it would produce at least headline numbers. I am surprised there is no activists or academics that trawl local police blotters for such info.

  5. How about the proportion of blacks among people that attacked police officers.

    Or the proportion of blacks in the people that were convicted of assault by a weapon?

    Anyone know what the black proportion in those populations are?

    • I haven’t been able to find any “assault with a weapon” statistics. Assault generally could be problematic as there’s a risk of racial bias in whether it’s a trumped up charge.

    • “Attacked a police officer” is one of those all purpose charges like resisting arrest. In general you have to be extremely careful about any CJ statistics you hear since they seldom talk about how they deal with multiple offender cases or how much discretion was involved. Since we have little idea how many people ever committed assault with a weapon, how many of those incidents were reported, whether police responded to the report at all, whether the report resulted in an arrest, whether an arrest resulted in a conviction and what the roles of race, gender and location were in influencing each of those proportions, it would be kind of meaningless to do a raw comparison. But anyone could look up the data in criminal justice abstracts.

    • There are a number of things required for this information to be relevant.

      1. Good data — valid inferences cannot be drawn from bad data.
      2. Detailed data — unless we know that the Police –> Suspect violence is occurring in situations in which there was actual Suspect –> Police attempted violence, then we have no basis to assume that the proportions are even relevant.
      3. Data on implicit biases of Police

    • Homicide offending rates might be a pretty good proxy for general dangerousness. According to the Obama Administration’s look at the data from 1980 through 2009, blacks were almost 8 times more like than whites to be homicide offenders.

  6. Isn’t the bias critique almost an insurmountable issue?

    If Sendhil offers to normalize by encounters, detractors say, well the cops could be having more unwarranted encounters with blacks.

    Fine, one could offer conviction rates or incarceration rates but then someone could bring up judicial bias or prosecutor bias. Or simply that the system convicts from only among those the police choose to present in the first place.

    You offer FBI crime stats but then someone says well the cops could have a reporting bias.

    Isn’t this a bit of turtles all the way thingy?

    • Those arguments were popular 40 to 50 years ago, but social scientists have consistently failed to find the much hoped-for major biases against blacks in crime statistics. Arrest rates, conviction rates, and victim interviews all give the consistent result that blacks commit a lot more violent crimes.

  7. Andrew: “First off, data are good”

    To which I would add: “Theory is also good”

    Getting into the data without theory is like searching for the source of the Nile without a compass. A horrible mess, or a luxurious garden of forking paths, depending on one’s view / goal.

    So why not start with police officer motivations for killing? Their beliefs, preferences, and constraints. And form that a definition of bias in relation to some theoretically defined (conditionally) unbiased rate.

    In short, I would say the discussion here is not about a denominator but, more fundamentally, about what it is we are talking about.

    • Models models models!

      A quick and dirty discussion of some issues:

      The population of black people in the US probably has various different proportions of people involved in various criminal activities than the population of white people. For example, the fraction of black people involved in embezzling more than $100k in a given year is probably orders of magnitude lower than the fraction of white people (though both are very low). However, the ratios of the fraction of black people to the fraction of white people involved in sales of street drugs is going to be very different.

      In any case, police officers are more likely to have first-hand data on how often they get involved with actual criminals by racial criteria, and their priors for whether a person they see on the street is likely to be a criminal, likely to be violent, or likely to be armed are going to be different from the average persons.

      Furthermore, for a police officer, they are going to decide how to behave based on their own loss function. If they turn out to be harassing a harmless person, then typically they go home at night safe and often don’t even get a complaint filed. If they turn out to be encountering an armed drug dealer and don’t take the right precautions, then they maybe go to the hospital or the morgue.

      Working in that environment, where they have particular information which is different from the average person’s, and they have highly asymmetric loss functions which are different from the overall population assessments (I care about how many police officers die, but I also care about how many innocent victims die, and I care MOST about whether I die. The police have a similar set of concerns but the frequency with which their concern for their OWN safety comes up is MUCH higher than mine)

      So, I think the main thing to be discussing is whether police officers are acting rationally based on good information, and reasonable risk assessment given their own different-from-average-person decision criteria, or whether their biases and risk assessments are highly out of line with reality. If the police are acting in a way that is near-optimal for their own welfare (even if that optimum behavior IS biased and “bad for society” in some sense), it’s going to be hard to change their individual behaviors, whereas if they’re acting on un-justified hearsay or poor threat assessments then more accurate information could potentially help them come into line with more reasonable actions.

        • That training seems to be a rational heuristic response to the asymmetry of the loss. When you assume everyone is out to get you, you take steps to protect yourself, and this leads to much lower chance that YOU will be injured or killed. But… that training as described seems to be too much of a shortcut/heuristic. Closer to realistic training would be better. For example if say <10% of traffic stops lead to violence then training that shows 20 or 30% leading to violence might make good sense, giving officers extra practice in dangerous situations makes sense, but portraying nearly 100% as dangerous doesn't, it sends a message that is out of line with reality by far too much. I suspect the real numbers are much lower than 10% too in most parts of the US. You can't train cops on fully realistic frequencies… because they'd have to do thousands of training stops before they got enough practice to know how to respond when the dangerous situation does occur. But you shouldn't be training them that EVERY SINGLE TIME they're in mortal danger. I mean, in the history of the US how many cops were actually shot with a double barreled shotgun by a 70 year old lady who gets out of her vehicle when told to remain inside?

          At the same time, the training that CITIZENS receive is terrible as well: "reality" cop shows indeed!!!

      • “For example, the fraction of black people involved in embezzling more than $100k in a given year is probably orders of magnitude lower than the fraction of white people”

        I’m not sure that there’s much evidence for that. The overall black conviction rate for fraud in general is several times higher than for whites.

        From the studies I’ve seen, price-fixing / antitrust violations are one of the few white-dominated crimes.

  8. … so the real issue of “bias” here is the bias of economic & academic analysts in automatically assuming official government data is true (FBI and police in this case).

  9. The original article attempts to address some of the issues raised by Sethi re: the denominator:, and in particular his comment that “If the very high incidence of encounters between police and black men is due, in part, to encounters that ought not to have occurred at all, then a disproportionate share of these will be safe.” From the original article:

    “If the major problem is then that African-Americans have so many more encounters with police, we must ask why. Of course, with this as well, police prejudice may be playing a role. After all, police officers decide whom to stop or arrest.

    But this is too large a problem to pin on individual officers.

    First, the police are at least in part guided by suspect descriptions. And the descriptions provided by victims already show a large racial gap: Nearly 30 percent of reported offenders were black. So if the police simply stopped suspects at a rate matching these descriptions, African-Americans would be encountering police at a rate close to both the arrest and the killing rates.”

    Moreover, I would question the assumption that an encounter that should not have happened at all is likely to be safe – prisons are full of people who were stopped because police had reasonable suspicion re crime X, and then uncovered evidence instead of crime Y. (Of course, if police are acting like the NYPD under the recent “stop and frisk’ policy, then they will be stopping lots of wholly innocent, and hence wholly safe, people).

  10. >For the entire country, 28.9 percent of arrestees were African-American. This number is not very different from the 31.8 percent of police-shooting victims who were African-Americans. If police discrimination were a big factor in the actual killings, we would have expected a larger gap between the arrest rate and the police-killing rate.

    >This in turn suggests that removing police racial bias will have little effect on the killing rate

    One thing I’ve realized is incredibly odd is how the original article talks about changes in rates, rather that the change in numbers.

    Think about it:

    If we are talking about moving from 31.8% to 28.9%, this might sound like a small change.

    But it’s only small if we think the size of the whole pool remains unchanged.

    This, of course, is not at all what people are talking about when they talk about reducing police bias. The idea isn’t to shift shootings away from black people and onto white people; the idea is to simply reduce the number of killings of black people.

    If that’s what we’re talking about, then a reduction from 31.8% of a large pool down to 28.9% of a smaller pool isn’t actually that small of a change. So I really don’t think it’s accurate to claim that “removing police racial bias will have little effect on the killing rate.”

    • No then the claim ought to be something like “Removing police trigger happiness will have an effect on the killing rate”.

      When someone speaks of “bias”, I take it to mean that the police are more likely to shoot a black person, out of an *irrational* or unjustifiable reasoning, far above what shooting rate we’d expect from the black proportion of the violent criminal population.

      • > When someone speaks of “bias”, I take it to mean that the police are more likely to shoot a black person, out of an *irrational* or unjustifiable reasoning

        Yes, exactly. So when we talk about removing the bias, we’re talking about lowering their propensity to kill black suspects, bringing it closer to their (lower) propensity to kill white suspects. The assumption here is that their propensity to kill white suspects is closer to a rational propensity. Are you questioning that assumption? Otherwise I don’t follow…

        >When someone speaks of “bias”, I take it to mean that the police are more likely to shoot a black person … far above what shooting rate we’d expect from the black proportion of the violent criminal population.

        No one is disputing that. We agree that ‘eliminating bias’ means bringing the 31.8% killing-rate down to the 28.9% arrest rate. My point is that when we talk about removing racial shooting bias, we’re talking about lowering this 31.8% figure by reducing the number of black deaths. Which nets to 28.9% of a smaller overall pool (since there are fewer black deaths in that pool). So 31.8% of a large pool becomes 28.9% of a smaller pool.

        >No then the claim ought to be something like “Removing police trigger happiness will have an effect on the killing rate”.

        This is of course also true.

        • @Jacob

          No, I disagree that bringing down 31.8% killing-rate down to the 28.9% arrest rate makes sense as a goal.

          Arrestees are a highly heterogeneous populations. It could very well be that black arestees are of the more dangerous kind.

          Even if black arestees are 20% more likely to draw a weapon on a cop then the 31%-28% discrepancy could be the natural outcome of an entirely rational response by a cop.

          What I’m saying is that I see no evidence of a systematic bias so far.

          • That’s fine. It’s a separate issue whether the higher shooting-to-arrest ratio for black people is driven by racism or by something else. My point was just that the difference between 28.9 and 31.8 isn’t as small as the original article makes it seem.

      • “The assumption here is that their propensity to kill white suspects is closer to a rational propensity.”

        (Just to clarify, I’m not saying that the rate at which police are killing white people is rational! Just that it’s closer to rational— it could certainly be lower still.)

  11. The problem is multi-layered. Take this from the Bureau of Justice Statistics (part of the DoJ), as of 12/31/2013: “Similar to 2012, non-Hispanic blacks (37%) comprised the largest portion of male inmates under state or federal jurisdiction in 2013, compared to non-Hispanic whites (32%) and Hispanics (22%). White females comprised 49% of the prison population compared to 22% black females. However, the imprisonment rate for black females (113 per 100,000) was twice the rate of white females (51 per 100,000).” If you look at these numbers, you could easily argue the percentage of arrestees who are African-American is low or reflects a greater “reality” of criminal conduct because a larger percentage is actually in state or federal prison. It seems worse if you look at violent crime: of all the “Black” state prisoners – the report uses Black and I put it quotes to emphasize that is their choice of word – “58.3% of all Black prisoners are in for violent crime, compared to 49.3% for whites. (Or another way: more whites are in prison for property crimes (24.5%) versus blacks in for property crimes (16%) and – surprisingly? – about the same rate for drug crimes.) BTW, in raw numbers there are more Blacks in prison for violent crime than Whites: 290k versus 228k.

    In other words, it isn’t just the denominator and one argument – or even what Daniel said about how police process both information and bias in their jobs – but many levels. I don’t know what these numbers mean. One can argue the system is harder on blacks and thus sends more to jail: just cite the numbers about the percentage and say it’s due to oppression. Say it’s due to whatever.

    All I think we can say with a relative degree of confidence is, without considering issues like under-reported deaths, that the percentage of killings by the police by race is roughly in line with these other measures. But that doesn’t say why the system this describes is like that.

  12. Andrew, thanks for posting this. Some good comments here.

    I wrote my post because I thought that one of Sendhil’s key claims (that elimination of police shooting bias would do little to reduce black deaths) was not supported by the evidence presented. This doesn’t mean that the evidence points to a large effect – we need much better tests to establish that.

    My main concern is with the killing of innocents who pose no threat at all to officers. Sendhil implicitly assumed that the ratio of risky to safe encounters was the same across groups and I tried to suggest why that might not be the case. I’ve been thinking about how we might test this.

    I think we can test a joint hypothesis of no police bias and equal rates of safe/risky encounters across groups. I think this would predict that conditional on being killed, the rate of armed/unarmed should be the same for both groups. The Guardian data allows us to test this:

    Haven’t had time to check but will do so. Another test is to see whether there are large difference in shootings by race of officer, especially among unarmed victims. But we don’t have data on officer race for police shootings as far as I know.

    • Rajiv: “I think we can test a joint hypothesis of no police bias and equal rates of safe/risky encounters across groups. I think this would predict that conditional on being killed, the rate of armed/unarmed should be the same for both groups.”

      Possibly but here is one possible (if perhaps not so probable) criticism.

      Police react, not only to whether a suspect they encounter is armed, but on how likely the suspect is likely to use the weapon.

      In turn the suspect may decide to use the weapon depending on their assessment of how likely the police is likely to shoot at him.

      In equilibrium it could be that dissonant pairs (e.g. cop, suspect of different race) are stuck in a prisoner’s dilemma.

      Suspecting prejudice, hatred or whatever they expect the other to shoot. And so their best course of action is to shoot first, which only reinforces the belief and the equilibrium. Being in a consonant pairs may act as a coordination device or convention to avoid the dilemma.

      If so the data should show that blacks will be shot at more often both when unknown to be armed (they could be armed and if so use it), and specially when known to be armed (they are more likely to use it).

      None of this is, by itself, some sort of bias, since each player is acting rationally.

  13. Did any of these articles discuss: when the cops interact with a man: what is the probability that this man is a cop killer given his race? Typically sophisticated analyses of events try to look at the decisions of a given agent via the lens of that agent’s interests. (Agency theory if you will.) As a first cut, I’d suggest that cops aren’t really concerned about who commits more crimes in general –- instead they are concerned about a very specific and taboo crime related to the killing of cops (and even more specifically the killing of them or their partners). Quote from The Economist a year ago “Roughly 29% of Americans shot by the police are black, but so are about 42% of cop killers whose race is known”. More transparency on this matter (and the converse of police shooting suspects) would be useful.

    As a second cut, though, you need another ruler to keep things in perspective. One possible ruler to use is the incidents of cops being killed on the job by crooks vs. car accidents. Despite what people feel on the matter, car accidents kill more cops on the job than criminals do. (The Economist had an article on this, though the link is proving elusive right now.) There’s more layers to the analysis of course.

    I am not sure what the term bias means in this case. Back to my first cut, if some group has a much higher base rate of being a cop-killer, is it bias for the cop to take that into account? I’d suggest no. Given the low incidence of death from crooks these days, though, it may be more appropriate for cops to suppress the base rate. That’s layer two of the analysis. Then there’s layer three… and so on.

    • I am afraid you are jumping levels before you even get started with your analysis. When the police engage, they are and should always be thinking about their own and the public’s safety by watching for weapons, aggressive movements, and other behavioral indicators of potential violence during the engagement at hand. This should be independent of race. If there is a disparate rate of incidence between racial groups for whether or not they pull a trigger, bash a person’s brains in with a nightstick, ground and pound, or choke them to death then one should ask ‘Why?’ The notion that ‘someone else who kinda looks like you and acts like you shot a cop’ is a reasonable explanation for how this might happen should be absurd on its face. The fact that it might not be suggests a number of possible alternatives, one being group level retaliatory intent.

      • @James

        That’s an interesting point that I’m always conflicted about.

        What you are saying is that it’s OK for a cop to form a mental correlation conditioned on past encounters that a bulge in the pocket or not showing hands is an indicator of increased risk.

        But OTOH, irrespective of what data show, any correlation on gender or race needs to be consciously purged from his mind? e.g. Say, conditioned on past data his heuristic says that older-Asian-lady-sitting-in-a-park is low risk, he should be trained to ignore that?

        • I am saying that it is essential to be situationally aware of what is happening in an engagement with the public and that there are better and worse ways to interact with people that include living within the constitutional rights of the suspects. It is proper to take note of a bulge, of aggressive movements, etc. If a policeman makes it personal policy to draw a weapon and fire simply based on these behaviors, there is a problem. Their should be an series of escalating actions on the part of the police and not a tendency to immediately draw their weapon and fire. I am saying that if one focuses on the present interaction and does not treat create the problem by treating a few loose cigarettes as if they were kilos of heroin or illegally selling weapons, then appropriate behavior on the part of police would be far more likely.

          Your questions suggest such an overly simplistic perspective on race that it is difficult to think you are not joking, but you do not seem to be. I am saying that if the lady on the bench is doing something threatening to someone else, then she should not be ignored, but if she is feeding pigeons then leave her the hell alone. If one is using race as a heuristic, then that is someone who needs some serious development of their critical thinking skills. And if we are giving power and weapons to people without those skills, then we need to think seriously about how to remedy that.

          • I don’t think my view is not simplistic and I wasn’t joking at all.

            I offer you a real examples from related domains:

            There are software products being sold and in use out there for flagging high risk passengers for screening at airports. They routinely assess risk using a number of available predictors e.g. age, sex, origin & destination airports etc. Is it OK or not OK to use things like nationality or race (inferred indirectly from names perhaps) as a risk predictor if it does turn out on the training data set that these variables have predictive impact?

            Another example: A recent blog post here described software Universities are using to flag academic fraud. If on the training data set such a software detects a correlation with race or nationality of the test-taker is it OK to use it or not?

            All of these could be called racial profiling and I guess they are. But if the racial profile does have predictive impact should we embrace it or ignore it?

            Look, I’m not suggesting some stupid working correlation for a cop that says black=crime and ignore all other cues. But when (if?) actual crime data indicates a huge bias towards a certain demographic, is it fair for the cop to consider this in his risk analysis in the context of the totality of the circumstance?

            • I think if someone is putting race into such algorithm’s they too need some work on critical thinking. We should be looking for causal indicators, not simply correlative.

              • There’s quite a bit of practical application that works on correlative indicators.

                When your email spam blocker blocks spam everyday is it working on purely causal indicators? How about Amazon when it tries to predict what items you might want to buy. Or Netflix when it decides what movies you might like.

                For predictive software out of bag prediction accuracy is important. So long as that is delivered will you throw out predictors simply because causality cannot be proven?

              • Yes, agreed. But, it is important to not forget that the correlative indicator is not the same thing as a causal attribute and may only be loosely related to it. And when potentially life altering decisions are being made with such crude data, the effects of such decisions should be taken into account. Whenever possible, the causal indicators should be chosen instead and contexts such as the one we are talking about should be required by funding agencies.

              • Commas are important:

                Yes, agreed. But, it is important to not forget that the correlative indicator is not the same thing as a causal attribute and may only be loosely related to it. And when potentially life altering decisions are being made with such crude data, the effects of such decisions should be taken into account. Whenever possible, the causal indicators should be chosen instead and in contexts such as the one we are talking about, should be required by funding agencies.

        • Great and thoughtful comments Rahul.

          In NZ we faced exactly this issue some years ago in a correctional setting. In NZ we have a national corrections database, which accounts for our whole correctional population and so have the luxury of not having state/federal/country crossovers to any real degree. We are a nicely contained sample. Also, we have data from many many years at quite a detailed level. In that context, we developed (1990s) a risk estimation system, which in general works quite well. This system had ethnicity in as an estimator. A government some years ago required that we remove it. Our best guess was that the variance it accounted for was about 3%, or less. The general view is that removing it as a risk predictor has made little realistic difference to the pragmatic purpose of using the risk tool to better target treatment or of considering suitability for parole.

          My second point then is that we have at all levels of the justice system a systematic bias against our indigenous people, NZ Maori. This is a finding seen in many other jurisdictions, notably Canada and Australia. For Maori, the rate of arrest was higher, rate of conviction higher, and that there was about 1 year longer on a sentence than non-Maori. While I was not allowed to report this finding back then, it nonetheless was fairly clear from the data. In Police data, there is a similar finding. They have an implicit ‘profiling’ system where race is a significant factor. Our correctional system has around 48% Maori ‘clients’; the population is around 18%.

          My third point is that, IMHO, most jurisdictions simply ignore reasonable evidence on the role of psychological priming on behaviour. There is an implicit view that officers are tabula rasa — this is simply wrong. In NZ we explicitly provide training on countering bias — which is useful in terms of any bias, not just ethnic bias. As in statistics, human beings need to overcome central tendencies to show change!

          So I do wonder if the disparities noted in the original article are really just outcomes of much ‘deeper’ social problems/variables around individual, ethnic, community, state and federal attitudes, policies and behaviour. In our situation here, despite years of attempting to change this apparent racial bias, we have largely failed.

          • Great to hear from an actual practitioner who has tackled these issues. Out of curiosity, is the NZ data or your model available online for someone to explore?

            The interesting question would arise, had the racial predictor been more significant, more necessary to getting good predictive performance. I wonder what would be the right thing to do then.

            Any thoughts?

            • Here is the link to one of the public articles I believe James O’Malley moved to Harvard to do epidemiology or similar — I didn’t know him but people who did suggest he never was completely comfortable with this methodology, which was early career. Dave Riley was Director and my former boss. The tool they developed enabled a scarce resource (psychological treatment and assessment) to spread further in a poorly funded area through targeting of risk and need, a la Andrews and Bonta.
              Here is a reasonable examination of the extent of the disparate outcomes by ethnicity to give as feel of the extent of the problem. What has always intrigued me is that the variance explained by ethnicity was so much less the the actual rate of Maori offending and incarceration.
              My view was that the racial predictor would always have been removed — largely due to the fact of our Treaty of Waiting (between the Crown and Maori from 1835) which specifies obligations and responsibilities of parties. The principles of Protection, Partnership and Participation have formed into law and policy that make ethnicity-based profiling simply illegal in most instances.
              If you wanted to look further, feel free to contact me directly.

  14. Perhaps a more relevant percentage in thinking about police shootings is not the percentage of arrestees who are black (29%) but the percentage of homicide offenders who are black.

    That number is 52%, according to a 2011 Obama Administration report covering 1980 to 2008 by Alexia Cooper and Erica L. Smith of the federal Bureau of Justices Statistics, “Homicide Trends in the United States, 1980-2008.”

    “Based on available data from 1980 to 2008—

    “Blacks were disproportionately represented as both homicide victims and offenders.…The offending rate for blacks (34.4 per 100,000) was almost 8 times higher than the rate for whites (4.5 per 100,000).”

    Using 52% as the best guess of the fraction of lethal, desperate people encountered by the police who are black would suggest that police are disproportionately shooting other races.

    My guess, however, is that cops are not actively discriminating by race against nonblacks. There’s a lot of evidence that “suicide-by-cop” is more common among whites than among blacks, so that skews the police shooting statistics against whites relative to their much lower homicide offending rates.

    • The problem with your analysis are the assumptions combined with incredible leaps of logic.

      1. The 52% and the 34% numbers are conviction rates which makes your leap to 52% of “lethal, desperate people encountered by the police who are black” a huge overestimate as even the 34% would likely be a substantial overestimate. It’s not clear to me why you dismissed the 34% number in favor of the 52% number other than an attempt to cherry pick favorable data and ignore unfavorable.

      2. You then use this gross overestimate combined with the exceedingly rare event of ‘suicide by cop’ to support your claim of ‘no bias’ which does not even necessarily follow as it is possible have more biased black shootings regardless of the percentage of black people murdered or who murder.

        • Perhaps a more relevant percentage in thinking about police shootings is… the percentage of homicide offenders who are black. That number is 52%… Using 52% as the best guess of the fraction of lethal, desperate people encountered by the police who are black…

          Can’t help myself…

          Perhaps a more relevant percentage in thinking about police shootings is the percentage of homicide offenders who are not black. That number is 48%. Using 48% as the best guess of the fraction of lethal, desperate people encountered by the police who are not black is insane, as is the above-quoted text.

          • I don’t get it. He is saying to take the population of murders as a proxy for the population of “lethal, desperate” people encountered by police. I can imagine responding “that’s not a great proxy” or “to a first approximation maybe, but there’s more too it than that”, but why would it be insane? Convicted murderers seems like a pretty good starting point for “lethal+desperate” to me.

      • @James

        Say we took the lower 34% number. That would still not indicate any anti-black bias, right? Given that only 31% of police shootings are black?

        More generally, your point about cherry picking is exactly what I was trying to say earlier: There’s so many possible Denominators, so many ways to normalize this against that any explanation of bias or non-bias on statistical grounds is very easy for detractors to critique. Sailer used % of homicide offenders who are black. One could use % of violent offenders who are black. Assaults perpetrated by blacks. Assault on a cop perpetrated by a black. Gun crimes committed by blacks. etc.

        PS. What makes you think the 34% is a “substantial overestimate”? Is there methodological grounds to think so? Just curious.

        • I disagree Rahul. My point was that the 34% number is certainly a substantial overestimate. Given the known bias in conviction rates (, it is likely closer to 24% than to 34%, which would indeed be suggestive of a problem. But, as I pointed out, there need not be such a statistical discrepancy for there to be a problem. One only need look at the news over the last year to see the significant numbers of incredibly racially motivated abuses perpetrated by law enforcement that have been going on for decades and decades, but not thought to be newsworthy.

          • A lot of the commenters might be surprised to discover that their assumptions about bias as a large cause of the differences in crime rates was the conventional wisdom a half century ago, and a lot of social science effort was devoted to coming up with unbiased crime rate by race — only to discover that the racial gaps in committing crimes were very real.

            In 2002, James Q. Wilson summed up what he’s learned from a lifetime of studying crime:

            “A central problem—perhaps the central problem—in improving the relationship between white and black Americans is the difference in racial crime rates. No matter how innocent or guilty a stranger may be, he carries with him in public the burdens or benefits of his group identity…

            “Estimating the crime rates of racial groups is, of course, difficult because we only know the arrest rate. If police are more (or less) likely to arrest a criminal of a given race, the arrest rate will overstate (or understate) the true crime rate. To examine this problem, researchers have compared the rate at which criminal victims report (in the National Crime Victimization Survey, or NCVS) the racial identity of whoever robbed or assaulted them with the rate at which the police arrest robbers or assaulters of different races. Regardless of whether the victim is black or white, there are no significant differences between victim reports and police arrests. This suggests that, though racism may exist in policing (as in all other aspects of American life), racism cannot explain the overall black arrest rate. The arrest rate, thus, is a reasonably good proxy for the crime rate.

            “Black men commit murders at a rate about eight times greater than that for white men. This disparity is not new; it has existed for well over a century. When historian Roger Lane studied murder rates in Philadelphia, he found that since 1839 the black rate has been much higher than the white rate. This gap existed long before the invention of television, the wide distribution of hand guns, or access to dangerous drugs (except for alcohol).

            “America is a violent nation. The estimated homicide rate in this country, excluding all those committed by blacks, is over three times higher than the homicide rate for the other six major industrial nations. But whatever causes white Americans to kill other people, it causes black Americans to kill others at a much higher rate.

            “Of course the average African American male is not likely to kill anybody.

            “During the 1980s and early 1990s, fewer than one out of every 2,000 black men would kill a person in any year, and most of their victims were other blacks.

            “Though for young black men homicide is the leading cause of death, the chances of the average white person’s being killed by a black are very small. But the chances of being hit by lightning are also very small, and yet we leave high ground during a thunderstorm. However low the absolute risk, the relative risk—relative, that is, to the chances of being killed by a white—is high, and this fact changes everything.

            “When whites walk down the street, they are more nervous when they encounter a black man than when they encounter a white one. When blacks walk down the street, they are more likely than whites to be stopped and questioned by a police officer…

            “The differences in the racial rates for property crimes, though smaller than those for violent offenses, are still substantial. The estimated rate at which black men commit burglary is three times higher than it is for white men; for rape, it is five times higher. The difference between blacks and whites with respect to crime, and especially violent crime, has, I think, done more to impede racial amity than any other factor. Pure racism—that is, a visceral dislike of another person because of his skin color—has always existed. It is less common today than it once was, but it persists and no doubt explains part of our racial standoff. But pure racism once stigmatized other racial minorities who have today largely overcome that burden. When I grew up in California, the Chinese and Japanese were not only physically distinctive, but they were also viewed with deep suspicion by whites. …

            “In spite of their distinctive physical features, no one crosses the street to avoid a Chinese or Japanese youth. One obvious reason is that they have remarkably low crime rates.”

            • Really? Where did he live? Anyone who uses the phrase ‘no one crossed the street to avoid ____” critical thinking cannot be trusted. Absolutism and sloppy reasoning such as this likely bleeds into the rest of Wilson’s analyses.

            • And anyone who thinks the existing data on statistical rates of offending is the only way to garner evidence for this problem needs to rethink their premises. I have never come across a metric that cannot be gamed by smart people.

      • My point was precisely to show what kinds of spurious conclusions we could be drawing all over the place by mining data sets selectively.

        In this sort of complex problem I think there’s a serious risk of “mathiness”. Just because we use some ratios doesn’t mean we really know much about whether or not there’s any significant underlying bias in police shootings.

  15. Many people here are citing rates of homicide or violent crime across groups as if this is a suitable benchmark against which to compare police killings. One needs to be really careful with this, because clearance rates for homicide vary so much across communities. None of the killings mentioned by Sendhil were related to a homicide or homicide investigation. See Danielle Allen on this point, and the book by Jill Leovy that she references:

    • Right, the clearance rates of homicides in black neighborhoods tend to be lower than in white neighborhoods, so possibly that biases downward the black share of homicide offenders and the black share of murderers is actually higher than the Obama Administration’s 2011 estimate of 52%. But, I’ll stick with 52% as the most authoritative estimate we have at present.

      By the way, here’s my review of Jill Leovy’s recent book “Ghettoside” on why it’s so tough to clear homicides in South-Central L.A.:

      “Leovy is something of a data nerd. Because her L.A. Times only covered about one out of every ten murders in Los Angeles County, she started the Homicide Report blog on the paper’s website.

      “In 2010, I analyzed the first 2,600 killings covered by her blog. Only 9 percent of Los Angeles County’s roughly ten million people were black, but they accounted for an imposing fraction of the dead. Among 15- to 29-year-old male victims of homicide in L.A. County, blacks outnumbered whites by a per capita ratio of 20.7 to 1, while Hispanics outnumbered whites 6.8 to 1. (According to the Obama Administration, the national ratios are lower but still eye-popping.)

      “Those were the ratios for homicide victimization. All evidence suggests that the homicide offending racial ratios are similar or worse. But it’s hard to pin down the exact figures, because so many homicides in Los Angeles never lead to an arrest. Leovy estimates in Ghettoside that only 38 percent of the murders of black males are “cleared” by an arrest.” …

      “In bringing South Central murderers to justice, the high-tech lab techniques featured on shows like “NCIS” often aren’t useful:

      “‘… there were few mysteries among Southeast cases. The homicides were essentially public events—showy demonstrations of power meant to control and intimidate people. They took place on public streets, in daylight, often in front of lots of people. Killers often bragged.'”

        • What’s bizarre about (1) considering how many murders are solved and (2) the race of the victim when thinking about estimating the race of the perpetrator?

          What’s bizarre is anyone who seriously thinks they win an argument just by paraphrasing “sailer is evil” enough times. Sailer isn’t evil, so what else do you got?

  16. Andrew, as blog master, is there any way you could ban/delete this racist troll Steve Sailer? He’s ruining what is otherwise an interesting conversation. Open debate is healthy, but that’s not what this crypto-fascist Malcolm Gladwell is interested in at all.

    • I don’t know if banning or deleting is the right response but he does seem more interested in demonizing a group of people than in seeking an honest answer to the question of whether eliminating police bias would have a significant impact on lives lost.

      My point about clearance rates was this: encounters between police and members of different groups are not proportional to homicide offending, so rates of offending do not help us determine the ratio of safe to risky encounters by group, and do not therefore help us answer the question at hand.

  17. “encounters between police and members of different groups are not proportional to homicide offending, so rates of offending do not help us determine the ratio of safe to risky encounters by group, and do not therefore help us answer the question at hand.”

    Sorry, but consider the obvious analogy to the disproportionate percentages of police shooting victims who are male. A much higher percentage of people who get killed by the police are male than males’ share of the population or even of people who get stopped by the police (i.e., women get stopped for traffic reasons quite a bit).

    But absolutely nobody seems to be concerned about this sex inequality in police shootings, in part because, obviously, men are much more dangerous than are women, on average. For example, according to the federal Bureau of Justice Statistics, men comprised 89.5% of homicide offenders for the years 1980-2008, which implies that men tend to be more dangerous in general.

    The black share of homicide offenders is higher (52%) than the black share of either arrestees (29%) or of police-shooting victims (32%) or of the population (13%).

    So, one natural question is why is the black share of police shooting victims so much lower than the black share of homicide offenders? Why are cops shooting so many white guys? Are the police biased in favor of handling blacks more gingerly than whites because of all the trouble they can get in for killing blacks?

    Maybe, maybe not. But I suspect a stronger reason is that suicide-by-cop is more of a white thing.

    Here’s a way to test this. It’s not conclusive, but you could check to see if the actual statistics are in the direction my suicide-by-cop theory would suggest. My impression from studying 2600 cases in Jill Leovy’s Homicide Report for the Los Angeles Times is that suicide-by-cop skews older.

    So, check the ages of shooting victims by race: my guess would be that whites who get killed by cops are more likely to be older.

  18. I do not believe that people with deep racial antipathies should be banned from comment threads on blogs. But I do believe that they should be banned from being cops.

    The implicit association test should be as central to screening and training of police officers as tests of general competence. It can be taken here:

    This blog has a large number of commenters with a sophisticated understanding of Bayesian statistics. Adding or subtracting one or two will not make much difference to the technical quality of the discussion.

    But this discussion we are having also requires knowledge of some history and sociology. I recommend Glenn Loury’s DuBois lectures (The Anatomy of Racial Inequality) or his Tanner Lectures (Race, Incarceration and American Values). From the latter:

    “I wish to suggest that if, with Patterson, we can see in American slavery not merely a legal convention but also a superstructure of justifying ideas defining and legitimating an order of racial hierarchy, then we should also be able to see that termination of the slave’s legal subordination could, in itself, never be sufficient to make slaves and their progeny into full members of society. The racial dishonor of the former slaves and their descendants, historically engendered and culturally reinforced, would have also to be overcome. I claim that an honest assessment of American politics in the post-civil rights era — our debates about welfare, crime, schools, jobs, taxes, housing, test scores, diversity, urban policy, and much more — reveals the lingering effects of this historically engendered dishonor.

    By “racial dishonor” I mean something specific: an entrenched if inchoate presumption of inferiority, of moral inadequacy, of threat to public safety, of unfitness for intimacy, of intellectual incapacity, harbored by observing agents when they regard (at least some of) the racially marked subjects. So, we have come from a history of racial slavery and institutionalized racial subordination. And, the principal venue in which the legacy of that history remains vividly apparent is in the realm of punishment.”

    • Is it an “unjustified presumption of an increased threat to public safety” if a city’s dataset reveals that 40% of violent crime perpetrators were black when, in fact, they were only 10% of the population?

      Is that presumption or fact?

      • If the data are accurate, then it would be a fact, though not satisfactorily explanatory of causes for such. In contrast, if the data are riddled with biased information based on what is chosen as report worthy, on who is falsely accused, on who is not accused, etc., then it would indeed be a presumption.

        From what I can tell based on the FBIs information as well as what has been uncovered by media is that we are currently solidly within the domain of presumption and need a lot of work and funding to move solidly into the domain of fact.

      • Good Lord, I give up. You have manufactured a quote. Loury is talking about a pre-analytical presumption, a disposition to believe, before one is confronted with data. Take an hour or two to read the lectures. You might learn something valuable.

    • None of that stuff matters. Put the Navy SEAL who killed bin Laden in any of these scenarios and he always gets it right, irrespective of how well he’d score on an implicit bias test or his knowledge of AA history. And I mean always; he’ll never shoot when he shouldn’t and always shoot when he should. So if we want to rid ourselves of incidents like Tamir Rice and Walter Scott, that’s the sort of selection and training we’ll need to subject cops to. It’s easy to blame the individual cops for these scenarios, but it’s our fault for putting un-vetted and under-trained people into these scenarios where a failure rate is wholly predictable.

      • John_H:

        Always is a strong word to use, especially on a Bayesian blog, but your point about selection and training is correct. Their focus would be in the moment and on the behaviors at hand. That said, I would not dismiss the importance of bias as it would still be relevant in how people are treated when apprehended rather than shot or whether to engage in the first place.

        • +1

          I have heard that a Canadian documentary called “Hold Your Fire” is in the works — it focuses on excessive use of police force in cases of mental illness, but the points of how people are treated when apprehended/whether to engage in the first place are relevant more generally.

Leave a Reply

Your email address will not be published.