Dale Lehman shares this article, “Police violence reduces civilian cooperation and engagement with law enforcement,” by Desmond Ang, Panka Bencsik, Jesse Bruhn, and Ellora Derenoncourt, which begins:
How do high-profile acts of police brutality affect public trust and cooperation with law enforcement? To investigate this question, we [look at] the ratio of police-related 911 calls to gunshots detected by ShotSpotter technology. Examining detailed data from eight major American cities, we show a sharp drop in both the call-to-shot ratio and 911 call volume immediately after the police murder of George Floyd in May 2020. . . . These effects persist for several months, and we find little evidence that they were reversed by the conviction of Floyd’s murderer. Together, the results illustrate how acts of police violence may destroy a key input into effective law enforcement and public safety: civilian engagement and reporting.
Lehman writes:
This is another in the long trend of interesting studies with many questions. The data and topic are interesting and of some importance, and the work seems competent enough. However, my first complaint is that the data is not provided – always worrisome to me, especially when I see questions I can’t answer without the data. In particular, the study focuses on the ratio of actual gunshots (subject to the adequately discussed caveats about measurement using the ShotSpotter data) to 911 calls. I am always hesitant to focus on ratios since the numerator and denominator are both moving. In this particular case, Figure A.I is the one that catches my attention. 911 calls mentioning “shots fired” actually increase after Floyd – but they don’t increase as much as the shots fired incidents. I’m not sure what to make out of this – whether it is a good measure of civilian cooperation. The overall 911 calls decrease (see the main graphs in the paper), but this suggests that it is other types of 911 calls that decrease. It would be nice to see this broken down by type of call (e.g., are the domestic disputes, robberies, or what?). It would also be nice to see Figure A.I for each of the 8 cities, rather than just the aggregate. These are the kinds of things that would be easy if the data were provided, but without that just leave me wondering. But, the author is at Harvard so I guess I should just trust him.
Lehman raises good questions, and I agree that it would help for the data to be available. Maybe some sort of multilevel model would make sense to manage the streams of data from different cities and different types of call, as they vary over time. Simple regression models of the sort fit in the above-linked paper can be fine as a starting point, but it’s good to dig deeper.
One problem when discussing research article is that there’s often a focus on a binary decision: accept/reject the journal submission, or if the paper has been published, it’s a like/dislike thing. We should be able to criticize a paper—in this case by asking for the data so that other researchers can look more deeply—without this implying that we’re trying to debunk it or shoot it down.
It’s all about division of labor. What it takes to collect these data, what it takes to think about looking at this comparison, and what it takes to get the most out of the data are three different skills. Open data allows us all to more forward more effectively.
I’d add that, like many important social questions, we shouldn’t rely just on a single set of data or study. As Dale says, this is an interesting and important question. The underlying theory also makes sense a priori: if I can’t trust the police not to shoot me or my friends, then I’m probably not going to risk bringing them into a situation by calling 911.
It just strikes me that in many aspects of social science, there’s a kind of “quantitative fever” where a study like this one is given more credence (at least in certain popular media) than, say, survey research on public attitudes or interviews with people from different communities. Of course, those other research methods aren’t the end-all either, my point is that understanding the issue requires integrating information from many different sources, not all of which might be big models of correlations in public data.
As a corollary, I suggest that while you’d learn a bit from trying out a bunch of different models and ways of operationalizing constructs using a single large open dataset, I’m not so sure that doing so really reveals much additional information about the underlying mechanisms (perceptions of police, how they form, and how people act on the basis thereof). You might get better descriptions of the structure in a particular dataset, but I think you’d learn more about potential causal factors by looking at entirely different methods and datasets. This is still pro-open science, obviously, since that’s how you enable integration across methods/data.
“Together, the results illustrate how acts of police violence may destroy a key input into effective law enforcement and public safety: civilian engagement and reporting.”
After a statement like that people are interested in the data? The infamous “may” is carrying far more than its usual unbearable weight. The statement isn’t a remotely plausible claim, regardless of the data. It’s as if nothing else were going on in society – like a massive pandemic and relentless public attacks on the police by leftish politicians. Sorry to say, stats people, no data will ever be able to substantiate this claim.
Even rigorous statisticians are still open to any kind of work that will pay even if it has zero hope of success – statistical ivermectin.
Without taking a position on the quality of the paper, I think the idea that it “isn’t a remotely plausible claim regardless of the data” sounds like pure ideology to me. Why wouldn’t widely reported police killings and other violent police misconduct make some people hesitant to call the cops? It makes ME hesitant to call the cops unless I’m pretty much certain that a real crime is being committed, and I’m an upper middle class white guy with a clean record.
“Police violence” (without any qualifier, just a free-floating universal) existed long before Floyd. Yet their analysis is of a single time point. Do you see the problem?
The question of “what are you measuring, exactly?” is raised by Andrew very often on this blog – just not in this post, apparently. I am reminded of a post I was reading just yesterday https://carcinisation.com/2020/01/27/ignorance-a-skilled-practice/ about how these sorts of dubious extremely narrow factoids get amplified and promoted into contextless universal platonic realities. Here we have a measurement of the effect of Black Lives Matters in a handful of specific Democratic cities in a single country in a single period in a single year after a single highly publicized event, which gets amplified into a universal “how acts of police violence may destroy a key input into effective law enforcement and public safety” conclusion, the even more simplified and factoid title “Police violence reduces civilian cooperation and engagement with law enforcement” (note the implicit quantifiers), which will get simplified even further when it gets tucked into a little “police violence discourages crime reports [44]” citation in future papers…
Dire:
Why do you say that the question of “what are you measuring, exactly?” is not being raised in this post? Questions of measurement are a central part of the post. Read it more carefully!
My interpretation of “may” in the sentence quoted above translates to, “is consistent with.” Nothing wrong with that–the results (methodological critiques aside) were in fact consistent with the experimental hypothesis. The results could’ve turned out inconsistent with it (again, assuming no forking paths or p-hacking), which hopefully would’ve resulted in a different conclusion (“may not” instead of “may”). So the study examines the hypothesis with falsifiability.
Yes, this is still only one set of circumstances, but that’s how science works: studies find or create a circumstance where a theory implies a prediction and test it, and those circumstances, predictions, and tests accumulate across studies in a way that tends to be consistent with the theory, or with a modification of it, or with its refutation. Information is gained, predictions are refined, but certainty is never achieved.
However, I fully agree with you that the title is careless in a way that’s both harmful and too common. There’s a pervasive perception among authors (and possibly reviewers and editors) that categorical statements sound more sciency and consequential. (They’re right, and that’s a problem.) Specifically, a more accurate title might read something like “Police violence against Floyd was associated with reduced 911 reports of gunshots across several urban cities.” That’s an inference, whereas the actual title is an opinion.
To be fair, my view, that scientific journals best serve science when they prioritize research conclusions over authors’ opinions, is itself only an opinion. But it happens to be consistent with the mission statement of most scientific journals.
“Why wouldn’t widely reported police killings and other violent police misconduct make some people hesitant to call the cops?”
You misunderstood my statement. The **idea** is not only plausible but certainly true in some larger sense. If the police go around harming innocent people as a standard practice, no one will want to call the police.
That’s not the issue. The issue here is *do these data show* that the Floyd incident specifically “destroy[ed] a key input into effective law enforcement and public safety”? I mean, here we are, in the begining of a pandemic, lock-downs, protests, all kinds of other things going on, and yet these authors claim that their data show a specific association.
As far as I’m concerned, unless they’re going to catalog and control for every event in every city, they don’t have a hope of demonstrating such an association.
The reason I call this “statistical ivermectin” is because statistics can’t ever show this claim is true or not true, because it can’t control effects of the myriad other events that occurred during this time period.
But in addition to that, there are several faulty claims embedded within the premise of the measurement. ”
First, “shots fired” isn’t the only kind of crime, is it? So is this is a valid proxy for “things the police need to be called for” or just a wild stab in the dark?
(In *physical science* people test the validity of a proxy before applying it – for example, there is ample work testing how stable isotopes are partitioned by natural processes and conditions, which allows stable isotope measurements to be used as a proxy for those processes. People don’t do that in social sciences because it’s not even possible to test the validity of the proxy, much less use in an application. Better to just grab some paint and throw it at the wall.)
Second, there have been **numerous** Floyd-like incidents that sparked large protests over the last ten years: Eric Garner, Trayvon Martin and Michael Brown and off the top of my head. Why would the Floyd incident have any more impact than the others? What makes this the sensible “test case”? The paper only refers to these widely known incidents in the context of police accountability, seemingly oblivious to the fact that these other incidents offer another test of its primary hypothesis. I mean: HUH? No mention at all? Since this particular incident happened contiguously with a once-in-a-hundred-years pandemic, its in very serious question immediately as a test case! To me the Floyd case is the worst possible choice. Why not do the obvious and test the other cases? And what’s the psychological effect of a continuous stream of cases?
Third, in light of the light discussion of previous cases in terms of police accountability, the paper seems oblivious to the fact that, one reason police might not be “held accountable” is that they might not have committed a crime! For example, three separate District Attorneys, two of which were black, have declined to bring charges against the officer that shot Michael Brown. That seems to suggest that he did not commit a crime, so there’s nothing to be held accountable for.
Fourth, I recall a news story in summer 2020 where a woman form Minneapolis complained that crime was everywhere and there were just flat out no police anywhere to be found. So the entire premise of this paper could be turned on it’s head: it could be that people didn’t call the police because they had disappeared from the streets. How did this study control for the **police response** to the Floyd incident? Not at all. In that sense, this paper makes the common assumption in social sciences that it’s hypothesis is the only possible hypothesis – an erroneous assumption.
So as usual it’s a disaster already and we haven’t even looked at the data (much less the statistics) which by any measure is hardly convincing.
But what’s funny is the introductory sentence to the discussion:
“Together, our results provide novel insight into the deleterious effects that high-profile acts of police violence may have on civilian crime reporting”
Conclusions drawn before the discussion!
1) (Fig IIa) there’s an obvious ongoing decline in the call to shot ratio before the Floyd incident, so how this can be ignored in the discussion and conclusions is totally beyond me.
2) (Fig IIb) there is a steady rise in shots detected preceding the “Floyd Boundary” so – what? Where’s the explanation?
3) (Fig IIb) Note the difference in scale for calls (0-50% change) and shots (0-200% change)!!! Since this is % change, why are the scales different? Because the change in calls is ***small*** compared to the change in shots. Imagine the “call” curve flattened to 25% its current height. It would be practically flat! Hilarious, who are the crooks who published this again??
I just noticed #3. That convinced me. This is junk science. But I shall persist in utterly devastating this garbage “research”.
4) (Fig IIb) the shots have a distinct rise at the “Floyd Boundary” but the calls don’t. They bump all over the place before and after the boundary. There’s also a big decline in calls at the “national emergency” boundry, with no concomitant rise in shots. Why? Sorry but these data are **FAR MORE** consistent with “the police disappeared” cause than the “Floyd’s death caused people to stop cooperating” cause. The pattern in shots is strong. The pattern in **calls** is very weak.
5) now to the city data (Figure III) which doesn’t support the thesis at all. In Minneapolis, where the event in question occurred, CPS rises to the “Floyd Boundary”, then falls more or less back to the earlier levels. Same for Richmond and San Diego. In fact most cities exhibit a high point in CPS *before* the “Floyd Boundary”. So what the falling CPS means **after** the “Floyd Boundary” is hardly clear.
6) which brings up the other obvious pink elephant: what about previous ***NON PANDEMIC*** years, (and previous “Floyd Like” events)? Sorry, but without showing solid year of pre-pandemic background, and most data in this paper showing an upside spike prior to the The reason I call this “statistical ivermectin” is because statistics can’t ever show this claim is true or not true, because it can’t control effects of the myriad other events that occurred during this time period.
But in addition to that, there are several faulty claims embedded within the premise of the measurement. ”
First, “shots fired” isn’t the only kind of crime, is it? So is this is a valid proxy for “things the police need to be called for” or just a wild stab in the dark?
(In *physical science* people test the validity of a proxy before applying it – for example, there is ample work testing how stable isotopes are partitioned by natural processes and conditions, which allows stable isotope measurements to be used as a proxy for those processes. People don’t do that in social sciences because it’s not even possible to test the validity of the proxy, much less use in an application. Better to just grab some paint and throw it at the wall.)
Second, there have been **numerous** Floyd-like incidents that sparked large protests over the last ten years: Eric Garner, Trayvon Martin and Michael Brown and off the top of my head. Why would the Floyd incident have any more impact than the others? What makes this the sensible “test case”? It happened contiguously with a once-in-a-hundred-years pandemic, which puts it in very serious question immediately as a test case! To me the Floyd case is the worst possible choice. Why not do the obvious and test the other cases? And what’s the psychological effect of a continuous stream of cases?
Third, I recall a news story in summer 2020 where a woman form Minneapolis complained that crime was everywhere and there were just flat out no police anywhere to be found. So the entire premise could be wrong: it could be that people didn’t call the police because they disappeared from the streets. How did this study control for the **police response** to the Floyd incident? Not at all.
So that’s just a disaster and we haven’t even looked at the data yet, which by any measure is **unconvincing**. My first complaint is that I hate aggregated data. Show us all the data so we can see the spread graphically.
But as for the data itself:
1) (Fig IIa) there’s an obvious ongoing decline in the call to shot ratio before the Floyd incident, so how this can be ignored in the conclusions is totally beyond me.
2) (Fig IIb) there is a steady rise in shots detected preceding the “Floyd Boundary” so – what? Where’s the explanation?
3) (Fig IIb) Note the difference in scale for calls (0-50% change) and shots (0-200% change)!!! Since this is % change, why are the scales different? Because the change in calls is ***small*** compared to the change in shots. Imagine the “call” curve flattened to 25% its current height. It would be practically flat! Hilarious, who are the crooks who published this again??
I just noticed #3. That convinced me. This is junk science. But I shall persist in utterly devastating this garbage “research”.
4) (Fig IIb) the shots have a distinct rise at the “Floyd Boundary” but the calls don’t. They bump all over the place before and after the boundary. There’s also a big decline in calls at the “national emergency” boundry, with no concomitant rise in shots. Why? Sorry but these data are **FAR MORE** consistent with “the police disappeared” cause than the “Floyd’s death caused people to stop cooperating” cause. The pattern in shots is strong. The pattern in **calls** is very weak.
5) now to the city data (Figure III) which doesn’t support the thesis at all. In Minneapolis, where the event in question occurred, CPS rises to the “Floyd Boundary”, then falls more or less back to the earlier levels. Same for Richmond and San Diego. In fact most cities exhibit a high point in CPS *before* the “Floyd Boundary”. So what the falling CPS means **after** the “Floyd Boundary” is hardly clear.
6) which brings up the other obvious pink elephant: what about previous non-pandemic years, (and previous “Floyd Like” events)? Sorry, showing only a month of pre-pandemic data – with a spike upward before the “Floyd Boundary”, meaning that the fall after the “Floyd Boundary” is could easily be a reversion to normal – makes this paper just not even close to demonstrating, showing or even suggesting anything about public confidence in police.
This is just a bunch of futzing around with R and mindlessly generating metrics and plots. It’s not science at all. I can’t really speak to the stats, but as usual that’s not necessary because the science is poor.
It’s really important to have the data available, and the title is too strong, but this paper is one of many that look at the relationship between community experience with/trust of the police (or criminal justice system more broadly) and their willingness to report crime. There are qualitative and quantitative papers, they look at geographic data, time series data, survey data, different kinds of crime, different kinds of neighborhoods, single instances (like Floyd) and pervasive ongoing practices (like stop and frisk). Likewise the impact of changing practices to increase trust has also been studied, sometimes with RCT. You may not like it, but it’s got pretty wide and consistent support.
>accept/reject the journal submission, or if the paper has been published, it’s a like/dislike thing. We should be able to criticize a paper—in this case by asking for the data so that other researchers can look more deeply—without this implying that we’re trying to debunk it or shoot it down.
It’s even possible that one table in the paper has errors, but everything else is fine!