Brandon Del Pozo writes:
Born in Bensonhurst, Brooklyn in the 1970’s, I came to public health research by way of 23 years as a police officer, including 19 years in the NYPD and four as a chief of police in Vermont. Even more tortuously, my doctoral training was in philosophy at the CUNY Graduate Center.
I am writing at the advice of colleagues because I remain extraordinarily vexed by a paper that came out in 2021. It purports to measure the effects of opening strip clubs on sex crimes in NYC at the precinct level, and finds substantial reductions within a week of opening each club. The problem is the paper is implausible from the outset because it uses completely inappropriate data that anyone familiar with the phenomena would find preposterous. My colleagues and I, who were custodians of the data and participants in the processes under study when we were police officers, wrote a very detailed critique of the paper and called for its retraction. Beyond our own assertions, we contacted state agencies who went on the record about the problems with the data as well.
For their part, the authors and editors have been remarkably dismissive of our concerns. They said, principally, that we are making too big a deal out of the measures being imprecise and a little noisy. But we are saying something different: the study has no construct validity because it is impossible to measure the actual phenomena under study using its data.
Here is our critique, which will soon be out in Police Practice and Research. Here is the letter from the journal editors, and here is a link to some coverage in Retraction Watch. I guess my main problem is the extent to which this type of problem was missed or ignored in the peer review process, and why it is being so casually dismissed now. It is a matter of economists circling their wagons?
My reply:
1. Your criticisms seem sensible to me. I also have further concerns with the data (or maybe you pointed these out in your article and I did not notice), in particular the distribution of data in Figure 1 of the original article. Most weeks there seem to be approximately 20 sex crime stops (which they misleadingly label as “sex crimes”), but then there’s one week with nearly 200? This makes me wonder what is going on with these data.
2. I see from the Retraction Watch article that one of the authors responded, “As far as I am concerned, a serious (scientifically sound) confutation of the original thesis has not been given yet.” This raises the interesting question of burden of proof. Before the article is accepted for publication, it is the authors’ job to convincingly justify their claim. After publication, the author is saying that the burden is on the critic (i.e., you). To put it another way: had your comment been in a pre-publication referee report, it should’ve been enough to make the editors reject the paper or at least require more from the authors. But post-publication is another story, at least according to current scientific conventions.
3. From a methodological standpoint, the authors follow the very standard approach of doing an analysis, finding something, then performing a bunch of auxiliary analyses–robustness checks–to rule out alternative explanations. I am skeptical of robustness checks; see also here. In some way, the situation is kind of hopeless, in that, as researchers, we are trained to respond to questions and criticism by trying our hardest to preserve our original conclusions.
4. One thing I’ve noticed in a lot of social science research is a casual attitude toward measurement. See here for the general point, and over the years we’ve discussed lots of examples, such as arm circumference being used as a proxy for upper-body strength (we call that the “fat arms” study) and a series of papers characterizing days 6-14 of the menstrual cycle as the days of peak fertility, even though the days of peak fertility vary a lot from woman to woman with a consensus summary being days 10-17. The short version of the problem here, especially in econometrics, is that there’s a general understanding that if you use bad measurements, it should attenuate (that is, pull toward zero) your estimated effect sizes; hence, if someone points out a measurement problem, a common reaction is to think that it’s no big deal because if the measurements are off, that just led to “conservative” estimates. Eric Loken and I wrote this article once to explain the point, but the message has mostly not been received.
5. Given all the above, I can see how the authors of the original paper would be annoyed. They’re following standard practice, their paper got accepted, and now all of a sudden they’re appearing in Retraction Watch!
6. Separate from all the above, there’s no way that paper was ever going to be retracted. The problem is that journals and scholars treat retraction as a punishment of the authors, not as a correction of the scholarly literature. It’s pretty much impossible to get an involuntary retraction without there being some belief that there has been wrongdoing. See discussion here. In practice, a fatal error in a paper is not enough to force retraction.
7. In summary, no, I don’t think it’s “economists circling their wagons.” I think this is a mix of several factors: a high bar for post-publication review, a general unconcern with measurement validity and reliability, a trust in robustness checks, and the fact that retraction was never a serious option. Given that the authors of the original paper were not going to issue a correction on their own, the best outcome for you was to either publish a response in the original journal (which would’ve been accompanied by a rebuttal from the original authors) or to publish in a different journal, which is what happened. Beyond all this, the discussion quickly gets technical. I’ve done some work on stop-and-frisk data myself and I have decades of experience reading social science papers, but even for me I was getting confused with all the moving parts, and indeed I could well imagine being convinced by someone on the other side that your critiques were irrelevant. The point is that the journal editors are not going to feel comfortable making that judgment, any more than I would be.
Del Pozo responded by clarifying some points:
Regarding the data with outliers in my point 1 above, Del Pozo writes, “My guess is that this was a week when there was an intense search for a wanted pattern rape suspect. Many people were stopped by police above the average of 20 per week, and at least 179 of them were innocent. We discuss this in our reply; non only do these reports not record crimes in nearly all cases, but several reports may reflect police stops of innocent people in the search for one wanted suspect. It is impossible to measure crime with stop reports.”
Regarding the issue of pre-publication and post-publication review in my point 2 above, Del Pozo writes, “We asked the journal to release the anonymized peer reviews to see if anyone had at least taken up this problem during review. We offered to retract all of our own work and issue a written apology if someone had done basic due diligence on the matter of measurement during peer review. They never acknowledged or responded to our request. We also wrote that it is not good science when reviewers miss glaring problems and then other researchers have to upend their own research agenda to spend time correcting the scholarly record in the face of stubborn resistance that seems more about pride than science. None of this will get us a good publication, a grant, or tenure, after all. I promise we were much more tactful and diplomatic than that, but that was the gist. We are police researchers, not the research police.”
To paraphrase Thomas Basbøll, they are not the research police because there is no such thing as the research police.
Regarding my point 3 on the lure of robustness checks and their problems, Del Pozo writes, “The first author of the publication was defensive and dismissive when we were all on a Zoom together. It was nothing personal, but an Italian living in Spain was telling four US police officers, three of whom were in the NYPD, that he, not us, better understood the use and limits of NYPD and NYC administrative data and the process of gaining the approvals to open a strip club. The robustness checks all still used opening dates based on registration dates, which do not associate with actual opening in even a remotely plausible way to allow for a study of effects within a week of registration. Any analysis with integrity would have to exclude all of the data for the independent variable.”
Regarding my point 4 on researchers’ seemingly-strong statistical justifications for going with bad measurements, Del Pozo writes, “Yes, the authors literally said that their measurement errors at T=0 weren’t a problem because the possibility of attenuation made it more likely that their rejection of the null was actually based on a conservative estimate. But this is the point: the data cannot possibly measure what they need it to, in seeking to reject the null. It measures changes in encounters with innocent people after someone has let New York State know that they plan to open a business in a few months, and purports to say that this shows sex crimes go down the week after a person opens a sex club. I would feel fraudulent if I knew this about my research and allowed people to cite it as knowledge.”
Regarding my point 6 that just about nothing ever gets involuntarily retracted without a finding of research misconduct, Del Pozo points to an “exception that proves the rule: a retraction for the inadvertent pooling of heterogeneous results in a meta analysis that was missed during peer review, and nothing more.”
Regarding my conclusions in point 7 above, Del Pozo writes, “I was thinking of submitting a formal replication to the journal that began with examining the model, determining there were fatal measurement errors, then excluding all inappropriate data, i.e., all the data for the independent variable and 96% of the data for the dependent variable, thereby yielding no results, and preventing rejection of the null. Voila, a replication. I would be so curious to see a reviewer in the position of having to defend the inclusion of inappropriate data in a replication. The problem of course is replications are normatively structured to assume the measurements are sound, and if anything you keep them all and introduce a previously omitted variable or something. I would be transgressing norms with such a replication. I presume it would be desk rejected.”
Yup, I think such a replication would be rejected for two reasons. First, journals want to publish new stuff, not replications. Second, they’d see it as a criticism of a paper they’d published, and journals usually don’t like that either.
From what I’ve seen in these links I don’t think a retraction is appropriate. But I also appreciate the quandary of how to write a comment – the idea that new (more appropriate) data be collected as a replication attempt seems to put a huge burden on Del Pozo – there may not even exist the data needed to examine this question. I think Del Pozo should take the editors up on their offer – write a comment about the fact that the paper suffers from a lack of construct validity. The editors may well reject that, but in my opinion it would be an appropriate comment to publish. If it were rejected as not providing enough new results, then I think criticism of the journal would be warranted. I am also curious about whether the referees raised any of these issues, but that is another whole set of standard practices that could be debated.
Dale:
I’d say that the substantive claim of the original paper should be retracted, in the sense that I don’t see any good evidence supporting that claim. But I agree with you that it doesn’t make sense for the paper to be retracted, because, in practice, retraction of a paper is pretty much only done if (a) there’s strong evidence of research misconduct, or (b) the authors want to retract.
I have the same feeling about the papers on beauty-and-sex-ratio or governor-election-and-life-span: the claims should be retracted, but I would not expect the papers to be retracted, given that there’s no reason to think there was any scientific misconduct (just bad statistical analysis, as explained here and here, but that doesn’t count as misconduct, as these are tricky statistical problems) and the original authors still think their claims are correct.
One of the things the journal says it entertains are letters of concern that travel with the paper. I see your point about a retraction, and I have come to think your points are reasonable as far as the entire article is concerned. But the study’s principal conclusion shouldn’t be presented as reliable knowledge. I’d argue the authors have a duty to see to that.
Thanks for providing some much careful thought and context.
Here is our critique in print: https://www.tandfonline.com/doi/full/10.1080/15614263.2023.2253350
> But I agree with you that it doesn’t make sense for the paper to be
> retracted, because, in practice, retraction of a paper is pretty much
> only done if (a) there’s strong evidence of research misconduct, or (b)
> the authors want to retract.
If a journal is going to make that the requirement for a retraction, then they should provide something else to indicate the papers that are incorrect.
It took 2.5 years for this to get published:
Is that normal for things to get published in this journal? It seems to me that delay implies quite a bit of back and forth. Ie, the editor believes it was vetted pretty harshly.
The data issue here reminds me of a paper I reviewed about salmon in the Central Valley of California that used US Fish and Wildlife estimates of the number of fish spawning in the various Sacramento/San Joaquin river tributaries. The problem is that the agency thought that a law called the Central Valley Project Improvement Act required them to make these estimates, even though there were no field data to base them on. I explained this in my review, and directly to the authors. The paper was turned down, and the authors got it published elsewhere. (There is a different data set covering a smaller number of streams that is based mainly on counts of dead fish — it has its own problems, but at least it is based on something.)
John,
I want to make sure I understand this. They (USFW) provided estimates of the number of fish spawning in various tributaries, in spite of there being literally no field data to base these estimates on? I know all I’m doing is repeating what you said; I just find it so hard to believe that if that’s really what you meant to say, could you confirm it? That’s crazy.
Phil,
It is crazy. Thinking about it more, I think it was the Bureau of Reclamation instead of the USFWS, but here is the story. The Central Valley Project Act (CVPIA) required something like “all reasonable efforts” to double the number of naturally spawning Chinook salmon as of some date I don’t remember (say 1980). However, nobody knew what that number was, so a couple of USBR biologist sat down with a list of all the tributaries and made their best estimates, aka made up numbers for the streams with no data. This got published in a draft report that was never finalized (so no need to respond to comments), but those were the basis for the official target. Then, to track progress toward the target, they needed numbers for these streams every year, even though there was still no field surveys on those streams. Everybody in the business understood that those were bogus numbers, so we didn’t use them. Probably you can still find those data on the web. As said in my review, riffing off an old Jewish joke about fish, those data are for reporting, not for analyzing.
I wonder how common this is in govt data in general. Crime, library utilization, public transport, education etc
it’s very interesting to read the cops’ insider perspective on the data and its limitations, but i don’t see the case for retraction. it seems like the paper provides fairly strong evidence despite the limitations of the data
Del Porzo claims that due to measurement error the date of opening of a strip club is measured imprecisely (it only measures the date registered, which might be weeks or months before it actually opens) and this therefore invalidates the analysis
Andrew agrees, and writes that “the substantive claim of the original paper should be retracted, in the sense that I don’t see any good evidence supporting that claim”
i’m curious how Andrew and Del Pozo explain the event study figure 5 of the original paper. if event time = 0 in the figure just represents some arbitrary date with no logical connection to the opening of the strip club, why do we observe a large and sudden decline right around it?
as Del Porzo and coauthors write “What the study has done is measure changes in police encounters with innocent people in the week after an entity has filed the paperwork necessary to apply for business licenses that will eventually allow it to open a strip club”. if this were the case, we would expect to see a null effect, since these are two totally unrelated outcomes. maybe the measurement error problem is not as poor as the cops believe it is? what’s the omitted variable that’s correlated with both the timing of paperwork filing and declines in stop and frisk measure?
Andrew’s claim that “I am skeptical of robustness checks” seems to avoid engaging with any of the robustness checks offered in the actual paper. it would be very convenient for critics of research if they could simply ignore all auxiliary analyses and focus on one single regression
Andrew also writes that “The short version of the problem here, especially in econometrics, is that there’s a general understanding that if you use bad measurements, it should attenuate (that is, pull toward zero) your estimated effect sizes; hence, if someone points out a measurement problem, a common reaction is to think that it’s no big deal because if the measurements are off, that just led to “conservative” estimates.” it is of course (and, i would add, taught in most econometrics courses) that in general arbitrary forms of measurement error don’t always bias estimates toward zero, but it seems like the specific form of measurement error that Del Pozo and coauthors are alleging would bias towards zero. am i missing something? ignoring the magnitude of effects, the timing of the effects are also quite stark
If Del Pozo and his colleagues are correct, it’s not a matter of measurement error. It’s a matter of not having any measurement at all of sex crimes or of when strip clubs open. And if you don’t have any measurement of these things, then you don’t have a paper that tells you how these things are related and you most definitively don’t have a paper that tells you how they are causally related.
So why do they have significant results? Either because 1) the garden of forking paths or 2) because police stops are weirdly related to when people get certain licenses (which they can’t use to open a strip club until they get a lot of other licenses.) My money would be on 1).
No one who knows about crime data would use police stops as a measure of anything other than the number of police stops. And that, in turn , is a function of things happening right then (e.g. the search for a suspect in a particular case) and public policy decisions, whether about where to deploy police resources, quotas given to police officers, political power of residents or commercial interests in the area. I read a lot of this exchange previously, and the thing that jumped out was the original authors’ total unfamiliarity with the vast amount of work that has been done in this area by scholars who are not economists.
The spatial distribution of crime, the impact of things like opening of new businesses (of any type), changes in police presence, community willingness to report issues are all widely studied. Not to mention, of course, the race, gender and other characteristics whose association with stops have been widely discussed.
Through the course of my career I used to fill out these reports, supervise the people who fill out these reports, evaluate these reports, and use these reports in analyses. They have nothing at all to do with the incidence of crime at the precinct level. There was a long time during the study period where police officers were given overtime to find reasons to write these stop reports. Using them as a measure of crime for a study is crazy.
How much “measurement error” is acceptable for a paper to not be retracted? The issues of measurement error on the IV and DV are suggested by the authors, but they are still underplaying the issue and not even looking in the right direction:
These are the measurement issues they acknowledge in the paper
On the IV: “measurement error in the explanatory variable might arise if these businesses are not registered in the Reference USA database. In this case, assuming that this measurement error is random would lead to attenuation”
The authors on the DV: “On the one hand, measurement error in the former could easily arise if we do not observe all the sex crimes committed in NYC. Measurement error is an issue in every crime
data set, and even more in data related to sex crimes. Measurement error in the crime economics
literature is mostly due to victims choosing not to report the crime (especially sex crimes). Thus,
we believe using the ‘stop-and-frisk’ data set minimises this concern since victims do not decide
whether or not to report the crime.”
They don’t even bring up anything that suggests they are thinking about how valid their measurements are at all, but rather they are only interested in how “fuzzy” their data are compared to the actual true concepts. I don’t want to reiterate the point of Mosker et al, but the points about construct validity are fatal in my opinion, but somehow not in the minds of many economists.
To me this is the same as Naomi Wolf misinterpretting “death recorded” in 19th century English court data as a death sentence rather than a pardon, which it actually was. The difference is the economist can point to their model still printing out the “right” result, so they can rest assured that the measurement was close enough. While someone like Wolf’s work is seen as being mostly her interpretation of relatively simple data, so when that data is shown to be invalid the whole house of cards falls down. But really that same logic should apply to the economists as well. But the choices and interpretations are more hidden and unacknowledged in quantitative social science.
On Naomi Wolf: https://nymag.com/intelligencer/2019/05/naomi-wolfs-book-corrected-by-host-in-bbc-interview.html
Thank you for stating this so clearly. I diverted a fair amount of time from my own research–in another field–to tackle this, and so there are limits to how much time we could spend on it. I appreciate these quotes from the paper.
Brandon:
Yeah, I just looked again at the letter from the editor of the Economic Journal. He wrote:
Yes, there is information in the data. What the editor missed is that the information in the data does not address the questions that the paper purported to answer.
For the editor to call this “imprecise measurement” is to miss the point. If I measure your weight on a noisy scale, that’s imprecise measurement. If I use your weight on a noisy scale to measure your height, that’s not imprecise measurement, it’s measuring the wrong thing.
P.S. I went to the editor’s homepage where it says, “My research interests concern macroeconomic dynamics and monetary economics.” Economists who work on macro have to be very aware of measurement problems. I remember this from taking an economics class in high school: the difficulty of measuring the money supply and the velocity of money. So it’s kinda funny that he was so cavalier about the measurement problems in the strip-club paper.
I think he got fooled by that whole robustness-checks thing. All journal editors in econ should be required to read this post from Uri Simonsohn.
In philosophy this is called ontological problem, winch is much different than an agreement on ontology and a difference in measurement. I must have tried to explain this to the editors no less than four times. This is the passage from my final email to the ed board of the journal:
“We are not pointing to noisy or imprecise measurements that would respond to a replication or additional analyses, we are saying there was an ontological problem with data selection that introduced a fatal category error from the outset. Peer review should have caught this, given how obvious it would be to a qualified reviewer. Unfortunately, As Dr. Gelman notes, there isn’t a ready means in the research process to address this concern post publication, so your response seems in keeping with prevailing norms. Hopefully this provides us with an opportunity for improvement.”
Sam:
You’re making the common error of taking a rejection of null hypothesis A as evidence for a favored alternative hypothesis B.
To put it another way, the data in the paper don’t measure what the paper says they measure, but they aren’t random numbers either. So, no, it’s not correct that “we would expect to see a null effect.” Just look at the time series in the paper of data on stops. There are some big spikes there.
Regarding the idea that noise should bias estimates toward zero: yes, but you’re forgetting the selection bias (“garden of forking paths”) whereby the large estimates are what are reported. Eric Loken and I discuss this in our paper from 2017.
Figuring out all the logical fallacies is like a brain teaser.
They seem to say the rate is 0.0313 per day, so a difference of 13% is 11 vs 10 police reports per year. I have more to say, but want to make sure that is correct first.
Well, I’ll say that 13% is based on a coefficient from an arbitrary model using only whatever data was available. So even if they did actually measure sex crimes and strip club debuts, they still can’t conclude what they did. If the same model also fits future data, then maybe it is worth figuring out why the police reporting correlates with strip club registration.
For example, when you want to open a strip club (which, admittedly, I have never done) I would think at some point you consider whatever “problems” are in that area that may threaten your business.
Andrew,
Very glad to see this paper discussed. I read it about half a year ago and thought “Now, that’s something that’s probably way too noisy to estimate.”
I’d disagree with one of your comments though. I do think there’s something here about this being a paper in economics. It seems to me that economists of a certain stripe (perhaps more than most people in most disciplines) are very happy to apply their methods for causal analysis to domains they’re not very familiar with. Inevitably, you end up with results like this.
In addition to my comment above, I’d also note that the date license in question is not a matter of classical measurement error but rather a biased measurement as the opening of the strip club (if it does eventually open) will always fall after the date of the license.
This.
There are lots of studies of the spatial distribution of crime, and that work involves intensive cleaning and thinking at the very detailed level. In NYC that would normally mean Census block groups. Then, you would spend a lot of time looking at how the block groups do or do not align with the precinct boundaries (in NYC this is overall pretty good). Then control for other geographic factors that are also associated with level of crime. Then control for social factors that are correlates of crime. Then add a spatial correlation structure.
One amusing thing that they say is that “women may move out of a precinct” if one of these establishments opens. Do they have any idea what that means? Precincts in NYC are huge, larger than many major cities, both in population size and geographic area. If those businesses were in residential areas (they are not, usually), you could move miles away and still be in the same precinct. The only exceptions would be the precincts that cover midtown Manhattan, which historically has a low resident population but huge day time population.
I don’t actually doubt that the opening of a new business in a place where there was no business before (e.g. unused industrial spaces, unused entertainment/restaurant spaces) does change the street level dynamics, from what prostitutes do to how police operate. And it could be that these businesses have a specific kind of impact. But I would predict it would be on prostitution, open street level drug use, theft and robbery (of customers). Some it would increase, some it would decrease. If they have lights on outside and bouncers and increase pedestrian traffic they might be associated with a decrease. Interesting empirical question.
The thing that bothers me is that they are using this to advance an argument that “rape is simply a substitute for consensual sex.”
And then they add something something about prostitution. Because prostitution is also a substitute for consensual sex? Or something?
Besides all of the problems with their paper as a study of the impact of specifically opening a “adult entertainment establishments” on street level police stops for “sex crimes” near by, much much worse is arguing that this is because the customers of those establishments would otherwise have been raping women on the streets in the immediate area of the establishments.
It also claims that ” crimes observed by the police” removes some kind of underreporting, but where in the literature on sexual assault does it say that leap makes sense? How many sexual assaults are directly observed by police? Who would argue for measuring the prevalence of sexual assault in this way?
Aside from the dodgy data, this study uses the awful strategy of applying OLS regression to the log of 1 plus the rate, a really common and completely awful thing in economics. It’s a little better than logging the rate knowing there are lots of zeros (also a common problem in economics) I guess, but why not just apply a Poisson regression to the counts? My guess is that would have put the whole issue to bed, because all the results would be insignificant.
They’ve basically taken every week when there were no offenses, and recoded them to have a single offense. Madness.
There are so many things wrong with the way econometricians and economists do stats, long before we get to the details of whether the things they actually measured are valid measurements.
Faustusnotes:
Use negative binomial regression. Never Poisson. See chapter 15 of Regression and Other Stories for explanation of this point.
It annoys me even if there’s no or very few zeros. And so completely unnecessary given the tools available to us – it’s not like running a negative binomial or Poisson regression takes the analyst any more or less time than doing OLS!
Yes this is one of the oddest things about the supposedly super sophisticated economists Even when I started grad school in the social sciences more than 40 years ago, when glm was not so easily available, we knew this was an issue.
In this case, there is a massive difference between 0 and 1. Among other things, as discussed in other comments, a 0 probably means no police presence on a routine basis and not much in the way of street traffic of the sort that allows for stop and frisk.
Haha Jeffrey Woolridge is out there telling everyone to always use Poisson regression
https://x.com/jmwooldridge/status/1603433022635876355?s=20
Somebody:
That’s not my experience at all. If someone could point me to his specific examples where negative binomial regression doesn’t work, I’d be interested in seeing them. I’m open to learning more on this one.
I’m not able to find any reasoning that goes beyond your standard economist reasoning (that is to say, nonsense).
Oooh, it’s asymptotically consistent, ooh. Variance of the estimator? Who cares? Uncertainty quantification? Well I can click on the robust standard errors button in stata
Somebody:
I haven’t seen the data examples but let me take a stab at what might be going on here.
In practice, the big difference between Poisson and negative binomial regression is that Poisson has a fixed variance and neg bin has a variance estimated from data. If you have overdispersion (which in my experience happens just about all the time in practice), Poisson regression will give standard errors that are too small.
So what we have often recommended is to do “overdispersed Poisson regression,” which can have different meanings. One way to go is to fit the negative binomial model and estimate the overdispersion parameter. Another way is to fit the Poisson regression, then use the standard deviation of the standardized residuals to compute an overdispersion factor, and inflate the standard errors accordingly. I’m sure there are other approaches too.
If you’re working in a Bayesian context, or embedding all this in a multilevel model, it will be good to have a generative model: trying to hack things with overdispersion corrections is a mess. So in Regression and Other Stories we just tell people to fit the negative binomial model right away.
If you’re not working in a Bayesian or multilevel or predictive context, and if you were planning to do some standard error correction anyway, then it might be fine to just fit Poisson and go from there. I don’t really know. In any case, I’d still like to see those examples where negative binomial regression fails. It could be that more stability is needed in the estimate of the hyperparameter.
Finally, all this reminds me of how much I hate twitter. In a blog, that dude could’ve given his full explanation. But in twitter he only has space to mention some points without any details.
hi Andrew, all,
may I ask, even if the problem is the registration date is wrong, would a permutation on the opening date help?
in the paper, they permuted on the number of openings (I dont get that), but if they randomized the opening date/registration date, would that address the concern?
after all, even if its true the registration date is not the same as the start of the business, but if we randomize the start date and randomization finds smaller effects, isn’t that evidence that **something** is happening in the week of registration?
Gabby:
There can be evidence that something is happening, but that “something” is connected to stops, not to sex crimes.
Stop reports were foremost driven by a desire by police officers to appear productive, irrespective of actual crime in the community. The threshold for them was the lowest for all of the activities’ they could perform that indicated productivity. No arrests, no other paperwork, no court appearances, no people or property to transport or process, just filling out a form about the size of a large index card because you purportedly suspected criminal activity did, or was about to, take place, stopped a person to investigate, and were wrong in your suspicions about 94% of the time.
If there was a wanted rapist in your borough and a man on the street broadly matched his description, if you stopped him and asked him for his name, that would generate a stop report which the authors here would then measure as a sex crime. If another officer a mile away stopped a person for the same reason the next day, that would be recorded as a second sex crime. If you saw a man who might have been following two women dressed for a night out on the town and asked for his name, that would generate a report of an investigation into what could have escalated to forcible sexual touching, and the authors would measure that as a sex crime.
Officers were paid overtime to patrol (mostly poor, minority) areas in the city and the lowest threshold way to show they were delivering value was for each of them to write 4-8 of these reports a night. Nobody would ever know the extent to which the reports were based on anything the barest of pretenses. And even if they weren’t, almost all of them in fact concluded that the person was innocent. This was all hashed out in a federal civil rights lawsuit the NYPD lost for these very reasons. This is what makes the premise of the study here ridiculous, apart from the fatal problem of business registration dates.
Personally, I’ve been told that some NYPD officers (always “I would never, but some of the other guys”) write completely made up tickets and reports during graveyard OT to look like they did something
This comment addresses the comment below, since I can’t directly reply to it. A made up ticket starts a court process where the ticket becomes the principal legal affidavit in the proceedings, and if a defendant doesn’t adjudicate it, it has legal penalties for a person’s license, etc. A stop report, on the other and, initiates nothing and never goes anywhere except black holes where economists and the ACLU occasionally use it as part of aggregate analyses; because the report names an innocent person, unless they sue or something, it never becomes part of a court process (and people who were never even stopped but just had their name borrowed aren’t even going to know this happened). If you were a police officer looking to fabricate activity with minimal risk, which one would you choose? I truly think thousands upon thousands of stop reports were populated with names taken off apartment building mailboxes, or by recycling the identities of people in the neighborhood with serious criminal records.
Neverminded, I see you type a comment above a reply and it posts below the reply. Thanks for bearing with me.
Does anybody else find these descriptions of police behavior disturbing? I’m not really naive – I don’t ascribe accuracy and truth to police reports. At the same time, police are professionals and I do believe, as with most professions, there is some unwritten code of conduct that applies. I’m sure there are some “stop” reports that are totally fabricated, particularly when monitoring is difficult and penalties for lying are low. But that does not automatically mean that most such reports are false. It seems to me that there is a leap taken here from “these reports are not reliable” to “these reports are worthless.” Perhaps we need to get those dishonesty researchers to work on this issue.
Dale,
Based on the people I know personally, it seems like like most people, most police officers are honorable by instinct and don’t fabricate reports, but know one or two officers who probably do, and also know that those officers are never caught. My suspicion is that:
1. The minority of officers that fabricate can be responsible for a lot of fabrication
2. It is nobody’s problem to catch them. Based on del Pozo’s response it seems there is no auditing process, and like with insurance no one involved is cost anything.
In any case, fabrication is only issue; it seem del Pozo pointed out that even legitimate report densities varies in a heavy tailed way with police strategy and focus rather than crime density, so in any case it fails construct validity.
Now, if you want to be disturbed, overtime scams seem to be accepted by everyone as just the thing to do. Not “scam” as in, break written laws, but allocating and doing as much bogus OT as possible for no reason just to burn through as much city budget as possible, knowing that it’s pointless, seems to be not even considered dishonorable up and down the chain of command.
@Del Pozo
I was told one way of doing it would be to pull the names from license plates off cars parked in the area
Then the relevant question is: does the police officer sign the report at the top or the bottom of the form?
Dale ftw.
Every officer I worked with or supervised was scrupulous about things that brought people to court or imposed a penalty on them. Now this doesn’t mean there weren’t transgressors who flaked evidence, tickets, or arrests onto innocent people, but I never met an officer who had a problem with firing and/or charging the officers who did that.
The thing about stop reports is their low, subjective threshold (ie, literally, suspicion that criminality may be, or was, “afoot,”) and the fact that their very purpose was to memorialize police contacts with innocent people *because* other reports associated with processes like tickets and arrests don’t, made them attractive to officers for use as an indication of effort or productivity without consequence or follow-up. As records of an encounter with an innocent person, the officer ipso facto wasn’t signing an affidavit or presenting evidence that would be used in court. So officers would see a person standing near an alley, have a short conversation with them, then fill out a stop report indicating they stopped a person for suspicion of burglary, but the suspicion was ultimately unfounded and the person refused to identify himself (now a record of a burglary in the study here, mind you). Or they would see, say, four kids on a corner who were probably selling drugs, and they would stop them all, detain them for a few minutes to let them know the cops were hip to their game, then let them go, and generate a stop report for each (again, a record of *four* crimes in the study here).
Arrest and crime reports were audited by designated units at headquarters by calling complainants and verifying facts, but how do you audit a report that a police officer stopped a man near an alley who was innocent? By having cops ask innocent suspects for their phone numbers so police auditors can call them back to be sure they 1) actually exist, and 2) were actually innocent?
That said, most of the stops were, in fact, legitimate in this sense: the officers believed something may have going on, and if they investigated, they might find evidence of a crime, or stop something bad from happening. And many stop reports were for people who matched the description of a suspect wanted for a past crime (again, each of these fruitless stops would be counted as a separate, *new* crime in the study here). In either case, a stop report was required. This generated millions of reports over the years that reflected everything from breaking up an impending robbery to stopping a person who was hanging out in his own apartment building lobby.
The point for this blog is that the actual underlying behaviors reflected by the report introduce a fatal category error for the study, one that is inescapable. In fewer than 1 in 20 cases were stop reports incidentally the record of a crime. That can’t power a study.
Brandon del Pozo? That’s a name I haven’t heard in a few years: https://www.sevendaysvt.com/OffMessage/archives/2019/12/12/burlington-police-chief-admits-he-used-an-anonymous-twitter-account-to-taunt-a-critic
So it turns out police chiefs are humans too.
I’d hate a job where I couldn’t get in some good anonymous jabs on Twitter.
Damn, that’s a shame, that’s pretty much all I do here. And seems the guy damn thoughtful for a police chief.
Could do a lot worse than make fun of someone on twitter
https://innocenceproject.org/news/florida-police-chief-sentenced-to-three-years/
If they have burglaries that are open cases that are not solved yet, if you see anybody black walking through our streets and they have somewhat of a record, arrest them so we can pin them for all the burglaries.
Somebody:
I followed the link, and it’s from 2018. It said he was sentenced to 3 years in prison. So I guess he’s out now. I hope he’s not working somewhere else, issuing fake tickets etc.