7% of adults have been present at the scene of a mass shooting? Believe it or not, there’s a lesson in probability theory here.
Matt Lerner points with skepticism to this new paper from JAMA, which reports:
In a survey study of 10 000 US adults, 7% reported having been present on the scene where 4 or more people were shot, which was more common among younger generations, males, and Black respondents. Two percent reported having been injured in a mass shooting–by being shot, trampled, or experiencing related injuries–which was more common among younger generations and males.
These findings underscore the extensive and often overlooked regular exposure to mass shootings in US society, which calls for targeted interventions designed to reduce violence.
Lerner writes:
I can’t see anything seriously wrong with the methodology on a quick skim (they used YouGov to generate…. some kind of sample) but obviously I just don’t believe it. A little BOTEC- there were 500 mass shootings in the US last year. If that’s been going on for fifty years and there’s 100 (mutually exclusive) people at each shooting, that’s 2.5m people, so still under 1% of adults. 7% ????
Agreed. The published paper is terrible, the kind of junk science that gets published because it enhances a political agenda.
From a statistical perspective the problem is obvious. What the survey tells us is not the percentage of respondents who had “ever been physically present on the scene of a mass shooting” or “were physically injured in the incident.” What it tells us is the percentage of respondents who say they had “ever been physically present on the scene of a mass shooting” or “were physically injured in the incident.”
That’s not the same thing! It’s well known that people overstate the frequencies of rare events. Whether this is misremembering, misclassification, or something else, I don’t know–but even a very small false-positive rate will destroy your estimate, if the true underlying frequency is low.
In a classic article in Chance magazine from 1997, “The myth of millions of annual self-defense gun uses: a case study of survey overestimates of rare events,” David Hemenway uses statistical reasoning to explain this bias in terms of probability theory. Misclassifications that induce seemingly minor biases in estimates of certain small probabilities can lead to large errors in estimated frequencies. Hemenway discusses this effect in the context of traditional medical risk problems and then argues that this bias has caused researchers to drastically overestimate the number of times that guns have been used for self defense. Direct extrapolations from surveys suggest 2.5 million self-defense gun uses per year in the United States, but Hemenway shows how response errors could be causing this estimate to be too high by a factor of 10.
Here are a couple more examples from Hemenway’s 1997 article:
The National Rifle Association reports 3 million dues-paying members, or about 1.5% of American adults. In national random telephone surveys, however, 4-10% of respondents claim that they are dues-paying NRA members. Similarly, although Sports Illustrated reports that fewer than 3% of American households purchase the magazine, in national surveys 15% of respondents claim that they are current subscribers.
Back to the published paper
The problem with the paper is not so much that it reports this survey result. It’s data! The problem is that they report it as if it’s telling us something useful about the number of people who have been present at, or injured in, a mass-shooting event.
Look again at their summary, which is written as if they have access to direct measurements:
These findings underscore the extensive and often overlooked regular exposure to mass shootings in US society.
The paper does have a Limitations section. Here it is, in its entirety:
Findings on the prevalence of direct exposure and associated correlates are subject to well-known limitations of survey research. Although survey prompts defined mass shootings and direct exposure, recall bias and cohort-specific experiences related to growing up in an era characterized by heightened awareness of gun violence could contribute to the generational gradient observed. Invariance in injury may mask differential forms of exposure by location and type of mass shooting, as well as the circumstances under which these incidents occur, including measurement of lifetime exposure (eg, shorter vs longer recall periods) and types of physical injuries (eg, shot vs trampled).
What a steaming pile of crap. Nowhere do they even consider that their estimate could be contaminated by false positives. They talk about “the generational gradient,” “differential forms of exposure,” and “types of physical injuries,” but never about their implausible and unsupported 7% and 2% claims.
Junk science presented as public health research
So, this is a bad paper. Bad papers happen. It could’ve been an ok paper, if they’d (a) reported their results as is without claiming to have learned about actual mass-shooting events, (b) addressed the problem of false positives, as discussed by Hemenway (1997), and (c) considered other data on the topic. On the other hand, without making those big, unsupported claims, they couldn’t have published it in a high-profile journal.
On the plus side, NPR doesn’t seem to have fallen for this one. The best media exposure this study has received seems to have been Axios, which I guess was taking a break from whatever the heck it is that they usually do at Axios.
The other thing that the paper has going for it–and I guess what got it published in JAMA–is that they’re framing a social problem as a public health problem:
This survey study offers evidence of wide-reaching direct exposure and experience with mass shootings in the US. Ongoing research and data collection are essential to understanding the full scope of mass shootings and their impact on US society. Future research should further investigate how direct exposure to these incidents shapes individuals’ health and well-being, with particular attention to the broader community and the potential for systemic interventions that promote safety and resilience.
Even setting aside the unaddressed data problems which make the paper entirely useless, this study was published by the Journal of the American Medical Association–and it has nothing to do with medicine. It pushes a particular political agenda in which social problems are framed as public health problems.
Yeah, JAMA gets the blame
Oh, you might say, the paper’s not really published in JAMA–it’s published in JAMA Network Open, a pay-to-play journal under the JAMA umbrella. To which I reply: No, if it’s got the JAMA brand, then JAMA has to take the responsibility for it. Reputation is a two-way street.
If JAMA keeps this up, they’re gonna get the reputation of Lancet as being willing to publish just about anything that fits their political agenda.
P.S. After writing the above I came across this post by Danielle Navarro regarding another example of unbelievable survey responses, and we also had this discussion a few years ago. All of these are respectable survey responses and it’s good to understand how people can get these wrong. The problem with the above-discussed JAMA paper is that it’s taking those numbers as representing external reality.
I actually have been, during a mardi gras parade. The parade just kept going to the next block quietly, then restarted, and it got no special mention on the news.
The streets were pretty packed but not completely. I would guess a thousand (maybe a few) people could be counted as present.
Events like that may affect the back of the napkin calculation. Also, who knows if it got included in the official 500 per year number.
There is an incentive for businesses and cities to “hush up” these events during tourist attractions. Imagine one outside an NFL game, you could easily get 10-100k people saying they were present.
I guess I was technically at one as well (never would have thought that if this post and comment hadn’t come up) at the Chiefs superbowl parade in 2024. One million people at the parade, 22 people shot. But if the survey was counting that sort of incident, where most people were blocks away from the shooting, it would really stretch any commonsense meaning of ‘present’.
From the paper it looks like that wouldn’t count: “in the immediate vicinity of where the shooting occurred at the time it occurred, such that bullets were fired in your direction, you could see the shooter, or you could hear the gunfire.” With that definition, seeing the shooter, hearing gunfire (I guess that’s much looser) or having bullets fired in your direction, the 7% seems absurdly high.
I was thinking only within a couple blocks. Hearing it could be more like 4-8 blocks in any direction. Then people will *think* they heard it after the fact. I could almost believe someone who enjoys nightlife and large events is near guaranteed to be “present” at one of these shootouts eventually.
There are 4500 shot fired in Phoenix every year and that’s just what get reported to the cops. My son’s friend from Eritrea was shocked to discover there was gunfire every night in his new neighborhood (and you think Africa is a mess). New Year’s Eve sounds like the Vietnam war. Actually there’s more gunfire and explosions. The newspaper and TV mostly ignore it. Bad for business. Don’t want to scare off TSMC and the tourists. The whole city ought to be suffering from PSDT … and maybe it does.
I agree that this is likely an overestimate. But the issue may be interpretation as much as misreporting. The number “100” for a guesstimate of how many are present at a mass shooting seems plausible for some definitions of present (in the same building?) but if we think of shootings on college campuses, someone might reasonably say they were present if they were anywhere on the campus – so potentially tens of thousands of people per incident rather than 100. High schools may have in the thousands of people as well. And parades or other public events even more. I don’t think it’s unreasonable for respondents to say they were present even if they were not very close to the shooting given how we define locations socially and how we think about risk (ie how scared or traumatized you might be from a shooting a mile away but that was on your campus while you were there). More generally, it seems like a badly worded question / badly thought out estimand.
This is a good point, I remember talking to one of our alums who was a grad student at Virginia Tech when the mass shooting happened there. But this is why when you do a survey you have to pretest the questions with many open ended questions/focus groups trying to understand what people think is meant by the question.
I think that the possible claim that through geographic and network effects a large number of people have felt personally impacted by mass shootings.
This was a pretty good example of this kind or crap being purveyed by purveyors of covid vaccine crap:
https://www.rasmussenreports.com/public_content/politics/public_surveys/died_suddenly_more_than_1_in_4_think_someone_they_know_died_from_covid_19_vaccines
Twenty-eight percent (28%) of adults say they personally know someone whose death they think may have been caused by side effects of COVID-19 vaccines, while 61% don’t and another 10% are not sure.
That polling was widely used to promote the idea of millions of “vaccine deaths.”
JAMA has a long history of publishing research on mass shootings and shooting generally, and it’s weird that you don’t think the leading killer of young Americans is a medical issue.
I think you should pay more attention to the survey, a yougov panel, a source of data that has corrupted a lot of social debates (see eg the material John Burns-Murdoch published in the financial times using this type of data). I suspect it’s highly non-representative but is presented as a proper survey. Response rates are very dodgy and poorly reported. If the sample over-represents parents, for example, they may be reporting a shooting at their school as “present at”, since they received texts and their kids were locked down. In that case one school shooting would have essentially 700 parents present …who are more likely to be selected in this sample than say a young New Yorker who has never seen a gun. These yougov data sets need to be dropped from all public health research!
Faustusnotes:
Regarding your statement that you think shootings are a medical issue, I looked up “medical” in Merriam-Webster and got these two definitions:
“1: of, relating to, or concerned with physicians or the practice of medicine
2: requiring or devoted to medical treatment.”
Most shootings are non-fatal, and I agree that the treatment of non-fatal shootings is a medical issue; one could also extend this to psychological treatment of people who are affected by the shootings of others or being shot themselves. All the “medical” aspect here seems to be in the treatment, not in the shootings themselves. That said, sure, I see your point. If it’s legit for JAMA to run an article on the prevalence of diabetes or cholera or whatever, it’s also legit for them to publish on the prevalence of mass shootings.
So I take back my statement that the paper has nothing to do with medicine. You have a good point.
The problem is that the next step is to consider everyone with resulting PTSD, or just an acute panic attack, to be counted as “injured” by the shooting. And sure, in some way that true, but it will dilute the original meaning of the term.
Andrew, JAMA also concerns itself with public health, health systems and policy, and covers death as well as disease. It published Chapman’s (awful) analysis of gun control and suicide death in Australia in 2016, and has also published work on health financing in the USAID (as do NEJM, the lancet and bmj). Almost no medical journal covers just “medicine”.
In my experience American research on gun control is universally bad, and JAMA has a strong bias in what it accepts. It doesn’t surprise me this would get through.
Anoneuid, ptsd *is* an injury, if you consider mental health to be a health issue that is. People who witnesses death and terror need follow up care and should be considered a public health consequence of shootings. But obviously with vague data like this, it’s possible a parent of a kid who was at a university when a shooting happened could claim to have been injured by distress. It’s probably pretty distressing! (I wouldn’t know, I live in a sane country with no guns).
My friend is in rehab after a stroke or whatever it was, this disturbs me. Am I injured by him?
Its a tower of babble kind of counter-productivity, the more the meaning of words is blurred the less likely the actual problem will ever be solved.
Anoneuoid, it’s a well known phenomenon , just because you don’t like it doesn’t mean it isn’t real: https://web.musc.edu/about/leadership/institutional-offices/communications/pamr/news-releases/2024/las-vegas-mass-shooting-survivors-continue-to-struggle-with-major-depression-ptsd
And it’s not unexpected really is it? Nor does it really have any bearing on whether or not the people queried in a dodgy data set have reason to report being involved in or harmed by a shooting. The shooting linked above killed 60 and injured 800, at a large concert. Presumably thousands more people witnessed it. It’s likely that every shooting has many more witnesses than victims. Depending on the response bias of the survey it’s possible that a loose definition of witness or victim could produce unexpected numbers.
Andrew, you could probably quickly copy paste some of your post here into a response to the paper. Correspondence related to papers is accepted up to two weeks after their physical publication. Push back on their dodgy data!
> Am I injured by him?
No, but you may well be injured for the reasons that Faustus describes.
“300 million people were injured by the 9/11 attacks.”
This is where its headed, and it is not good for effective communication at all. Its like the opposite of using latin for precise medical jargon. It is fascinating, in a disturbing way, to see the language destroyed to fit whatever agenda is at play here.
Anoneuoid –
This is where its headed,
Let’s add reductio ad absursum to your (long) list.
The hilarious thing is I thought I already was doing that: “The problem is that the next step is to consider everyone with resulting PTSD, or just an acute panic attack, to be counted as “injured” by the shooting”
Turns out we are already well past that absurd step.
Andrew: Consider a jocular term for being shot: “lead poisoning”. Surely you’d agree that leaded gas causing lead in the air, or lead in pipes causing lead in the water supply, would be legitimate topics for public health research. Not only the specific treatment of people who get seriously sick, but more general effects overall. Why, then, not a pellet of lead inserted into a body at high speed? (of course the effects here aren’t due to chemical itself, rather the speed of the pellet, but that doesn’t strike me as a medical distinction).
This isn’t to defend any specific piece of research. But it strikes me as very reasonable for medical topics along the lines of “How many cases of this are there overall, and what are the associated effects of each case?”.
Not directly pertaining to this post, but I’m pretty sure the majority of shooting deaths are suicides, especially with the youths, which is pretty obviously a mental health issue. Arguably also a gun availability issue.
Is the war in Ukraine a health issue? It’s weird if you don’t, because it’s the leading killer of Ukranian men, not to mention Russian men.
I think the distinction here is not subtle.
“I can’t see anything seriously wrong with the methodology.” This sentence describes a common problem with using statistical methods. If one uses linear regression to fit quadratic data, there is nothing wrong with the methodology. There is ground truth here – about 500 mass shootings last year involving 4 or more victims (according to AI). There are 200 million or so adults, so 4 million people are estimated to have been injured in a mass shooting based on their findings. The only conclusion about the paper is that it is reporting its own sampling bias.
> The only conclusion about the paper is that it is reporting its own sampling bias.
It’s seems unlikely that the explanation is sampling bias (i.e. actual victims being over-represented in the sample).
The lizard constant strikes back
If being present is “more common among younger generations, males, and Black respondents” and being injured is “more common among younger generations and males” was there any accounting for males who fit those categories through military exposure (Middle East, National Guard in recent USA disturbances) or other occupational exposure (police and firemen in Chicago or other areas where shootings are numerous)? I don’t do statistics for a living; but, as a consumer of these types of articles in my occupation (law) I’m always interested in quality control aspects of the materials I use.