Geoff Holtzman writes:
In April 2021, The Guardian published an article titled “Gun Ownership among Black Americans is Up 58.2%.” In June 2022, Newsweek claimed that “Gun ownership rose by 58 percent in 2020 alone.” The Philadelphia Inquirer first reported on this story in August 2020, and covered it again as recently as March 2023 in a piece titled “The Growing Ranks of Gun Owners.” In between, more than two dozen major media outlets reported this same statistic. Despite inconsistencies in their reporting, all outlets (directly or indirectly) cite as their source a survey-based infographic conducted by a firearm industry trade association.
Last week, I shared my thoughts on the social, political, and ethical dimensions of these stories in an article published in The American Prospect. Here, I address whether and to what extent their key statistical claim is true. And an examination of the infographic—produced by the National Shooting Sports Foundation (NSSF)—reveals that it is not. Below, I describe six key facts about the infographic that undermine the media narrative. After removing all false, misleading, or meaningless words from the Guardian’s headline and Newsweek’s claim, the only words remaining are “Among” “Is,” “In,” and “By.”
(1) 58.2% only refers to the first six months of 2020
To understand demographic changes in firearms purchases or ownership in 2020, one needs to ascertain firearm sales or ownership demographics from before 2020 and after 2020. The best way to do this is with a longitudinal panel, which is how Pew found no change in Black gun ownership rates among Americans from 2017 (24%) to 2021 (24%). Longitudinal research in The Annals of Internal Medicine, also found no change in gun ownership among Black Americans from 2019 (21%) through 2020/2021 (21%).
By contrast, the NSSF conducted a one-time survey of its own member retailers. In July 2020, the NSSF asked these retailers to estimate demographics in the first six months of 2020 to demographics in the first six months of 2019. A full critique of this approach and its drawbacks would require a lengthy discussion of the scientific literature on recency bias, telescoping effects, and so on. To keep this brief, I’d just like to point out that by July 2020, many of us could barely remember what the world was like back in 2019.
Ironically, the media couldn’t even remember when the survey took place. In September 2020, NPR reported—correctly—that “according to AOL News,” the survey concerned “the first six months of 2020.” But in October of 2020, CNN said it reflected gun sales “through September.” And by June 2021, CNN revised its timeline to be even less accurate, claiming the statistic was “gun buyers in 2020 compared to 2019.”
Strangely, it seems that AOL News may have been one of the few media outlets that actually looked at the infographic it reported. The timing of the survey—along with other critical but collectively forgotten information on its methods are printed at the top of the infographic. The entire top quarter of the NSSF-produced image is devoted to these details: “FIREARM & AMMUNITION SALES DURING 1ST HALF OF 2020, Online Survey Fielded July 2020 to NSSF Members.”
But as I discuss in my article in The American Prospect, a survey about the first half of 2020 doesn’t really support a narrative about Black Americans’ response to “protests throughout the summer” of 2020 or to that November’s “contested election.” This is a great example of a formal fallacy (post hoc reasoning), memory bias (more than one may have been at work here), and motivated reasoning all rolled into one. To facilitate these cognitive errors, the phrase “in 2020” is used ambiguously in the stories, referring at times to its first six months of 2020 and at times specific days or periods during the last seven months. This part of the headlines and stories is not false, but it does conflate two distinct time periods.
The results of the NSSF survey cannot possibly reflect the events of the Summer and Fall of 2020. Rather, the survey’s methods and materials were reimagined, glossed over, or ignored to serve news stories about those events.
(2) 58.2% describes only a tiny, esoteric fraction of Americans
To generalize about gun owner demographics in the U.S., one has to survey a representative, random sample of Americans. But the NSSF survey was not sent to a representative sample of Americans—it was only sent to NSSF members. Furthermore, it doesn’t appear to have been sent to a random sample of NSSF members—we have almost no information on how the sample of fewer than 200 participants were drawn from the NSSF’s membership of nearly 10,000. Most problematically—and bizarrely—the survey is supposed to tell us something about gun buyers, yet the NSSF chose to send the survey exclusively to its gun sellers.
The word “Americans” in these headlines is being used as shorthand for “gun store customers as remembered by American retailers up to 18 months later.” In my experience, literally no one assumes I mean the latter when I say the former. The latter is not representative of the former, so this part of the headlines and news stories is misleading.
(3) 58.2% refers to some abstract, reconstructed memory of Blackness
The NSSF doesn’t provide demographic information for the retailers it surveyed. Demographics can provide crucial descriptive information for interpreting and weighting data from any survey, but their omission is especially glaring for a survey that asked people to estimate demographics. But there’s a much bigger problem here.
We don’t have reliable information about the races of these retailers’ customers, which is what the word “Black” is supposed to refer to in news coverage of the survey. This is not an attack on firearms retailers; it is a well-established statistical tendency in third-party racial identification. As I’ve discussed in The American Journal of Bioethics, a comparison of CDC mortality data to Census records shows that funeral directors are not particularly accurate in reporting the race of one (perfectly still) person at a time. Since that’s a simpler task than searching one’s memory and making statistical comparisons of all customers from January through June of two different years, it’s safe to assume that the latter tends to produce even less accurate reports.
The word “Black” in these stories really means “undifferentiated masses of people from two non-consecutive six-month periods recalled as Black.” Again, the construct picked out by “Black” in the news coverage is a far cry from the construct actually measured by the survey.
(4) 58.2% appears to be about something other than guns
The infographic doesn’t provide the full wording of survey items, or even make clear how many items there were. Of the six figures on the infographic, two are about “sales of firearms,” two are about “sales of ammunition,” and one is about “overall demographic makeup of your customers.” But the sixth and final figure—the source of that famous 58.2%—does not appear to be about anything at all. In its entirety, that text on the infographic reads: “For any demographic that you had an increase, please specify the percent increase.”
Percent increase in what? Firearms sales? Ammunition sales? Firearms and/or ammunition sales? Overall customers? My best guess would be that the item asked about customers, since guns and ammo are not typically assigned a race. But the sixth figure is uninterpretable—and the 58.2% statistic meaningless—in the absence of answers.
(5) 58.2% is about something other than ownership
I would not guess that the 58.2% statistic was about ownership, unless this were a multiple choice test and I was asked to guess which answer was a trap.
The infographic might initially appear to be about ownership, especially to someone primed by the initial press release. It’s notoriously difficult for people to grasp distinctions like those between purchases by customers and ownership in a broader population. I happen to think that the heuristics, biases, and fallacies associated with that difficulty—reverse inference, base rate neglect, affirming the consequent, etc.—are fascinating, but I won’t dwell on them here. In the end, ammunition is not a gun, a behavior (purchasing) is not a state (ownership), and customers are none of the above.
To understand how these concepts differ, suppose that 80% of people who walk into a given gun store in a given year own a gun. The following year, the store could experience a 58% increase in customers, or a 58% increase in purchases, but not observe a 58% increase in ownership. Why? Because even the best salesperson can’t get 126% of customers to own guns. So the infographic neither states nor implies anything specific about changes in gun ownership.
(6) 58.2% was calculated deceptively
I can’t tell if the data were censored (e.g., by dropping some responses before analysis) or if the respondents were essentially censored (e.g., via survey skip logic), but 58.2% is the average guess only of retailers who reported an increase in Black customers. Retailers who reported no increase in Black customers were not counted toward the average. Consequently, the infographic can’t provide a sample size for this bar chart. Instead, it presents a range of sample sizes for individual bars: “n=19-104.”
Presenting means from four distinct, artificially constructed, partly overlapping samples as a single bar chart without specifying the size of any sample renders that 58.2% number uninterpretable. It is quite possible that only 19 of 104 retailers reported an increase in Black customers, and that all 104 reported an increase in White customers—for whom the infographic (but not the news) reported a 51.9% increase. Suppose 85 retailers did not report an increase in Black customers, and instead reported no change for that group (i.e., a change of 0%). Then if we actually calculated the average change in demographics reported by all survey respondents, we would find just a 10.6% increase in Black customers (19/104 x 58.2%), as compared to a 51.9% increase in white customers (104/104 x 51.9%).
A proper analysis of the full survey data could actually undermine the narrative of a surge in gun sales driven by Black Americans. In fact, a proper calculation may even have found a decrease, not an increase, for this group. The first two bar charts on the infographic report percentages of retailers who thought overall sales of firearms and of ammunition were “up,” “down,” or the “same.” We don’t know if the same response options were given for the demographic items, but if they were, a recount of all votes might have found a decrease in Black customers. We’ll never know.
The 58.2% number is meaningless without additional but unavailable information. Or, to use more technical language, it is a ceilingestimate, as opposed to a real number. In my less-technical write-up, I simply call it a fake number.
This is kind of in the style of our recent article in the Atlantic, The Statistics That Come Out of Nowhere, but with lot more detail. Or, for a simpler example, a claim from a few years ago about political attitudes of the super-rich, which came from a purported survey about which no details were given. As with some of those other claims, the reported number of 58% was implausible on its face, but that didn’t stop media organizations from credulously repeating it.
On the plus side, a few years back a top journal (yeah, you guessed it, it was Lancet, that fount of politically-motivated headline-bait) published a ridiculous study on gun control and, to their credit, various experts expressed their immediate skepticism.
To their discredit, the news media reports on that 58% thing did not even bother running it by any experts, skeptical or otherwise. Here’s another example (from NBC), here’s another (from Axios), here’s CNN . . . you get the picture.
I guess this story is just too good to check, it fits into existing political narratives, etc.
Another strike against polls. Of course, this debacle refers to a very badly done poll – most are done better. But that leaves open the question of whether polling, on net, is a positive or negative influence on policy decisions. I can’t help but think of a parallel: many statistical analyses are done poorly, as we have amply seen on this blog. Of course, many are better than the worst examples. What is the impact, on net, of statistical analyses for policy decisions?
Put me in the camp that the net effect is positive – otherwise I am wasting most of my energy. I’m not as convinced about polls, however. But of course I haven’t invested any of my energy in doing or analyzing polls. It may be my own biases that make me treat statistical analysis differently than polling. The one common feature I see is that, in both cases, measurement is a key problem. I don’t find polls able to measure what they purport to measure most of the time. And many statistical analyses suffer from the same problem (but I still think that many measurement issues are endemic to polls more than to many other data sources).
There are definitely issues with polls. The other issue is TV analysts like Harry Enten who need to fill airtime with a narrative and use polls out of their context or when the effect size is small or other methodological issues are present.
Not clear how this gets better any time soon. Audiences eat it up.
Quote from above: “What is the impact, on net, of statistical analyses for policy decisions?”
I think these types of questions should be asked much more. I recently pondered such questions about psychological science in general. I sincerely wonder whether psychological science has had a net-positive effect on society and for people.
If I think about the many possible sub-optimal research, and associated conclusions, that have been produced by psychological science over the last decades, or perhaps even before that, I wonder about it all.
It’s one of the reasons I have recently started to think about the importance of learning about, and honing, more possibly solid and sound things like logic and reasoning. I often wonder what society would look like when students at high school and university would have had an education filled with logic, reasoning, debates, etc.
I think these things are extremely important, and I simply note that I have never been taught any of it to the best of my recollection. Not at high school, and not at university. This is still incomprehensible to me…
Quote from above: “I think these things are extremely important, and I simply note that I have never been taught any of it to the best of my recollection. Not at high school, and not at university. This is still incomprehensible to me…”
I have mentioned this, in a few different ways, in some of my most recent manuscripts. I know of a paper by Mulnix (2010) who mentions that far too many college undergraduates have yet to achieve proficiency in thinking critically, and that many students are unable to recognize even the simplest of evidential relationships between statements (p. 11). Perhaps this can be seen as an indication for the possible need of better education in logic and reasoning and argumentation.
By mentioning this (also now, here, on this blog), I also reason that it might contribute to someone thinking some more about this, and perhaps even performing some basic research concerning this. I reason that if I did not have (if I remember correctly of course) a specific mandatory course in logic, reasoning, and/or argumentation at university during my Research Master in Behavioral Science for example, I reason others might also not have.
Perhaps if someone would investigate this some more, it would become clear that a substantial part of students at university do not receive mandatory courses concerning logic, reasoning, and/or argumentation. That could be something worthwhile to note, also in relation to several of the possible problematic issues in academia and research.
I don’t think there is any question “statistical analysis”* has become a net negative, and is, in fact, the biggest threat facing our civilization and species. Fisher was right:
Fisher, R N (1958). “The Nature of Probability”. Centennial Review. 2: 261–274.
* NHST is actually pseudo-stats, but the people using it for everything and constructing philosophies like evidence-based-medicine around it don’t know that.
The Centennial Review, volume 2, 1958 was quite an issue. A few articles after Fisher’s, On Probability, there is, Analogy in Science, by Robert Oppenheimer, and a few articles before is another article by Fisher who in discussing the data says:
What is not quite so much the work of a good citizen is to plant fear in the minds of perhaps a hundred million smokers throughout the world— to plant it with the aid of all the means of modern publicity backed by public money, without knowing for certain that they have anything to be afraid of in the particular habit against which the propaganda is to be directed.
Fisher, R N (1958). “CIGARETTES, CANCER, AND STATISTICS”. Centennial Review. 2: 151-166.
Nice quote from Fisher! Now edit it for covid vaccines, or any vaccines, and you have the present day!
Jdk:
Indeed, one problem I have with traditional statistics education is to take these historical figures and turn them into heroes. Fisher was indeed a great researcher and came up with an amazing number of impressive methods in areas including logistic regression, maximum likelihood, randomized experimentation, and statistical genetics. He also made some mistakes. I just hate that scientist-as-hero thing.
It isn’t that Fisher is a hero, it is that he was in a position to, and then mananged to, correctly predict the future. Because he was key to popularizing this whole thing that grew into som BS he didn’t anticipate at first.
There are some nice discussions of issues with (nonprobability sample) polls among the discussants of this recent piece in the Harvard Data Science Review (including Andrew): https://hdsr.mitpress.mit.edu/pub/ejk5yhgv/release/4
OK, so 58.2% is a bit dodgy, and the real number is likely to be close to zero.
But now that we have an actual number, we can just slap a -58 on the NSSF for “house effect” and their data should be fine for any subsequent statistical roll-up, amirite?
“As I’ve discussed in The American Journal of Bioethics, a comparison of CDC mortality data to Census records shows that funeral directors are not particularly accurate in reporting the race of one (perfectly still) person at a time.”
I’m sure that most of the criticisms Mr. Holtzman levels against the survey are correct, but the article from the above quote and the reference from the footnote in the article both show very high accuracy (97-98%) when identifying someone as “black”. It is the “other non-white” racial categories and intersectional ethnic categories that show lower levels of agreement between Census records and death certificates. The references suggest that retailers can accurately identify customers as “black”.
One thing I think *is* probably true is that *the fraction of people purchasing guns who were black* went up a lot. This is not the same as the wording by this trade study, but is probably true based on other sources.
Similarly, the fraction of people purchasing guns who were women went up A LOT, and that was not based solely on trade industry bullshit.
https://www.voanews.com/a/usa_gun-ownership-steadily-increasing-among-us-women/6207787.html
They cite “preliminary data from harvard school of public health” but I can’t actually find which data they were referencing. Is it so damn hard for media to just provide a link to the stuff they’re discussing? Terrible.
In any case, demographics of gun buyers were very atypical during the period 2020-2022 or so.
I personally think that’s a good thing as the “typical” gun purchaser in the past 20 years has been some kind of rural conservative white male Trump supporter possibly with previous military affiliation and it’d be good if they were not the only armed group in America.
“In any case, demographics of gun buyers were very atypical during the period 2020-2022 or so.”
how the hell do you know this? you yourself don’t cite any data.
I cite an article that cites data (albeit in a poor way, by name and source only rather than link), I did at one point find that information and I remember a number of other sources discussing the topic, but you’re right I didn’t directly link to primary sources, in part because Google’s enshittification is making stuff harder and harder to find.
https://www.norc.org/research/library/one-in-five-american-households-purchased-a-gun-during-the-pande.html
Gives some discussion of the demographic changes. This was a significant topic of discussion about 12-16 months ago.
Another article which cites percentage women etc
https://www.acpjournals.org/doi/full/10.7326/M21-3423
Unfortunately we get caught up in political tropes about everything these days. So too many conclusions end up being partisan. My unfortunate experiences with guns, definitely not a random sample, suggests that our young are heavily armed these days and the weapons are too often procured through channels that are not in any way surveyed. The armaments problem is so pervasive in the first place how in heaven’s name could it be increasing.
This is really stunningly awful journalism. Despite being at a university where journalism is a surprisingly popular major, I have no idea what is taught there, and whether the journalism faculty, like the rest of us, bang their heads against the wall about the pervasive ineptness of journalists to do anything with numbers.
While my experiences are purely anecdotal, I’ll note that I’ve had several grad students from our Communications department take some of my more technical courses (including Bayesian stats) and they were outstanding. Of course, they self-selected, but it was with the support/advice of their advisors. So my guess is that journalists at the academic level do indeed appreciate good quantitative reasoning and, like us here on the blog, get frustrated at how poorly it is done by people in the real world.
I’m a huge fan of The Guardian (and voluntarily donate to it every year [it’s a non-profit]). So it pains me to no end to see it associated with the usual suspects. I just filed a complaint and asked for a correction. Let’s hope (at least for my sake) that I’m not wrong about The Guardian.
.”Or, for a simpler example, a claim from a few years ago about political attitudes of the super-rich, which came from a purported survey”
No ausurvey rvey, training set or AGI is able to cope with the Rochester and stupidest man in the world.
https://www.huffpost.com/entry/elon-musk-pizzagate-twitter-x_n_65660d49e4b03ac1cd177636
Kt2:
Presumably he (or one of his employees, if that’s who’s posting this stuff) is not so stupid as to believe pizzagate; he just thinks it’s politically useful to take that position, or maybe he thinks it’s a funny joke, or maybe it’s a mix of the two: he thinks it’s a funny joke and he also likes the political implications of allying with Alex Jones etc. People joke about violence all the time. I guess you could say that anyone who would want to ally with Alex Jones and promote dangerous conspiracies is, at some deeper level, stupid even if he doesn’t believe the conspiracy theories. Ultimately I don’t know where to draw the line between stupid, malevolent, and disturbed. Lots of people have major emotional problems; we just hear more about this guy’s problems cos he’s famous and has been propped up by the news media for a few years and we’re in a time when certain business leaders are treated as heroes in the manner of athletes, musicians, and movie stars.
When it comes to rich guys being stupid, my favorite example remains this one, as it’s just dumb, it doesn’t have hatred mixed in.
! Nor my defiled spell checker. Shannon’s Entropy? Or ghost in the machine.?