Part 1: 2020
Back in the dark days of April, 2020, a team of 42 social scientists published an article that began:
The COVID-19 pandemic represents a massive global health crisis. Because the crisis requires large-scale behaviour change and places significant psychological burdens on individuals, insights from the social and behavioural sciences can be used to help align human behaviour with the recommendations of epidemiologists and public health experts. Here we discuss evidence from a selection of research topics relevant to pandemics, including work on navigating threats, social and cultural influences on behaviour, science communication, moral decision-making, leadership, and stress and coping.
As I wrote at the time, the author list includes someone named Nassim, but not Taleb, and someone named Fowler, but not Anthony. It includes someone named Sander but not Greenland. Indeed it contains no authors with names of large islands. It includes someone named Zion but no one who, I’d guess, can dunk. Also no one from Zion. It contains someone named Dean and someone named Smith but . . . ok, you get the idea. It includes someone named Napper but no sleep researchers named Walker. It includes someone named Rand but no one from Rand. It includes someone named Richard Petty but not the Richard Petty. It includes Cass Sunstein but not Richard Epstein.
As befits an article with 42 authors, there were a lot of references: 6.02 references per author, to be precise. But, even with all these citations, I wasn’t quite sure where this research can be used to “support COVID-19 pandemic response,” as promised in the title of the article.
And the article got me angry. I’ll give some details in a moment, but here I just want to say that (a) I assume those 42 authors were sincerely trying to help the world, (b) their help was framed in terms of claims that social-science research by them and their friends would be helpful, and (c) I don’t think it was. Indeed, I’d argue that their work was counterproductive to the extent that it distracted policymakers from the real issues.
As I wrote back in 2020, my trouble with that 42-authored article is that many of its claims are so open-ended that they don’t tell us much about policy. For example, I’m not sure what we can do with a statement such as this:
Negative emotions resulting from threat can be contagious, and fear can make threats appear more imminent. A meta-analysis found that targeting fears can be useful in some situations, but not others: appealing to fear leads people to change their behaviour if they feel capable of dealing with the threat, but leads to defensive reactions when they feel helpless to act. The results suggest that strong fear appeals produce the greatest behaviour change only when people feel a sense of efficacy, whereas strong fear appeals with low-efficacy messages produce the greatest levels of defensive responses.
I’m not saying that people shouldn’t do research on this topic, I’m just asking what can you do with this sort of thing? In what way does this “support COVID-19 pandemic response”?
Beyond the very indirect connection to policy, I’m also concerned because, of the three references cited in the above passage, one is from PNAS in 2014 and one was from Psychological Science in 2013. That’s not a good sign!
Looking at the papers in more detail . . . The PNAS study found that if you manipulate people’s Facebook news feeds by increasing the proportion of happy or sad stories, people will post more happy or sad things themselves. The Psychological Science study is based on two lab experiments: 101 undergraduates who “participated in a study ostensibly measuring their thoughts about “island life,” and 48 undergraduates who were “randomly assigned to watch one of three videos” of a shill. Also a bunch of hypothesis tests with p-values like 0.04. Anyway, the point here is not to relive the year 2013 but rather to note that the relevance of these p-hacked lab experiments to policy is pretty low.
Also, the abstract of the 42-author paper says, “In each section, we note the nature and quality of prior research, including uncertainty and unsettled issues.” But then the paper goes on to unqualified statements that the authors don’t even seem to agree with.
For example, from the article, under the heading, “Disaster and ‘panic’” [scare quotes in original]:
There is a common belief in popular culture that, when in peril, people panic, especially when in crowds. That is, they act blindly and excessively out of self-preservation, potentially endangering the survival of all. . . . However, close inspection of what happens in disasters reveals a different picture. . . . Indeed, in fires and other natural hazards, people are less likely to die from over-reaction than from under-reaction, that is, not responding to signs of danger until it is too late. In fact, the concept of ‘panic’ has largely been abandoned by researchers because it neither describes nor explains what people usually do in disaster. . . . use of the notion of panic can be actively harmful. News stories that employ the language of panic often create the very phenomena that they purport to condemn. . . .
But, just a bit over two months earlier, one of the authors of that article wrote an op-ed titled, “The Cognitive Bias That Makes Us Panic About Coronavirus”—and he cited lots of social-science research in making that argument.
So what is it? Is “panic” a concept that’s been “largely been abandoned by researchers because it neither describes nor explains what people usually do in disaster,” or is it a concept that is so useful that it should be the subject of a widely-circulated op-ed?
I don’t think social science research has changed so much between 28 Feb 2020 (when that pundit wrote about panic and backed it up with citations) and 30 Apr 2020 (when that same pundit coauthored a paper saying that researchers shouldn’t be talking about panic). And, yes, I know that the author of an op-ed doesn’t write the headline. But, for a guy who thinks that “the concept of ‘panic'” is not useful in describing behavior, it’s funny how quickly he leaps to use that word. A quick google turned up this from early 2016: “How Pro Golf Explains the Stock Market Panic.” I guess that, since the Democrats were in power at that time, he had to talk about volatility in the market as an illogical “panic” rather than a rational response to economic conditions.
All joking aside, this just gets me angry. These guys have the influence and media access that allows them to go around promoting themselves and their friends with the PANIC headline whenever they want. But then in their scholarly review article, they lay down the law and tell us how foolish we are to believe in “‘panic.'” They get to talk about panic whenever they want, but when we want to talk about it, the scare quotes come out.
Don’t get me wrong. I’m sure these people mean well. With decades of focused effort they’ve climbed to the top of the greasy academic pole; their students and colleagues tell them, week after week and month after month, how brilliant they are. We were facing a major world event, they wanted to help, so they do what they can do. But sometimes the thing that you’re professionally trained to do, isn’t much help.
It’s like, I dunno, suppose you’re really fast and strong, maybe you’re an NFL cornerback! And then some emergency comes up, and a loved one is in the hospital. You want to help. But all the speed and strength and coordination in the world won’t do any good here. The only way you can contribute is through other means, like holding someone’s hand or paying the medical bills or calling the family or whatever. You can help, but your professional skills have no relevance here. This sort of thing comes up all the time; there are areas of life that are orthogonal to each other. But people don’t always want to hear it. They want to help out, not just in the same way that others might help, but by using their special talents.
Interlude: The social-science establishment
We all like to use the term “establishment” to refer to powerful or influential people who we disagree with. But that’s not fair. “Rogue economist” Steven Levitt is part of the establishment (sorry, Steve!). I’m part of the establishment too. I can post grumpy blog entries every day between 9 and 10am eastern time until the end of my days and beyond, and I’m still an MIT-and-Harvard-educated Ivy League professor living a comfortable life and with more influence than most.
Indeed, back in 2020, I expressed agreement with another member of the establishment, University of Chicago political science professor Anthony Fowler, who, in a snappily-titled piece called, “Curing Coronavirus Isn’t a Job for Social Scientists,” wrote:
The public appetite for more information about Covid-19 is understandably insatiable. Social scientists have been quick to respond. . . . While I understand the impulse, the rush to publish findings quickly in the midst of the crisis does little for the public and harms the discipline of social science. . . . Even in normal times, social science suffers from a host of pathologies. . . . A global crisis only exacerbates these problems. . . . and the promise of favorable news coverage in a time of crisis further distorts incentives. . . .
Well put. So what does it mean that I’m the establishment, Anthony Fowler’s the establishment, and we’re arguing that those other 42 people in the establishment are wrong?
The key point, I think, is that the Gang of 42 is arguing, not just for specific ideas–indeed, I argued above that their ideas are not so coherent–but in support of the idea of the social-science establishment. That’s what they mean by saying, “insights from the social and behavioural sciences can be used to help align human behaviour with the recommendations of epidemiologists and public health experts.”
Now, don’t get me wrong, I think the establishment has its uses–somebody has to run the journals, standardize the curricula, etc., indeed when I’m writing textbooks, my coauthors and I are making our bid to be part of that establishment, and I’m happy when that happens (as with Bayesian Data Analysis) and frustrated when we don’t fully succeed (as with Regression and Other Stories).
But in the case of covid policy, I think Anthony Fowler was right that social scientists should’ve backed off. And there I see a big problem with the social science establishment, which is that one of its major roles is . . . to promote itself. So to ask the establishment to chill out, to ask Cass Sunstein to, for just once in his life, not make a confident pronouncement or write a book telling everyone else how to think and behave, to tell policymakers that the solutions to their social-behavior problems will not be found in the latest issues of Psychological Science and PNAS . . . that’s a counter-establishment move.
Part 2: 2024
In the meantime, the rise of the political right has brought us what might be called a counter-establishment or new social science establishment that has all of the same problems of the old social science establishment–no, that’s not right, let me say it avoids some of those problems but has introduced its own problems–so we still have a lot to struggle against.
The old establishment had power pose and the collected works of Brian “pizzagate” Wansink. The new establishment has what might be called “podcast science”–cold showers and miracle cures–along with a soft spot for conspiracy theories such as the original Pizzagate story. The old establishment had the scientist as hero; the new establishment has the rich guy as hero. I don’t like either of those narratives.
There is some overlap: both the old and the new establishment are suckers for crude gender and racial essentialism as well as various goofy fallacies of measurement (the Implicit Association Test for the old establishment, finger measurements for the new establishment). Conceptually I don’t see much difference between ridiculous claims in Psychological Science about ovulation and voting, and the sorts of extreme gender essentialism that we hear about from podcasts–either way, it’s what we’ve called “schoolyard evolutionary biology.”
The people who say, “Invite Cass Sunstein to more parties and he’ll help with our public health problems” are not the same people who say, “Cryptocurrency will solve our economic problems,” but I see a similarity: in both cases there’s a group with some political power and intellectual influence wielding some crude theories and spending lots of time promoting themselves.
If you look at the present post, I have a lot more to say in the “2020” section than the “2024” section. Part of that is the benefit of hindsight–much of the first section above is copied from my posts from May 2020, and I like what I said there–but most of it is that I’m a lot more familiar with the world of bad science in academia than I am with the world of bad science in the partisan conservative media, even though it is the latter that has more influence now. I’ll have to leave it to others to pick up that particular baton; here I just want to register my objection to well-meaning but useless or counterproductive social science establishments of all kinds, and I want to suggest that the lessons we have learned (or should’ve learned) from 2020 about distrusting the establishment should apply to the new establishment as well.
My message is not “don’t trust anybody,” and it certainly isn’t “don’t ever trust an Ivy League professor” (as that would lead us straight to the paradox of the liar). And social science theories can be valuable for policy, including theories associated with the left, right, and center. But . . . hmmm, I’m not quite sure how to finish this paragraph. I’m not completely sure what my message here is, or should be. So let me just stop here. A luxury I have since I’m blogging, not writing a news article or journal article or book that needs to come to some coherent conclusion.
> it certainly isn’t “don’t ever trust an Ivy League professor” (as that would lead us straight to the paradox of the liar)
I’m not sure it would be a paradox. It is possible to use science without trusting individual scientists (science can be useful even if we don’t trust the scientists). We simply have to let your post stimulate us to think about these issues and then make up our own minds. Why would we need to trust you, to find your writing useful?
Thomas:
I’m not asking my readers to agree with everything I write, and I agree with you that I should present my reasoning clearly enough that readers can follow along (and find places where they disagree). But I do expect some level of trust that people don’t think I’m lying. See here for a discussion of how annoyed I was when someone called me “disingenuous”–which is another way of calling me a liar, or saying he didn’t think I was being sincere.
This is very interesting. I was thinking about it more in terms of trusting scientists on the facts. I’ll have to think about this a little more, but my gut says that science can be useful even if we don’t presume scientists are sincere.
There was a huge temptation back in 2020 to make a novel contribution to science in a way that virtually guaranteed instant impact, recognition, and possibly even policy implications. I tamped down on said temptation myself when I realised that I would probably be doing more harm than good as someone who was not an infectious disease epidemiologist. However, I can certainly understand the impulse when publishing in that period gave you everything universities are pushing these days around “impact” (media appearances, citations in policy papers, and so on).
I think these 42 authors – no doubt among a wide variety of other scientists of various persuasions with no business publishing articles on pandemics – were of the mind that this was helpful information at the time. It must be difficult to deal with being simultaneously a big shot and also almost completely irrelevant to the present conversation.
Like you, I refrained from putting out any analysis. I explored some modeling but realized that I had nothing to contribute that wouldn’t just add to the confusion – lack of enough subject matter knowledge as well as issues with data quality. So much stuff was getting distributed and reported, that I didn’t view any “contribution” I could make as helping. Now, these authors apparently felt differently – and they have reputations that could justify that. However, as Andrew points out, their “contributions” really didn’t live up to that. I’d say they relied on their reputations as proof that they were being useful in place of actually being useful. Hubris I’d say. Some people need to learn that their credentials don’t automatically make their views valuable.
I would agree with the hubris tag. In my experience there is a temptation once you have achieved some level of expertise to think that your brilliance is universally generalisable. Andrew Huberman is a good example.
A typo: “But, just a bit over two moths earlier”
Somehow, I think this particular typo, substituting “moths” for “months” has happened a few times before on this blog. Makes me wonder if there is a field of study devoted to seemingly trivial typing errors when blogging. Perhaps more than one field of study. In other words,
“I’m not saying that people shouldn’t do research on this topic, I’m just asking what can you with this sort of thing?”
Typos fixed; thanks.
The astute reader will no doubt observe that the last sentence of my reply, a copy paste of the original, contains another typo:
“I’m not saying that people shouldn’t do research on this topic, I’m just asking what can you with this sort of thing?”
It’s the publication establishment that causes the probem:
… For example, the mystery of what the tumor suppressor p53 actually does seems only to deepen as the number of publications about this protein rises above 23,000. The notion that your work will create more confusion is not particularly stimulating, which made me look for guidance again. …
https://www.cell.com/cancer-cell/pdf/S1535-6108(02)00133-2.pdf
Off topic of Andrew’s post, but I’m curious if the “simple language that experimental scientists can use to introduce themselves to formal descriptions of biological processes” mentioned in your paper was ever developed.
I mostly agree with this post, but we should be careful about two-sidesism, no? I don’t think the autism claims of RFK jr should be seen as counterbalancing Cass Sunstein’s various claims about panic, the dangers of group discussion in venues like juries, etc. Both can be overreaching when measured against a more exacting standard, but the differences in degree matter. It takes some expertise to unpack the social science pastiches of the centrist establishment, but the errors of the right wing version are glaring and practically announce themselves. It’s like the difference, to use the technical philosophical language, between error and bullshit.
Even granting that the two sets of claims have different epistemic status, the corrective on both sides is the same: humility.
When the pandemic hit, I asked myself how I could use my expertise to help society. I talked with a friend of mine from grad school. We put together a proposal and got funding to the tune of $30-50k or something to do some studies. The idea was to create a system for pooled testing. Remember we had only PCR based testing, and each reaction cost a fair amount. Some people were charging like $50k for a single test (that was more of a grift than anything) but individual tests in the $100-200 range were common initially. And we were doing hundreds of thousands of tests a day in the US, so billions of dollars were being spent rapidly.
My strategy was to work out ways to use pooled testing to screen large groups of people to prevent transmission in hospitals/schools/institutions. We worked out the math, we worked out a testing strategy, we got a hospital to built out and test in real actual PCR chemistry pooling at 1, 5, 10, 20, 40 people per pool. We proved that it worked in reality. The hospital that did the test for us even began screening their interns using 10x pooling!
Then it came time to try to develop this into a service. We looked for funding. Cedars Sinai was interested at first, but they were too busy charging $200/test to cut into their profit by doing something vastly more efficient. UIUC had funded our initial work, but they had their own saliva based individual test they had developed and were planning to run thousands of them per day on their entire population, and sell the service to others. They wanted us to go away. Also one guy there started making vague threats about owning the rights to stuff we hadn’t even developed yet, I guess later he was removed for doing something wrong (fraud or embezzlement or some misuse of funds or something). Red Cross of Phillippines desperately wanted to use our techniques, but Duterte had earmarked the money for individual diagnostic testing, and we only promised a probabilistic answer about who should isolate. The NYT called us and wanted interviews. They acted like they were excited about the possibility of pooled testing, and we explained the benefits to them, and then the reported wrote a scathing hit piece about how pooled testing was bullshit.
In the end, the people who made a real contribution were the Saliva Direct people. Scientists got together, they designed a cheap test based on saliva that could be run for like $2 or less and they distributed it freely. With our pooling technique on top of that we could test entire hospitals daily and give probabilistic answers about who should not come to work for something like 5 cents a person…. but literally no-one wanted it. no one cared about keeping people safe. people cared about making the maximum amount of cash hand over fist by the least efficient means possible (ie. burning the most cash for the least amount of work). CEOs of meat packing plants were running betting pools on who in their plants would be disabled by COVID or whatever.
In the end after thousands of hours of work to try to make my society safer and keep essential work going I got paid something like $10k, so less than minimum wage in CA.
I’ve always been distrustful of institutions and rules and capitalism. But over the last few years since then I’ve read up about these things more intensely, and now I’m a full blown Anarchist. Literally the only things that actually worked were the anarchist solutions: people cooperating at human scale in groups. Ignoring the bullshit about institutions lying about the effectiveness of masks to hoard them for medical workers, ignoring the bullshit about whether it met the institutional criteria for “airborne” and just treating the disease as an airborne pandemic that could be mitigated by masking, air filtration, reduced intensity of congregation, etc. Some anarchists got together and created the COVID tracking project. I know Erin Kissane from Mastodon, she and a ragtag group of like 20-40 people made the data work for 8 billion others, at basically zero cost to anyone, except enormous personal cost to them (overwhelmed by how much everyone was relying on them without supporting them). The vaccine was made by a de-facto anarchist who got shit on by institutions over and over again (Kariko, who eventually won the Nobel prize). Shit that got done was almost entirely in spite of institutions. The role of institutions at BEST was to mitigate the brain damage of other institutions. And yes, to manufacture the vaccine in large quantity eventually, a process that would have been even bigger and more quick if Kariko’s invention paid for by taxpayers weren’t proprietary to the 2 big pharma companies.
Stanford idiots went public with claims about how the pandemic was already over based on some bullshit surveys in Santa Clara. Entire institutions funded to predict health stuff built completely shit models based on fitting a gaussian curve to a data bump and claimed the pandemic would affect a total of a few thousand or ten thousand people…. something like 4M people died since 2020 of COVID related issues in the US alone. Almost the only useful data sources were just competent people organizing themselves in little groups and ignoring the bullshit.
So, that’s what finally radicalized me more than I already was naturally. And now we’ve got a fascist govt arresting judges and deporting legal residents to gulags and talking about rounding up neurodiverse people and sending them to “health farms” (concentration camps). Congrats institutionalists. You voted that shit in.
Discuss amongst yourselves. I wont’ be responding further here.
“Stanford idiots went public ……”
And who is the new head of NIH??!!?!!??
Ummmm . . . this guy?
It’s all so dystopic. I vividly remember that Santa Clara study, your analysis, coding it up myself and concluding the obvious, that they couldn’t possibly be correct. I lost almost all respect for Ioannidis based on how he rushed to push out their foregone conclusion to the alt right media sphere, and never once seems to have conceded how wrong they were. Then came the whole ‘Great Barrington’ chicanery, ringlead and downstream of the same people with the same foregone conclusions. I suppose I’m not as radicalized as Lakeland as a consequence, but the Covid crisis for me was an unmasking of how deep the rot had gotten and of the inability of our body politic to manage a complex crisis requiring nuance and adaptation.
Chris:
Yeah, and the frustrating thing is that at just about any point–including now–Bhattacharya could apologize for his misrepresentations and retreat to reasonable positions that I think would still comport with his political and scientific views. All the overclaiming and misrepresentation are just so avoidable. I feel the same way about the academic psychologists who lie about their critics.
My current understanding of their bad behavior is as follows:
1. One of their core beliefs is in the importance of authority. From their perspective, they and their friends are in charge, and it’s improper to criticize them without permission. The fundamental principle is that they are owed deference at all times, and criticism from the outside can just not be tolerated except on their terms. So what, to me, would be a minor retraction, for them to say, “Hey, we messed up on that, sorry!”, would represent to them a devastating loss of status.
2. They are big-picture people. They care about principles and policy and deep ideas, not details. I get bothered when they misrepresent or lie about things, but they think I’m just being picky. This is related to our discussion the other day about advocates who think of the details of research as a sort of paperwork, something they just have to get through as part of the bigger picture of amassing credentials and influence.
Given who’s running around in the National Academy of Sciences, it’s no surprise that someone exhibiting this sort of behavior is running the National Institutes of Health. They disagree with each other politically but they can unite on the principles of deference to powerful people and the dismissal of details.
Daniel,
That is a rather infuriating story to read and must have been an even more frustrating experience for you. But I have to take issue with one thing you say : “no one cared about keeping people safe.” That can’t be right. Presumably you cared about keeping people safe, for example. Surely you don’t think you’re the only one.
And “literally the only things that actually worked were the anarchist solutions: people cooperating at human scale in groups”, that’s not right either. There are working COVID vaccines that were created, tested at least somewhat, and delivered at enormous scale in about 1/5 the time of normal vaccine development. You’re saying this was done by anarchists but even if — I say _if_ — the key inventor of the technology was an anarchist, this achievement took a huge system that was definitely not a bunch of anarchists. (I don’t know anything more about Karikó than is present in her Wikipedia article; there is no mention of her being ‘a de-facto anarchist.’ It is indeed clear that she got shit on by a bunch of institutions and individuals. Actually it is quite a remarkable story.)
You’re correct that there was a ton of bullshit around the pandemic but it’s just not true that all of the institutions were hopelessly incompentent or compromised. It is true that _most_ of the institutions were hopelessly incompetent or compromised, but I’m not sure how surprising that is supposed to be.
Basically, I’m with you on your theme but I think you are painting with much too broad a brush.
I doubt that Karikó would consider that she made the vaccine. She developed technologies for reducing the immunogenicity of nucleosides that allowed the use of mRNA-based therapies. She wouldn’t have been able to do this if it wasn’t for the institutions within which she researched, and nor would the mRNA vaccines (or the very successful adenovirus based DNA vaccines like Oxford-Astrazeneca) have been possible without the massive wealth of existing expertise on virus’s, coronavirus, viral structures and mechanisms, viral epitopes, and all of the associated molecular and cellular biology methods required that exists within institutions like the NIH.
Karikó is a truly inspiring example of dogged persistence in pursuing one’s goals in the face of difficulties, major career setbacks and absence (or worse) of institutional support. If one wishes to find a contrast between fighting relentlessly to achieve success as opposed to giving up and complaining on the sidelines about injustices she could hardly be bettered.
When I worked in the US quite a few years ago I came to the conclusion that two of the three American institutional “jewels in the crown” were the NIH and the National Parks Service – both under attack now.
I read that New York Times “scathing hit piece about how pooled testing was bullshit”. It was nothing of the sort. It pointed out perfectly reasonable issues and problems with pooled testing in practice, especially once the covid rate became higher, and explained why it was sometimes being used, but wasn’t adopted more.
I’ve personally never seen an example of the lone anarchist fighting “the man” and being right. Although I’m sure it does happen sometimes. Somewhere. Yet every single one of them is convinced if it just wasn’t for the corrupt institutions they’d be lauded as the next Jesus Christ/Einstein or whatever.
Daniel is this the piece you meant?
https://www.nytimes.com/2020/08/18/health/coronavirus-pool-testing.html?smid=nytcore-ios-share&referringSource=articleShare&sgrp=c-cb
Yes, that’s the piece. I said I wasn’t going to respond, but it’s worth it to make clear some background that you couldn’t know from reading the piece.
What isn’t at all clear in that piece was just how much conversation we had in which the author asked about the problem of high testing positivity rate, and the problems it would cause for pooled testing, and we repeatedly and explicitly explained how high testing positivity came from only people who felt they were sick or exposed being tested. We advocated screening entire populations within various organizations. In the matter that UIUC eventually did (they were responsible for some huge fraction of *all testing* in the US at one point, they tested their entire students and faculty daily or several times a week).
Of course, UIUC was doing individual testing with individual diagnostic results using a saliva based test they developed and wanted to commercialize (which later was made perhaps much less commercially relevant by the Saliva Direct consortium putting out a freely usable test procedure that cost $1 to run or something).
Instead of testing 10000 students daily with 10000 tests like UIUC did (which as we predicted had extremely low positivity rates), we would advocate something like testing 10k students daily with 4 96-well plates (pooled to about 26 people per pool for example). Through repeated testing either by entering individuals into more than one pool, or by using serial test on adjacent days, we could work out say three groups of people… those who were very safe to circulate… those who needed to come in for a test tomorrow and should otherwise limit their circulation… and those who should stay home right away for an extended period.
Having that kind of Low, Medium and High risk group classification would have vastly limited the risk of circulation while cutting the cost to provide the testing by a factor of maybe 100x over what UIUC was actually doing (both by cutting the cost to run the reactions, as well as the cost of material handling, as well as the personal cost of people standing in long lines for hours to get their samples taken.
We repeatedly explained this, and the author feigned excitement and then wrote that piece in which she essentially said that pooled testing was useless because test positivity rates were 5-10%
Note that not that long after that piece the cost of individualized testing was dropped by a factor of 4 or something because indeed Qwest and soforth got permission from the FDA to do internal pooling, and then re-testing of pools that showed up positive, reducing the number of reactions they were running by a factor of 4 or something.
So, I consider the piece to have been a hit piece which was later proven wrong by the fact that the entire country relied on pooled testing shortly thereafter.
Daniel,
Dude, that is not a hit piece no matter how wrong it may have been. The reporter spoke with a lot of experts, one of which was you, and they rightly pointed out many challenges and issues with using pooling, especially when the positivity rate is high. The fact that the reporter didn’t wholeheartedly adopt your view does not make it a “hit piece.”
I knew that many schools and other institutions adopted pooled screening but was unaware that it was used for “the whole country.” You’d know better than I but it does seem like pooling would not be worth the extra labor if the positivity rate is above some level because…well, suppose it’s over 50%, what could you gain? I dunno where the threshold would be, 20% or 10% or 5% or 2%, any of those seem plausible to me without having done or seen any calculations about it. I also don’t know how much extra labor it would take, how much more record-keeping, how much time you save or how much reagent you save, etc. You’re going to say “yes Phil, you don’t know those things but I do, I did the calculations and I knew it was the right thing to do”…. But the reporter had you telling them that and other experts telling them something else.
Possibly wrong, definitely not a hit piece.
Phil, the issue was that she asked about each of these issues and we told her what the issues were. She apparently misrepresented the point of view that she was going to portray in the piece to us in order to get our cooperation or at least led us to believe that our opinion had substantially informed the direction of the piece, and she came out with a piece that was eventually shown to be factually wrong (qwest used pooling nationwide already). My conclusion is that she had no intention of changing her mind, that she was interviewing us so as to provide cover for a preconceived opinion, our input didn’t matter and that her conclusions were set in stone before talking to us. If not a hit piece it was at least an editorial in disguise. (Note I don’t have NYT and can’t reread the piece so I’m going on memory. I also suspect they changed the title.)
You might forgive a reporter for doing this because they don’t have the expertise, and relied on the consensus of scientists but she was a PhD in biochemistry or something similar and had the expertise herself to evaluate the ideas. My impression is she had worked out some naive view along the lines of yours “if the positivity is high enough you can’t do pooling” and then went looking to push that agenda. But we had discussed the issue of basically if 20% of the population has COVID on any given day, the pandemic would be over in a week, because 80% of people would have it within 10 days, that high positivity occurs because we only test people who are sick or exposed and that testing broadly would always have lower positivity. If you figure 100% of people got COVID in 3 yrs as shown in ourworldindata then on avg there were like 0.09% of people getting new cases per day. Even in surges 5% of people positive was rare. That was like NYC in the first surge.
Here’s the first thing I found searching for quest specimen pooling. July 2020 they were doing it nationwide. It formed the basis of all testing eventually because its more cost effective.
https://www.questdiagnostics.com/content/dam/corporate/restricted/documents/covid-19/specimen-pooling-faq.pdf
Say ~2% technical false positive rate. Testing 26 people individually gives 1 – (1-0.02)^26 ~ 40% chance of one false positive. While pooling gives ~2%.
So even with *no virus at all* pooling will drastically reduce the positivity rate.
That NYT article wasn’t even anti-pooling. It pointed out some issues with wider adoption and made no claim one way or another whether is should or could be increased or decreased.
Phil sent me a gift link and I reread the piece. Note at the bottom that the TITLE of the piece was originally far more dismissive. “Pooled Testing, Once Seen as Important Tool, Is Now Useless in Much of US”
I thought it probably had expanded from its initial publication, but the content of the piece didn’t change as much as I thought based on wayback machine. Memory is definitely fallible.
I’ll revise my assessment. The piece studiously avoided saying any of the ways in which pooled testing would best be used that we discussed with her. For example by expanding the number of people being tested to screen large populations and prevent the spread of the virus rather than exclusively as a diagnostic tool to say “yes or no this person has the virus”. It then published with a dismissive title implying people were wrong to think it was an “important tool” and that it was instead “useless”. And provided multiple examples which countered the narrative of the original title (and probably that’s why they changed the title).
So, rather than a hit piece is was more of a lie by omission of the vast ways in which pooling could expand people’s ability to prevent the spread of the virus focusing instead entirely on the one thing it was probably least usable for: diagnosing people most of whom were referred by a doctor because they were already showing symptoms.
Omitting all information about the most important use of the tool is hardly a neutral stance though.
My experience is that the part that tends to rile up domain expert consultants is the headline. The dirty secret is that the headline also tends to rile up the journalists and reporters. Journalists and reporters write the text, the headline gets A/B tested. This to me is the most absurd standard practice of modern newspapers, and basically a way for the business to run-around their own journalistic standards.
somebody: wouldn’t surprise me at all.
But to give some background info on the sort of thing that was left out of that article. I certainly didn’t give this exact example but I did try to explain this concept to her, and I sent some basic charts with some use-cases in it.
Let’s imagine a town similar in size to Pasadena near where I live. Let’s for round numbers say it has 100,000 people in it. These people need to get dentistry done, surgeries, haircuts, allergy shots, cancer treatments, maybe go to schools, teach school, go to university… etc.
Now, suppose to prevent spreading this disease we want to test these people every 3 days or so. So we need to test about 100,000 / 3 = 33333 people a day. Using individual tests, at prices similar to what were charged at the time, let’s say thats $120 a test, and a cost of therefore $1440/mo/person individually.
Positivity rate in the full population averages 0.005 say at the moment (remember above I showed 0.09%/day was the overall average rate or so based on ourworldindata timeline graph).
Now, we pool people into 40 person pools. there are 96 wells per plate, so we need to do 8.6 plates per day. Let’s round that up to 10.
So, we hire 10 people and each one does ONE plate per day at $50/hr for 8 hours (this is surely overestimating cost).
So the labor cost to do our 10 plates is 8*50*10. There’s an additional $2/well in materials maybe. Let’s round up to $6000. This is for 33,333 people, so the cost per person is now $0.18 per test, and we’re testing 12 times per month, so a monthly cost of $2.16/person/mo a factor of like 500x reduction from the individual testing we were doing.
Even if you do NOTHING smart about it. You’ll get 1-(1-.005)^40 = 18% of wells positive. If people who are in positive pools come back and test tomorrow and only isolate if they get two positive wells at once… you are now isolating ~3% of the population, catching essentially EVERYONE with the disease, and soon the positivity rate would be near 0 as the spread would be nearly completely stopped.
Dentists and doctors and employers and whatever simply require that you have a negative pool in the last 2 days, or if your pool is positive that you stay home one day until you either get a negative pool the next day, or continue to stay home for 10 days to quarantine if you get 2 positives in a row.
FDA doesn’t even regulate this, it’s not required to have a CLIA lab certificate, because you don’t provide individual diagnosis! All of this could have been implemented widely around the country without anyone’s permission…
But uniformly, the thing everyone worried about was “is this legal? can we do this? will the FDA give its stamp of approval? Will the CDC? Can we get sued?” … etc etc. etc..
Permission culture. Bosses. Threats of violence* (ie. lawsuits, bankruptcy) for doing this was what killed it.
NYT didn’t want to touch that with a 10 foot pole. They essentially portrayed the problem as one of reducing the cost of individual diagnosis by studiously avoiding discussing the real benefits.
I think the NFL player example is clearer if you keep the issue as the pandemic, not a sick loved one which introduces immediate personal stakes. Both NFL cornerbacks and social scientists experienced the global pandemic. But NFL players didn’t go around tackling policymakers using their “speed and strength and coordination” (hilarious image). But, social scientists sure did go jamming their regression and survey toolkits into every opportunity the pandemic created.
The authors of the mentioned study about the potential of social science to inform policymaking during the COVID-19 pandemic later published a review of the original claims. In case someone is interested:
Ruggeri, K., Stock, F., Haslam, S.A. et al. A synthesis of evidence for policy from behavioural science during COVID-19. Nature 625, 134–147 (2024). https://doi.org/10.1038/s41586-023-06840-9
Lukasz:
Thanks. I followed the link and read the abstract, and . . . color me skeptical. But full credit to the authors to have the connections to get this sort of paper published in Nature!
Not just 42, but 80 authors! How could it not be terrific?!
In all seriousness, I have some friends and colleagues in the mix on this one, and I think there’s some good work on show. Perhaps even the missing remainder of your last paragraph Andrew…
Many of these researchers operate in a tradition that I think has some potential to deliver on the promise of the behavioural sciences: the social identity approach. Their mid-COVID effort is here:
https://uk.sagepub.com/en-gb/eur/together-apart/book275359
The difference I think is that this perspective adopts a properly social view on people, rather than the ‘individual +- social’ model that arguably dominates the field. It’s the line of effort that gives me hope that social psychology, my field, can be saved. To use your language Andrew, it keeps me from descending from social science scepticism to nihilism.
Let me know if that’s too abstract, and naturally I’d be interested in what people here think.
I glanced at the article and I see problems. Here is just one (randomly chosen) insight that was investigated: “Cultures accustomed to prioritizing freedom over security may also have more difficulty coordinating in the face of a pandemic.” That hardly seems surprising to me. And, I suppose “insights” like this can be of value to policymakers. But the claim is so general and open to interpretation that I really don’t see it offers anything more than a platform on which to place “experts” into the policy making process. In other words, I think it is really a claim that these “experts” should be consulted and used in making policy. Nothing wrong in that, since I think expertise should be used. But without a clear definition of how freedom and security are measured, I don’t see this insight as providing anything concrete, other than telling decision makers that these experts should have a seat at the table.
The other insights seem similar to me. They all seem somewhat obvious, somewhat true, and lacking in concrete details. They strike me as designed to get a place in the decision making process more than offering any real concrete advice about what to do. They sound like pleas saying “you should involve these subject matter experts because they have useful knowledge” but without providing any real useful knowledge. Is it really useful to know that individual freedom can inhibit responses to a pandemic? Would decision makers have failed to understand that without these experts? The claims look like any social science textbook: plausible ideas, somewhat backed by evidence, but not specific to the issues at hand – a platform for getting a place at the table, not a recipe for action. Maybe nothing is wrong with that, but it feels like a invitation to overestimate the contributions of behavioral science to policy making.
Everything is obvious once someone points it out.
Andrew, did you see this article* that came out later in 2020?
https://www.pnas.org/doi/10.1073/pnas.2009787117
This one also has insights from behavioral science, but these at least seem useful. I guess we could say that these authors had the benefit of some hindsight as the pandemic was further along and that they could cherry pick the ideas that fit the problems b/c they could better see what the problems were, but the 10 insights are based on work that generally far predates the pandemic.
Anyway, this is just to say that there is some decent quality research that has/had policy implications for a pandemic response.
*Yes, I know, PNAS, but if you look beyond that…
Adam:
This article you link to is a lot more interesting than the 42-authored piece discussed in my post. It’s thought-provoking and fun to read. That said, I don’t find it persuasive. All its claims seem to me to be stretching credulity. The whole thing reminds me of an article I read in Omni magazine many years ago, featuring a bunch of what they called “partly-baked ideas.” The one I remember was to take some plant such as a Venus flytrap that opens and closes its leaves once per day, keep it in an inside environment with lighting control, and gradually decrease the period of night and day from 24 hours to 23 hours to 22 hours to shorter and shorter, until eventually the light is flickering on and off multiple times a second and the leaves are flapping so fast that the plant elevates off the ground. Clever, but I don’t think it’ll happen. That’s how I felt about the ideas in that PNAS article.
The linked paper is not any worse than the other stuff, but it’s not any better either. Stuff like this:
“How can we use our knowledge of dual inheritance theory to make cultural evolution take place faster and at a larger scale than ever before—even so fast that it can keep pace with the genetic evolution of the virus? One way is to focus on the three ingredients that define a Darwinian process—selection, variation, and replication—and manage them at a systemic scale. In other words, we must identify a systemic target of selection (such as reducing transmission of the virus), orient variation around the target (including monitoring unplanned variation and controlled experiments), and identify and replicate best practices. Cultivating cultural evolution in this manner must be done realizing that such practices are likely to be sensitive to context, requiring adaptive adjustments at a finer scale (73), and that measuring efficacy will be difficult and not entirely objective.”
This is gibberish.
Now imagine social scientists REALLY helping. In this scenario, they would have pointed out at the beginning of the pandemic that the right-wing echo chamber was sure to manufacture a conspiracy about the “China virus.” Knowing that this was going to happen no matter what, the government could have manipulated hearts and minds so that the conspiracy was beneficial to public health efforts. For example, the CDC could have announced that people living in big cities were at higher risk, so all the early vaccine doses were going to the big cities first. Next, key right-wing influencers could have been hired to claim that the big-city-first approach was intended to kill off Republican voters in rural areas, and that those Republicans should DEMAND early access to the vaccines. Since the big-city libs were going to get vaccinated anyway, a public nudge like this could have increased vaccine uptake by 20%.
Stuff like that.
Matt –
. For example, the CDC could have announced that people living in big cities were at higher risk, so all the early vaccine doses were going to the big cities first. Next, key right-wing influencers could have been hired to claim that the big-city-first approach was intended to kill off Republican voters in rural areas, and that those Republicans should DEMAND early access to the vaccines. Since the big-city libs were going to get vaccinated anyway, a public nudge like this could have increased vaccine uptake by 20%.
I’m hoping you’re not being serious. First, I find it hard to believe that you think they should have promoted completely made up and knowingly contrary to fact public messages. Even if it were a good idea, the only way it could be done is through a vast agreement to conspire to deceive the public. Even if you think that’s a plausible scenario and that people wouldn’t object morally, it would expose people to huge risks (of the conspiracy to lie) that at least some wouldn’t take..
Second, I think your theoretical reaction among RWers is highly unlikely. Yes, just being oppostional to librul institutions of science is a motivating factor for RWers, but there are other reasons for their baseline resistence o vaccination. As an example, Alex Jones has been anti-vax for years. Not even his fear-mongering about “globalists” would have erased that baseline opposition to vaxes.
I think that multiple social scientists did point that out. That said, probably other people who use the term “social science” to mean “Andrew’s tag social science” probably are not doing much scholarly reading in the field.
That said, anything talking about “dual inheritance” in the context of “cultural evolution” is not part of the social science establishment.
Joshua wrote:
“I’m hoping you’re not being serious.”
Yeesh. It was an attempt at irony. There were a lot of pathological behaviors exhibited by various demographic groups. In hindsight, some seem inevitable, like the defection from public health measures by people who choose defection as a way of life. Most of those same folks parroted stupid things they heard about vaccines but were all in on monoclonal antibodies when they were sick themselves, despite the antibodies also being an unproven product pushed by Big Pharma, etcetera ad nauseum.
I’m not really pushing the idea that psychologists should have anticipated any of these goofy human behavioral responses in advance. But the point I was trying to make is that if social scientists had been able to help, it would have been by predicting how humans with various views would respond to public health measures, and if we had those predictions and had heeded them and responded accordingly, lives would have been saved. Instead, we got a lot of stuff from social scientists about fear and loneliness, the latter being mostly driven by a dearth of new content on Instagram.
Matt: I think those “goofy human behavioral responses” were very well known to psychologists, or even anyone with a bit of history. I remember quite a few articles about parallels to the 1918 flu, with anti-maskers back then too. The problem is that it’s hard to publish a paper basically saying here-we-go-again. And while some media market exists, it’s rather limited. Someone would mostly have to go to places often regarded as very ranty partisan-left, which is not all that good if a person wants to be considered a respectable intellectual.
Suffering midlife crisis, but perhaps statistics and social science don’t have much insight to offer. Social interactions are irreducibly complex, thus only AI black box models have any hope of providing satisfactory predictive accuracy. We run studies with Bayesian or frequenting methods but all the signal is drown by noise. Articles that have me regretting my graduate studies in social science and stats: two cultures by Breiman and the bitter lesson by Sutton. Also taleb’s criticism of stats. My counter argument has been, perhaps stat models can’t compete with LLMs but at least they offer interpretability. Seems more and more like copium these days. People should drop stats and go straight to CS though that too will be automated soon
Social scientists _should_ have stepped up. Evidence to the UK covid inquiry shows that the UK government, and I believe others, were not sufficiently aware of the costs of the lockdown that they were imposing. Even at the time I noticed that psychologists who had founded careers on claiming to distinguish the effects of subliminal cues and microagressions on human behaviour were telling us that being forced to wear a mask had no effect whatsoever on human interactions or on childhood development.
“were telling us that being forced to wear a mask had no effect whatsoever on human interactions or on childhood development.”
They would have been correct had they been talking about Japan, which has pretty much no problem whatsoever with mask wearing (and saw a rather lower Covid fatality rate).
I love the passive aggressive rhetorical strategy of making “factual” statements in the form of describing studies that were not done that you assume would agree with you.
I didn’t mean for there to be anything passive in that response: keeping kids and their families safe from a deadly disease with a couple of really simple interventions (ventillation and masking) that actually work is such a no-brainer that the incredible stupidity of the anti-mask folks should be blatantly obvious. But they whimper about “childhood development”. If you care about people dying, your choice is going to school with masks vs. not going to school. Clearly not going to school is going to be worse for “childhood development”. You don’t need a friggin’ study to see that. (Of course, the “childhood development” folks probably are fine with kids bringing Covid home. (Also, Covid was the leading transmissible cause of death among children for a while there: the idea that Covid doesn’t kill kids is dead wrong.))
So, no. Nothing “passive” in my opinion here. Just anger at the folks (anti-mask and others) who resulted in millions of excess deaths in the US.
The actual situation is that many people have very (I mean extremely) low standards for medical interventions.
Because of that group we all suffer and die early because there is no incentive to produce interventions that actually work.
These low standards are a big problem, and it is getting bigger. Even the most obvious safeguards are now being dropped:
Injecting hydrophobic drugs without aspirating (making sure it doesn’t go directly into a vein) is now normalized. Literal elder abuse is now considered “medicine” to a significant part of the population.
Stuff like this is why US life expectancy peaked in 2014. People won’t stop supporting junk healthcare. Just have high standards (even like back in ~1950) and we will have actual cures for disease, things were headed in such a great direction.
And then, these same low standards people want to force their crappy (even actively harmful) “medicine” on everyone else. Its a very bad sign for civilization.
Sorry I actually misunderstood what you are saying.
Andrew –
Tangentially related main thrust of your OP, but you say this:
One of their core beliefs is in the importance of authority. From their perspective, they and their friends are in charge, and it’s improper to criticize them without permission.
I was listening to this podcast
https://open.spotify.com/episode/23zZDnXEBcE7NADS9F4jSg?si=H70st3jPTrKwnTM7EHi_3Q
And it helped to bring some things into focus.
This podcast offered what I thought is useful distinction between authoritarianism and patrimonialism. Authoritarianism relies on strong state capacity—think efficient systems and controlled media, like trains running on time. Patrimonialism, however, thrives in undermining state capacity and low-trust environments with weak institutions, where power flows through personal networks. So the trains are unreliable and no one trusts the media or scientific institutions. Think of a mafia “family” loyal to a patron.
The behaviors you describe —demanding deference, dismissing criticism as nitpicking—fits patrimonialism. Leaders in “legacy media,” institutional science, and public health (e.g., NIH) form a nexus that operates like such a network. Their authority stems from prestige and loyalty within this “family,” not from robust, trustworthy institutions. Admitting errors would undermine their status. In a low-trust system, protecting the network’s image trumps accountability, explaining their resistance to retracting misrepresentations.
This process links the science-specific focus of your post with the directly connected political goings on, that correctly link Bobby Kennedy and Battacharya (and Makary) to the Trump admin that empowers them as “made men” to carry the analogy
The Podcast conversation with Jonathan Rauch was interesting Joshua, thanks.
Two things resonated with me:
1. The discussion (around 36 min in) of the diminishing of Christianity in the US and its partial resurgence in the form of an evangelical white church that is partly an adjunct of the Republican party and has become associated with the whole MAGA thing. And how replacement of religious faith with secular political alignment has had a pernicious effect on wellbeing especially amongst parts of society that would in the past have been more likely to align with more “traditional” forms of Christianity. Though like the podcasters I’m non-religious I can acknowledge the nice discussion of the probability that faith may have answers to questions that secular alternatives can’t provide.
2. The way that individuals that may be essentially liberal in outlook can end up inadvertently supporting illiberal agendas through a combination of an inability to avoid absorbing some of the massive agenda-led misrepresentation and criticism of individuals and institutions with which they would normally align, and a sort of unhelpful sense of right and wrong that induces them to participate in the criticism of targets thrown into the limelight by those agendas. It relates IMO partly to the above – replacement of personal faith and morality that helps one deal with adverse outcomes, with the adoption of angry sectarian positions so that small-p political dissatisfactions are easily manipulated by large-P political agendas ( a typical populist strategy).
I’m thinking for example of the massive amount of agenda-led anti-science stuff that goes on which intersects with a bunch of small-p positions (“I didn’t get my last two grants funded so I’m glad the NIH is being attacked” or “I felt badly treated by my advisor in grad school and so I support the right-wing anti-academia schtick”).
Chris –
If I read you right, I agree. I tend to get uncomfortable with a lot of the criticism of “mainstream science” that comes in the form of criticism of peer-review and the like. Much of the frequent commentary on the “replication crisis” fsllaninti that category. Not that there aren’t obvious merits to a lot of that criticism; but as I see it, much of that criticism ends up empowering negative outcomes. Important institutions and norms are judged against unrealistic, binary standards, and unfalsifiable claims. It replaces imperfect institutions with batshit crazy conspiracy theories that function mostly to channel power a “patron” network. That’s what we have with Bobby Kennedy and Battacharya and Makary leading out scientific institutions as “made men” of the Trump family. As flawed as our scientific institutions and norms have been, this is a much worse outcome.
How can fear appeals be used in policy? Maybe not directly as law or policy with a small “p”, but in terms of response, the findings are useful for health messaging and media campaigns. Is that not obvious? A fear appeal that is too arousing may lead to avoiding that message altogether – sorta counterproductive for such communication approaches.