What about that new paper estimating the effects of lockdowns etc?

A couple people pointed me to this article, “Assessing Mandatory Stay‐at‐Home and Business Closure Effects on the Spread of COVID‐19,” which reports:

The most restrictive non‐pharmaceutical interventions (NPIs) for controlling the spread of COVID‐19 are mandatory stay‐at‐home and business closures. . . . We evaluate the effects on epidemic case growth of more restrictive NPIs (mrNPIs), above and beyond those of less restrictive NPIs (lrNPIs).

We first estimate COVID‐19 case growth in relation to any NPI implementation in subnational regions of 10 countries: England, France, Germany, Iran, Italy, Netherlands, Spain, South Korea, Sweden, and the US. Using first‐difference models with fixed effects, we isolate the effects of mrNPIs by subtracting the combined effects of lrNPIs and epidemic dynamics from all NPIs. We use case growth in Sweden and South Korea, two countries that did not implement mandatory stay‐at‐home and business closures, as comparison countries for the other 8 countries (16 total comparisons).

Implementing any NPIs was associated with significant reductions in case growth in 9 out of 10 study countries, including South Korea and Sweden that implemented only lrNPIs (Spain had a non‐significant effect). After subtracting the epidemic and lrNPI effects, we find no clear, significant beneficial effect of mrNPIs on case growth in any country. . . .

The article was also discussed in this news report, which situates the above findings in comparison to other studies of anti-coronavirus policies.

I’m in contact with Seth Flaxman and Samir Bhatt, who earlier this year published a study claiming that lockdowns are effective (see some debate here) to ask them what they thought of this new study. Here’s what they said:

Flaxman:

At this point, there have been so many second (and third) lockdowns that effects are clearly visible with the naked eye. Check out Ireland—and try to guess what week lockdown came into effect. More broadly, I’ve noticed a trend and I wonder if your readers have other examples—it’s sort of the opposite of what the replication crisis has usually focused on: I’m seeing studies with small sample sizes and / or noisy measures, in which researchers fail to find a significant effect (perhaps because they were careful in adjusting for multiple comparisons!). But then the story that’s told (in the media) is a strong conclusion equating “lack of evidence for an effect” with “no effect.” One example is the Danish mask study; discussing it in BMJ, Abbasi uses the terms “inconclusive” and “negative,” writing that the study should have been interpreted as inconclusive (lack of evidence), rather than negative (no effect). I’d be curious if others have useful ways of framing this issue or examples. It’s not an easy issue, to be sure–we’ve struggled with it as well! In addition to Bendavid et al. and the Danish mask study, other examples are: (1) reporting by the ONS of infection rates among teachers in England critiqued by Dr. Sarah Rasmussen and (2) this reanalysis of early data on children by McConway & Spiegelhalter.

Bhatt:

The modeling approach using growth rates is reasonable, there is something I don’t understand where they do some kind of mediation analysis on “country pairs,” and there is no acknowledgement of collinearity in NPI timings or how the NPIs were chosen. I would like to flag an important issue overlooked by the authors and other researchers – the use of case data in the first wave. For example, South Korea tested very widely from the get go, the UK tested only in hospital suspected covid cases (from early March). Leave aside these two not having comparable ascertainments rates, the UK case data is more correlated with hospitalisations than actual cases, which means its lagged. This is why we used deaths in our first paper, but we now use both. Their results are very different to the similar (but I think) more robust analysis done by Chris Dye (https://www.medrxiv.org/content/10.1101/2020.06.26.20131144v2). This work would have been much more powerful if they used data post-June for second lockdowns in Europe. More broadly, the difference between mandatory and voluntary NPIs is very important, but one that is still very difficult to answer given current data. We need knowledge of adherence. This exposes once again how important behavioural surveys are.

It seems clear that stopping people from circulating will reduce transmission. Whether this is done by a lockdown order or by people voluntarily staying at home as much as possible, or by some mix of these . . . well, the virus doesn’t care what the law is, it just cares what people do. From a causal inference standpoint, there’s no “effect of lockdown.” Rather, there’s an effect of lockdown compared to the alternative. If lockdowns are done when people are already locking down, maybe they’ll have no epidemiological effect—but, then again, people are already locking down, hence no economic effect either. On the other hand, if the disease spread can be reduced with milder measures, that’s great. That was one of the points of the earlier Flaxman et al. paper, that certain steps such as school closings didn’t actually do much to stop the spread of the virus. On the other hand, if you don’t close schools, you’re sending school employees to work every day risking their health, so there are issues of governance and trust and shared risk to consider.

I’m just babbling now . . . Ok, my point is that, like Flaxman, I’m super-skeptical of the claim that lockdowns etc. have no effect. It’s fine to be against lockdowns, and there are lots of good anti-lockdown arguments that don’t require the claim that they have no effect. You can make the cost-benefit argument, or the trust in government argument, or go full Hoover and deny there’s any reason to be concerned about a pandemic. You can argue that most of the people are gonna get exposed anyway so we should just get it over with, or you can argue that targeted policies could be more effective: go for the superspreader events, not for people who are already being careful. It’s complicated because policies are designed to effect behavior, but behavior changes on its own, and policies are sometimes implemented in response to behaviors. For example, a governments can institute a lockdown for two different, opposite reasons: (1) because the disease is spreading like wildfire and people aren’t altering their behavior, hence there’s a need, or perceived need, for drastic measures to slow the spread, or (2) because the disease is spreading and people are scared, pulling their kids from school and not going to work, and a lockdown or something like it will solve the coordination problem. This is not to say that lockdown or whatever is the best policy in either case, just that empirical policy evaluation and decision making is difficult in such settings. And that would be the case even without our polarized political environment.

179 thoughts on “What about that new paper estimating the effects of lockdowns etc?

  1. I have the same reaction to these NPI studies as I had last March/April studies on modeling the spread of COVID. Every day a new model (or more than 1) comes out. They are all complex and are all studying a worthwhile topic. But I feel like the data is just not there to support any of their conclusions. Lockdowns are like many legislative acts – it is easy to define the time at which an official action is taken, but there is considerable anticipation in advance of that action. There is also endogeneity where voluntary actions, infection rates, and government actions are all happening at the same time. There is still the difficulty of getting accurate measures of the infection rate (less so for death rates, but even that still has some measurement issues). The actions themselves are difficult to measure. Some places have mask mandates, others use moral suasion to urge people to behave responsibly. And, the effectiveness of official actions and pleas is likely to vary across cultural settings (even within countries, but certainly across countries).

    All of these issues are now new, nor are they unique to COVID. However, I feel that the cumulative effect of all these measurement issues, combined with the compressed time and need for such analyses, is creating an atmosphere where modeling progress is elusive and highly uncertain. Again, nothing new – this is how science works – but the time frame in which we are looking for results does not match up with the time needed to reduce some of these data problems. The consequence, for me at least, is that I am reluctant to look into any of these studies in detail. If the data is not publicly available I don’t look at them at all. When the data is available, I’ve occasionally looked at it, but the measurement questions arise so quickly that it isn’t clear to me how/whether to proceed with my own analysis.

    When you put all this into a political climate where people have strong opinions about whether these policies are effective, necessary, and worth the cost, it isn’t clear what you get. Something worse than the sausage that politics usually produces? To be clear, I am not advocating that research stop on these issues – this is very important work and should continue for a long time (it will have many implications for future understanding). But what I think is needed is some buffer between these research efforts and the impending decisions that need to be made. I guess I’m not convinced that the little light that these studies can shed on the subject will lead to better short-term decisions that we need to make. My own preference would be for better descriptive data to inform these decisions, and less reliance on the complex modeling that is being attempted.

    • The fact is that we are in a completely different position with respect to the virus than with most social science questions. We have a very good understanding of the mechanism of disease spread. The disease is cause by a virus, the virus is spread primarily through respiratory droplets and aerosols. The evidence for aerosols was there from the choir practice etc as far back as March… If we want to keep the virus from spreading we need to keep healthy people from coming in contact with the virus. It’s as simple as if you want people not to have greasy hands you need to keep them from touching greasy surfaces.

      “Do lockdowns work” is the wrong question, the question is “does keeping people from breathing air with virus in it or having direct contact with respiratory droplets from the moths of others prevent them from getting the illness?” The answer has always been 100% yes and no amount of statistical bullshit was going to change that. We have literally overwhelming evidence for the viral cause and the viral spread from over a hundred years of biology. Some bullshit study about effectiveness of stay at home orders with noisy proxy measurements has never had any weight at all other than political.

      Keep people from contacting the virus and you keep them from getting sick. End of story.

      • Nobody has ever doubted the fact that if everyone in the world could go 20 days without leaving their own bedroom the virus would disappear. Saying that keeping people from having contact with each other is the cure for a pandemic is like saying just buy ten thousand shares of a stock that’s going to double in price next week and you’ll be rich.

        Given that your so-called solution is, you know, impossible it becomes 100% a political (or policy) question as to what real-world steps should be taken to deal with virus as well as possible while allowing society to continue to function.

        But I totally agree with you that these statistical models are not informing anyone of anything w.r.t. identifying the best policy choices. Apparently, one full year into the pandemic there is STILL no data available to allow a real-world understanding the disease’s interaction with human behavior.

        • To the extent that contacts between people decrease the disease spread is much slower. But this ABSOLUTELY IS (or was) being doubted by MANY people in this country. There were reports from nurses in South Dakota about people dying in hospitals who were insisting that the virus doesn’t really exist.

          We couldn’t just wipe out the virus but there was never a need to argue about whether restrictions on motion reduced the spread. The reality was we needed to monitor spread through proper surveillance, and respond with localized lockdowns in regions where spread was growing. This shouldn’t have been controversial, but with a president who insisted that “the Chinese Virus” would “just go away” and it “was like a flu” or whatever. to hell with that idea, we got Sturgis motorcycle rally, and etc.

          The idea that “nobody has ever doubted” that we could treat the spread through dramatically reduced social contact is definitely very counter to the reality we experienced all year. Hell people have been attacked bodily for asking customers to put on masks before entering a store. Less virus in the air = less chance of transmission but that was definitely widely “doubted”

        • I very much doubt that masks AS ACTUALLY WORN IN THE REAL WORLD have any meaningful effect on the chance of one customer in a store passing COVID to another. I wear a mask because what the heck it probably helps a little but the real world efficacy of mask usage is another thing that’s treated as so obvious (supposedly) that it admits no analysis. I just detest the way everyone confuses “obvious” with “my own preconceptions” and nobody bothers to collect actual data and analyze it. You in particular I think are much too intelligent to continue stating qualitative “less virus = less chance of transmission” memes as some sort of substitute for quantitative inference supported by objective data.

        • I fully support the idea that we are doing a TERRIBLE job of masking the public. Right from the start we should have been cranking out as many N95s as possible and importing and vetting KN95s as well. People wear their masks with their nose sticking out etc… We know masks can be effective because there are experimental tests that extablish how well these masks work when used properly.

          Meantime we’ve got craploads of people sewing masks when what’s needed is N95s and KN95s and some kind of maybe not quite as good but at least well controlled mask: https://www.theatlantic.com/health/archive/2021/01/why-arent-we-wearing-better-masks/617656/?utm_source=pocket-newtab

          Other countries have made and handed out these things… we didn’t 100% because under Trump we were essentially a failed state culminating in letting just any old bunch of yahoos run rampant over our house and senate floors.

          We don’t need studies on how well “real world” mask usage made out of grannies old underpants works. we need effective goddamn masks and a mandate that’s enforced.

        • I mean, we act like there aren’t thousands of well educated engineers who **specifically design respiratory protection for a living**. We know what effective masks look like, and we know what we’re supposed to do to wear them correctly, and there are whole industries that know how to message/train people who don’t have tons of science education to use them properly (such as say welders or agricultural equipment maintenance workers etc) we don’t need studies where we regress super noisy measures like “presence/absence of mask mandate laws” against “covid spread” we need the damn masks and the messaging. There’s no shortage of capacity, or knowledge! there’s shortage of political will!

        • I’m not convinced (at all) that masks need to be N95 or better in order to have a substantial effect:

          1. Viruses are very small, but droplets are not (including virus-containing droplets that, floating freely in air, would quickly evaporate and become small). A couple of layers of moderately tightly woven fabric can remove a lot of virus.

          2. I think we all recognize that inhalation is the main mechanism for infection, but as far as I know it may also be possible to get sick by touching a contaminated surface and then touching your eye, nose, or lips. By catching droplets, masks reduce the contamination of surfaces.

          3. Masks reduce the amount of air (and particles) exchanged between people standing at or near conversational distance. When you speak or even breathe, your breath typically carries at least a few feet in the direction you’re facing. Normally, you and the grocery store clerk are exchanging a fair amount of air. But if you’re each wearing a mask, you are each inhaling much less of the air the other person has just exhaled. In a confined space such as an elevator this doesn’t matter, you each create an expanding cloud of recently-exhaled air that soon fills the space. But in a brief encounter in a fairly open space this can be a large effect. I haven’t tried to quantify it but it’s certainly not negligible, especially in a situation in which there is a huge difference between R0 = 0.9 and R0 = 1.1.

          I think item 3 may be the largest of these effects. If so, perhaps transparent face shields would be just as effective as masks (or even more effective).

          In any case I don’t think it takes N95 masks to have a beneficial effect.

        • Phil, I actually agree with everything you say and for precisely the reasons you mention. Also I know that testing these things in a laboratory is possible and we know protocols for testing, so we should have done those tests. But none of that argues against N95, it just means we should still use the “less good” masks as well, but if we had N95 or KN95 or even “COVID masks” manufactured to a decent tolerance with an ad-hoc testing protocol behind them designed to quantify and assure effectiveness at some level, we wouldn’t be better off. We should absolutely have put a lot of effort into all of that. We didn’t.

        • To Anoneuoid’s question, “In what country did everyone wear N95s correctly?”: It could have been the US, if we had actually done the work to make it happen.

          I defer to others on the efficacy of different types of mask, but the broader point is the fact that we fucked up by not even trying should not deter us from continuing to try things.

        • The thing is you don’t have to wear a N95 correctly every moment of the day in order for it to be better than you wearing 3 layers of socks over your face incorrectly all day.

          N95s when used in real danger situations like Hospitals can be reused 50 times by heating them to 180F for 20 mins in dry heat (I can’t find this citation but I did a few weeks ago for my sister, you can definitely google this one up if you like). Or by just leaving them in a paper bag for 3 days… so you can rotate through them. I had a box of 20, I use them every time I go out to places where I could be with other people. I go out fairly rarely, and reuse them a lot, I’m confident that I’m better by far than wearing a couple layers of cotton fabric. When not used for infectious materials they’ve always been reusable (like gypsum dust etc, it’s always been allowable to reuse them until they start to become “clogged”). I’ve gone through like 3 of them since the start of the pandemic.

          The fact that not everyone will use an N95 correctly is not proof that we should all just not use masks, or just let everyone use some old socks or whatever stupid thing they’re doing. N95 or KN95 or even some kind of validated tested “paper mask” designed ad-hoc for the pandemic (which is what some other countries have done) would be a lot better than what we have, which is a ton of people doing random stuff and then wearing them stupidly.

        • > To Anoneuoid’s question, “In what country did everyone wear N95s correctly?”: It could have been the US, if we had actually done the work to make it happen.

          I notice you failed to name anywhere it was done.

          Theres a video of someone picking up dog crap with their mask and putting it back on in the US. What sort of mandate or training do you think will prevent that type of behaviour?

        • To Anoneuoid’s question, based on the Atlantic article I linked above:

          “Not all countries have this problem. Taiwan massively scaled up its manufacturing of masks at the start of 2020, such that by April every citizen received a fresh supply of high-quality masks each week, and the distribution system was regulated by the government. Taiwan’s COVID-19 death rate per capita is more than 1,000 times lower than that in the U.S. Hong Kong has been distributing patented six-layer masks (the efficacy of which has been laboratory tested) to every citizen. Singapore is on at least its fourth round of distributing free, reusable, multilayer masks with filters to everyone—even kids, who get kid-size ones. In Germany, Bavaria has just announced that it will be requiring higher-grade masks. If all of these places can do this, why can’t we?”

          Did they get everyone to wear N95s correctly all the time? No. Did they get people to wear much more effective and quality controlled masks most of the time? Hell yes, and did it help, hell yes.

        • Thanks.

          A quick search shows Taiwan, Hong Kong, and Singapore all closed their borders in January. This was being called racist or xenophobic by politicians in the US and Italy.

          I’d guess that played a bigger role in slowing the spread than masks. So I don’t think we can conclude from that passing out these masks did much.

          I’d also guess everyone who is going to use an N95 even close to correctly in the US is already doing it. It is much more uncomfortable than the surgical or makeshift masks so I bet you would just get people poking hidden holes in them and such.

        • Phil: For your point 3, are you talking about aerosols? It seems there’s consensus now that it’s a relevant mode of transmission although there’s probably some debate about exactly how important it is. Maybe I’m understanding wrong but under your point 3 wouldn’t N95 make a big difference? They’re designed specifically to prevent stuff like aerosols, no? And perhaps it doesn’t matter if say you’re just going to the grocery store for 30 minutes but if you work there then it seems like there could be a lot more exposure risk from aerosols so quality of masks is much more important.

          At any rate, I do think N95s are a good idea because there really isn’t much downside cost at all and a ton of potential benefit, even if you disagree on the likeliness of the benefit. Precautionary principle is important here, unless you think the probability of benefit is super low. As Tufekci mentioned in her article, KN95 masks are pretty much the same price as cloth masks, and they also fit better (which, side note for anyone who wears glasses, this helps with the fogging up issue). I don’t think they’re more uncomfortable either. I might be wearing it “wrong” too but it might still be an improvement over the cloth stuff (and definitely not worse).

        • Phil: I can’t believe you’re still talking about surfaces and droplets. The evidence against these being anything other than a minor mode of transmission is overwhelming.

          Just make a physical model of the virus moving through a party by each mode of transmission. Aerosols a can blow and diffuse across the room and increase in concentration with every breath of an infected person.

          But even if a man sneezes in his hand and grabs the bathroom doorknob, and even if someone else grabs it just minutes later, every single thing the person touches will wipe more contaminated material off their hands. It gets thinner and thinner and less and less dangerous with each touch. And still no one knows how it causes and infection from there – licking the hands? Really the more you think about it you’ll see that things you grab with your hands – the side of a beer can, cup handles, door knobs, etc – just don’t wind up in your mouth. If there is even an oral path to infection.

          The reality is that if you model every droplet and hand touch, I think you’ll see that the odds of a virus infecting a significant number of people at a party in this way are extremely small. The only reasonable path for such rapid spread is aerosol.

          The more I think about this the more I think that the flu is probably also primarily transmitted by aerosols and not through surfaces. That’s why it spreads so quickly through the entire population.

        • I don’t think surface transmission is an *essentially* irrelevant mode of transmission. It’s just that of all the advice, the easiest for people to understand is “if you’re in the environment around other people or receiving packages etc, don’t touch your face until after you wash your hands”. Furthermore, the protection is directly to the practitioner, so there’s a strong incentive to comply.

          I see people all over the place doing the hand cleaning and not touching stuff.. So if everyone is complying, then yeah, you’re not going to see much transmission this route.

          That doesn’t mean “surfaces don’t transmit the virus” or “you can safely pick your nose after opening a door to the store”. It just means this mode is the easiest to disrupt.

          People should continue to wash hands and not touch their faces. If they do that, then surface contamination will continue to be the least important mode of transmission. if people give up on all that, suddenly we’ll see people getting surface transmission more often.

          As I understand it, several outbreaks in NZ have been thought to be most-likely surface transmission, so it’s clear that this is possible.

          https://www.rnz.co.nz/news/national/427446/rubbish-bin-the-likely-source-of-covid-infection

        • Hong Kong will place tens of thousands of its residents in a lockdown to contain a new outbreak of the coronavirus, the first such measure the Chinese-ruled city has taken since the pandemic began, a local newspaper reported on Friday.

          “Persistently high and spreading infection [in the areas] and sewage surveillance suggest the outbreak is not yet under control, and many silent sources still exist within the area,” a source was quoted as saying.

          https://www.reuters.com/article/us-health-coronavirus-hongkong-idUSKBN29R0CA

        • Michael J,
          Sorry about the slow response, I had somehow missed your comment.

          Yes I’m talking about aerosols and yes I think N95 masks are better at removing them than a typical homemade cloth mask will be. I certainly encourage people to wear good masks. But Daniel seemed to be saying that cloth masks are essentially completely ineffective, and I don’t think that’s the case at all.

        • Another thing about masks which I noticed at least for myself is that by hiding parts of your mimic, they discourage you from non-essential talk with strangers, because communication becomes a bit more tedious without those non-verbal elements. If I’m with a friend and we go to a shop where we have to wear masks, I also think we usually don’t talk that much in the shop and continue the conversation afterwards. So it would be interesting if masks also reduce spread by reducing communication in general a little bit.

        • N-95 (surgical) masks tested to 0.3 microns (manufacturer)

          COVID coronavirus measured at 0.12 micron (https://www.gisaid.org/fileadmin/_processed_/csm_betacoronavirus_Wuhan_Jan_2020_d88d5d2f4a.png) with active virions down to 1/5th of that

          human droplets size range down to largely ambient air

          (smallpox 0.2 micron, ebola 1.0 micron)

          N-95 masks are certified against penetration by active bacteria, which are very much larger than the C-19 virus particle

          Trying to catch sand particles with a tennis net …

          Some reply to this by saying masks are to stop other people from being infected by trapping your exhalations, as if they only worked one way or something. Nanny state control, I suspect.

          1 micron = 1000 nanometres = 0.001 mm)

        • I have worked with multiple people who are involved in testing. The highest concentration in saliva any of them had measured was about 10^9 virions per milliliter.

          A 1 micron diameter sphere has a volume of 4e-12 ml, so the expected number of virions in 1 micron droplets is 0.0042

          It’s generally considered that “aerosols” are less than 5microns. The volume decreases like the cube of the diameter. Particles in the range of 0.3 to 3 microns are the ones of most concern, and they are effectively filtered by N95 masks.

        • That’s not how N95 masks work and that’s not how the viruses exist in aerosols. The viruses are in droplets containing salts and protein and, initially at least, some water. These droplets stick to mask fibers when they collide with them. Large droplets collide not because of sieve size but because inertia makes them deviate from the air streamline. Small droplets collide because Brownian motion makes them deviate from the air streamline. In between (~0.3 micron) is the hard range, but electrostatic charges in good N95s help capture those.

        • Thanks for explaining this, better than I had been planning to. One might still ask, how likely is a droplet/aerosol it collide with a fiber? I just measured one of my (industrial, not medically-rated N95 masks at 0.9 mm, the center layer at 0.22 mm. So, a 0.12 micron coronavirus is attempting to negotiate a mat of fibers several thousand times thicker than its own diameter. It’s not like trying to catch sand in a tennis net, ianl, it’s like trying to run through a kilometer of forest, off-trail, blindfolded.

      • My take is a mixture of yours Dale’s. The virus follows a reasonably clear mechanistic process of spread. So if everyone was in a personal bubble for 2 weeks, it would be largely gone.

        But, policy != behavior. We’ve seen prohibition and the drug war fall flat. The arguments I’ve heard (but not really bought into), is that lockdowns force people who want to socialize to do so in more dangerous ways. Rather than meeting in the park, they crowd into apartments and close all windows and doors to keep the neighbors from hearing as much. Rather than a reduced capacity bar, they go to packed speakeasies. Virus spread is intrinsically tied to human behavior, and that complicates analysis of policy.

        Still, I tend to believe first-order effects of restricting certain actions should be given larger weight than those second-order effects of rule-breaking being more dangerous. Having a prior that the effect of lockdowns is zero feels silly, given what we know of disease spread.

        But, the data is way too noisy and even if not, just like with prohibition and the drug war, ones opinion often hinges on issues besides direct effectiveness. I feel like often people have the compulsion to argue over the scientific measures (covid spread, global temperature rises, harms of marijuana use, etc) and act like that alone dictates correct policy (lockdowns, carbon tax, drug war), even when there’s many other factors that need to be considered (morality, compliance, economic costs, other unmeasurables).

        • “Rather than meeting in the park, they crowd into apartments and close all windows and doors to keep the neighbors from hearing as much.”

          Or that people crammed in walmarts b/c other businesses are closed, causing the spread to worsen. I was really worried about that in March/April.

        • The main argument against this is that there haven’t been to my knowledge any outbreaks traced to a retail establishment.

          As I’ve argued again and again, big box stores with their 30-50ft ceilings are an extremely unlikely place for people to become infected. The virus in aerosol form would disperse upward and ventilate out. Also people are continually moving through stores, not gathering in one corner and breathing there for hours, allowing aerosol concentrations to increase.

        • The big clusters of employee cases in LA county are Target, Costo, Home-Depot.

          The county updates these lists ASAP and the pattern became clear as the numbers began to get very large last month.

          It is much more difficult to ascertain — apparently — whether alongside these reported clusters of *employees* there are latent/silent unreported clusters of *customers* associated with these same sites.

          It may be simply that these business are reporting the counts directly to the county; so the county is not actually in the business of *discovering* these clusters by backward projection in time from generic case reports.

          However, this is my point: there are indeed sizeable clusters of infected *employees* associated with certain categories of sites (large box department stores) and on any reasonable hypothesis at all, it should be assumed therefore that these sites are also generating clusters of infected *customers*

        • Thanks Ron, I knew you were following this. It would surprise me a lot if big box stores weren’t a major source of cases contrary to Jim’s thinking. There’s just a lot of people going to them, and hence a lot of possibility for transmission.

        • It is actually more likely, the reverse. That is, on-floor employees of large retailers come into contact with hundreds or thousands of customers per day. If there is community spread, they have a much higher probability of being in the proximity of an infected person than even an employee of a large factory or warehouse, where high levels of automation means few workers per 100m^2.

        • “It is actually more likely, the reverse. ”

          I don’t believe this is true. The employees cluster in break rooms and lunch rooms smoking areas. These areas are typically far more confined than the selling floor of the store. This was documented here even in a local hospital.

          The contact with customers isn’t long enough and the space isn’t confined. Look at any CDC-documented outbreak: confined space, lengthy exposure.

        • Ken Schulz: you could be right too — I actually had not thought of it until you point that out.
          Jim: I agree with the point you raise.

          What is sorely needed (and it is astonishing that were still trying to address this critical question via “thought experiment”) is backward projection from case-reports to sites/settings of likely clusters of exposures. Not just the precious few carefully observed closed populations from which conclusions can be teased with the degree of exactitude that merits publication.

          The box-stores show up on the County List because they are obligated to report confirmed cases to the county.

          There are presumably mountains worth of confirmed cases which have been reported and within which are perhaps traces of narrative. Some of these traces could be like gold to be mined. Perhaps not. Perhaps the case reports are by and large vacuous — if they’re anything like my doctor’s notes. Too bad.

          Details useful for backward projection to site of transmission ought to be mandatory under the circumstances. After all the county does attempt to do contact tracing. That means they already ask for contact information. If they’d ask as well, for a concise history of the patient’s whereabouts (shopping, eating etc) in the recent period; then the discovery of clusters becomes feasible — e.g. on specific airplane flights; in specific supermarkets; and so on.

        • “employees cluster in break rooms and lunch rooms smoking areas”
          Who’s minding the store?
          More seriously, I would expect this to predict more clusters in assembly-line production, where the line shuts down and everyone goes to lunch at the same time. Good retail managers stagger lunches and breaks to make sure the floor is covered.

        • Someone can cough or sneeze then expose everyone who walks through the area for the next few hours:
          https://en.wikipedia.org/wiki/Airborne_transmission

          This was why they failed to eradicate measles as planned. It was assumed you needed direct contact between the two people, until they figured out one kid with measles could cough in the hall and infect the entire school as they walked through it to change classes.

          That is why sending everyone in a community to a few centralized locations to get supplies is a bad idea. Much better to have many smaller stores.

        • Anoneuoid, your first statement is highly speculative with reference to Covid-19, and unsupported by the Wikipedia citation. Actual infection by the method you describe would be dependent upon the proportion of virus aerosolized (most of what sneezes and coughs expel is in the form of droplets which fall within minutes), the rate of dilution (dependent an volume, air currents, and diffusion), and viral-load dependency.

        • Addressing the second question, studies have shown that SARS-CoV-2 is present and infectious in small aerosols. Samples of airborne particles less than 5 um in diameter contain viral RNA, although the presence of RNA does not establish that infectious virus is present. Using a sampling technique that maintains viability, airborne infectious virus was detected at >6 feet from patients. Laboratory experiments suggest that SARS-CoV-2 in aerosols remains viable for up to 16 p hours, with a half-life for viability of 0.5-3.3 hours in one study.[5]

          https://pubmed.ncbi.nlm.nih.gov/33458756/

          Seems plausible to me. Does it make sense to implement very costly (in the trillions of dollars) interventions without ruling out whether they may actually contribute to the spread of the virus?

        • Anoneuoid –

          > Does it make sense to implement very costly (in the trillions of dollars) interventions without ruling out whether they may actually contribute to the spread of the virus

          Does it make sense to talk about “cost” of interventions when such an assessment is necessarily based on unsupported counterfactual assumptions? What is the “cost” of interventions if (1) you can’t disaggregate the impact of the virus itself from outcomes directly attributable to interventions and (2) you don’t know what would have happened absent the interventions (i.e., those same outcomes being described as “costs” may well have been worse)?

    • +1

      It seems that all of the recent advances in our ability to analyze Big Data should have allowed this one to be far better characterized than the 1918 Spanish Flu pandemic, and yet…

      • And yet … fear, dread and wishful thinking in the spring kept us from getting down to business. I distinctly remember friends of mine saying — in effect — they’d found such-and-such a ‘professor’ on such-and-such a You-tube program who told them exactly what they wanted to hear: the whole thing would ‘peter out’; the excess-mortality rates in the flu season x or y years ago were worse; that the medical establishment is corrupt; and that Dr. Snow, by the way, didn’t *really* “cure” cholera; and the poor, misunderstood souls angry about masks and restaurants, who’d stormed the Michigan statehouse, like it was Beirut, didn’t mean anything by it.

  2. My question is why do health experts require RCTs before recommending cheap and safe interventions like correcting vitamin deficiencies but not expensive and dangerous interventions like lockdowns and early intubation?

  3. I see that the discussion has now shifted away from the fields of statistical modeling and causal inference to the realm of “it’s so obvious we don’t even have to model it”. That is fine, but then why publish scientific papers at all?

    At the risk of sounding petty, I’d like to bring up the fact that in the last post of this type, Seth Flaxman himself commented that he would be open to talking about the statistics of his paper. I summarized some of the main issues that had already been brought up by various commenters in the same thread by the time he commented – each of them serious – in a comment at https://statmodeling.stat.columbia.edu/2020/12/25/flaxman-et-al-respond-to-criticisms-of-their-estimates-of-effects-of-anti-coronavirus-policies/#comment-1631540 . There has been no response from any of the authors, though I did write Flaxman an email pointing to it.

    As for the new claim that “effects are clearly visible with the naked eye” – their eyesight must truly be amazing, as they are able to see through a smoothed (7-day-rolling average) lagging indicator (new cases) of a complex process and are able to determine not only correlation but also causation! Truly impressive, my eyesight doesn’t let me see the counterfactuals :)

    In all seriousness though, the important question isn’t whether lockdowns “work” – they obviously do reduce transmission rates by at least some epsilon > 0 – the question is how well they work, and what alternatives there are to lockdowns. If the goal is to have as few covid-19 cases regardless of the cost, then of course everyone should lockdown. The question (to me) is how many cases can be avoided by lockdown, and whether those benefits outweigh the costs. And these questions are far from obvious.

    The claim that lockdowns do no harm if people are already locking down doesn’t seem true to me, given that e.g. (a) lockdowns affect also “safe” activities (think jogging at night in Germany), and (b) tend to produce resentment more than voluntary measures would.

    I do agree with Flaxman’s comments on the Danish mask study, though on his more general point, I think it would be weird if every other field were plagued by people reading into weak data, except for epidemiology. If people aren’t rushing to conclusions, maybe the data is just unusually hard to read into?

    • Nathanael:

      Nobody here is saying, “it’s so obvious we don’t even have to model it.” That’s coming from you; nobody else in the discussion is saying this. Flaxman and Bhatt are pointing to the challenges of asserting no effect based on low-quality data.

      Estimating the effect of lockdowns or saying they can be effective is not the same as supporting particular lockdowns. I don’t think it’s a good idea to ban people from outdoor exercise, and I myself was resentful when the city locked the basketball courts and took down the hoops.

      • It is pretty clearly a problem of underdetermination. It cannot be overcome by more math so I don’t see why people keep trying.

        These models don’t even account for testing rate or the details of what triggers a test or the details of the test (thresholds, type of test, etc).

        Where I’m at the positivity rate approximately doubled over the holidays then dropped back down. The tests meanwhile dropped by about 1/2. This triggered more restrictions of course.

        Interesting this corresponded to all the regularly tested students going on vacation then returning to class for more asymptomatic testing.

        • The “details of what triggers a test” is the piece that is never going to be in a large scale dataset and it renders any so-called analysis of positivity rates worse than useless.

          It is of a piece with the people who make RDD phone calls with a sub-1% response rate and have no real idea why that 1% of people answered the survey while the other 99% did not. Without that crucial piece of information, which is never available, weighting the responses can’t possibly render the results generalizable to that missing 99%.

        • Test details matter too. Now there is a push to make the definition of a case more restrictive:

          WHO reminds IVD users that disease prevalence alters the predictive value of test results; as disease prevalence decreases, the risk of false positive increases (2). This means that the probability that a person who has a positive result (SARS-CoV-2 detected) is truly infected with SARS-CoV-2 decreases as prevalence decreases, irrespective of the claimed specificity.

          Most PCR assays are indicated as an aid for diagnosis, therefore, health care providers must consider any result in combination with timing of sampling, specimen type, assay specifics, clinical observations, patient history, confirmed status of any contacts, and epidemiological information.

          https://www.who.int/news/item/20-01-2021-who-information-notice-for-ivd-users-2020-05

          As this guidance is slowly adopted we will see a reduction in cases. I predict 100% of this reduction will be attributed to vaccines and new mask mandates, etc. 0% will be attributed to changing the case definition.

        • I recall the early years of the AIDS epidemic where case definitions were constantly morphing from year to year or even rapidly. It seems to be an inevitable aspect of emergent diseases. And as you say, the constantly shifting definitions leave room for various parties to claim efficacy (or non-efficacy) of putative interventions by cherry picking their time frames.

        • From the BMJ page:
          “Nothing, however, can be learnt from the arbitrary, haphazard, and confusing imposition of different interventions …. Interventions—such as population testing, vaccines and vaccination schedules, mask mandates, school closures—and even lockdowns—could all be introduced in a manner which allows for the gathering of evidence as to their effectiveness.”
          Not in a free society. Interventions will always be arbitrary and haphazard relative to any research design, because these are policy decisions and must be made by individuals who are answerable to the voters.
          We will progress more by pursuing better measurements. We have some understanding of the modes of transmission and mechanisms of infection, but imagine for a moment that we could fit volunteers with miniaturized laboratories that continuously or intermittently measured particle sizes and counts and viral concentrations in air volumes, viral loads within body tissues, and the individual’s location. Perhaps they would wear a miniature Street View-type camera with image processing to record proximity and duration of human interactions. Imagine what we could learn about the relative risks of activities and environments.
          So, how close could we come today to such a study?

      • ‘ Nobody here is saying, “it’s so obvious we don’t even have to model it.” ‘

        OK! I’ll step up the the plate! :)

        It’s 100% obvious – data or no – that lockdowns should be and are effective. We know that if you don’t put gas your car, the motor can’t run. We also know that if you isolate people from one another, a transmissible disease can’t spread. (and yes, we can be highly confident that spread from package delivery is extremely low if it occurs at all).

        The *serious* question about lockdowns has never been “are they effective”? It has always been:

        1) are they worth the cost
        2) are there alternatives that would obviate the need for lockdowns

        And, at this point, in retrospect:

        3) was any serious attempt ever made to find any other alternative?

        So while I guess it’s a free country and people can gather data and publish papers on whether or not there is salt in the ocean, it’s not a very useful endeavor.

        I think we did make some progress on (3): we established that the original expert line against masks was wrong and that masks are effective *to some degree*. We failed, I think thus far, to clearly establish to *what* degree, although I confess I’ve lost track of what’s going on.

        I think we also made some progress – if only through misapplied brute force – on the question of surficial transfer. Last July I went to the now bankrupt Guitars Etc and found that I had to wait in line over an hour to buy guitar strings and play a few guitars because
        everything I touched had to be thoroughly cleaned afterward.

        I’m not aware that we made any progress on understanding the interactions of air volume, room design and ventilation in indoor spaces, which could probably be worked out to a high level of certainty. Hopefully actual scientists and engineers are working on this problem somewhere.

        hmm…well I don’t know if that’s a sensible comment or not but I have other things to do so I’ll leave it there.

        • Jim – I was going to write a bunch of “erudite” crap above, but I just read your response here — and I think you “hit the ball out of the park” !!

        • Thanks! I wish that we as a country had hit the ball out of the park on the virus situation.

          I just scheduled the first parental vax. It’s six weeks out. Almost there.

        • I’m not aware that we made any progress on understanding the interactions of air volume, room design and ventilation in indoor spaces, which could probably be worked out to a high level of certainty. Hopefully actual scientists and engineers are working on this problem somewhere.

          I haven’t been following this much but from my understanding the air physics stuff has been solved for quite some time. The bottleneck is what happens to the virus. Like I think there are still questions about how long viruses can live in aerosols and under what settings. And then there’s the whole how much virus do you need to come into contact with to get infected? (I’m guessing this varies quite a bit from person to person and chance making it even more difficult but I’m not a virologist or anything so who knows). And these questions are very difficult but I do remember that some scientists were working on it.

        • “the air physics stuff has been solved for quite some time.”

          Yes, that sounds likely!

          However, in terms of making any sensible recommendations re: covid, I haven’t heard anything; and I realize making direct measurements is challenging to say the least. Just the same I think there are enough CDC type investigations into outbreaks to back out some useful constraints.

          I guess there are some capacity limits being utilized. Just the other day I drove down a trendy bar street. Every establishment has multiple tents, which I presume are there to expand capacity beyond the much reduced indoor capacity.

        • Linsey Marr is working on all aspects of the aerosol problem including ventilation. You could follow her stuff to see what’s up. For rooms without HEPA filters, CO2 monitoring does a good job of following a proxy for contamination. Filters of course reduce contamination but not CO2.

      • Andrew:

        When Flaxman writes both

        a) “At this point, there have been so many second (and third) lockdowns that effects are clearly visible with the naked eye” with a reference to a n=1 sample (Ireland)

        and

        b) “I’m seeing studies with small sample sizes and / or noisy measures, in which researchers fail to find a significant effect”

        it’s a bit hard to see how to reconcile the two without suggesting that the naked eye is better at statistics on small sample sizes than most other researchers are. The “it’s so obvious we don’t even have to model it” was my reading of this situation. Though really this part of my comment was the least important one, it was perhaps also influenced by the fact that I don’t think Flaxman et. al’s Nature paper is convincing as as model.

        >Estimating the effect of lockdowns or saying they can be effective is not the same as supporting particular lockdowns.

        In principle I agree. In practice, most lockdowns – by their very broad nature – will have some suboptimal results. The perfect lockdown is one which prevents only those interactions that would have otherwise spread covid, but this kind of lockdown would no longer be called a lockdown and is not really feasible as a strategy.

        • I think when Flaxman refers to “small sample sizes” he means studies that involve a small number of people. There have been some small case-control studies in which groups were asked to modify their behavior in various ways, and have been inconclusive. He is not suggesting that the “naked eye” would have found a conclusive result in these either.

          He also doesn’t claim that Ireland is the only country in which it’s obvious that lockdowns have worked, although (1) it’s presumably the best example he knows, and (2) he didn’t provide any other examples.

          I’m not saying he is or isn’t right, I’m just saying it’s not hard to reconcile his points a and b.

        • I am not happy with the “it’s obvious just by looking” argument – lockdowns will be put in place when it is bad, so they will *look* effective even if the disease is following a wave pattern naturally (because of seasonality or some other factor).

          (This doesn’t mean that the lockdowns weren’t effective – just that IMO that’s a poor argument.)

        • confused –

          > lockdowns will be put in place when it is bad, so they will *look* effective even if the disease is following a wave pattern naturally (because of seasonality or some other factor).

          The flip side of that is that they’ll be implemented in situations where the precipitating conditions are differentially worse. Therefore while in a relative sense they’ll have the biggest impact in an absolute sense they’ll be associated with locations that experience the highest mortality.

        • Yes, that too.

          I am honestly not sure a lot of this will be “knowable” for years; I still think there needs to be some explanation as to how *all* the high death rates are in Europe and the Americas before any explanation can be fully convincing. East Asia can be attributed to policies, but it strains belief that that applies to every single country in the Old World tropics, regardless of government form, resources, etc.

          Poor reporting could be a factor, but there are many less-developed nations in the New World tropics which nonetheless report high death rates.

        • Very interesting. It makes sense, too – in places where most people spend most of their time in climate-controlled indoor environments I wonder how much actual weather can really explain even cold/flu seasonality.

          I don’t think vitamin D can be a significant factor in the Old World vs New World tropics thing, though. There may be yet another unknown factor in there.

  4. jim – it’s hard for me to tell who you are agreeing with or disagreeing with!
    In any case, I do believe its obvious that most of the NPI efforts are successful, to some degree. The questions you ask, however, are important. How effective and at what cost? What I question is whether the modeling efforts can hope to provide useful information within the time scale that we must decide things. What I take from Daniel’s response is that he believes our short term policies should be clear, and we shouldn’t look to the analyses to decide what to do. I tend to agree, though ultimately I think it will be interesting and worthwhile to understand more about both the effectiveness and costs of various policies.

    What I object to is what I see as a mismatch between what the modeling efforts can reveal in a timely manner, and based on the poor data we have, and the time frame in which we need to act. If there is really the mismatch that I perceive, then I think it is counterproductive for these models to be used in the public debate about policies. It just becomes finding a model that supports the prior belief you had (such as the conflicting mask studies). The result is that a debate about civic values becomes a debate about the models, and that debate will just work for a very small minority of the population.

    • Dale, I think I pretty much agree with you. :)

      I just wanted to go on record pointing out that we don’t really need to model whether isolation is effective at stopping the transmission of disease, and to point out that even at this point after almost a year of dealing with this there’s really **alot** of confusion about what actually works beyond lockdowns.

    • Great comment, Dale, I agree in many respects! I guess there is no doubt that social distancing policies are overall effective. The focus should be more on the comparison of the effectiveness (and the costs) of different policies. More and more modeling studies are attempting to do this comparison. Whether modeling studies are the right way to attempt this comparison is another issue (see my third point).

      A note on the “lockdown debate”: I have no doubt that lockdowns are effective. I think what people sometimes doubt is whether the additional effect of a stay-at-home order is large enough to justify the costs. Unfortunately, Flaxman don’t estimate the additional effect of a stay-at-home order, but the overall effect of a lockdown including various policies such as venue closures, gathering bans, and stay-at-home orders. Thus, when other studies claim they cannot replicate the strong lockdown effect from Flaxman, they often fail to acknowledge the different definitions of a lockdown.

      Finally, I understand that people have reservations regarding the contribution of modeling studies to the policy debate. Indeed, data is often poor and there are limits to the number of adjustments that you can make within your model. There are obviously many factors that influence the effectiveness of policy measures, most notably adherence, that are just very difficult to measure and can thus hardly be taken into account. Doing these modeling studies myself, I understand these limitations but I still think that the modeling studies can add to the discourse about the effectiveness of various policy measures. The emphasis is on “add”. No individual modeling study and not even all modeling studies together can inform the policy debate alone. They should be considered jointly with other work from different angles. It is unfortunate that the modeling studies on the effectiveness of policy measures currently seem to get more attention in the public debate than other work…

  5. I’d like to switch the discussion to the general sort of analysis “Assessing Mandatory Stay at Home and Business Closure Effects on the Spread of COVID 19” represents. What struck me about its methodology is how similar it is to a type of study that shows up a lot in economics. You have an outcome variable of interest, a bunch of countries with different policies and a bunch of time periods. So you do a dif-in-dif or something similar to see if there is evidence of a particular policy “working”.

    This type of work drives me crazy.

    First, of course, there are typically serious measurement issues regarding the outcome variable, the policy variable and whatever “controls” are used, and those issues differ by country. The more diversity there is in the underlying generators of your data, like processes operating in different countries (including as we now know the possibility of different viral strains), the more likely it is these issues are producing bias and not just noise.

    Second, the apparent underlying assumption, contra my point #1, that there is a common process at work in all the countries and time periods, and all we have to do is estimate “it”, is nuts. A classic example is the effect-of-public-debt-on-growth literature, which didn’t even distinguish between countries whose debt was denominated in their own currencies and those whose wasn’t, or whether they were hard or soft currency countries, or whether they had chronic surpluses or deficits in their current account. (Even these distinctions are not necessarily decisive in themselves, but may be mediated through other aspects of a country’s economic structure and performance.)

    Third, there is elaborate question-begging at work. The simplifications necessary to model such a diverse array of countries and experiences are justified on the basis of an a priori model, and the model is justified, if at all, by the “success” of prior studies in parameterizing it. The circularity of this process apparently escapes most practitioners.

    Finally, what are usually lacking are detailed studies at the level of single episodes/cases to identify credible processes in those instances. This work is crucial both to filter out implausible results from assumption-ridden cross-sectional studies and to select relationships that can be tested in larger data sets. Think of Japan in the public debt studies, or how a close analysis of the timing of mandatory lockdowns in a single jurisdiction might help us understand their potential endogeneity wrt to the course of the pandemic.

    I’m not against any particular type of analysis, but against tunnel vision work whose priors blind it to potential sources of error.

  6. > and there is no acknowledgement of collinearity in NPI timings or how the NPIs were chosen.

    Well, yes.

    I have yet to see analyses that take into account the bi-directionality in the causal interactions between severity of disease, stringency of NPIs, and costs/benefits as outcomes.

    While I guess any modeling can have some value – any attempt to assess the efficacy of an intervention w/o controlling for the conditions that precede the NPIs and thus necessarily interact with the outcomes, has to be quite limited in value.

  7. > This is not to say that lockdown or whatever is the best policy in either case, just that empirical policy evaluation and decision making is difficult in such settings.

    Well said. So then it comes down to Bayesian priors like: should we act if we’re not sure it’s really, really bad? See Huemer’s In Praise of Passivity: https://spot.colorado.edu/~huemer/papers/passivity.htm

    This brings up questions of ethics, which are rarely discussed because they’re _even harder_ than questions of NPI timing data.

    I think a middle ground between the “leave us alone unless it’s really bad crowd” and the “let’s just act and hope the little data we gather will help tune our approach” crowd is the following:

    Unless things are extremely, imminently bad and most people agree (e.g. alien invasion), then let’s first run really, really good RCTs and look at the effect sizes. The ethical conundrums of such RCTs are outweighed by the ethical implications of active involvement. From Huemer’s paper:

    “when the state actively intervenes in society–for example, by issuing commands and coercively harming those who disobey its commands–the state then becomes [23] responsible for any resulting harms, in a way that the state would not be responsible for harms that it merely (through lack of knowledge) fails to prevent. Imagine that I see a woman at a bus stop opening a bottle of pills, obviously about to take one. Before I decide to snatch the pills away from her and throw them into the sewer drain, I had better be very certain that the pills are actually something harmful. If it turns out that I have taken away a medication that the woman needed to forestall a heart attack, I will be responsible for the results. On the other hand, if, due to uncertainty as to the nature of the drugs, I decide to leave the woman alone, and it later turns out that she was swallowing poison, I will not thereby be responsible for her death. For this reason, intervention faces a higher burden of proof than nonintervention.”

    • So where does vaccine & drug regulations fall in your view? Would the FDA suspending them during covid be like leaving the woman be, or like snatching the pills her (by changing from standard practice)? Faster vaccine release without FDA review/mandatory clinical trials, faster vaccine evaluation through challenge trials, and who knows what snake oils could’ve been promoted (although I suppose some of that happened anyway).

      • I think Huemer’s paper summarizes the heuristic well:

        “Political leaders, voters, and activists are well-advised to follow the dictum, often applied to medicine, to “first, do no harm.” A plausible rule of thumb, to guard us against doing harm as a result of overconfident ideological beliefs, is that one should not forcibly impose requirements or restrictions on others unless the value of those requirements or restrictions is essentially uncontroversial among the community of experts in conditions of free and open debate.”

        Snake oils are a real thing, and are just one variable in a multivariate equation of sociology that we don’t know very well.

        • > Political leaders, voters, and activists are well-advised to follow the dictum, often applied to medicine, to “first, do no harm.”

          Policy-makers actually have no such option here as “do[ing] no harm” here in any meaningful sense. That frame is a useless and unrealistic dichotomy.

          Policy-makers broadly have two choice. One is to not act and just let harm, which is certain to occur, occur without any attempt at mitigation. The other is to act with the potential of worsening the harm.

          There are, of course, degrees within those two options and the options don’t really exist as a dichotomy either.

          Simplistic framing such as “so no harm” can serve a purpose as an intellectual exercise, but when applied to advance a particular action (or non-action) in the real world they don’t really provide insight but only function as a rhetorical device.

        • I think the questions of whether there is an ethical difference between inaction and action; and whether the coercive aspect of *government mandatory* action changes the ethical balance – are ethical questions that statistics, etc. can’t really help with.

          Personally, I do think there is a significant difference, the latter especially, but that’s not something I can support with numbers :)

        • Yes – I should have mentioned that. Although I don’t like framing it as “ethics” so much as political orientation – which can easily shift issue by issue (like how the insurance mandate went from basic ethical accountability to Obama’s unethical tyranny).

        • Well, political orientation of course affects the decisions people actually make.

          But I think the question of what decision *should* be made absolutely is an ethical one.

          Not that ethics and politics don’t have significant overlap, of course – but if one wants a question like “what role should government have in medicine and health” to be anything other than a purely two-sided political fight, IMO it has to be approached as an ethical question.

          Now to accomplish anything, you have to deal with the political realities; but what goal you want to accomplish in the first place IMO should be an ethical question.

        • confused –

          > of course – but if one wants a question like “what role should government have in medicine and health” to be anything other than a purely two-sided political fight, IMO it has to be approached as an ethical question

          My point is that “ethics” is a bit of a moving target. With reference to your example, government involvement in a woman’s medicine and health (i.e., what medical procedures she can have free access to) stirs different beliefs in a lot of people that differ from their views on government involvement in requiring peope to wear masks in the middle of a pandemic.

          I think “ethics” is too global a term. I think my “ethics” are probably quite similar to many people who disagree with me on many ways that “ethics” play out in a polarized context. In other words, I would generally agree with many that disagree with me in particular contexts, in that I don’t want government overreach into personal lives. That is my “ethic” just as it is theirs.

          So I think it has to be approached as a stakeholder dialog question. You approach it from an “ethical” question and, IMO, you’ll get the partisan zero sum food fight that predominates now.

        • >>My point is that “ethics” is a bit of a moving target.

          IMO, that is exactly the problem, though. It is necessary to work from some framework of fundamentals to avoid it being purely “political”.

          IMO much of the deterioration of political discourse is because people *don’t* think about fundamentals, but address only immediate surface issues as they come, in a “sound-bite culture”.

          >>I don’t want government overreach into personal lives. That is my “ethic” just as it is theirs.

          Well, no one wants “overreach” – by definition overreach is bad. My point is there needs to be an ethical framework, a definition of what the government’s role *should* be, for “overreach” to be meaningful, more than a ‘weaponized’ political term.

        • Re healthcare workers: yes, to a degree. There is a practical aspect, sure.

          But I was really talking about the question of government intervention — are the risks/harms of action versus inaction on an equal ethical plane, or does the mandatory/coercive aspect of government action raise the ethical standard, IE require a higher “burden of proof” that action is required?

        • confused –

          > IMO, that is exactly the problem, though. It is necessary to work from some framework of fundamentals to avoid it being purely “political”.

          I don’t see how you’d avoid it being political. Do you want the president to issue a mandate that no one should politicize it? :-)

          > IMO much of the deterioration of political discourse is because people *don’t* think about fundamentals, but address only immediate surface issues as they come, in a “sound-bite culture”.

          I think that you’re creating a false distinction between what is and what isn’t political. This question is necessarily political and you can’t create some de-politicized ethical framework. The “ethical” questions sit in a political context. I’d rather deal with them through stakeholder dialog frameworks – where you work out the politics in a non-tribal framework. Of course, that’s pie-in-the-sky.

          > … or does the mandatory/coercive aspect of government action raise the ethical standard, IE require a higher “burden of proof” that action is required?

          That doesn’t work for me – because I think that in this context the distinction between action/inaction is to a significant degree arbitrary. To some degree it would necessarily depend on the magnitude of the “action,” and it definitely depends on the magnitude of the problems created by “inaction.” And I think that the issue is being gamed (or at least leverage) a lot by partisans to push an agenda. It’s tough to get to a good convo because so many “freedom fighter’ are running around in three-cornered hats fighting against the outrageous tyranny of being asked to wear a mask when they go into the supermarket.

        • >>I don’t see how you’d avoid it being political.

          Well, perhaps political vs ethical isn’t exactly quite the right distinction here… It’s more about case-by-case decisions (building up possibly unintended precedents) vs. decisions flowing from application of *a consistent set of principles* to specific facts/situations/issues.

          Forming a set of principles in itself, if they are to be applied to policy decisions, would be “political” in itself, of course, just as (say) writing the Constitution was.

          And I think we have a lot of political divides that are “unnecessary” in the sense that the two sides’ goals are not as fundamentally contradictory as either side believes, and both goals could be attained simultaneously by discarding unexamined assumptions shared by both sides. (E.g. poverty, wages, etc. – I think both sides’ political “plans” are based on a view of the economy which has been made quite obsolete by technological change over the last 40 years or so.)

          Some of this is I think a result of “compromises” made to solve past problems, which unintentionally introduce “une

          For example I think that we could make far more effective progress against poverty and improve healthcare outcomes more, at far less taxpayer cost and less government/bureaucratic expansion, than current government programs accomplish – but the whole system (including both parties and all the not-terribly-partisan structure of the bureaucracy) is full of excluded middles and unexamined assumptions that

          >That doesn’t work for me – because I think that in this context the distinction between action/inaction is to a significant degree arbitrary.

          I’m not sure I understand what you mean there. Either a rule/mandate/whatever is introduced or it is not.

          (Do you mean that inaction is not considered practical/politically viable, so it becomes an argument between more and less aggressive actions in practice?)

          >>And I think that the issue is being gamed (or at least leverage) a lot by partisans to push an agenda.

          Absolutely – but the misuse of the issue doesn’t obviate the core ethical question; if anything, it makes it more critical IMO.

          No decision will make everyone happy – but it is quite valuable to be able to say why you came to that decision.

        • Ugh, that is rather garbled. Sorry. Those middle paragraphs should say:

          “Some of this is I think a result of “compromises” made to solve past problems, which unintentionally introduce “unexamined assumptions” into the system which no longer apply when a different set of problems arises decades later.

          For example I think that we could make far more effective progress against poverty and improve healthcare outcomes more, at far less taxpayer cost and less government/bureaucratic expansion, than current government programs accomplish – but the whole system (including both parties and all the not-terribly-partisan structure of the bureaucracy) is full of such unexamined assumptions and excluded middles. To get qualitatively better outcomes, one would have to look outside the box (e.g. not just argue about who should pay healthcare costs, but deal with why healthcare is so costly in the first place).”

        • confused –

          > And I think we have a lot of political divides that are “unnecessary” in the sense that the two sides’ goals are not as fundamentally contradictory as either side believes, and both goals could be attained simultaneously by discarding unexamined assumptions shared by both sides.

          That’s the basic principle of stakeholder dialog. You separate positions from interests and look for synergies (or common interests).

          > I’m not sure I understand what you mean there.

          Basically that during a pandemic when deaths are necessarily an outcome either way, inaction is effectively an action. So the relevant question isn’t action vs. inaction but the costs and benefits of each

        • >>Basically that during a pandemic when deaths are necessarily an outcome either way, inaction is effectively an action. So the relevant question isn’t action vs. inaction but the costs and benefits of each

          Well, that’s one side of the question I was talking about – but the fact that you think the answer is clear/one side is clearly right doesn’t make it not an ethical question.

          The other side being that the coercive nature of government action means that you don’t start from a position of “ethical equipoise”, if you will, and so it’s not simply a matter of cost/benefit – taking coercive action requires a higher “standard of proof”.

        • > but the fact that you think the answer is clear/one side is clearly right doesn’t make it not an ethical question.

          “you” as in me? Or the royal “you?”

          No, I don’t think there’s a clear answer. It depends on how you evaluate the context, and politics necessarily affects that evaluation

          > and so it’s not simply a matter of cost/benefit – taking coercive action requires a higher “standard of proof”.

          The standard of proof for me is the quality of evidence of costs and benefits, mixed with politics. Coercion in a democratic government can mean taking action prescribed by a voting public. That’s the problem with all this wringing of hands and gnashing of teeth about interventions being coercive:there’s (not fully conclusive) evidence that a majority want interventions. So what is “coercive” there? To not act and allow a minority to determine to just let people die? Add to that, those who don’t want “coercion” are represented by a minority in congress even though they are a minority of the public. Add to that, they have a historical legacy of disproportionate power where they benefited from coercion on a large scale.

          It’s complicated.

          I’m not a fan of applying some abstracted moral calculus to decide. I’d rather decide based on what information you can glean about the context. Sure, you can talk about some idealist or abstract moral calculus. Why not? In fact I think you should. But I don’t see applying it as some kind of moral emperative that takes some kind of priority.

        • >“you” as in me? Or the royal “you?”

          All I meant is that what you are writing sounds like you are saying that action/inaction are ethically equivalent –

          eg

          >>Coercion in a democratic government can mean taking action prescribed by a voting public. That’s the problem with all this wringing of hands and gnashing of teeth about interventions being coercive:there’s (not fully conclusive) evidence that a majority want interventions. So what is “coercive” there? To not act and allow a minority to determine to just let people die?

          To me that implies that you aren’t drawing a distinction between action and inaction, ie deaths allowed by inaction are “just as much the government’s fault” as deaths caused as a side-effect of action. Am I misreading you?

          I’m not saying that is wrong. But it is a *different question* from costs vs benefits. One is susceptible to numerical analysis, the other is ethical and comes down to axioms, IMO.

          Personally I think that taking “abstract moral calculus” (I would say “fundamental principles”) into account is the only way to avoid one-decision-at-a-time, one-crisis-to-the-next accumulation of bad precedents. There’s always the temptation to say, yes, we have X and Y limits, but THIS crisis is the REALLY big one, so we need to ignore that…

          Now, I’m not saying that the actions taken to deal with COVID were wrong (I’m reserving judgment until the crisis is over and we can compare final results). But this has *clearly* happened in political/economic issues, leaders or institutions claiming more power to deal with problem X, and never giving it up… “Feature creep” applies very strongly to governments.

        • confused –

          > To me that implies that you aren’t drawing a distinction between action and inaction, ie deaths allowed by inaction are “just as much the government’s fault” as deaths caused as a side-effect of action. Am I misreading you?

          Kinda. It depends on the context. That’s what I mean when I say you look at the costs and benefits.

          As for action/inaction, consider the trolly problem, or more specifically the “bystander at the switch.”

          https://en.wikipedia.org/wiki/Trolley_problem

          Maybe at 5 people standing on the track it’s one problem. With 10 million people on the track it could look like a different problem. Maybe it’s one person standing on the track that would get hit by the train if you do nothing and 10,000 who would get hit if you throw the switch. Maybe one group is children and the other seniors. Mabe some are people incarcerated for murder. Add in an element where it’s government that is being “coercive” in some manner by acting. Then throw in political elements (maybe one is a group of Nazis shouting “Jews will not replace us, carrying torches and nooses and promising to lynch people).

          I don’t see how a generic ethical calculus about government action having an inherent higher ethical bar is of much use. It would depend on context, IMO.

    • “when the state actively intervenes in society….the state then becomes [23] responsible for any resulting harms, in a way that the state would not be responsible for harms that it merely fails to prevent. ”

      Baddabing!!! great.

      • There is a saying that deciding not to act is also a decision. I don’t see that action by government leads to more responsibility than deciding not to act.

        With regards non-pharmaceutical interactions and RCTs. Arms of the trial would need to be different geographic regions so as to compare government-imposed lockdowns to voluntary curtailment of activity. One arm of the RCT would then be a region where :
        a) people are not to wear as mask
        b) people are to go about their daily work/school/social life without any efforts at distancing
        c) buildings are required to maintain their usual ventilation rates and air filtering practices

        This seems hugely problematic.

        • And imagine trying to make the participants and evaluators of that study *blind* to the treatments!
          It would be a procedure requiring stellar talent — of the order of Jonathan Swift — to put together.
          Maybe some members of our academy (who crank out these so-called “NPI” studies) have the talent?
          Maybe the same stellar academicians can — in their spare time — teach dogs to walk around on two legs; to paraphrase Dr. Johnson, “it is done badly; but it is remarkable that it can be done at all1”

        • There is no expense to great when it comes to lockdowns, etc. Why is that an issue for getting evidence on how well they work? For $100 billion dollars I am sure we can get a decent RCT on this. Probably for 1/100th of that.

        • The participants will have to be blinded to the different “treatments”. That means — I presume — that some scheme must be devised whereby those in the locked-down group do not realize that they are confined to their living-rooms.

        • The vaccine trials suffered the same problem. Everyone knew they got vaccinated due to pain at the injection site, etc.

          Didn’t stop that.

        • Are you telling us you think that the vaccine trials were just as preposterous an exercise as the hypothetical two-town-trial?

        • Are you telling us you think that the vaccine trials were just as preposterous an exercise as the hypothetical two-town-trial?

          All RCTs are flawed. You still haven’t pointed out any reason such a study would be preposterous. All the complaints so far are standard for any RCT except the cost, which would be high relative to other clinical trials… but so is the cost of implementing the intervention so that makes sense. Do you really think anyone would notice if $10 trillion was spent vs $10.1 trillion? It is a rounding error at that point.

          Personally, I *do not* consider RCTs to be a gold standard. But the people making these decisions do.

          I would just like to know why RCTs are required for safe and cheap interventions like correcting vitamin deficiencies but not expensive and dangerous interventions like shutting down businesses everywhere and putting people on ventilators ASAP.

        • Anoneuoid –

          > but not expensive and dangerous interventions like shutting down businesses everywhere and putting people on ventilators ASAP.

          I’ll agree that “dangerous” could apply since sure, there are risks involved. Not implementing NPIs is also “dangerous.”

          As for “expensive.” That categorization rests on a counterfactual assumption that isn’t supported by the information we have. We see a correlation with harms and NPIs. But we also see a correlation with those same harms where there is a relative lack of NPIs, particularly if you control for precipitating conditions w/r/t the state of the pandemic. Saying that interventions have an “expense” implies that you have some way of assessing what would have occurred absent the NPIs, and that the differential costs would be lower w/o them. You don’t have the information to do that.

          “Expensive” assumes causation and you’re conflating correlation and causation.

        • > You just set up two communities in the middle of nowhere and people live there for a few months.

          A few follow-up comments and now I’m beginning to wonder. Was that a serious suggestion?

        • People seem to be quite serious about shutting down businesses around the world and printing ~ten trillion dollars to pay for it as cases and deaths grow anyway.

          Nothing unreasonable about devoting 1% or less to see if that actually works.

        • It’s clearly an important issue – worthy of significant effort to study.

          Believing that isn’t mutually exclusive with believing that trying to set up an experimental design, where you create two communities to meaningfully model enough predictive variables to inform questions about the efficacy of NPIs, is extremely implausible.

          IMO, if you want to get a quick look at the efficacy of NPIs, look longitudinally at localities that have implemented them: The associations with changes in infection rates, along with a plausible theory of mechanism, makes a pretty strong if not definitive case. This is making decisions about risk in the face of uncertainty. Perfect understanding doesn’t exist.

          Stop looking for meaningful answers from comparisons across localities – just waaaay too many confounding variables to make such an analyses particularly instructive as to the efficacy of “lockdowns” as a broad category. Not saying don’t to them – it may have some value. But it can’t answer big questions.

          Do cluster some analyses to see if you can gain some insight into the relative effects of different interventions.

          The question of cost/benefit analysis regarding the negative outcomes correlated with NPIs is a really tough one. Where would the causality lie – with the NPIs or the pandemic itself (where the impact could be worse absent the NPIs). I haven’t seen any discussion of how that could effectively be assessed. So sometimes you have to make decisions about risk in the face of uncertainty. Such is life.

          For me, it’s about hedging against obviously massive downside risk, where healthcare systems are overwhelmed and hero healthcare providers are overly stressed and lack sufficient resources.

        • Believing that isn’t mutually exclusive with believing that trying to set up an experimental design, where you create two communities to meaningfully model enough predictive variables to inform questions about the efficacy of NPIs, is extremely implausible.

          Non-representativeness is a standard problem with RCTs. They are still considered a “gold standard” anyway.

        • > Non-representativeness is a standard problem with RCTs. They are still considered a “gold standard” anyway

          (1) not all RCTs are created alike
          (2) because some are poorly done, doesn’t mean that they can’t be a useful methodology
          (3) even if they couldn’t, it wouldn’t make your proposal meaningfully informative.

        • It would be just as informative as a typical RCT.

          One town goes out to restaurants, etc whenever they want. The other gets in trouble for doing that.

          What happens would inform us of whether that measure meaningfully changes the spread of the virus under those conditions.

        • Anoneuoid –

          > It would be just as informative as a typical RCT.

          That’s a totally unqualified statement. You describe some completely undefined experimental paradigm and deem that it would “just as informative” using some completely undefined standard for what is “typical.”

          But “typical” is a useless standard anyway. Even if the “typical” RCT isn’t informative because of unrepresentative sampling, there could be a non-trivial number that are informative due to representative sampling. So even if your proposed study is no worse than a putative “typical” RCT, that wouldn’t make it informative whereas good RCTs could be.

          >One town goes out to restaurants, etc whenever they want. The other gets in trouble for doing that.

          >> What happens would inform us of whether that measure meaningfully changes the spread of the virus under those conditions.

          But controlling for one, isolated, potentially predictive variable would be more or less useless w/o examining for interaction effects with multiple other potentially predictive variables that could not plausibly be controlled. The point is that you can’t realistically just go out and create two communities that would be representative on a national scale.

        • But controlling for one, isolated, potentially predictive variable would be more or less useless w/o examining for interaction effects with multiple other potentially predictive variables that could not plausibly be controlled

          Same with every RCT.

          The point is that you can’t realistically just go out and create two communities that would be representative on a national scale.

          No RCT achieves this.

          Please don’t continue unless you have a counterexample to share. It should be easy for you to find one single RCT that achieves what you claim is required.

        • > Please don’t continue unless you have a counterexample to share. It should be easy for you to find one single RCT that achieves what you claim is required.

          Lol.

          Now prove to me that you can create such communities. Don’t continue unless you can prove that you can do so. It should be easy enough for you to find one study where they experimentally created two communities that served as representative sampling for assessing incredibly complex social and medical phenomena.

          I will readily agree that there aren’t likely any RCTs that can fully control for every possible protective variable.

          But that’s an absurd standard. There are plenty of RCTs that control for enough predictive variables to be informative. Your notion of creating two experimental communities that simulate enough meaningful variables to be instructive as to be able to assess the effects of multiple vectors of NPIs, let alone one such community and just one vector, seems absurd to me at face value.

        • So you cannot find one RCT that meets your criteria. But this one is ridiculous because it would fail to meet your criteria.

          As usual it is just endless waste of time paragraphs riddled with logical fallacies.

        • Anoneuoid –

          > So you cannot find one RCT that meets your criteria. But this one is ridiculous because it would fail to meet your criteria.

          I can find plenty of RCTs that meet my criteria (controlling enough for enough predictive variables to be meaningfully informative).

          As to your standard of whether the “typical” RCT meets that standard – it’s hard to say unless you define “typical.” But the question of whether the “typical” RCT meets that standard is completely irrelevant to the question of whether your proposed study would be.

          And yes, the one you’re proposing seems extremely implausible because it seems to me it would be highly unlikely to control well enough for enough predictive variables, let alone interaction effects, to be meaningfully informative.

          I assumed initially that your proposal was a joke. Apparently not.

        • Then why don’t you link to it?

          Don’t write more empty paragraphs, link to one single RCT that meets your criteria.

        • Anoneuoid –

          > Don’t write more empty paragraphs, link to one single RCT that meets your criteria

          Because you’re playing a silly game.

          But if you link to a single study in the enite history of interventional science that even remotely resembles the one you’re proposing (and that is meaningfully informative), I’ll be happy to Google around to find one RCT conducted in the entire history of interventional science thst is resonvky considered to be informative.

          Of course all RCTs are imperfect and of course there are general problems with the paradigm.

          But your evangelical belief that there are none in the history of interventional research that have ever been reasonably informative is, IMO, absurd.

        • But *you* know the solution! And they’re not listening to *you*. What a plight. The plight of the misunderstood and suppressed prophet (or prophetess).

        • Anon:

          No, I’d just like to see expensive and dangerous interventions held to *at least* the same standard as cheap and safe ones.

          In reality the standards should be much greater for dangerous/expensive interventions under any rational system of decision making.

          And no, I don’t expect that to happen at all. This problem has been festering for a long time now, no reason to expect it will stop:

          “We are quite in danger of sending highly trained and highly intelligent young men out into the world with tables of erroneous numbers under their arms, and with a dense fog in the place where their brains ought to be. In this century, of course, they will be working on guided missiles and advising the medical profession on the control of disease, and there is no limit to the extent to which they could impede every sort of national effort.”

          Fisher, R N (1958). “The Nature of Probability”. Centennial Review. 2: 261–274. http://www.york.ac.uk/depts/maths/histstat/fisher272.pdf

        • “I would just like to know why RCTs are required for safe and cheap interventions like correcting vitamin deficiencies but not expensive and dangerous interventions like shutting down businesses everywhere and putting people on ventilators ASAP.”

          You could seek out crowds if you like — if you can find them — and you will find out if it was a good idea or not. There are is a noble tradition of self-experimentation in medicine. Take the challenge and report back.

        • Is this a joke? There are entire states of people doing that. The states where everyone is going to the same superstore since they closed all the small ones seem to be worse off if anything.

        • You could seek out crowds and report the course of your illness; which you would treat with your favourite remedy (which you seem to be touting). There is a noble tradition of self-experimentation in medicine. We look forward to your report of your own clinical course.

        • You could seek out crowds and report the course of your illness; which you would treat with your favourite remedy (which you seem to be touting). There is a noble tradition of self-experimentation in medicine. We look forward to your report of your own clinical course.

          This does not sound scary to me. I have been living as normal as possible this entire time (just wear a mask if people say so) and have traveled across the entire country. I’ve flown half a dozen times, been to pubs, parties, large weddings, etc. Nothing of note happened to me or anyone else including my 60+ yr old parents who I stayed with for an extended time.

          I was even at Mardi Gras in the spring when covid *did* look very scary based on the info coming out of China…

          Personally, I think I had it in early November of 2019 when I did get very sick and could barely get out of bed for a week. That is when I got interested in vitamin C because a friend gave me some ascorbic acid powder and I felt immediately better. Before that I thought it was BS like most people who never actually look into it. If there is ever an at-home covid antibody test I will try it.

        • Actually, forgot to mention a friends sister did test positive (fever for two days) and I wanted to have her sneeze on me then I’d live in a tent for two weeks but everyone else was too scared so it didn’t happen.

    • “Unless things are extremely, imminently bad and most people agree (e.g. alien invasion), then let’s first run really, really good RCTs and look at the effect sizes.”

      Nooooooo! By the time “most people agree” that an epidemic is imminently extremely bad, it is typically too late to do much about it. That’s the dilemma of exponential growth. By the time it is obvious you are facing trouble, an enormous number of deaths that might have been prevented by early action are baked in and inevitable. Also, how on earth do you do RCTs of “lockdowns?” If nothing else, there are so many different parameters of what is banned under the “lockdown” and how it is enforced, and what levels of adherence are obtained that pretty much the entire world would have to participate to get a reasonable sample size in each of these strata! But that also doesn’t work because epidemics are typically not synchronous around the world: like the real COVID-19 epidemic they peak in different places at different times. But the intensity of the epidemic at the time of the study would also have an impact on the apparent effectiveness of different kinds of lockdowns.

      Much as I generally prefer to see RCTs, or perhaps other high-grade evidence, before policies are adopted, there are pragmatic limits to this approach. And the management of an epidemic of a highly transmissible disease with a high mortality just does not lend itself to this approach.

    • > . Before I decide to snatch the pills away from her and throw them into the sewer drain, I had better be very certain that the pills are actually something harmful.

      Nir a very useful analogy, IMO. In the current situation, we know that harms are going to occur. So it’s not like the woman with the polls.

      We don’t know the exact magnitude of the harms that will occur. And we don’t know with exactitude that interventions will result in less harm than the harm that would occur if the interventions weren’t to take place.

      So this becomes a situation of risk analysis in fhe face of uncertainty. For me, it books down to hedging against what would be the highest downside risk. In this case, overwhelmed hospitals, hero medical workers over-stressed and under-resourced, people suffering the same kinds of losses they’d suffer with the interventions – loss of jobs and economic collapse, etc.

      • Joshua said, “So this becomes a situation of risk analysis in the face of uncertainty. ”

        Duh. Isn’t risk analysis always in the face of uncertainty? If there’s no uncertainty, there’s no risk.

        • If my sewer drain is clogged by roots and I cannot unclog it myself, it is certain that unless I call a plumber I will lose the use of the whole house. If I call the plumber, it is certain the drain will be unclogged; and it is certain too, that I will pay dearly for the the work. If I do not call the plumber I will pay dearly too; for having done nothing and thus losing the use of the house. The costs are certain; the outcomes are certain. The only thing that is possibly uncertain is what I shall do. Though if my bank-balance allows and I don’t intend to sell the house (or camp out in the yard) then I suppose I must pay the plumber. The uncertainty, in other words, lies not in the outcome, but in the action I shall take! You could say that what is uncertain is actually my evaluation and balancing of the (certain) costs. What is uncertain is the concrete consequence of my latent disposition towards worldly affairs.

        • That is a pretty good analogy.

          1) Did plumbers use RCTs to figure out the chosen solutions?

          2) Do they try the cheapest and safest methods first?

          3) Was there a time plumbers tried stuff like dumping gasoline down the drain and lighting it on fire to clear the roots?

          What sort of decision making process did the plumbers use to decide on what to do?

        • 4) The plumber says two weeks to clear the roots, a year later he is still working on it and your drain is even more clogged than when they started working on the problem. At what point do you look for a different plumber?

        • This sometimes happens with the sewer drain problem! They dig the whole place up, halfway out the street too. Have to pull a city permit for that. Then at some point they declare their equipment just isn’t up to it. One should not be too hasty in selecting the first one who comes along and says don’t worry the thing’s in-the-bag. Well, there’s a not insubstantial degree of uncertainty after all, in the prospect of completion. Automobile repairs sometimes are a similar sump. Medical treatments and procedures are the most frightful. We mustn’t be too hasty; nor can we chary either. Better at peace with oneself and with the world first; before submitting to the knife. Well, it must be admitted, it’s all the same in the end. Tend to the soul, my friend; and the drain will sort itself out.

        • rm –

          > The only thing that is possibly uncertain is what I shall do.

          That’s an interesting way to look at it, however…

          > The costs are certain; the outcomes are certain.

          W/r/t NPIs and the pandemic, that most certainly isn’t the case.

      • While I agree with your general point (more than I would have a few months ago), I’m not totally sure about this phrasing:

        >> overwhelmed hospitals, hero medical workers over-stressed and under-resourced,

        I would argue that the main cost is the additional deaths due to these things (IE – people who could otherwise have been saved). I am not very comfortable with valuing the stress put on healthcare workers as more significant than deaths in the general population!

        (I’m not saying that’s what you meant… but the wording kind of sounds like it.)

        • confused –

          > I would argue that the main cost is the additional deaths due to these things (IE – people who could otherwise have been saved). I am not very comfortable with valuing the stress put on healthcare workers as more significant than deaths in the general population!

          I think you and I may have discussed this difference previously.

          I recognize that there’s an element of embracing “inequality” there – ut I’m comfortable with it because it is through their sacrifice that I am able to mitigate my level of risk. For me, that merits a higher degree if value when evaluating distribution of potential costs.

        • >>I think you and I may have discussed this difference previously.

          I guess so. And I’m a bit less uncomfortable with it than I was earlier, because…

          >>it is through their sacrifice that I am able to mitigate my level of risk.

          …this seems a bit more true now that more treatments are available; early on, I wasn’t entirely convinced that the availability of care made all *that* much difference to my (fairly low to start with due to age & health) level of risk.

          But it still bothers me – perhaps irrationally – because early on I saw a lot of exhortations to be careful based on this that felt very “hero-worship-ish”. As well as somewhat disingenuous – telling the young people everyone was saying were being careless they’ll die because of lack of hospital beds is not very convincing if they’ve seen what the hospitalization rate in their age group is…

          And I do think the extreme emphasis on preserving capacity early on led to a lot of undesirable results (COVID patients staying in nursing homes rather than hospital beds on the one hand, and insufficient concern in places/times where hospital capacity was not particularly stressed on the other hand).

  8. Did anyone see a convincing explanation for why it was appropriate for the new Bendavid et al. paper to select only the countries they did? e.g., why they included Sweden but not its immediately comparable neighbors like Denmark, Norway, and Finland which implemented lockdowns and had covid death rates a fraction of Sweden’s. Or why they would include South Korea, a virtual-island nation (its one land border is impenetrable) with massive tracing and isolation, for use in comparisons involving the US and The Netherlands.

    Contrast the selection in Bendavid et al. to that in the Brauner et al. study in Science last month:
    https://science.sciencemag.org/content/early/2020/12/15/science.abd9338

    Any other aspects of a study seem secondary if the data have been curated to produce the conclusion.

    • Yeah, I think South Korea’s effective-island status is under-valued. The nations which seem to have done really well due to policies, e.g. SK, NZ, Australia, Taiwan, Singapore, with the possible* exception of China, are islands or near-islands (SK, Singapore).

      *I am not sure how much I trust China’s numbers, especially with regards to some of the parts of China which I have read have huge human rights issues (e.g. among Uyghurs). Even Singapore had a big explosion of cases among migrant workers who were largely missed (though few deaths, due to healthy worker effect).

  9. “Check out Ireland—and try to guess what week lockdown came into effect”.

    Ireland is a bad example to use because the early-January spike is artificial. The tests were heavily backlogged due to Christmas and then NPHET reported a huge amount of “cases” after this backlog cleared. The spectacular rise and fall is an artefact of this, not lockdown. So the real epidemic curve is far less pronounced than the data Flaxman points to suggests.

    Admittedly there was a rise over Christmas which is now in decline. However, this is not evidence that “lockdown” is responsible. People met up over Christmas and were always going to lay low over January regardless of government intervention. I know this is what all my friends and family were going to do.

    Counterfactually, cases would have declined anyway in January because of the backlog being cleared and people voluntarily social distancing. That this decline coincided with a lockdown is irrelevant. I can’t believe I am having to write such a basic statistical inference point on this blog! What strange times we live in.

    • I think the issue is one of semantics, does “Lockdown” mean “governments pass a lockdown law” or does it mean “people actually stay home and avoid contact with other people” ??

      I think much of the argument comes from failure of people in the argument to recognize that they’re not talking about the same thing. Some people are saying “if you stay home you and people in your community won’t get sick” and other people are saying “if the government passes a law requiring everyone to stay home, it won’t do anything because people are going to go out and be in contact and get sick anyway”

      These can easily be simultaneously both true. but if the two sides call their thing “lockdown” then this degrades to “lockdown works, no lockdown doesn’t work, yes lockdown works, no it doesn’t”

      • “I think the issue is one of semantics, does “Lockdown” mean “governments pass a lockdown law” or does it mean “people actually stay home and avoid contact with other people” ??”

        This point can’t be made strongly enough.

        I live in north Orange County, CA, just a short walk from the LA County line. The state regulations on what businesses may operate and mask wearing are the same in both counties (at this time). In Orange County the Sheriff’s office announced early on that they would not enforce any of these rules. If memory serves, the LA County sheriff made a similar statement, although the City of Los Angeles (a proper subset of the county) is using its various administrative agencies to do some enforcement.

        Back when the rules were set out, before and through the holidays, on my daily exercise walks, I observed that I was nearly always the only person wearing a mask. I started keeping track, and recorded mask usage at 5% in round numbers. But since the New Year, the local news is constantly reminding people that the death toll around here is skyrocketing and the ICUs are full. There has been, as far as I know, no change in enforcement. Yet since the New Year, mask wearing has risen to about 40% within eyeshot of my walks. (I’m taking the same route at approximately the same time every day.) I don’t think this is a rigorous study by any means, but I think it is clear that there can be huge divergence in place and time about the actual degree of adherence to lockdown rules, and that adherence can rise spontaneously in response to dismal news.

        • I made an early morning run to Lowe’s to pick up a couple things last week. Of the dozen or so customers I saw in the store, maybe half of us were wearing masks.

          Later that same day I saw someone riding alone on a motorcycle, wearing a full helmet and wearing a mask under the helmet’s face shield.

          The law clearly says everyone in that Lowe’s was to be wearing a mask, period. And no law says someone riding down the road on a motorcycle should wear one under their helmet. I was struck by how those two observations on the same day encapsulate the futility of these stupid models trying to tease out the effects of a “lockdown” by simply tracking various ever-changing government edicts.

        • Yeah, IMO this is pretty important. Government measures may “not do much” either because people were already being careful, or because they are largely ignoring the measures…

          IE, even if we are sure that social distancing, mask wearing, or whatever works, there remains the question of *what effect the government measures have on the actual amount of social distancing/mask wearing/whatever occurring in practice*.

          And this may vary depending on the local culture’s degree of “individualism” or “conformism” or whatever.

        • confused –

          Yup. It’s really frustrating (for me) that so many are engaged in these discussions in a manner that doesn’t explicitly acknowledge that issue.

      • Agree, except that I would say it is a matter of carefully defining one’s measures, rather than ‘semantics’. which I am afraid that many people take to mean ‘a quibble’. As Andrew said in the OP, “well, the virus doesn’t care what the law is, it just cares what people do.”

    • Robbie: Thanks for the insightful comment. It’s always interesting when concrete real life specifics are incredibly relevant to how one has to interpret data.

  10. As someone living in Ireland, it seems strange to refer to our experience here as a “n=1 sample”. There are ~4 million people in Ireland, covid has been here for almost a year, and the effect of the three Irish lockdowns on covid rates are clearly visible to the naked eye: in each case infection rates went from increasing to declining ~10 days after lockdown was introduced.

    When you are talking about samples, you need to name the reference population. What is the proper “reference population” to use in considering the effects of introducing a lockdown? To me the reasoning would go something like this: we predict that introducing a lockdown will change the covid trajectory significantly within 14 days. We have had ~300 days since covid started, 3 lockdowns, and 3 significant changes in covid trajectory. The chance of one of those changes happening within 14 days of a lockdown announcement, purely by coincidence, is 14/300 ~ 0.05; the chance of three such changes happening within 14 days of a lockdown announcement is 0.05^3 ~ 0.0001, around 1 in 10,000. This association can be seen “with the naked eye”.

    It may be that there was some other cause that produced both the lockdown and the subsequent change in infection trajectory (the most likely one being that both the government and the people were responding to the rate at which the infection was spreading in the population, or to reports from the scientific advisory board on covid). But there is something to be learned even from this “sample size of 1”.

    • fin: I’m assuming you’re replying to my comment.

      The “n=1” refers to the number of data series being looked at. I guess we could also do some partitioning and look at the number of lockdowns (n=3 perhaps?) but I don’t see why the number people in Ireland would be relevant to the statistical questions about this specific data series, which is the sum case numbers from one country, Ireland.

      Nobody disputes that lockdowns have some correlation to cases going down, and (almost) nobody disputes that they probably ’cause’ some of the decline in case numbers. But we can imagine a counterfactual where Ireland didn’t lock down but everybody was looking at the explosion of case numbers and became more careful, with the result being a decline in case numbers almost like the one seen after the lockdown. And “people being more careful” isn’t the only variable that will correlate with case numbers (just like lockdowns) that could have an effect.

      I don’t understand what you are trying to say with “reference population”, and your statistics are obviously wrong but you address this in your next paragraph. Imagine someone lighting a candle every night in order to “cause” the sun to go up the next day. They do this every night, and the correlations are really amazing – the sun does indeed go up every time they light a candle. But it would be a mistake to conclude that lighting the candle caused the sun to rise. On the other hand, I feel like the case would be much stronger if the candle were forgotten one night, and the sun then did not rise the next morning. In the same way, I feel like there is much more to be learned about causality from Sweden – which did not lock down – than from all of the amazing correlations we see in places like Ireland.

  11. Nathanael –

    > I feel like there is much more to be learned about causality from Sweden – which did not lock down – than from all of the amazing correlations we see in places like Ireland.

    Comparing across countries introduces all manner of confounding variables, many of which are effectively eliminated by looking longitudinally at what has happened in association with NPIs in a give country. And of course, you also have the very same problem with correlation vs causation when you make comparisons across countries as you do when you look at what happens over time in association with NPIs in one country.

    Comparing Sweden to other countries is a breeding ground for confirmation bias. There are many factors that are unique or somewhat unique to Sweden and, actually, it’s quite possible that many of the outcomes subsequent to NPIs in other countries are closely paralleled by outcomes in Sweden – such as reduction in mobility.

    What do you think you can learn about causality from NPIs (enacted in other countries with usually very different variables/predictors) by looking at Sweden?

    • I agree that there are all kinds of confounding variables. I don’t think you eliminate confounding variables by looking longitudinally at a country – after all, there are variables like “weather” or “public opinion” that don’t stay constant.

      For what it’s worth, I don’t think Sweden tells us much – after all, determining causality after the fact is not really possible. However, if one compares the predictions of e.g. the early Imperial study for Sweden with what actually happened, it is clear that those modeling this did not know what was going to happen. I’m not making any strong claims here, because I don’t feel like I have any good knowledge about what is going on.

      Sure, there may be factors that make Sweden unique. But if Sweden has some factors that let it bring down covid cases without a lockdown, maybe other countries have them too?

      • Nathanael –

        > I agree that there are all kinds of confounding variables. I don’t think you eliminate confounding variables by looking longitudinally at a country

        I wasn’t quite suggesting that you would. I was suggesting that you would have a pathway to do analysis without introducing a vast set of confounding variables, such as baseline health differences (such as # of comorbidities), differences in pop density/distribution, SES differences, race/ethnicity differences, age profile differences, access to healthcare differences, broader cultural differences, differences in % of multi-generational households, differences in relative number of grandparent-caregivers, differences in ability to work from home, differences in ability to take paid leave from work, differences in healthcare policies w/r/t caring for people once infected, differences in quality of healthy care, broader scale differences in climate, amount thst people use public transportation, etc.

        > – after all, there are variables like “weather” or “public opinion” that don’t stay constant.

        Of course, all of those variables I mentioned differ across communities within countries as well as across countries. But you’re suggesting a comparison across countries. Estimating average impact within countries doesn’t work well if you’re generalizing from the average to specific communities. The same problem exists. But if you’re going to evaluate the efficacy of NPIs at a country-wide level – which after all is what we’re talking about – then looking longitudinally within countries has fewer significant confounds.

        > However, if one compares the predictions of e.g. the early Imperial study for Sweden with what actually happened, it is clear that those modeling this did not know what was going to happen.

        So what is your goal? Is it to make some kind of point about errors of previous modeling, or is it to determine our best understanding. I’d say that to the extent that earlier modeling – necessarily done when there was very limited information – was wrong, piling on with more poorly conceived modeling that tries to make comparisons across countries without controlling for myriad, very important predictor variables, does nothing to resolve the earlier problems.

        > Sure, there may be factors that make Sweden unique. But if Sweden has some factors that let it bring down covid cases without a lockdown, maybe other countries have them too?

        I find that question pretty meaningless because, as people discussed above, the definition of “lockdown” is so vague. In terms of many developments, the biggest difference between Sweden’s “non-lockdown” state and the “lockdown” state of the other Nordic counties seems primarily to just be a difference in the degree to which interventions were officially mandated. While that’s not a meaningless distinction, it’s implications to the efficacy of interventions is limited.

        The notion of “bringing down” infections, which implies some kind of intentional causal mechanism, seems particularly ill-suited when you look at Sweden, as an isolated country. It actually seems even worse when you do look at it cross-nationally when you compare to the countries that seem (to me) likely to present the fewest confounds, or confounds of limited magnitude of difference; e.g., it’s closest Nordic neighbors.

        • Yeah, I think that some of the early modeling had issues not related to the effectiveness or otherwise of NPIs.

          I am not really sure why Sweden gets focused on so much by US writers; there are several US states which did even less than Sweden. Though even inter-state comparisons within the US are pretty fraught, especially if there really is a seasonality effect (I thought the summer surge/wave in the Southern US was pretty strong evidence against it, as while relatively warm-climate TX FL AZ etc. still have winter flu seasons, but this current surge/wave really does look like a classic fall wave).

        • hey confused –

          Good to “see” you.

          For me the biggest issues are (1) the vagueness of the starting definitions of how people are approaching the analysis. As Daniel mentions above, people are disagreeing violently without even agreeing what it is that they’re disagreeing about. For me, that’s a HUGE red flag for motivated reasoning – in this case ideologically mrocsrsd reasoning. I think it’s highly unlikely that the strongly predictive signal of the ideological predisposition of the analysts views on the efficacy of NPIs is just purely coincidental. Nor do I think it is determine by one side or the other employing categorically superior science and,

          (2) The lack of consideration of the relationship between the precipitating conditions and the stringency of the interventions – of course one would expect, generally, more severe interventions where the pandemic was most out of ixnreok, and thus you can’t just compare outcomes as if the precipitating conditions weren’t ksefdlyboresixifc did outcomes independent of the interventions implemented, and

          (3) the lack of attention to how people are working from largely unsupported counterfactual presumptions about what would have happened absent interventions,

          (4) the willingness to compare across vastly different conditions to asses the efficacy of interventions withiir controlling for confounding variables

          (5) the lack of good cluster analysis to help look at interactions and mediation/moderation effects between and among different interventions.

          I think that basically this whole situation is one big clusterdf*ck.

          I started out with thinning I had two biggest issues but couldn’t get out without 5, and if I thought about it in sure I’d come up with more even bigger ones.

        • >>I think that basically this whole situation is one big clusterdf*ck.

          Yeah.

          It’s pretty critical IMO that people are not even talking about the same thing; all the statistical problems are bad enough, but if you’re not even asking the same question…

          There’s a big difference between “social distancing/masks have X effect” and “government-mandated measures have X effect”, since you’d need to account for how measures are reflected in actual on-the-ground behavior.

          I would like to see a study on how — just for example — mask *mandates* affect *the prevalence of mask wearing in practice*.

  12. Lots of good discussion, raising good points, here. But we need to remember that our views here are not representative of the population in general. We’re mostly talking in terms of what makes sense to us. I recently got involved in an online discussion in which one participant insisted that masks were not effective, because her definition of effective was “effective 100% of the time”. Anything less than that meant that it was not worthwhile to wear a mask. I don’t think she’s the only person in the world who thinks this way — indeed, I think that most of us in this discussion are aware that intolerance of uncertainty is a big problem. We need to take that into account when considering effects of interventions.

    • I believe some of this is a “wizard-of-oz” syndrome. I began perceive it fulminating among some of my friends this spring. It has several features. When the ostensible wizard fails in person to live up to the supplicant’s great and naive expectations — he was after all seeking a wizard who’d put everything back in order — he turns on ‘authority’ in general: “they all speak out of two sides of their mouths; they say This on Tuesday and they say That on Wednesday; they’re *all* charlatans; etc”. The third phase of the syndrome is, “I’ve found a (charlatan on you-tube) who tells me such-and-such and it contradicts completely what all you fools in the thrall of ‘legitimate’ authority believe; sorry you!”

    • Martha –

      > I recently got involved in an online discussion in which one participant insisted that masks were not effective, because her definition of effective was “effective 100% of the time”. Anything less than that meant that it was not worthwhile to wear a mask. k. I don’t think she’s the only person in the world who thinks this way

      +1

      I come across that constantly in online discussions about COVID.

      I’d guess it’s more prevalent online, as it seems to me to be largely a function of ideological investment (mixed with intolerance for uncertainty and motivated reasoning), but I’m reasonably sure it’s found aplenty in the offline world as well.

      • I don’t think that has so much to do with disbelief in the vaccine’s effectiveness as the idea that flu is no big deal, so people just don’t bother to get it.

        Certainly pre-COVID some years I just forgot, unless family members reminded me.

      • > About half of American adults get vaccinated for influenza in a typical year

        I had to check the number, from a European perspective it seemed implausibly high. I wonder if it’s worth it.

        • Carlos:

          I’m pretty sure that it’s worth it, given the low cost of the vaccine and the high benefits, both in reducing flu among people who get the vaccine and in reducing the spread among people who don’t get it. I mean, sure, you can wonder-if-it’s-worth-it for just about anything, but this one seems like a very easy case.

        • Yeah, I’m not sure what the disadvantage would really be, in most cases.

          A totally non-expert, wild thought: I wonder if the apparently low efficacy of flu vaccines is because non-vaccinated people aren’t *really* immunologically naive to flu?

          I’ve read that flu pandemics’ tendency to be less “biased toward the elderly” than seasonal flu might be due to past exposure to similar flu viruses, decades before; so if some level of immunity persists — well, probably practically everyone has had either the flu or a flu vaccine at *some* point in their life.

          So I wonder if flu vaccines would show better efficacy if the control population had never been exposed at all?

        • It also likely reduces severity among those who get it anyway, and associated hospitalizatons, ICU admissions and deaths.

        • I think this is true – but I think the reason why many people don’t bother to get it is the idea that healthy non-elderly adults have essentially zero risk of hospitalization/severe illness from flu in the first place, before vaccination.

          (I have no idea what the actual incidence of flu hospitalizations among non-elderly adults is. I’d imagine it is fairly low, but probably higher than the practically-zero people would assume.)

        • Oh, sure, I absolutely think it is worth getting… I am talking more about why people don’t (in practice), not denying that they should.

          (I don’t think most people know that there are tens of thousands of flu deaths in an average year… most people seem to think the flu is inherently no big deal, not helped by the fact that a lot of random mild illnesses not actually influenza get called “flu”, eg “a stomach flu”.)

        • Very late comment, but worth making I think. When I was considerably younger I got a case of he flu and within hours was in a wheelchair – a relatively rare side effect (I think it was called myositis) where the muscles become inflamed to the point that they won’t support your weight. I spent a week in the wheelchair and then slowly recovered fully. I’ve had an annual flu shot (27 years running now) ever since. So, I do think the cost/benefit analysis of he flu shot is relatively clear.

          However, and I think this is an important point. My absolute conviction derives from my personal experience (n=1). It is not uncommon for personal experience to trump statistics – it happens every time, and is human nature. We all know it is not a “rational” basis for making decisions, and certainly not for policy decisions (as opposed to personal decisions, such as whether or not to get a shot). But it is so very hard to put aside personal experience and rely on “the evidence,” particularly when we know “the evidence” is not quite what it appears to be (subject to so many forking paths, selection problems, poor statistical practice, etc.).

        • In Europe it’s recommended only for some specific subgroups so maybe it’s not so clear-cut. (The previous word brings to my mind another prophylactic measure that it’s very popular in the US but not in Europe outside of some specific subgroups.)

  13. Can any Stan people help with this paper:
    https://www.nature.com/articles/s41598-021-81419-w

    They fit a model to European data to determine the “surge date”, which I believe is this:

    a0 + N_c*exp(t*(a + (b – a)/(1 + exp(-0.252*(t – t_c)))) )

    The excel file in the supplements shows this:

    =I$1 + I$2*EXP( ( I$3+I$4/(1+EXP(-I$7*(A6-I$5))) )*A6 )

    Does this look like the correct likelihood for stan?

    y ~ normal(a0 + N_c * exp( t .* (a + (b – a) ./ 1 + exp(-.252 * (t – t_c))) ), sigma)

    I was trying to quickly fit it to some US state data to see if the correlation with latitude held, but all the t_c values are ~1 instead of near the visibly apparent “surge date”.

    • Posting this made me look closer and I found a dropped set of parentheses. I was focused on the elementwise multiplication/division and missed it. This was very helpful thanks.

      Should be:

      y ~ normal(a0 + N_c * exp( t .* (a + (b - a) ./ (1 + exp(-.252 * (t - t_c)))) ), sigma)

  14. Hi Andrew,

    Wondering if you could comment on this paper, which suggests that R < 1 before lockdown was implemented in the UK. It seems to loosen assumptions made by a paper driving policy in the UK, while also criticising many of the modelling assumptions.

    Alex.

    Was R < 1 before the English lockdowns? On modelling mechanistic detail, causality and inference about Covid-19
    S.N.Wood, E. C Wit, 2021
    https://www.medrxiv.org/content/10.1101/2021.02.03.21251112v1.full.pdf

    • Alex:

      I’ve not read all the background literature, but I’m supportive of the general claim that R can go very low just based on voluntary behavior. One reason for lockdowns is that they are a coordination mechanism which empowers people to do self-lockdown behaviors that they might find more difficult to carry out without an official rule.

Leave a Reply to rm bloom Cancel reply

Your email address will not be published. Required fields are marked *