Struggling to estimate the effects of policies on coronavirus outcomes

Philippe Lemoine writes:

I published a blog post in which I reanalyze the results of Chernozhukov et al. (2021) on the effects of NPIs in the US during the first wave of the pandemic and, if you have time to take a look at it, I’d be curious to hear your thoughts.

Here is a summary that recaps the main points:

– The effects of non-pharmaceutical interventions on the COVID-19 pandemic are very difficult to evaluate. In particular, most studies on the issue fail to adequately take into account the fact that people voluntarily change their behavior in response to changes in epidemic conditions, which can reduce transmission independently of non-pharmaceutical interventions and confound the effect of non-pharmaceutical interventions.

– Chernozhukov et al. (2021) is unusually mindful of this problem and the authors tried to control for the effect of voluntary behavioral changes. They found that, even when you take that into account, non-pharmaceutical interventions led to a substantial reduction in cases and deaths during the first wave in the US.

– However, their conclusions rest on dubious assumptions, and are very sensitive to reasonable changes in the specification of the model. When the same analysis is performed on a broad range of plausible specifications of the model, none of the effects are robust. This is true even for their headline result about the effect of mandating face masks for employees of public-facing businesses.

– Another reason to regard even this result as dubious is that, when the same analysis is performed to evaluate the effect of mandating face masks for everyone and not just employees of public-facing businesses, the effect totally disappears and is even positive in many specifications. The authors collected data on this broader policy, so they could have performed this analysis in the paper, but they failed to do so despite speculating in the paper that mandating face masks for everyone could have a much larger effect than just mandating them for employees.

– This suggests that something is wrong with the kind of model Chernozhukov et al. used to evaluate the effects of non-pharmaceutical interventions. In order to investigate this issue, I fit a much simpler version of this model on simulated data and find that, even in very favorable conditions, the model performs extremely poorly. I also show with placebo tests that it can easily find spurious effects. This is a problem not just for this particular study, but for any study that relies on that kind of model to study the effects of non-pharmaceutical interventions.

– To be clear, as I stress in the conclusion, this doesn’t mean that mask-wearing doesn’t reduce transmission, because this paper evaluated the effect of mandating mask wearing, which is not the same thing. It may be that, as another study recently found (though I have no idea how good this paper is), mandates don’t really matter because people who are going to wear masks do so even if they’re not legally required to do so.

Anyway, since you disagreed with my harsh take on Flaxman et al.’s paper about the effects of NPIs in Europe during the first wave, I was curious to know your thoughts about this other study.

I replied that I agree with Lemoine’s general point that it’s very hard to untangle the effects of any particular policy, given that so much depends on behavior. Another complication is the desire for definitive results. From the other direction, I see the value of quantitative analyses, as some policy choices need to be made.

Lemoine responded:

On the need to make policy choices and what it means for what should be done with quantitative analyses, I think it’s a very complicated issue. I was a hawk on COVID-19 before it was cool and, back in March, I was in favor of the first lockdown. I changed my mind after that because I became convinced that, whatever their precise effects (I think it’s impossible to estimate them with anything resembling precision), they couldn’t be huge otherwise we’d see it much more easily (as with vaccination) and they generally needed to be huge in order to have a chance of passing a cost-benefit test. One reason I came to deeply regret my initial support for lockdowns is that I have since then realized they have become a sort of institutionalized default response, which is something I think I should have predicted but didn’t, so this has taught me the wisdom of requiring a much higher level of confidence in social scientific results before acting on them. (I’m French and here we have been under a curfew and bars/restaurants have remained completely closed between last October and May of this year!)

In response to my question about what exactly was meant by “lockdown,” Lemoine pointed to his post arguing against lockdowns and added:

I [Lemoine] think it has been a problem in those debates on both sides, but it’s not really a problem in Chernozhukov et al. (2021) since they look at pretty specific policies. My impression is that, when people talk about “lockdowns”, they have in mind a vague set of particularly stringent restrictions such as curfews, closure of “non-essential businesses” and stay-at-home orders. In any case, this is what I’m referring to when I use this term, though in my work I usually talk about “restrictions” and state my position as the claim that, whatever the precise effects of the most stringent restrictions (again things like curfews, closure of “non-essential businesses” and stay-at-home orders) are, they are not plausibly large enough for those policies to pass a cost-benefit test when you take into account their immediate effects on people’s well-being, because even when I make preposterous assumptions about their effects on transmission and do a back-of-the-envelope cost-benefit analysis the results come out as incredibly lopsided against those policies. This is still vague but I think not too vague. In particular, I don’t think mask mandates of any kind count as “lockdowns”, nor do I think that anyone does even the fiercest opponents of those mandates.

I did not have the energy to read Chernozhukov et al.’s paper or Lemoine’s criticism in detail, but as noted above I am sympathetic with Lemoine’s general point that it is difficult to untangle causal effects of policies—and this difficulty persists even if, like Chernozhukov et al., you are fully aware of these difficulties and trying your best to address them. We had a similar discussion a few years ago regarding the deterrent effect of the death penalty, a topic that has seen many quantitative studies of varying quality but which, as Donohue and Wolfers explained, is pretty much impossible to figure out from empirical data. Effects of policies on disease spread should be easier to estimate, as the causal mechanism is much clearer, but we still have the problem of multiple interventions done at the same time, interventions motivated by existing conditions (which can be addressed statistically, but results will be necessarily sensitive to details of how the adjustment is done), effects that vary from one jurisdiction to another, and unclear relationships between behavior and policy. For example, when they closed the schools here in New York City, lots of parents were pulling their kids out of school and lots of teachers were not planning to keep showing up, so the school closing could be thought of as a coordination policy as much as a mandate. And then there are annoying policies such as closing parks and beaches, which nobody really thinks would have much effect on disease spread but represent some sort of signal of seriousness. And the really big thing which is people lowering the spread of disease by avoiding social situations, avoiding talking into each others’ faces, etc. From a policy standpoint it’s hard for me to hold all this in my head at once, especially because I’m really looking forward to teaching in person this fall, masked or otherwise. One of the points of a statistical analysis is to be able to integrate different sources of information—a multivariate probability distribution can “hold all this in its head at once” even when I can’t . . . ummm, at this point I’m just babbling. Speaking as a statistician, let me just say that it’s important to see the trail of breadcrumbs showing how the conclusions came from the data, scientific assumptions, and statistical model, starting from simple comparisons and then doing adjustments from there. I think the sorts of analyses of Chernozhukov et al. and Lemoine should be helpful in taking us in this direction.

P.S. Ethan Bolker shares this letter he sent to the Notices of the American Mathematical Society which he thought would be relevant to our discussion:

62 thoughts on “Struggling to estimate the effects of policies on coronavirus outcomes

  1. “I replied that I agree with Lemoine’s general point that it’s very hard to untangle the effects of any particular policy, given that so much depends on behavior. Another complication is the desire for definitive results. From the other direction, I see the value of quantitative analyses, as some policy choices need to be made.”

    I think the problem here is when we see that policy choices need to be made, and we attempt to inform those choices with quantitative methods that aren’t fit for the task, i.e. we’re dealing with problems that are too noisy for statistics, namely statistical methods for causal inference, to be of any use.

    In these sorts of situations, I think we’re better off admitting that and leaving those choice to be made on the basis of theory, etc. I appreciate the fact that you mention that any deterrent effect of the death penalty “is pretty much impossible to figure out from empirical data.” I think this is probably a similar situation.

    • Josh –

      > In these sorts of situations, I think we’re better off admitting that and leaving those choice to be made on the basis of theory, etc. I appreciate the fact that you mention that any deterrent effect of the death penalty “is pretty much impossible to figure out from empirical data.” I think this is probably a similar situation.

      I agree.

      What we see here is Phillipe, like many others making grand and emotional appeals to objective analyses of the costs and benefits of “lockdowns,” decry “the most radical violations of human rights in the West since the end of WW2,” when they haven’t even actually defined what “lockdowns” mean, but even more importantly IMO, haven’t even addressed the issue that ANY putative analysis of the differential efficacy of “lockdowns” as an intervention should, NECESSARILY, at least attempt to examine the counterfactual comparisons of what would have been different had the “lockdowns” not occurred.

      For example, accompanying the closing of non-essential businesses in this country were unprecedented financial assistance and extended unemployment and bans against evictions, and the like. Would that assistance have been provided if “lockdowns” hadn’t been mandated? Who knows? What would have happened had that assistance not been provided? Had “lockdowns” not been mandated, would teachers have been fired for not coming in to work because they didn’t want to risk infecting their 80-year old parent that lives with them? What would the ripple or possible multiplier effects have been if “lockdowns” actually did reduce transmission at least to some degree, and as a result w/o the “lockdowns” that occurred hospitals were more overrun than they were and more healthcare workers got sick and were thus unable to supply needed assistance to the many people who got sick and recovered?

      Phillipe’s reasoning, as is that of many other anti-“lockdown” advocates, is further accompanied by a totally unrealistic expectation that interventions should have been “justified by sound epidemiological and moral reasoning” and not implemented “because they are playing it by ear” and that contrariwise, the implementers of “lockdowns” “have fallen prey to the hysteria that has taken over both the media and the experts that advise them.”

      Looking beyond the interesting view that “moral reasoning” didn’t occur in the decision-making process (how does one make such a determination?), the problem as I see it is that (1) the cost-benefit analysis that Phillipe demands should accompany any such public health policy is an incredibly complex task if it’s to be done well, requiring careful control for many very complex confounding variables that are near impossible to control for, necessarily demands longitudinal data collected over a significant period of time, requires all kinds of sophisticated methods for sensitivity analysis and assessing clustering effects and interaction effects and moderator and mediator effects.

      And that (2) Phillipe demands “epidemiological reasoning” but doesn’t seem to really understand what’s involved in that, and instead thinks that his “back of the envelope” attempts to perform such an analysis in some way prove “radical violations of human rights.” IMO, this is a situation of “fat tails” of perhaps low probability but “high damage function” risk in the face of extreme uncertainty situated in a fast-moving and incredibly complex environment. And some dude gets out his pencil and his envelope and makes such, sorry, grandiose assessments, falsely believing that his assessments somehow suffices whereas others are insufficient.

      That isn’t to say that people shouldn’t perform assessments or discuss the flaws in the assessments of others. Just that they should be at least somewhat circumspect about their conclusions that their own assessments suffice, and that as you say, there are basic questions here about “fit for the task” no matter who’s doing it.

      (To explain the rant-level of my rhetoric – I come at this from being long involved in on-line arguments about climate change, which is often subject to the exact same kind of unrealistic and self-inflated notions of “fit for the task” as Phillippe displays on this topic. It gets my dander up).

      And btw, nice name.

      • Hi Joshua,

        Regarding your parenthetical remarks, I think the difference here is that in climate science, extremely high-quality models are available. Whereas when it comes to predicting and modeling this virus, the best “establishment” research is stuff like Chernozhukov et al, the Imperial College models, etc, which are themselves no more reliable a source of information than back-of-the-envelope calculations by statistically literate laypeople.

        On the issues you bring up about the impacts on social assistance and overwhelmed hospitals, I agree entirely with your points, but these were issues that presented early in the pandemic. I would say (along with Lemoine’s original opinion) that early lockdowns were reasonable conceived of as part of an overall response, but after Spring 2020 when the precedent for aid had already been set and the risk of overfull hospitals and PPE shortages was past, the situation was much different and the arguments for lockdowns were much weaker.

        • Dave –

          Certainly there is more sophisticated and more thoroughly researched modeling available for climate change, and modeling for climate change has the advantage of being focused on relatively well-understood manifestations derived from basic physical laws and not the vagaries of human behavior, but it’s still and issue with a lot of uncertainty w/r/t the range of outcomes and likewise is a matter of risk assessment in the face of those huge uncertainties.

          Not surprisingly, just like with climate change, with the pandemic we have politically and ideologically oriented discussants who are completely sure that they’ve modeled the costs and benefits of interventions in ways that prove the stupidity of “experts.” Well, the “experts” that they disagree with politically, that is.

          I don’t think it’s mere coincidence that with a high degree of skill we can predict the conclusions someone will draw from their “lockdowns” modeling by asking about their basic political or ideological orientation.

          I would certainly agee that in somw theoretical sense we can offer conjecture thar the relative costs and benefits of interventions would shift over time, and that what might have been fully justified initially would lose justification as the “far tails” of the potential risk were motivated. But I think that determining exactly what that trajectory of the relative shift over time would have been is an enormously complicated task, and one that requires a high degree of circumspection and humility. Not the kinds of certainty and moral posturing that we have seen so frequently.

        • I agree with a lot of what you say; probably the difference between us is that we fall on different sides when it comes to our best guess as to when one could be confident that the tail risk of opening is low enough to be acceptable. On the issue you raise here:

          >I don’t think it’s mere coincidence that with a high degree of skill we can predict the conclusions someone will draw from their “lockdowns” modeling by asking about their basic political or ideological orientation.

          It’s perhaps worth mentioning that I’m a liberal Biden voter who has always voted straight-ticket Democrat for high office. I was convinced in Spring 2020 that the way forward for the US was test, trace and isolate, and that the restrictions of that time would pay off in the future as the United States ended up like Australia. (I still think Australia’s handling of the pandemic was excellent, up until the point when they collectively decided that they didn’t want to be vaccinated and preferred NPIs over vaccines.) Anyway, I’ve now concluded that that was all a pipe dream and nation-wide contact tracing and suppression in the US and Europe would be a logistical impossibility even if it had been politically possible.

          Like you, I hate the moralistic tone that the anti-“lockdown” side often takes. But they do have a point that freedom of assembly and movement are important liberties that shouldn’t be interfered with unless the expected benefits are very great. In an acute emergency, that burden of proof shifts somewhat. But once months’ worth of data were in hand and the benefits of restrictions remained unclear at best–and the best possible outcome, contact tracing and suppression, was clearly never going to happen–at that point civil liberties needed to be taken more into consideration. Not necessarily as an overriding factor, but as a non-zero factor.

        • Dave –

          I agree that we pretty much agree, including that the difference is probably that we fall out in a different point along the trajectory of exactly where the tail risk should drive decision-making.

          I’ll note that as a liberal Biden voter, you are not an “anti-lockdown” advocate. I probably don’t agree that with the political will we couldn’t have significantly lowered the arc of the pandemic through testing, tracing and isolation – but I do think that the necessary political will could never have been likely given that it would have required more embracing the balance between individual “freedom” and community welfare as non-zero sum. Probably the biggest problem there was the failure at testing early on, and as to how “political” that problem was in its nature is a bit hard to discern.

          In all fairness, it isn’t only the anti-“lockdown” side that has an unfortunately moralistic tone, and I do agree that it IS important to keep an eye on concerns about limiting freedom of assembly, etc. However, I probably see that issue as more of a trade-off than you. For example, IMO, freedom of assembly at some level should include being free to go into a supermarket where the management has determined that their employees and my welfare can be infringed upon by customers who won’t just put on a mask.

        • I will also add that the vaccines shift the risk avoidance calculus to a completely different landscape than where it was 18 months ago or a year ago or 6 months ago. And they also throw in yet another “freedom” spanner into the works.

        • Just to split hairs a bit: I’d say that I am now a (moderate) anti-lockdown advocate, I’m just not a single-issue voter. (And if I were a single issue voter, this wouldn’t be the issue.)

          Good discussion–thanks!

        • Hi Dave –

          My response (and another comment below for Philippe) are stuck in moderation. If they don’t get out of blog prison tomorrow I’ll try reposting.

      • Cost benefit analysis of mass Covid lockdowns are not difficult because on any objective and reasonable assessment they easily fail a cost benefit test, by a lot. So there’s no need to get into the complexities of making precise estimates (not really possible in any case), you just have to bring a cost benefit mindset to the issue.

        And if you can’t see the enormous differences between this situation and the climate case I question your expertise in or basic knowledge of climate science and modeling.

        • MS –

          > Cost benefit analysis of mass Covid lockdowns are not difficult because on any objective and reasonable assessment they easily fail a cost benefit test, by a lot.

          I couldn’t possibly count the numbers of times I’ve seen essentially that very same argument with respect to the existential “cost” of mitigating climate change.

          > And if you can’t see the enormous differences between this situation and the climate case I question your expertise in or basic knowledge of climate science and modeling.

          Well, not much more that I can say there other than that along with many people intimately linked to the discussion of climate change, I think there are VERY strong parallels in how the public discussion has played out with respect to the “cost/benefit” analysis needed to assess the wisdom of interventions.

          Indeed, as I noted above I could just replace a few words in your comment and post it on climate blogs and many people would just say “same old, same old.”

      • Joshua, your assessment of Philippe is hilariously distorted that it indicates nothing more than that you cannot have possibly read his analysis on these topics or, if you have, that you cannot possibly have understood it.

        And if the benefits of a policy are so difficult to determine that it is effectively impossible to know whether it hurts or harms on net, then it should not be implemented, since even implementation alone has obvious costs. “We can’t possibly know” does not imply “therefore I can choose as I please.”

    • When the benefits of a policy are impossible to determine, it is probably better simply not to implement that policy, since costs of implementation alone outweigh when benefits are essentially a wash.

  2. Two thoughts. First, isn’t it contradictory to stress (correctly) that the effect of lockdowns on case rates (or some other outcome of interest) should take into account voluntary distancing, masking, isolation etc., which vary according to the latent factors that also influence propensity to lock down, and then assess the economic cost of lockdowns as the total impact of closures and disemployment without taking into account voluntary behavior? Why should we be so sure that, properly measured, restrictions wouldn’t pass a cost-benefit test?

    Second, restrictions/lockdowns are likely to be most effective when part of a larger package that includes travel restriction, testing and contact tracing and “real” (not pro forma) quarantine. These interact, don’t they? Countries that came close to extinguishing Covid did all of them to some extent. By themselves restrictions slow down the aggregate pace of transmission by limiting it in some circumstances while it persists in others, but that deceleration can make more individual-level interventions feasible.

    • Peter:

      I agree on both points. On your first point, perhaps there is a problem that for economic impacts, it can seem natural to use very direct accounting-type calculations, which build in hidden assumptions about causality.

    • To his credit, Lemoine’s argument doesn’t rely on estimates of the more purely economic costs of lockdowns, which he admits are “highly uncertain,” but rather on assessing the immediate effect on well-being (e.g., not being able to go outside during the curfew hours,as in France). Obviously there’s still plenty of room for disagreement on how large these direct effects are, but I agree with Lemoine that, over a year into the pandemic, it would be good for governments to be more explicit about their assumptions regarding those costs—how many months of a curfew outweigh a year of life saved, etc.

      • Right, you could say that I implicitly assume lockdowns have no economic costs whatsoever, but note that I also don’t impose any particular assumption on their immediate effect on people’s well-being. I sometimes make such an assumption for illustrative purposes, but as I explain in another comment in response to Peter Dorman, I also compute an upper bound on how large this reduction of well-being can be before lockdowns cease to pass a cost-benefit test given extremely optimistic assumptions about their effect on the number of deaths. Anyone can then decide for themselves whether the immediate reduction in well-being caused by lockdowns could plausibly have been smaller than this upper bound, but it’s so incredibly low that I find it very difficult to believe that anyone could genuinely believe it was.

    • First, isn’t it contradictory to stress (correctly) that the effect of lockdowns on case rates (or some other outcome of interest) should take into account voluntary distancing, masking, isolation etc., which vary according to the latent factors that also influence propensity to lock down, and then assess the economic cost of lockdowns as the total impact of closures and disemployment without taking into account voluntary behavior?

      I don’t think it would be contradictory, but I agree it would be stupid, which is why I didn’t do that :-) There is a common argument that, if lockdowns really don’t make a huge difference on the number of cases/deaths because people voluntarily change their behavior in a way that reduces transmission even in the absence of a lockdown, then lockdowns can’t possibly be that costly, but that’s obviously fallacious. Indeed, the fact that, even without very stringent restrictions, people voluntarily change their behavior in such a way that it has roughly the same effect as very stringent restrictions would have doesn’t imply that people change their behavior in the same way they would have been forced to do if very stringent restrictions had been in place and, in particular, that such voluntary behavioral changes had the same impact on their well-being as those restrictions would have. For instance, people in France have been under a curfew for more than 6 months, but it’s very dubious that it had more than a small effect on transmission and not even clear that it had any effect. But it definitely had a huge effect on their well-being. The claim I’m making is that very stringent restrictions didn’t buy you much bang (i. e. reduction in transmission) for your buck (i. e. reduction in well-being) because the behavioral changes that really contribute to reducing transmission occur even in the absence of very stringent restrictions while the behavioral changes that such restrictions force people to make although they wouldn’t have made them otherwise don’t reduce transmission that much.

      I think one reason why people are inclined to commit this fallacy is because they see the costs of restrictions as limited to the economic costs, but if you actually read my back-of-the-envelope cost-benefit analysis I don’t even take into account the economic costs of restrictions and just consider their immediate effects on people’s well-being. Moreover, I’m very careful not to commit the mistake you flagged, by talking about the impact of well-being that a lockdown would have relative, not to a situation in which there is neither a lockdown nor a pandemic, but a situation where there is a pandemic and even some restrictions in place, just far less stringent than what pro-lockdown advocates were asking. Finally, instead of making some arbitrary assumption about the immediate effect of very stringent restrictions on people’s well-being (relative to a situation where there is still a pandemic but far less stringent restrictions), I make some ridiculously optimistic assumption about the effect those restrictions would have on the number of deaths and ask how large the immediate effect they have on people’s well-being can be at most for them to pass a cost-benefit test. And the result is that, even when I make a preposterous assumption about the effect of very stringent restrictions on the number of deaths, they would have to reduce people’s well-being by something like 1% to pass a cost-benefit test. With more realistic assumptions, that upper bound would likely be even smaller. Now, if someone tells me that he doesn’t believe that, relative to the situation in a country like Sweden where there were relatively few restrictions, putting the entire population under a 6pm, 7pm or 8pm curfew, keeping all bars and restaurants closed, etc. for 6 months (as France did after the second wave), reduces people’s well-being by more than 1% on average, than there is nothing I can say to convince that person that lockdowns were not worth it, but I don’t believe many people really believe that.

      On your second point, I think it’s more plausible that borders closure (even without lockdowns) played a crucial role in allowing some countries to contain the pandemic, but as I argue at length in my post against lockdowns I don’t think it was a realistic possibility for many countries. European countries in particular would have faced an insurmountable coordination problem in my opinion and even if they had somehow succeeded in surmounting them the cost would have been much larger than for e. g. Australia given how tightly integrated they have become. But perhaps more fundamentally, in my post, I focus on the options that European countries had last Winter, when incidence was very high everywhere. So as I explain in the post, even if you think that it would have been possible for Europe to replicate Australia and New Zealand’s success during the summer of 2020 (which again I don’t think it could have), it doesn’t really matter to my argument because that train had already sailed. To be clear, I also don’t think that Australia and New Zealand’s approach was good policy and I think that in the long-run they will come to regret their decision, but that’s a whole other story and I haven’t made that case in writing anywhere. Of course, you can still disagree with me on all that, but I just wanted to point out that I did not commit the mistakes that you were attributing to me, even if I agree that many anti-lockdown activists probably did.

      • Maybe the question should be reformed to identify under what conditions do various suites of interventions provide net benefits (and by how much). March 2020 and now, and New Zealand and Paris are different.

      • I haven’t read your work, so I don’t know how you put health and social restrictions in the same metric in order to do a cost-benefit analysis—but that would take us on a tangent.

        If you are basing your argument on the specifics of the French curfew, I agree; it made little sense from a public health standpoint but was extremely restrictive of people’s ordinary lives. On the other hand, closing indoor bars and restaurants was a good call. Once aerosol transmission was established, it became clear that indoor gatherings, especially maskless (as eating and drinking demand), are unwise. In both of our countries restrictions often seemed to be motivated by moralistic judgments instead of rational policy-making.

        I would encourage you to consider unemployment as a quality of life as well as specifically economic issue. Loss of a job is immensely stressful and often has consequences for years to come, on children as well as the unemployed themselves. As you may know, there is quite a bit of research on this. The evidence, e.g. Nordic comparisons, suggests that the employment consequences of the pandemic were primarily the result of the health threat and not the specific restrictions imposed by governments; this is an application of the point about distinguishing between the total cost of behavioral change and the marginal cost of mandatory restriction.

        As for the potential synergy between different NPI’s, I agree that the EU was hobbled by its institutions. It lacked a common public health apparatus and decision-making entity, and even Schengen is so ruptured that a common border interdiction was out of the question. Meanwhile the social safety net side of policy, the support for individuals and businesses so they could adapt to public health constraints and make it through the pandemic, was different everywhere. There have been no true success stories in Europe. The US, on the other hand, could have done far better, and in that sense its failures are even more deplorable.

        • Stop the presses! I just came across this recent NBER paper that uses a multidimensional framework to compare health outcomes and well-being indicators between two groups of countries, the “eliminators” whose policies were aimed at eliminating community transmission and the “mitigators” whose goal was just to flatten the curve. They found the eliminators did better across the board. I haven’t read the paper closely, but I suspect there may be issues with the way they instrumented for membership in these two groups, east Asian geography and SARS deaths from prior epidemics. These countries are distinctive in many other respects as well, which muddies causal interpretation. The inter-Nordic comparison fares better on this count, however. (That’s the closest thing we have to a natural experiment, IMO, since aside from its peculiar political choice Sweden is not so different from Norway and Denmark.)

          I mention this because the metric problem I referred to above is addressed here, rightly I think, by preserving the distinctiveness of different types of outcomes.

      • When you estimate the impacts of Covid on well being, do you account for the impact of the distress from falling ill, the distress from having an ill family member, and the distress from losing a family member or close friend?

        Also, do people take into account that in areas without restrictions, some behavioral change is likely induced by observing other jurisdictions imposing restrictions and medical people arguing for restrictions?

        • I’m skeptical that people’s voluntary behavior is affected that much by inferring the seriousness of the pandemic from what other jurisdictions choose to do, but even if it is, that shouldn’t change the calculus for many policymakers. If you’re the government of (say) the Netherlands, the signal “this is super serious because lots of countries are imposing restrictions” is going to remain regardless of what you choose to do, so you can “free ride” on it.

          As for the other effects, yes, he does discuss the effects on those who fall ill but survive and argues that they’re likely dwarfed by the decrease in well-being due to loss of life even with heroic assumptions about how bad things are for survivors. If you disagree, I’d be interested to see some quantitative spitballing of those effects that’s both plausible and alters the ultimate conclusion.

      • Philippe –

        > For instance, people in France have been under a curfew for more than 6 months, but it’s very dubious that it had more than a small effect on transmission and not even clear that it had any effect.

        You’re claiming that you can measure the impact of “lockdowns” on “well-being” when in fact you have no way of doing that, not the least because you can’t measure how much differently “lockdowns” changed people’s behaviors compared to how much they would have changed absent the “lockdowns.”

        Not to mention you think you’re assessing their differential cost, but are only doing so by making entirely unsupported counterfactual assumptions. What would “well-being” have been with a raging pandemic and no formal economic support for people who lost jobs because no one went to the movies or for people who would get evicted because the couldn’t pay rent or people who lost their homes through foreclosure? Nyiu font know shy of that yet your conclusions imply certainty that you do.

        You’re conflating correlation with causation, i.e., that because well-being suffered concurrent with “lockdowns,” therefore “lockdowns” (and not the pandemic itself) were the cause. Not that I’m suggesting it’s an either/or situation, but merely that you need more respect for uncertainty.

        • “You’re claiming that you can measure the impact of “lockdowns” on “well-being” when in fact you have no way of doing that, not the least because you can’t measure how much differently “lockdowns” changed people’s behaviors compared to how much they would have changed absent the “lockdowns.” ”

          This is why the there is such desperate need for the alternate universe machine favored in science-fiction. So then it could be known what would have happened in France had they done nothing differently, *but* for their “lock-down”. Absent such a miraculous intervention, we are forced to admit all our actions and reactions are but mere suppositious ephemera; plus or minus minute fragment of knowledge maybe squeezed out of the unhelpful data by ever more fanciful and intimidating mathematical acrobatics. Oh, well; I know where to put my money — in that sci-fi startup that’s announced the “alternate universe machine” is soon to be a reality. Just imagine, if it were; maybe I could dial up the universe of my choice. But in sleep … dreams.

        • Or alternatively, people could just stop trying to make a scientific argument that they’ve bent reality into the shape their ideology would prefer, as Phillipe has done

          And instead accept what they don’t know and talk about the ranges of uncertainty.

        • “And instead accept what they don’t know and talk about the ranges of uncertainty.”

          My science-fiction friends assure me there is a parallel universe in which this is so!

      • Is the claim that lockdowns have little effect biased by omitting the effect on regions that don’t enact them. This occurs because seeing some region lockdown sends a signal to other regions that the problem is serious.

        Also there is a cost of being sick, possibly with permanent damage and to having friends and family die.

      • Not sure how you measure and value a % quality of life change for a year. It is probably non-linear in that a 10% decrease in the quality of life for a year may be less costly to people than a .1% chance of losing the rest of your life.

        For many people the impact of restrictions on quality of life may have been positive. Mask mandates made them more comfortable in certain places and made hospitals less crowded for people needing other treatments. Many people refused to eat indoors at restaurants even when permitted with restrictions on capacity.

      • Philippe –

        > For instance, people in France have been under a curfew for more than 6 months, but it’s very dubious that it had more than a small effect on transmission and not even clear that it had any effect.

        This misses an important point – which is not that you need to know how much people’s behavior changed as the result of mandates to measure the impact on transmission, but that you need to know how much mandates changed behavior to measure the outcomes you’re attributing to them.

        You’re claiming that you can measure the impact of “lockdowns” on “well-being” when in fact you have no way of doing that, not the least because you can’t measure how much “lockdowns” changed people’s behaviors compared to how much they would have changed absent the “lockdowns.”

        You think you’re assessing a differential cost of “lockdowns,” but are only doing so by making entirely unsupported counterfactual assumptions. What would “well-being” have been with a raging pandemic and no formal economic support for people who lost jobs because no one went to the movies, or for people who would get evicted because they couldn’t pay rent, or people who lost their homes through foreclosure? You know none of that yet your conclusions imply a certainty that you do.

        You’re conflating correlation with causation, i.e., that because well-being suffered concurrent with “lockdowns,” therefore “lockdowns” (and not the pandemic itself) were the cause. Not that I’m suggesting it’s an either/or situation, there’s good reason indeed to assume that “lockdowns” had some negative differential effect (to be balanced against potential positive effects) but merely that you need more respect for uncertainty.

        • “You think you’re assessing a differential cost of “lockdowns,” but are only doing so by making entirely unsupported counterfactual assumptions. What would “well-being” have been with a raging pandemic […]”

          Yep. The calculation starts with the direct effects of getting COVID set to zero, while the effect of being forced to cooperate for the greater good of society is devastating to the well-being of certain types of people. And it’s just so, no measurement necessary.

        • Matt –

          > …is devastating to the well-being of certain types of people.

          Indeed, that is also a manifestation of hidden assumptions. Obviously, the “well-being” effect would be varied across strata of society, and wouldn’t exist as some kind of differentiated and aggregated mass. To suggest it is, IMO, is pretty meaningless. Not entirely unlike talking about COVID’s IFR without discussing the enormous age stratification.

        • …undifferentiated…

          It is materially important that polling suggests that there’s a positive correlation between those who would likely to be the most affected and support for interventions. At least in the US. In the very least it’s important to note that there are differences.

          IIRC, Andrew touched on that in one of his posts…

          By appointing themselves as arbiters of what’s true and right, and as protectors of “freedom,” IMO, anti-intervention advocates display a rather discernable sense of entitlement.

        • > You’re conflating correlation with causation, i.e., that because well-being suffered concurrent with “lockdowns,” therefore “lockdowns” (and not the pandemic itself) were the cause.

          He actually isn’t—the effects on “well-being” that he’s considering are not some inferred indirect effects, but the actual direct effects: For example, in the case of a curfew, the effect is “not being able to go outside after 6 pm,” which is obviously caused by, and not just correlated with, the curfew, and the question is then how many hours or days of curfew-less time would you be willing to give up given particular (very optimistic) assumptions about their effect on cases/deaths/hospitalizations/long-term health.

          I’m sure lots of “anti-lockdown” writers are guilty of the many errors you’re accusing Lemoine of, but if you read his posts you’ll see that he either avoids most of them or explicitly deals with them. (For example, contrary to your criticism above that he’s conflating various policies under the term “lockdowns,” he seems to be a lot more careful than you to distinguish between different interventions, as briefly adverted to in his exchange with Andrew above). You may have no interest in reading them (they’re long!) but you should be aware that it just makes you sound wholly unconvincing to those who actually read the posts, even those who don’t agree with them.

        • Raghav –

          > He actually isn’t—the effects on “well-being” that he’s considering are not some inferred indirect effects, but the actual direct effects: For example, in the case of a curfew, the effect is “not being able to go outside after 6 pm,” which is obviously caused by, and not just correlated with, the curfew, and the question is then how many hours or days of curfew-less time would you be willing to give up given particular (very optimistic) assumptions about their effect on cases/deaths/hospitalizations/long-term health.

          How do you measure the “direct effect” of a 6 PM curfew on “well-being” as differentiated by myriad other factors, and the interaction and mediator/moderator effects of myriad other factors, be they the result of government policies or merely behavioral changes that would have taken place in the absence of the 6 PM curfew? If there is a 6 PM curfew and people aren’t out on the streets at 6:01 PM, do you know whether their absence is the result of the curfew or the mere fact that people are staying home because they don’t want to get infected? Sorry, but the notion of measuring the “direct effect” of specific policies such as a curfew is necessarily an immensely complicated task, requiring as I said elsewhere all manner of sensitivity analyses and controls for interactions and moderators and mediators. And then you have the whole issue of timing – at what point in the trajectory of the pandemic were the initiatives implemented and how does that specific point along the trajectory affect the outcome measures? Seems to me that your mathematical equation is absolutely guaranteed to confirm your initial “prior.”

          I’m not saying don’t do some analysis – just have some humility about the enormity of the task and in promoting your conclusions.

          > …he seems to be a lot more careful than you to distinguish between different interventions, as briefly adverted to in his exchange with Andrew above).

          I’m not promoting conclusions with respect to this kind of analysis, but saying that “it’s complicated”.

          > You may have no interest in reading them (they’re long!) but you should be aware that it just makes you sound wholly unconvincing to those who actually read the posts, even those who don’t agree with them.

          I’ve read some and didn’t come away as impressed as you – but since you’re convinced of the integrity of his conclusions, then maybe you can explain what he showed in a bit more detail how those questions I’ve raised are answered in his measurement of the “direct effect” of a 6 PM curfew in some general framework, let alone as applied to specific contexts in which they were implemented. Or take any specific intervention of your choice where he explained the “direct effect” on “well-being,” and offer an explanation for me – it doesn’t have to be that one.

        • As Lemoine notes above, he didn’t attempt to actually come up with an estimate of the “direct effect” of the curfew (i.e., the loss in well-being attributable to not being able to go out past 6 pm when, in the absence of a curfew, one would have wanted to go out despite the presence of the pandemic). What he instead does is to pose the question of how small that loss in well-being would have to be in order to justify the curfew (or other intervention), even if you make heroic assumptions about the effectiveness of those interventions. If you think that loss is very small (as it would be if no one had any desire to go out on or after 6:01 pm on account of the pandemic)—or if you think it’s impossible to estimate with even the very low degree of precision necessary—then the argument won’t move you.

          The point is not to estimate that loss in well-being with anything resembling rigor, but simply to show how small that loss of well-being would have to be—and it turns out that it would have to be quite small, to a degree that’s surprising (at least to me). The Slate Star Codex guy put together this Guesstimate model, the second of which deals with Sweden during the first wave, and where the confidence interval for the lives saved by lockdown includes the scenario where the lockdown prevents every single case in that country (which I think most people would agree represents a heroic assumption with respect to the efficacy of NPIs, though maybe you disagree). Since you think the result is guaranteed to confirm your initial priors, feel free to substitute in your own priors and see whether, in your view, the person-lockdown-months needed to save a QALM is worth it. (Naturally, this back-of-the-envelope model doesn’t touch on a number of relevant considerations, like the distributional impact of interventions, but I think that for many people it will not simply confirm their priors. I also think that at this point in the pandemic, democratic governments should be publishing more sophisticated versions of models like these in order to make explicit the assumptions behind the policies that they’re maintaining in place.)

        • Raghav –

          Ok, I’m willing to go with an obvious observation that I’m far from the smartest person in the room here, and that I’m missing something that’s quite obvious, but I don’t see how you get to this:

          …he didn’t attempt to actually come up with an estimate of the “direct effect” of the curfew.

          From all of this:

          ===============

          Indeed, while they are in place,restrictions reduce people’s well-being because they prevent them from doing many things they would like to do.

          or

          What I want to do is compute the upper bound of the immediate effect a lockdown would have on people’s well-being* in order for the benefits of that policy to outweigh the cost

          or this:

          since we have assumed that a 2-month lockdown followed by a gradual reopening over another period of 2 months would save 150,000 years of life over 4 months and Sweden has a population of 10.2 million, a lockdown would have to reduce people’s well-being by at most 150,000 / (10,200,000 * 4 / 12) = 4.5%

          or

          Moreover, keep in mind that, in order to estimate this upper bound of the reduction of well-being caused by a lockdown, I have made preposterous assumptions about how many years of life a lockdown would have.

          or

          But I have also ignored a lot of things on the costs side of the ledger, since I have only taken into account the immediate effect of restrictions on people’s well-being,

          or

          but as I have argued above it’s nowhere as large as pro-lockdown advocates claim and the effect they have on people’s well-being alone is enough to make them completely irrational from a cost-benefit perspective

          ==============

          What I see there is a definitive assertion that he knows for sure (and has proven) that the sign of the effect of “lockdowns” on “well-being” is negative, and not only that, but he must have also (1) disaggregated the effect of “lockdowns” from the effect of the pandemic itself on well-being in order to know that the putative negative effect isn’t from other sources and (2) that actually he’s run some sort of model to assess the counterfactual scenario and that in fact it’s not possible that “lockdowns” actually INCREASED well-being over what would happened in their absence, (because he knows somehow that absent the lockdowns “well-being” wouldn’t have been even WORSE).

          I’ll just take that first one again:

          Indeed, while they are in place,restrictions reduce people’s well-being because they prevent them from doing many things they would like to do.

          This is an assumption, as far as I can tell, that necessarily relies on some kind of quantification of a putative NEGATIVE effect on “well-being” as measured against any potential POSITIVE effect of the intervention on people’s well-being. As far as I can tell, there has to be an assumed (or implied) calculation in there somewhere.

          So I see you and he telling me that I”m wrong, and that he isn’t doing any such calculation – but yah, I don’t see how he isn’t doing calculations, despite having been told (mostly through argument by assertion) that I can’t believe my lyin’ eyes.

          > The point is not to estimate that loss in well-being with anything resembling rigor, but simply to show how small that loss of well-being would have to be—

          This doesn’t help my confusion: How do you know whether something is larger than the minimum size that the loss would be if, you haven’t measured its size?

          I appreciate your attempt to explain and would love to be able to understand what you’re saying but I”m willing to drop it here so as not to tie up the blog any further.

          > I also think that at this point in the pandemic, democratic governments should be publishing more sophisticated versions of models like these in order to make explicit the assumptions behind the policies that they’re maintaining in place.)

          I would certainly agree that this issue should be studied further – so at least we have that point of agreement.

          Thanks for the SlateStarCodex link.

        • >>This doesn’t help my confusion: How do you know whether something is larger than the minimum size that the loss would be if, you haven’t measured its size?

          It seems to me that this boils down to just semantics about what ‘measuring’ something means, as opposed to giving a ballpark estimate. Raghav and Philippe are using ‘measure’ to mean a specific determination of quantity, as in: I can know a dog won’t fit into a mouse hole even though I haven’t measured the dog’s size, because I can make a reasonable ballpark estimate of the dog’s size based on my background knowledge.

          Philippe is saying that he can make a reasonable ballpark estimate that the loss of well-being due to restrictions outweighs the gain in well-being from lives saved. because the former would have to be less than ~1% in order to be outweighed by the latter.

          Is he right that this is an obvious ballpark estimate? The right test is exactly the one he suggests: Would a reasonable person prefer to live 99 healthy days under pandemic conditions without the restrictions, or 100 healthy days under pandemic conditions with the restrictions?

          For me, the restrictions where I lived didn’t really cost me any well-being, because I have no children and I was able to spend the pandemic working remotely in a beautiful rural area, and I had no interest in taking the risk associated with things like indoor dining. So I would prefer the 100 days with restrictions. But I have many friends with children who would have strongly preferred the 99 days without restrictions, because of the chaos and disruption from school closings. That sort of experience is much more typical, and so I’m inclined to say that Philippe’s ballpark estimate is correct even for US restrictions. And if the restrictions in my area had included an 8 pm curfew, I certainly would be inclined to take the 99 days!

        • Philip:

          To continue this discussion, one thing that irritates many people (including me) are stupid rules, such as telling people to wear masks in non-crowded settings outdoors, or shutting down parks and beaches, or removing the hoops on the local basketball courts. I can understand the political reasons for such rules—people want something to be done—but that doesn’t stop me from being annoyed!

        • Dave –

          > It seems to me that this boils down to just semantics about what ‘measuring’ something means,

          Maybe…but…

          > Philippe is saying that he can make a reasonable ballpark estimate that the loss of well-being due to restrictions outweighs the gain in well-being from lives saved.

          But this is an assumption that depends on an assertion that the effect of the restriction is negative, as opposed to positive, and in fact that the size of that negative impact can be measured with some degree of accuracy. How do you know that the impact of well-being on a curfew, say, is negative? Let’s take a curfew because there’s rioting. Should we assume the effect is negative because the rioters feel “restricted”? Phillipe is assuming that the net effect of “well-being” resulting from interventions is negative, based on a simplistic logic that people’s movements are restricted. Maybe for some, the restriction placed on movement has a positive impact on well-being. Maybe you think that’s illogical – which is fine, but it seems to me that it’s a counterargument that needs to be addressed. I get that anti-lockdown folks assume it’s negative because it seems negative to them – but in fact many people have been advocating for the interventions precisely because (I assume) in their judgement the effect on their well-being will be positive. Positive well-being is not one and the same as “deaths avoided” or “transmissions averted.”

          I’d say it’s plausible that the effect of restrictions may well be positive. But more to the point, we really can’t even get close to measuring whether the effect is negative or positive, let alone measure, estimate, ballpark, whatever you want to call it, the magnitude of that negativity.

          > For me, the restrictions where I lived didn’t really cost me any well-being, because I have no children and I was able to spend the pandemic working remotely in a beautiful rural area, and I had no interest in taking the risk associated with things like indoor dining. So I would prefer the 100 days with restrictions. But I have many friends with children who would have strongly preferred the 99 days without restrictions, because of the chaos and disruption from school closings.

          Polling seems to suggest that among the people most heavily impacted by interventions, there is net positive support for the stronger interventions. For example, many people who were really struggling early-on with whether to stay home to take care of children and potentially get fired, or send their children to school when they thought it might well lead to their children getting infected and bringing the virus home to seniors in their pods. The ability to have legal cover to stay home from work w/o getting fired, and in fact to collect unemployment would be a positive effect on well-being from an intervention (in comparison to the meaningful scenario – the real world alternative – as opposed to some abstract or theoretical comparison.

          I’d guess the interplay is multi-factorial, but I think you’re making some assumptions there about public opinion that might not stand up to scrutiny.

        • You’re claiming that you can measure the impact of “lockdowns” on “well-being” when in fact you have no way of doing that, not the least because you can’t measure how much “lockdowns” changed people’s behaviors compared to how much they would have changed absent the “lockdowns.”

          I make no such claim, nor have I committed the basic mistake you keep attributing to me, as I have already explained. Since I don’t know how I could possibly explain that any more clearly than I already have, I don’t see the point of discussing this with you. If you think I have made the mistake you attribute to me, then you either haven’t read my post or you haven’t understood it, but what is certain is that I did not make this mistake and that I was in fact very careful to avoid it.

    • Dean:

      One of my favorite examples along these lines is the variation in smoking rate across countries and over time. Smoking is popular in some places and times but not others. It can be really hard for people to quit smoking, but smoking rates can drop across an entire country. So, paradoxically, this is a behavior that is in many ways easier to change in aggregate than individually.

  3. >From the other direction, I see the value of quantitative analyses, as some policy choices need to be made.

    Why would you expect a quantitative analysis do better than just “gut feeling” given that it seems like quantitative analysis in this case is merely an exercise in p-hacking?

    • Matty:

      Even what is called “gut feeling” involves quantitative analyses. If someone says, “Hey, Sweden didn’t lock down and they didn’t do so much worse than otherwise similar countries,” that’s a quantitative analysis. It’s an analysis that maybe can be improves, but it’s not just a gut feeling. Similarly if someone has a gut feeling that masks should make a big difference, there’s an implicit quantitative analysis involving the spread of particles, etc.

      • > Even what is called “gut feeling” involves quantitative analyses.

        I agree, though at this point it feels like you’re shifting the goalposts because I don’t read “I see the value of quantitative analyses” from your post above as “I see the value of gut feeling”. Fact is that “studies” like the ones thoroughly debunked by the blog post are used as evidence all over the world that policies are “following the science” and thus given epistemological status far beyond gut feeling. Courts in Europe are routinely told that lockdowns are “following the science”; many of the measures would be unconstitutional if the government could not refer to “scientific evidence” like these studies that they work. As someone who reads these kinds of studies, do you understand that this state of affairs is frustrating? We have a system with clear rules limiting government power, and it is abused by the scientists who claim far more confidence for their “findings” than is appropriate.

        • To clarify my point – you (and many others) seem to treat this as an exercise in finding the “best policy”, but many governments in fact do not have a mandate to institute the “best policy”. For example, Germany famously bans utilitarian tradeoffs involving lives. There are therefore two questions to ask:
          (a) What policies have which outcomes?
          (b) Is the evidence overwhelmingly clear?
          Though I am not convinced, I understand that formal quantitative analyses might sometimes give slightly better answers to (a), but they almost never reach the threshold (b); public perception (and by the courts) however is that science is always overwhelmingly clear.

        • Matty:

          You seem to be arguing with someone other than me! You write, “you (and many others) seem to treat this as an exercise in finding the ‘best policy.'” I don’t see why you say this. The phrase “best policy” appears in only one place in this thread, and that’s in your comment.

        • Matty,

          My comment was not intended to refute your perspective in any major way. I cede that the quality of explanations is so variable. It is frustrating b/c, as has been alluded to before, there are career & institutional constraints on scientists and officials to pose the evidence that can produce better measures. That is why the role of independent scientists is so critical.

      • > Even what is called “gut feeling” involves quantitative analyses.

        Likewise, saying that interventions reduce well-being involves a quantitative analysis, although I’ve been told above that it doesn’t.

        As in when Philippe says:

        …(again things like curfews, closure of “non-essential businesses” and stay-at-home orders) are, they are not plausibly large enough for those policies to pass a cost-benefit test when you take into account their immediate effects on people’s well-being…

        and then says:

        …but note that I also don’t impose any particular assumption on their immediate effect on people’s well-being.

Leave a Reply

Your email address will not be published. Required fields are marked *