Further comments on “Assessing mandatory stay‐at‐home and business closure effects on the spread of COVID‐19”

A few weeks ago we discussed a recent paper from Bendavid et al., “Assessing mandatory stay‐at‐home and business closure effects on the spread of COVID‐19.”

Lonni Besançon writes:

Together with Gideon Meyerowitz-Katz we found that the study was fairly limited and did not properly highlight its limitations nor did it discuss several conflicting results from past work. We posted our concerns on PubPeer and we thought that you might be interested in these and that your readers might want to chime in.

Unrelatedly, he writes:

You might remember that I [Besançon] contacted you about our paper “Open Science Saves Lives” and a call for co-signature. I just wanted to let you know how it went. We have collected 371 verified signatures on our call for more transparency and on our analysis of the lack of transparency in COVID19 papers which is not too bad I guess. Based on all the comments we obtained we created a second version of the paper as a preprint and this is now under review.

16 thoughts on “Further comments on “Assessing mandatory stay‐at‐home and business closure effects on the spread of COVID‐19”

  1. One problem is the outcome measure. A decline in overall virus cases can also conceal increasing numbers of outbreaks and community spread involving variants.

  2. Mixed feelings about their review…

    Yeah some of their points are *obviously* true, but other points just don’t make any sense. Not the best, and definitely not the most fair review I’ve seen about this paper.

  3. Off topic:

    This study reported 5.5 days to primary endpoint in the vitamin c group vs 6.7 in standard of care. That means recovering one day earlier on average, something the original study was powered at 80% to detect. The actually performed study was stopped early with less than half the expected sample size:

    Due to slower than expected enrollment, an interim analysis was conducted at approximately 40% of expected enrollment (214 of 520 patients).

    […]

    We assumed that the usual care group would achieve a 50% reduction in symptom severity in a mean (SD) of 6 (3) days and that at least 1 of the other 3 study groups would achieve a 50% reduction in a mean (SD) of 5 (3) days. Assuming a sample size in each of the 4 groups of 130 patients, a 1-way analysis of variance would have 80% power (2-sided α of .05) to detect a difference in means of 1 day with a common SD of 3 days.

    https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2776305

    When will researchers be held accountable for making stats 101 errors like concluding no benefit from this type of study?

    Despite the popular use of vitamin C and zinc to fight off or lessen the severity of viral colds and flu, the new study, published Friday in JAMA Network Open, found the two supplements were of no benefit to people isolating at home with Covid-19. In fact, the findings were so unimpressive that the study was stopped early. “Unfortunately, these 2 supplements failed to live up to their hype,” wrote Dr. Erin Michos of John Hopkins and Houston Methodist’s Dr. Miguel Cainzos-Achirica, in an accompanying editorial.

    https://www.cnn.com/2021/02/12/health/vitamin-c-zinc-covid-19-wellness/index.html

    It is clear these people don’t have slightest clue what their results mean or the purpose of the steps they performed. Why are they being paid for this and held up as experts?

    • From that commentary:

      In recent years, misinformation has permeated health discussions, including those involving COVID-19 therapies.

      https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2776299

      There are the two authors of this commentary, eleven authors on the paper, then probably three reviewers and an editor. That is 0/17 people who know what a p-value, statistical significance, or a power analysis means. Yet they use them for everything, interpret the results incorrectly, and then lecture others about spreading misinformation.

      In what kind of world do we live where these people are not all resigning in shame for such an extreme display of incompetence?

  4. Peer-reviewed critique of the study:

    https://onlinelibrary.wiley.com/doi/abs/10.1111/eci.13518

    From a summary of criticisms:

    > 4/ The analysis does not correct for the relationship between interventions and case counts.

    IMO, so totally obvious, it’s really shocking that it got past the authors.

    Clearly, the precipitating conditions would predict the severity of the interventions (on average) and also the kinds of outcomes independent, to some degree, of the efficacy of the interventions. To not control for that is mind-blowing – especially given Ioannidis’ stature as an epidemiologist.

      • Hi Joshua and thanks for pointing out to our letter and the twitter thread about it.

        There are so many things that are absolutely mind-blowing about the limitations of the study and the fact that many of them are not even mentioned (+ the very strong conclusions of the article).

        What is actually really problematic is that Ioannidis’ name is used to erase/destroy all possible criticism of the study. Many people online have pointed out that our letter has 0 value because we are not as good as he is… Can’t care much about this TBF but this goes to show that this article of theirs will be used no matter how valid our critics are (last time I checked their article had been shared 27k times on twitter only).

        I do not know if Bendavid et al. are ever going to reply to our letter. I certainly hope so, but I somehow do not see that happening.

  5. Around April, 2020 New Zealand experienced a total lockdown for about four weeks. In January, 2021 in New Zealand, we were sitting on our deck with friends in a COVID free country. We were close together and we were not wearing masks. An educated and talented friend returned from his choir practice. He announced loudly that “lockdowns don’t work.” Surprised, I looked up and replied, “You have just come from choir practice. You no doubt had a party afterward. You did not have to wear a mask and there was no need to social distance. You weren’t afraid of catching COVID. How can you say that lockdowns don’t work?”
    Afterwards I traced his misinformation to this article: “Assessing Mandatory Stay-at-home and Business Closure Effects on the Spread of COVID-19” published in the European Journal of Clinical Investigation (EJCI). The article had obviously gotten around, been accepted at face value by some, been diluted by various web sites, and found its way to a number of irresponsible news sources.
    The article, based on a deeply flawed model, concluded that “…we do not find significant benefits on case growth of more restrictive non-pharmaceutical interventions.” Of course they couldn’t find the benefits of a properly managed lockdown. Here’s why:

    In the words of one of the authors: “I cannot say that our results are THE truth, because it is very difficult to model several measures acting at the same time…” He is absolutely right and yet they depended on a model based on nearly an infinite number of measures many of which were based on data that was not accurate.
    They used politicized data from countries like Iran.
    The time period the authors picked, at the very beginning of the pandemic, failed to consider the mismanagement of those early lockdowns, wide spread enforcement failure, and the long term outcomes.
    They mentioned the import of how “risk communications motivate personal behaviors” but then seemed to ignore their own observation. In cultures such as South Korea’s, people did not need to be told to wear masks and keep their distance. They practiced a de facto lockdown. Even with a high level of compliance the Moon administration eventually tightened restrictions, banned church gatherings and large outdoor protests, and shut down nightclubs and bars.
    The study focused on subnational regions. This is absurd when people can move freely between regions taking COVID with them. The outcomes in the USA are the best example of the disastrous effects of subnational controls. If you don’t believe in the importance of border closure, read on about events in New Zealand.
    Sweden was used as an example because of successful early results despite the fact its COVID response proved to be no more successful than other European countries in the long run.
    Some countries included in the model kept changing the rules. Lockdown rules have been changed 64 times in the U.K.! No wonder their enforcement was so poor and the study model so inaccurate.
    The data were cherry picked in the worst possible way. The authors sought publication even though, in their words, “underlying data and methods have important limitations.” Specifically, the authors failed to include countries like New Zealand, Australia, and China that have been able to achieve a near COVID free status despite continuous assaults by the virus. In China’s case that was achieved through their authoritarian government’s rigid enforcement of protocols. But, in New Zealand, a country that puts a great emphasis on personal freedom, the lockdown was managed differently.

    New Zealand’s success illustrates the hopeless inaccuracy of the authors’ model and proves their conclusion that lockdowns don’t work is wrong.
    So, let’s dump their conjured up model and look at a real world experiment. Here’s why New Zealand succeeded:
    There was strong, insightful, central leadership. Prime Minister Jacinda Ardern addressed the country regularly. Her government acted quickly and decisively. She continuously emphasized that we had to all work together to stamp out COVID. A democracy in crisis cannot succeed unless everyone is willing to be part of the team. In the freest country in the world we had to sacrifice our individual desires for the good of us all. We accepted that. Compare this with the approach in the United States where poor central leadership led to chaos and disaster.
    New Zealand’s lockdown was country-wide, absolute and complete. Travel was severely restricted with roadblocks. The only exceptions were businesses like grocery stores and pharmacies where social distancing and masks were required.
    The borders were closed. The import of this cannot be over emphasized. Citizens coming home to flee the pandemic were isolated in 5-star hotels which could no longer cater to tourists. Numbers allowed in were limited by the rooms available. A surprising number of the people isolated at the tourist hotels were COVID positive. There were problems initially with escapes and poor management by private security businesses. Leadership was then turned over to Air Commodore Darryn Webb who handled the effort with military precision and much greater firmness. There have been on-going problems, mostly with the complexities of transforming luxury hotels into effective quarantine facilities but, overall, the approach has worked despite the fact that the spread of the virus from just one case getting through isolation was frighteningly fast. Again, real world results have shown how inaccurate the models were that predicted border closure was not important.
    Leakage of COVID into the community was dealt with quickly by testing, contact tracing and ring fencing the environment. When COVID spread into a Pasifica community a second regional lockdown was necessary. Again, strong central leadership, the efforts of local elders, contact tracing and personal cooperation were emphasized and the transmission chain was broken quickly.
    Contact tracing has been facilitated with QR codes at every business and, lately, blue tooth communication between cell phones. When a case was discovered in the community the infected person’s phone revealed where they had been so others could check their phones and see if they had overlapped. Privacy was assured but the emphasis was still on community protection. Compliance tended to lag when there are no community cases for months at a time but quickly rebounded when the virus resurfaced. More education is needed about the importance of using the QR codes at all times because we are under an unrelenting assault on our borders.

    People point to New Zealand’s success and say it’s because the country is isolated and small. But this is not true at all; just look at China, a very large and centralized country that has successfully used lockdowns to control the virus. In March, 2020 the President of the USA was told by the prominent hedge-fund manager Bill Ackman that most businesses could survive a 30 day lock-down and the borders needed to be closed. Ackman summarized: “No one defaults, no one forecloses. A 30-day rent, interest and tax holiday for all.” He added: “The shutdown is inevitable as it is already happening, but not in a controlled fashion which is extending the economic pain and amplifying the spread of the virus.” New Zealand followed Ackman’s advice. The United States did not. New Zealand’s economy is in good shape today but the American economy is not. The differences in the two countries’ outcomes are obvious. In the real world, contrary to the conclusion of the EJCI article, it is the complete lockdown that succeeds and the partial one that fails.
    Because of the success of public health procedures, antibiotics and vaccines the human race has lost its generational memory of pandemics. We have forgotten the dangers of infectious disease and are not prepared for the “big one” when it will surely come along. This pandemic has been a test that mankind has failed miserably. Because of the infiltration of the virus into their populations, it is too late for many countries to utilize the strategies that have been so successful in New Zealand. When a far more serious virus emerges we need to be prepared so we do not repeat these mistakes. We have to study why countries like New Zealand, Australia, Taiwan and China have succeeded while so many other countries have failed.
    Misinformation like this article in the EJCI has been allowed to proliferate through the media and has been politicized. The article should never have passed peer review. It was obvious that it would be misused, and so it has. The authors and the EJCI should repudiate what they created for the good of us all.

Leave a Reply to Joshua Cancel reply

Your email address will not be published. Required fields are marked *