An open letter expressing concerns regarding the statistical analysis and data integrity of a recently published and publicized paper

James Watson prepared this open letter to **, **, **, and **, authors of ** and to ** (editor of **). The letter has approximately 96,032 signatures from approximately 6 continents. And I heard a rumor that they have contacts at the Antarctic Polar Station who are going to sign the thing once they can get their damn fur gloves off.

I like the letter. This kind of thing should be a generic letter that applies to all research papers!

I’ve obscured the details of the letter here because I don’t want to single out the authors of this particular paper or the editor of this particular journal.

If the paper really does have all the problems that some people are concerned about, then maybe the journal in question will follow the “Wakefield rule” and retract in 2032. You thought journal review was slow? Retraction’s even slower!

A journalist who’s writing a story about this controversy asked me what I thought, and I said I didn’t know. The authors have no obligation to share their data or code, and I have no obligation to believe anything they say. Similarly, the journal has no obligation to try to get the authors to respond in a serious way to criticisms and concerns, and I have no obligation to take seriously the papers they publish. This doesn’t mean all or even most of the papers they publish are bad; it just means that we need to judge them on their merits.

P.S. The above link no longer works, so here’s another copy of the letter from Watson et al.

P.P.S. In this news article, Stephanie Lee notes:

One of the biggest concerns of the letter’s signatories was that the authors had not released their code or data, even though the Lancet has signed a pledge to share COVID-19–related data.

52 thoughts on “An open letter expressing concerns regarding the statistical analysis and data integrity of a recently published and publicized paper

    • In France, a sort of cult has developed surrounding Raoult, who probably did more than anyone else to promote hydroxychloroquine as a treatment for COVID-19. His supporters will attack anyone who criticize their guru, accusing them of being lobbyists for the pharmaceutical industry, etc. I really doubt those people would stop just because a study was published in a prestigious journal though. It’s really amazing that something that, one would think, is a rather technical debate has become a culture war issue. I know that in the US the treatment was endorsed by Trump, so one might be tempted to explain it by this fact, but it’s the same thing in France where this explanation is not available. That being said, I think the Lancet study is really bad and that scientists should not let politics interfere with post-publication peer review, so I’m glad so many prominent researchers signed this open letter.

      • It’s really amazing that something that, one would think, is a rather technical debate has become a culture war issue.

        That bad PACE trial that’s been mentioned on this blog before also became a “culture war issue”. Only that time it was the researchers making false claims of having received death threats!

      • More on Raoult can be found at

        https://www.nytimes.com/2020/05/12/magazine/didier-raoult-hydroxychloroquine.html:

        “According to one survey, by late March, Raoult had become one of France’s most popular “political personalities,” with particular appeal on the populist extremes. Votives bearing his image were being sold in Marseille, and on some evenings, at 8 p.m., a battalion of municipal garbage trucks assembled on the roadway outside his hospital, where the drivers leaned on their horns in loud and furious tribute. A hundred-foot banner, painted by a club of local soccer fans and strung up near the entrance, read, “Marseille and the world behind Prof. Raoult!!!”

        “He is skeptical, for instance, of the utility of mathematical modeling in the realm of epidemiology. The same logic has led him to conclude that climate modelers are no more than “soothsayers” for our “scientistic era” and that their dire predictions are mostly just an attempt to expiate our intense but irrational feelings of guilt.”

        “He believes it to be unnecessary, in addition to being unethical, to run randomized controlled trials, or R.C.T.s, of treatments for deadly infectious diseases. If these have become the accepted standard in biomedical research, Raoult contends, it is only because they appeal to statisticians “who have never seen a patient.” He refers to these scientists disdainfully as “methodologists.”

        “I’ve invented 10 or so treatments in my life,” Raoult told me. “Half of them are prescribed all over the world. I’ve never done a double-blind study in my life, never. Never! Never done anything randomized, either.”

        • Or maybe he just has a very low opinion of typical math models and RCT’s. There are so many worthless ones out there that they can discredit the whole enterprise. It may be rational for doctors to fall back on case studies, given how easily someone who doesn’t understand math can be fooled by a veneer of rigor.

        • What is medicine’s 5 sigma?

          4-11-15

          by: Richard Horton, Ed., “The Lancet”

          “A lot of what is published is incorrect.” I’m not allowed to say who made this remark because we were asked to observe Chatham House rules. We were also asked not to take photographs of slides. Those who worked for government agencies pleaded that their comments especially remain unquoted, since the forthcoming UK election meant they were living in “purdah”—a chilling state where severe restrictions on freedom of speech are placed on anyone on the government’s payroll. Why the paranoid concern for secrecy and non-attribution? Because this symposium—on the reproducibility and reliability of biomedical research, held at the Wellcome Trust in London last week—touched on one of the most sensitive issues in science today: the idea that something has gone fundamentally wrong with one of our greatest human creations. * The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness. As one participant put it, “poor methods get results”. The Academy of Medical Sciences, Medical Research Council, and Biotechnology and Biological Sciences Research Council have now put their reputational weight behind an investigation into these questionable research practices. The apparent endemicity of bad research behaviour is alarming. In their quest for telling a compelling story, scientists too often sculpt data to fit their preferred theory of the world. Or they retrofit hypotheses to fit their data. Journal editors deserve their fair share of criticism too. We aid and abet the worst behaviours. Our acquiescence to the impact factor fuels an unhealthy competition to win a place in a select few journals. Our love of “significance” pollutes the literature with many a statistical fairy-tale. We reject important confirmations. Journals are not the only miscreants. Universities are in a perpetual struggle for money and talent, endpoints that foster reductive metrics, such as high-impact publication. National assessment procedures, such as the Research Excellence Framework, incentivise bad practices. And individual scientists, including their most senior leaders, do little to alter a research culture that occasionally veers close to misconduct.

          http://www.thelancet.com/pdfs/journals/lancet/PIIS0140-6736%2815%2960696-1.pdf

          Drug Companies & Doctors: A Story of Corruption

          Marcia Angell,

          New England Journal of Medicine tenure[edit]
          Angell joined the editorial staff of The New England Journal of Medicine (NEJM) in 1979. She became Executive Editor in 1988, and served as interim Editor-in-Chief from 1999 until June 2000.[3] The NEJM is the oldest continuously published medical journal,[4] and one of the most prestigious; Angell is the first woman to have served as Editor-in-Chief of the journal since it was founded in 1812.[3]
          https://en.wikipedia.org/wiki/Marcia_Angell

          1-15-09

          http://www.nybooks.com/articles/archives/2009/jan/15/drug-companies-doctorsa-story-of-corruption/

          NEJM editor: “No longer possible to believe much of clinical research published”
          http://ethicalnag.org/2009/11/09/nejm-editor/

        • Regarding the fees paid by drug companies to the FDA:

          We know who put the regulatory fees there in the first place: the people who were outraged that companies sell drugs to make profit. Now they’re outraged – yep – that companies sell drugs to make profit. But the solution is suddenly just the opposite.

        • Covid-19: The game is over?!

          https://jdmichel.blog.tdg.ch/archive/2020/03/24/covid-19-the-game-is-over-305275.html

          The world’s leading expert on communicable diseases is Didier Raoult. He’s French, looks like a Gaul out of Asterix or a ZZ Top who left his guitar by the side of the road. He runs the Institut Hospitalier Universitaire (IHU) Méditerranée-Infection in Marseilles, which employs over 800 men and women. This institution possesses the most terrifying collection of “killer” bacteria and viruses in existence, and is one of the world’s leading centers of expertise in infectiology and microbiology. Professor Raoult is also ranked among the top ten French researchers by the journal Nature, both in terms of the number of his publications (over two thousand) and the number of citations by other researchers. Since the turn of the millennium, he has studied the various spectacular viral epidemics and has established close scientific contacts with the best of his Chinese colleagues. His achievements include the discovery of treatments (using chloroquine, in particular…) which are cited today in all handbooks on infectious diseases throughout the world.
          On February 26th, he published a resounding video on an online channel (including the word “tube”) saying: “Coronavirus, game over! »

          The reason for his enthusiasm? The publication of a Chinese clinical trial on the prescription of chloroquine, showing suppression of viral carriage in a few days on patients infected by SARS-CoV-2. Studies had already shown the efficacy of this molecule against the virus in the laboratory (in vitro). The Chinese study confirmed this efficacy on a group of affected patients (in vivo). Following this study, the prescription of chloroquine was incorporated into the treatment recommendations for the coronavirus in China and Korea, the two countries that have been most successful in controlling the epidemic…

        • I would not trust an article that says : “His claim of a 100 percent cure rate shocked scientists around the world.” ; which is obviously wrong.
          See one of his article for details : https://www.sciencedirect.com/science/article/pii/S1477893920302179

          Regarding randomized controlled trials, he believes they take too much time to complete, and that, in the context of the covid-19 crisis, it is necessary to go faster. He also thinks it is unethical not to give a medicine to someone while believing this medicine can save his life.

          He is popular because, while most governments advise not to treat patients, he advises to treat the patients at the onset of the disease with of drugs (hydroxychloroquine, azithromycin) that the thinks have good chances to cure them (or to prevent their disease from becoming serious), and little side effects.
          Here is an article endorsing his approach :
          https://academic.oup.com/aje/advance-article/doi/10.1093/aje/kwaa093/5847586

        • There is some circumstantial evidence that says that early treatments are better with antivirals. See lopinavir/ritonavir, apparently not useful in late stage COVID-19 (NEJM), and *possibly* (study without a placebo) more useful within 7 days of symptom onset (Lancet, earlier this month).

        • Yes, it is now widely accepted among practitioners that anti-virals (be it hydroxychloroquine, or other ones) are only helpful at the onset of the disease, in order to prevent complications.

          And when patients have respiratory difficulties, and need to be hospitalized, then it’s too late for anti-virals to help.

          At that stage, here is what may work, depending on the symptoms :
          – immunomodulators (hydroxychloroquine, tocilizumab…), against cytokin storm
          – antibiotics (azithromycin, ceftriaxone…), against bacterial infection
          – anti-coagulants (heparin…), against thrombosis

          At the moment, many studies are done in hospitals (so, at a late stage of the disease), and look at the effects of anti-virals (that’s indeed the case of the Surgisphere one) ; and that’s a problem in my opinion. A that stage hydroxychloroquine may help because of its immunomodulatory effect, and azithromycin as an antibiotic ; but if there is a thrombosis, they won’t win the battle.

        • I’m not interested in debating the evidence on hcq, but to be clear, vl, hydroxychloroquine is most certainly NOT an antiviral drug.

        • The “100% cure rate” is not a claim by the NYT, but a quote from attorney and Fox News host Gregory Rigano, who is the one claiming that Didier Raoult has a 100% cure rate, because he somehow is in correspondence with Raoult and Raoult supposedly gave him a copy of his study with a 100% cure rate mentioned. No such study exists or was ever produced, of course. However, Raoult is claiming lately a 99,5% cure rate for his patients who were on average only 48 years old because he got all the rather young hypochondriacs who came for free testing. One of the worst hit countries, Italy, has almost exactly the same cure rate for this age range, 99,38%.

      • Emotions are running really high because of COVID. It’s definitely not just an US thing. The 5G conspiracy theory has been big in Europe as well…

  1. Nicely written post, Andrew. The letter speaks for itself.

    One hting though– I hate letters that are mass-signed. They are much less persuasive than if 10 good scholars sign them. We all know how stuff gets passed around for signatures and people just sign on to signal which side they;re on or because a friend asks them. There’s not enough accountability if you’re one of 1,000 signatures. Also, numbers are meaningless, especially nowadays. Anybody can get 1,000 goofball professors at minor universities to sign onto anything.

    It’s nice to let lots of people sign just so they can show solidarity. But hte “real” signantories shoud be listed separately. I only care about (a) people I’ve heard of, (b) people I konw personally, and (c) people from departments I know are good. The rest is just fluff.

    • I agree with your concerns about mass-signed letters. However, I don’t want to care only about (a)-(c). That is a blueprint for the elitism that is helping to undermine science. The NEJM worships Harvard and publishes plenty of bad stuff from “good departments.” Alright, I go along with (b), people I know personally (but that is a small set). But (a) and (c) just convey power to those with the “right” credentials and lengthy publication records, and we’ve seen where that gets us.

  2. Just for the record:

    Re: “If the paper really does have all the problems that some people are concerned about, then maybe the journal in question will follow the “Wakefield rule” and retract in 2032. You thought journal review was slow? Retraction’s even slower!”

    I don’t know what the so-called “Wakefield rule” has to do with the topic of this blog other than some vague connection with the retraction of flawed research. Wakefield’s research may have been maligned and retracted, but it wasn’t because it was flawed or faked.

    Andrew Wakefield is commonly understood to have somehow or other produced faulty results in his research with his colleagues regarding the relationship of an MMR vaccine with severe intestinal issues experienced by a small group of children who had been vaccinated with the MMR vaccine. In fact, his colleagues’ medical licences, which had been revoked, were reinstated upon appeal by the Queen’s Bench judge following a review of the issue of whether or not Wakefield et al were conducting research or practicing medicine and treating the children involved. If they were conducting research they needed an ethics review of their project by their institution, which review didn’t take place. If they were practicing medicine no such review was required.

    The judge ruled the doctors were practicing medicine and didn’t need an ethics review. The study was never found to have been faulty or fraudulent. Wakefield couldn’t afford legal representation, so he was not a party to the appeal.

    England and Wales High Court (Administrative Court) Decisions
    Walker-Smith v General Medical Council [2012] EWHC 503 (Admin) (07 March 2012)
    URL: http://www.bailii.org/ew/cases/EWHC/Admin/2012/503.html
    Cite as: [2012] EWHC 503 (Admin)

  3. > The letter has approximately 96,032 signatures from approximately 6 continents.

    Is this a joke I am not getting? I think that’s the number of patients in the study.

  4. This is interesting. James Watson is the lead statistician on an ongoing trial of over 40,000 patients and hydroxychloroquine. He’s also taking the lead with all of these criticisms, but has not really shown anything truly substantial. Ths open letter is a disgrace to him and the people who signed it. This isn’t how peer reviewed science happens and here we have a junior “stats guy” trying to get his five minutes of fame. Shame on him.

    I am more trouble by the obvious conflicts of interest that this James Watson has. He stands to lose quite a bit from this Mehra publication. So far what I have seen are vocal tweets from fringe groups who think global warming doesn’t exist, people with major conflicts of interest (@Watson), and groups that are financially invested in hydroxychloroquine.

    I am disgusted by all of this.

    • I come to this blog because I find it to have contributors who are serious, smart, and a rare place to seriously exchange ideas and generally avoid ranting and trolling. So, I find these recent posts most disturbing. The Lancet piece interests me because it is potentially important, is influential, and raises many interesting questions about data and analysis. Then, it is pointed out that James Watson (who I know nothing about) may have his own conflicts of interest. But it is pointed out by “Angelo” – who may or may not have conflicts of interest. And, I also see posts from “pogohere” with links to provocative stories about Raoult – which are a couple of months old, and I can’t honestly tell if these posts are denigrating the opposition to HCQ as a COVID treatment or pointing out how wrong the earlier claims have proven – in other words, I can’t quite tell which “side” this anonymous commentator is taking.

      “I am disgusted by all of this.” Me too, but you are part of the problem at this point. I don’t like anonymous comments though I understand the potential need to be anonymous. But once we start attacking perceived conflicts of interest, I think anonymity is no longer productive. Since I don’t know any of these people (anonymous or otherwise) personally, I have no way to gauge what to make of the comments. I try to focus on the issues raised, the facts presented, and the quality of the arguments. But it is getting more difficult daily. This is a very bad sign for those of us that believe is science, data, evidence, and ultimately truth. And it does a disservice to Andrew’s blog, which has been a rare bright spot in the devolution of public discourse.

      • Dale:

        I agree. Over the years, we’ve had commenters with strong and sometimes controversial political views, but in their comments here they’ve tended to engage in focused discussion.

      • The conflict of interest game is what people resort to if they can’t attack the data, method or argument.

        If you’re investing your life savings, would you rather trust your money to someone with skin in the game, or an external observer with nothing at risk? It’s not an easy question to answer. Both positions have advantages and disadvantages. The “skin” person has something to lose if they are wrong; on the other hand “observer” would presumably be less likely to misrepresent issues for his/her own gain. But the “skin” person might be motivated to do more detailed and careful research; on the other hand the “observer” might be more skeptical and less likely to get sucked into confirmation bias.

        The route around all this is to analyze the investment on its merits independent of the advisor’s claims about it.

        And, when all is said and done, review the data and see who obtained the best results.

        • By “conflict of interest” who are you talking about? Angelo’s post above? I mean, people have been pointing out that one of the Lancet authors is the CEO of the data company that provided the data. So if James Watson is invested in hydroxychloroquine research, it may be relevant indeed.

        • Mr.:

          I think it’s ok to point out conflicts of interest. I also agree with Jim that this is just one piece of information. If you look at the Sponsors list on this blog, you’ll see that I have a zillion conflicts with everything. Watson has acknowledged his conflict; see his Disclaimer here.

        • Andrew:

          There’s nothing wrong with pointing out conflicts of interest. It’s just a waste of time. You can’t solve anything by finding out conflicts of interest because they don’t have any predictable relationship with the results. Moreover, you can’t possibly know all the potential conflicts of interests or their relative weights in a given situation (the Rumsfeldian unknown unknowns). So if you want something to talk about conflict of interest discussions are an endless maze of things to talk about. But it’s just talk. It won’t lead to a solution because it can’t lead to a solution.

          In the end the way we find out of some proposed medication or mechanism or whatever works is by multiple independent tests. That’s the only way science has ever been successful and it’s the only way it ever will be successful.

        • Jim –

          I mostly agree – except to the extent where people hide or simply aren’t proactive about stating conflict of interest. Failure to announce potential conflict of interest is a methodological failing that is likely to (although admittedly, not necessarily) indicate a more general methodological sloppiness.

        • Jim:

          Conflicts of interest are not particularly relevant for evaluating a scientific claim. But conflicts of interest and, more generally, social/economic/political structures, can be relevant for understanding what went wrong, if we do see problems with the claim. For example, if PNAS publishes a series of bad papers on himmicanes, air rage, etc., we can evaluate the papers one at a time and see their problems. But then when we want to understand how did they happen to publish these papers and never apologize, then it can make sense to look into the journal’s review process.

        • Joshua:

          I agree in principle that a direct financial interest in a scientific result should be disclosed. But in the end doesn’t a direct financial interest have as many incentives for honest, forthright and high quality work as it does for poor or fraudulent work?

        • jim –

          > But in the end doesn’t a direct financial interest have as many incentives for honest, forthright and high quality work as it does for poor or fraudulent work?

          Yah. I agree with that, also. They cynical suspicion of the integrity of scientists, and the assumption that they are largely driven by self-interest rather than by an intrinsic drive to solve problems, is faddish but I find rather reactionary.

          At the same time, just to reiterate, I think all serious scientists should know about the basic requirement of proactively divulging conflicts of interest. So in that sense I do think that it can serve as a very rough way to assess the probabilities with respect to which scientists are serious.

        • Andrew:

          Well, with regard to himmicanes and fishy smells, acceptance and even embracing of these results was so widespread it’s hard to know who to blame, but we already know the broader incentive structure there.

          In the end himmicanes might be a good thing: it’s clear now that purportedly “dispassionate observers” with absolutely no direct financial incentives have ample and powerful indirect incentives to produce fraudulent work.

        • “it may be relevant indeed.”

          Or not. But in the end if we simply check the results and find out if they’re good or bad, then we don’t have to concern ourselves with myriad possible opposing motivations.

    • The data in the Lancet piece is claimed to be from a previously unknown database of 670 hospitals on six continents who have mysteriously installed QuartzClinical – software that was launched last year and has no reviews or references other than by its owners – and allowed their data to be pooled for research. QuartzClinical is provided by a firm with 3 employees. This is not just improbable, it doesn’t even pass a laugh test.

      When it was pointed out to the authors that the claimed cases in Australia was more than actually existed at the time, they said they’d missclassified a hospital as Australian that was really in Asia. No Australian federal or state health department knows how data from hospitals could get to the authors.

      Taking those two paragraphs above together, it is difficult to think of a plausible explanation other than that the data is completely fabricated.

  5. @Andrew you are to be commended for maintaining your equipoise and balance throughout this. Have you contacted the authors for comment? Or for that matter, I heard that there was some sort of edit being made. Does anyone know what this is all about?

  6. James Watson should state that he has a conflict of interest since he works for MORU Tropical Health Network in Thailand. MORU is conducting a large-scale clinical trial for chloroquine and hydroxychloroquine called COPCOV, which is funded by a COVID-19 Bill & Melinda Gates Foundation, Wellcome and Mastercard Therapeutics Accelerator grant. Seven signatories work for MORU. The rest seem to be directly connected to the COPCOV trial, conducting their own chloroquine or hydroxychloroquine, or receive funding from Wellcome or Gates, or have past connections to Watson (ex. Caroline Buckee). https://www.tropmedres.ac/covid-19/copcov

  7. I love your point about making this a “generic letter” for all studies. Every paper should provide de-identified data to the extent possible, regardless of whether there are potential problems with the findings. Unfortunately, I doubt the signatories to this letter would be willing to adhere to that standard generally. Call me cynical, but I suspect a large proportion of the signatories have published studies without also publishing the data, without remorse or regret. The standard they are applying here is probably more like: Data publication should be mandatory only when a) potential errors are found in the publication, b) the results are controversial, and c) the paper is influential and could have grave consequences.

    Were I an author on the paper, I’d start circulating a letter in which signatories pledge to release data AND code (there’s no HIPAA for code) for all their papers, past and future, and see how many people sign both letters.

    • Thanks. I didn’t expect the number of deaths for each continent to change in S-3 (apart from Asia/Australia), at least the total is still the same.

Leave a Reply

Your email address will not be published. Required fields are marked *