Evidence distortion in clinical trials

After seeing our recent post, “Seeding trials”: medical marketing disguised as science, Till Bruckner sent me this message:

I’ve been working on clinical trial transparency issues for over two years now, first for AllTrials and now for TranspariMED, and can assure you that this is only the tip of the iceberg.

This report by Transparency International, Cochrane and TranspariMED lists multiple instances in which selective and misleading reporting of clinical trials have been used by pharma companies to inflate apparent effectiveness and conceal harms (I was the lead author):
https://docs.wixstatic.com/ugd/01f35d_def0082121a648529220e1d56df4b50a.pdf

This study of mine explores case studies in which such evidence distortion has lead to patient deaths and waste of public health funds on a large scale:
https://media.wix.com/ugd/01f35d_0f2955eb88e34c02b82d886c528efeb4.pdf

Sadly, none of this is new to people working in the field of clinical trial transparency. Even more sadly, policy makers have yet to wake up to the potential public health and fiscal benefits of enacting and enforcing effective transparency rules for clinical trials.

For example, the FDA has so far failed to impose a single fine on companies that break a 2007 law requiring (some) clinical trial to publicly post their results within a year. Wider benefits aside, collecting these fines could net the US taxpayer $400 million (and counting). See:
https://fdaaa.trialstracker.net/

More on this topic can be found at:
https://www.transparimed.org/resources

I hope to read more on clinical trials by you in future. I share related news on Twitter using #AllTrials.

I asked if I could post this and he said Sure and added the above chart based on the findings of Turner et al. (2008), along with some more material:

Major new development last week:
https://www.transparimed.org/single-post/2018/09/13/New-study-shows-that-89-of-clinical-trials-run-by-European-universities-violate-transparency-rules

See also this press release:

Sanction those who fail to comply with clinical trial transparency

Quote from me [Bruckner]:

“Institutions conducting research in human volunteers should not be allowed to violate the rules with impunity. Policy makers have yet to wake up to the potential public health and fiscal benefits of enacting and enforcing effective transparency rules for clinical trials. The FDA has so far failed to impose a single fine on companies and universities that have violated the FDA Amendment Act. Collecting these fines could net the US taxpayer over $670 million in direct revenue. The indirect benefits would be far higher, because forcing the results of all trials out into the open would give us a far better picture of the real effectiveness of expensive new drugs. In many cases, new drugs are probably no better or even worse than far cheaper generic alternatives, but due to widespread evidence distortion, especially in scientific journals, doctors stilll prescribe them, and insurers still pay for them. Both patients and taxayers pay a steep price. It’s sheer madness.”

– said Dr Till Bruckner, founder of TranspariMED, an initiative to end evidence distortion in medicine

WARNING
The $670 million figure quoted above is based on FDAAA Trials Tracker data, which is updated every day. You may have to adjust the number upwards. Check quickly here:
https://fdaaa.trialstracker.net/

Some intriguing facts and figures:
https://www.transparimed.org/single-post/2018/05/08/Most-US-universities-still-lack-clinical-trial-registration-and-reporting-policies-new-study-finds

Unreported clinical trial of the week: Triamcinolone acetonide injection vs oral prednisolone for treatment of rheumatoid arthritis flare ups (2007-006729-28)


https://www.transparimed.org/single-post/2018/08/23/Continued-violations-of-key-integrity-rule-fuel-evidence-distortion-in-medicine-%E2%80%93-new-study
https://fdaaa.trialstracker.net/sponsor/national-cancer-institute-nci/

Human interest story:
https://www.theatlantic.com/health/archive/2016/01/clinical-trial-unpublished-results/423540/

Context:
https://www.statnews.com/2018/01/09/clinical-trials-reporting-nih/

And, the next day, one more item:

FDA released this just two hours ago. Hard to keep up in this field.
https://www.transparimed.org/single-post/2018/09/20/Breaking-FDA-to-sometimes-impose-fines-on-some-institutions-that-fail-to-post-clinical-trial-results

I have not had a chance to look at any of this, but I thought it could be of interest to you.

And, a few days later, after I told Bruckner that this post would be appearing in Feb, he added:

Quick word of warning: If you plan to include a $$$ figure on uncollected FDA fines, this number will have substantially increased by February, and will need updating just before you publish.
You can find the latest figure here:
https://fdaaa.trialstracker.net/

23 thoughts on “Evidence distortion in clinical trials

  1. So with this one simple trick, literally any results become positive results.

    Sounds like what newspapers do routinely. If there are 10 stories that support your narrative and 10 that don’t, report all the supportive stories and only 1 of the non-supportive stories. Then voila, the narrative is 10-to-1 in your favor. Next, write 10 articles on each favorable story and only 1 on the unfavorable story, and your ratio is up to 100-to-1 without telling a single lie. Add dishonesty to the mix and the sky is the limit for your ratio.

  2. Andrew began this post with
    “After seeing our recent post, “Seeding trials”: medical marketing disguised as science, Till Bruckner sent me this message:”

    Unlike real clinical trials, some of which fail to post their results, seeding trials are eager to publicize. To understand the reason for the discrepancy consider Joe Orton’s famous play of the 1960s, “Loot,” in which a home owner demands a warrant from the intruding police inspector; the inspector counters that since he is from the water board, no warrant is necessary. Similarly, even in the unlikely event that IRBs could be strengthened, a seeding trial organizer might paradoxically claim that any IRB has no jurisdiction because an IRB is set up to deal with real clinical trials, not bogus ones such as a seeding trial!

  3. Without expressing an opinion about whether the fines are a good or bad idea, the claim that fines represent foregone “direct revenue” to the government is, for me, a big strike against this guy’s credibility. The cost of enforcement can easily put the government in the red. In the end, fines and compliance costs always fall on consumers; in this case, that may well be the government itself.

    • I agree that the idea of financing the government through fines should be a non-starter. As should the cost of enforcement. And the idea that fines and compliance costs always fall on consumers assumes a degree of certainty unknown to social science. The social world is a world of uncertainty.

    • Jordan:

      I followed the link and went around on twitter for awhile and came across this which you kinda linked to: Fast food science is a shit sandwich, by Bradley Love. I assumed this was another swipe at Brian “Pizzagate” Wansink, which I think has been a bit overdone, but whatever.

      But then I read the post. Love wasn’t talking about “fast food science” (in the sense of scientific research about fast food) at all! He was criticizing bloggers. Apparently he’s angry that his published research is criticized in public.

      I disagree with some of what Love writes in this post. To start with, he writes that, after his research was criticized, he and his colleagues “were obliged to respond to defend ourselves.” Why do you need to “defend yourselves”? Indeed, why consider that you need to be “defended”? It’s about the research, not the people. Responding to criticism should not have to be defensive.

      I do agree with this remark from Love: “there was no actual scientific discourse on Twitter.” I hate twitter. But I don’t think he should fault a blogger for the fact that later twitter discussion is no good.

      He also refers to open science discussion as “rapey.” Huh? What’s with the escalation.

      One thing that bothers me here is that Love is no stranger to the news media; his webpage lists dozens of media mentions, ranging from Nature News and Views and Psychology Today to press releases and even the Daily Mail. That’s fine; I do lots of press too. The point is, if we’re talking about shit-sandwich fast-food media channels, you’d have to put the Daily Mail high on the list. Actually, I think NPR is often pretty bad. And don’t get me started on press releases. Again, fine: if the Daily Mail wanted to do a study of my research, I wouldn’t try to stop them either (although maybe they won’t, now that I’ve written the above!).

      The point is that Love seems to have no problem with fast-food media when it conveys a positive, possibly uncritical, message about his research. But when the message is skeptical, all of a sudden his scruples are aroused. Criticism is, in his word “rapey.”

      I don’t agree with Love’s attitude. When news media are only allowed to report what Love wants them to report, we’re missing a lot.

      I don’t know anything about Love’s work. The paper he wrote may be wonderful; the criticism that he disagrees with may be completely misinformed. What I do know is that there’s been lots of bad research published, work done at many prominent institutions, work that’s received millions of dollars of government funding, that’s been published in top journals, that’s received uncritical coverage all over the news media, and with problems that were only revealed after painstaking external review, done on blogs or elsewhere by outsiders who had no connection to the original work, and who performed the criticism based solely on published information, without the involvement of the original researchers. I also know that in many of these cases, the criticisms were serious, and valid, but the response of the original researchers was to deny all problems.

      Thus, whatever can be said about the particular case of this research by Love, I cannot agree with his larger claim that outside criticism of published research on blogs is the equivalent of low-quality fast food. Perhaps a better analogy is if we learned that certain famous three-star restaurants were serving rotten food, and they were all reviewing and promoting each others’ restaurants, and it took some outsider food bloggers to reveal the problems. Again, I’m not saying Love’s research has these problems—I have no idea whatsoever—I’m just saying that these problems arise all the time, and I think it’s great that outsiders are willing to put in the work and stick their necks out and criticize published research. My published research gets criticized by outsiders—sometimes rudely!—and I appreciate that criticism, as I make mistakes all the time.

      • I have a lot of thoughts about his blog post, but not sure it’s worth the time since most people outside of a certain social club find his posts ridiculous already.

        The first thing to notice is how his post was received versus Schimmack’s similar post:
        https://replicationindex.com/2018/03/20/z-curve-vs-p-curve-break-down-of-an-attempt-to-resolve-disagreement-in-private/

        There are some differences, Love is responding to a publicly posted review while Schimmack is responding to reviews he posted himself. Schimmack received so much backlash he stepped down from his own journal shortly thereafter (I assume that wasn’t a coincidence).

        It seems to me that the same people who blasted Schimmack as yet another angry #OAbro are the same people celebrating Love’s response.

        If you haven’t seen it I think Simine’s response (and responses to her response), are spot on:
        https://twitter.com/siminevazire/status/1083533474332430336

        For what it’s worth, in the recent post by Love, where he is upset by the response that his blog post about open peer review received, I don’t think “rapey” is directly referring to open peer review or scientific criticism, but rather to the overall response he felt he got after his first blog post which could maybe be summarized as “open peer review is how science should work, if you don’t like it you aren’t doing science right.”

        Knowing who is in his lab, and what they are tweeting, and what tweets they are liking does provide some more context however.

        For example, they love to compare open science to fascism.
        https://twitter.com/o_guest/status/1058734140747583488

        And it seems that the lab members are friends with the person who runs a parody account who has compared open peer review to rape right after Love’s first blog post:
        https://twitter.com/OmnesResNetwork/status/1099105102940512256

        So long story short I think this is just a guy upset his paper got rejected, and then upset not everyone agreed with his take on open peer review, and he and his lab are using surprisingly strong rhetoric which I don’t think would fly for any other group.

        • Oh, and I forgot to add. Love said this in his first blog post: “This is inhuman.” in reference to Niko emailing him about his review.

          I’m just glad David Allison has another example for his next PNAS blog post.

        • An interesting development is Stuart Buck’s attempts to get to the bottom of the “rapey” description.
          It starts here:
          https://twitter.com/StuartBuck1/status/1100090452001386496
          And continues:
          https://twitter.com/StuartBuck1/status/1100117107046576133

          It seems the Love lab is claiming someone, maybe Niko, said something along the lines of “sit back and enjoy it”, which the lab interpreted as “rapey”.

          However, when Stuart asked for the exact tweet they wouldn’t provide it, and the best he could do was find a tweet from Niko that included “Consider it. It’s more productive and fun.”

          I find this interesting because I’ve been seeing this phenomenon over and over again. When someone is criticized they first paint themselves as the victim, then they exaggerate or lie about their “bully” to generate sympathy. And then if someone comes in and asks for evidence for their claims of bullying they are met with “how dare you ask a victim for evidence”, “if someone says they were bullied they were, end of story”.

          I don’t think asking for details of an event is equivalent to accusing the self-proclaimed victim of lying, I’d just like to see what happened. Similarly, I don’t think asking someone to post their data is the same as saying you think their results are fraudulent. In the case of publications, open data is important to see if the analysis of it was biased or not since we now know how severe p-hacking is and how a researcher’s bias can influence the results. Similarly, how people interpret events is also biased, and while the person being criticized may interpret things a certain way, their version of what happened is only one interpretation of what happened. Should their version be weighted more heavily? I guess most people would say yes, but the problem I have with this is everyone could just claim they are being bullied and no one would ever have to respond to criticism.

        • Jordan:

          Yes, that sounds about right. Sometimes when people cry Fire there really is a fire, and then they can easily supply evidence. Other times, not so much.

          One advantage of social media (even twitter, much as I hate it!) is that it can be tough for liars and exaggerators to keep this up on social media, because people can call them on it in real time.

          I think it’s easier to lie, exaggerate, and mislead in one-way media such as scientific journal articles, traditional newspaper/magazines/radio/TV, and public talks, where you can emit falsehoods at will (if you’re on good terms with the editor/publisher/host of your journal/newspaper/venue) and it can be difficult or impossible for people to correct the record in ways that are so visible, except in rare cases such as Pizzagate where a huge effort on the part of many people was required to correct the record (and it’s still not corrected, as Wansink’s work is referred to in many books etc.). That himmicanes study keeps appearing as fact in the news media! Etc.

        • You need to read those twitter feeds you linked very closely. This tantrum is the kind of thing you’ll be seeing more of in the future.

          ‘Does it (open review) enforce rather than reduce power structures?’
          ‘reviewers cannot judge scientific merit independent of gender’
          ‘I’ve ended up thinking of this as a problem of capitalism’
          ‘Reimagining open science through a feminist lens’
          ‘we must constantly interrogate whose interests openness is serving and whose is it neglecting and stay vigilant of the ways in which openness may amplify or bridge power asymmetries.’
          ‘Open peer review assumes that everyone accepts criticism in a friendly way and that there is no such thing as bias, which is demonstrably untrue.’
          ‘replication police’

          The reason they are defending themselves is because this is a turf war. They are taking territory, simple as that.

        • There is no scientific standard except:

          1) Demanding independent replication of any observations
          2) Deriving meaningful predictions from any explanations for the observations and checking against new data.

          This has been exceedingly rare for a long time and has allowed people who hate science like this to proliferate throughout academia.

          People need to get used to the idea that everything since about 1950 needs to be rechecked before taken seriously (most will not their risk sacred cows though).

      • “Thus, whatever can be said about the particular case of this research by Love, I cannot agree with his larger claim that outside criticism of published research on blogs is the equivalent of low-quality fast food.”

        The word “publish” essentially means make available to the public. So if something is made public, it’s de facto available for public criticism — and people have a right to publish criticism (which happens to be easier and more affordable now that we have the web and blogs) . If you don’t want your work to be subject to public criticism, then don’t make your work public!

      • “Rapey.” Amazing.

        The last refuge of scoundrels is no longer patriotism, it is Social Justice. I’m kind of surprised he didn’t call criticism “lynching.”

  4. Jordan Anaya,

    Why does you Twitter “about” say “Hispanic first generation academic”? Are you asking for pity? You think it is a major accomplishment? Why? Jewish Nobel Prize winners who went to City College were “first generation academics”. Susskind’s father was a plumber. Hispanics are not stupid either. Please, remove that info, it’s embarrassing

    https://en.wikipedia.org/wiki/Leonard_Susskind

    https://en.wikipedia.org/wiki/List_of_Nobel_laureates_affiliated_with_the_City_University_of_New_York

    • It’s there as a kind of irony. I recently saw someone from my high school talk about how they were unprepared for the coursework at Berkeley because the high school ranks poorly in average test scores. It’s true that the average test scores at the high school are low due to the demographics of the school, but I actually felt the AP courses were really good, and I thought the courses at Berkeley were easy in comparison.

      Anyways, my point is people are always looking for excuses. I don’t think this person would have done any better at Berkeley had they gone to a different high school. I think intersectionality is a silly way to qualify someone’s accomplishments since you can’t know how hard someone’s path was from a few variables, and my Twitter bio is a way to weaponize intersectionality against these people.

      • I have gotten the impression parts of academia are less about science, and intelligence, and reasoning, etc., and more about politics, influence, money, and certain world views.

        It is quite sad if that’s indeed the case, and i wonder what the possible solution for this could be.

        I personally think it’s beyond saving at this point.

        • > more about politics, influence, money, and certain world views.
          Don’t believe it could ever be otherwise – the only way to mitigate this is to allow/encourage all views to be aired.

          That mitigation is stifled by for instance selecting some revised comments from Simine Vazire’s link, accuse the reviewer that they missed the point of your paper (wasn’t written clearly as could be), impute uncharitable motives and intentions to the reviewer, suggest it would be more useful to the community if reviewers and authors corresponded to reduce or clarify points of contention (which does more to hide the real contentions and how they might be resolved as well as slows this down until they may not matter much), etc.

        • “Don’t believe it could ever be otherwise – the only way to mitigate this is to allow/encourage all views to be aired.”

          I do believe it could be different. Aside from that, i would like to add that i reason 1) allowing/encouraging all views to be aired could be a very important thing but not the only thing that possibly needs to happen for academia to be(-come) about science. I reason it should, for instance, also 2) be made eplicitly clear (over and over and over again) that specific scientific values and principles need to be upheld at all times and by everyone, and 3) that every allowed/encourage view gets presented at, and in, an equal manner (e.g. not having one view published in a top journal and receiving media attention, and another view only on a pre-print server without media attention).

          I feel these possibly important additional 2 things are not communicated, or perhaps even realized, enough. In fact, many of recently proposed “improvements” all run counter to these 3 things in my reasoning. For instance, what’s the use of setting up a pre-print server (where everyone should be able to publish their views) when you are still publishing your papers in the “best” journals, hereby not really changing anything and still being a part of the posibly messed-up publication system.

          And, sure i hear folks talk about how they “value criticism”, but they in turn do not engage in the discussion and/or don’t do anything with the criticism. This is often not even corrected by others. Talk is cheap, and actions speak louder than words. I have heard and read many times, even from the most supposedly “scientific” and “i’m all about improvement” folks, that saying A and doing B seems to be just fine for certain “special” people. This is part of why i think parts of academia are beyond saving.

          Anyway, just sticking to your point about making sure to allow/encourage all views to be aired. I fear this is exactly what is currently not happening. In my view and reasoning, even many of currently proposed processes, and “improvements”, give power and influence to an increasingly smaller number of people. With some of them, it may look like “all views are being aired” but i reason they are in fact not because there is always a certain “filter” involved with “special responsibilities” (e.g. committees, editors, senior author, etc.)

          Take for instance this recent blog about writing a paper in a group: http://deevybee.blogspot.com/2019/02/the-paper-in-day-approach.html. To me, it may look like you are giving everyone a chance to express what they want to say or investigate but in my reasoning you are not really doing that because these group processes will probably let 1) either the total group decide what gets into the paper and how, and/or 2) let a specific individual (senior author) decide what gets in the paper, and/or 3) take away the individual’s process, and opportunity, to express their view without possible compromises.

        • As a side note, but related to my points made above:

          1) I tried in the above mentioned blogpost about writing a paper in a group (http://deevybee.blogspot.com/2019/02/the-paper-in-day-approach.html) to publish a (critical) comment at least several days ago by now. This has not shown up there however. This could be due to an error, perhaps an i did something wrong, but i just simply note it here. Regardless, blogs can of course be manipulated easily concerning the comments they allow, which is why blogs per se are not necessarly “good” or “better” or a “solution”.

          2) The above mentioned blog used a “guest author”, which i have been annoyed with before. To me, this is similar to editors letting certain “special” people publish in their “top” journal. Again i point to the possible issues i wrote above concerning making sure every view is allowed, but also that every view gets presented in an equal manner.

  5. This might be a case where the use of a blockchain would be good. If the data recorded was indelible, transparency could be assured. Then I guess the issue would be how do you ensure the correct information gets recorded… Just a thought I guess.

Leave a Reply

Your email address will not be published. Required fields are marked *