A small, underpowered treasure trove?

Benjamin Kirkup writes:

As you sometimes comment on such things; I’m forwarding you a journal editorial (in a society journal) that presents “lessons learned” from an associated research study.

What caught my attention was the comment on the “notorious” design, the lack of “significant” results, and the “interesting data on nonsignificant associations.” Apparently, the work “does not serve to inform the regulatory decision-making process with respect to antimicrobial compounds” but is “still valuable and can be informative.”

Given the commissioning of a lessons-learned, how do you think the scientific publishing community should handle manuscripts presenting work with problematic designs and naturally uninformative outcomes?

The editorial in question is called Lessons Learned from Probing for Impacts of Triclosan and Triclocarban on Human Microbiomes, it is by Rolf Halden, and it appeared in a journal of the American Society for Microbiology.

I do find the whole story puzzling, that Halden describes the study as small and underpowered, while also “presenting a treasure trove of information.” The editorial almost like a political effort, designed to make everyone happy. That said, I don’t know jack about the effects of triclosan and triclocarban on human biology, so maybe this all makes sense in context.

The “underpowered treasure trove” thing reminds me a bit of when food researcher and business school professor Brian Wansink told the story of a “failed study which had null results” (in his words) which at the same time was “a cool (rich & unique) data set” that resulted in four completely independent published papers. Failed yet wonderful.

10 thoughts on “A small, underpowered treasure trove?

  1. I’ll try to find time to write more later, but if you’re ever looking to dive into a treasure trove of terrible studies, full of noise-mining, over-hyped results, p-hacking, and more, studies of the gut microbiome are a good place to look. On the other hand, it’s a wonderful subject — I work in it — and there’s a lot of very good, fascinating, and important work out there.

    About “presenting a treasure trove of information” — this would be easier to be happier about if more papers simply presented information, rather than drawing unsubstantiated conclusions from them. Sadly, that’s not acceptable, for understandable reasons.

    • “if you’re ever looking to dive into a treasure trove of terrible studies, full of noise-mining, over-hyped results, p-hacking, and more, studies of the gut microbiome are a good place to look. On the other hand, it’s a wonderful subject — I work in it — and there’s a lot of very good, fascinating, and important work out there.”

      If your community doesn’t limit the amount of BS being produced you can’t expect other people to waste time looking through it for diamonds in the rough though. At some point it becomes easier just to throw it all out and start over if that is a topic of interest.

        • It shouldn’t be fine… that was a major factor in me getting away from biomed. I judged that it would be rational for people to ignore my work even if I did a good job. And if it isn’t you who is familiar with the area and recognizes a problem that can direct the community, who will it be? Personally I wasn’t willing to take that responsibility, but hopefully others can. If not, it will all just be redone from scratch by future generations.

  2. Are we trying to have it both ways? We don’t want investigators to overly focus on statistical significance, but we also don’t want them to say anything unless there is a p-value < 0.05.

    What's the better alternative?

    OTOH I do agree that it's impressive to get four papers out of a "failed" non-significant study.

    • Sean:

      I don’t know who is the “we” you are referring to, so let me clarify that I completely disagree with your statement, “we also don’t want them to say anything unless there is a p-value < 0.05."

  3. Clinical trials commonly involve measuring many outcomes and covariates. While a particular treatment may not affect the outcomes, there remains a wealth of knowledge in the interrelationships among the outcomes and covariates, which may otherwise not justify a dedicated trial. There are obviously potential problems in how these studies are presented.

    • “a wealth of knowledge” sounds much too strong. However, the data may point to *possible* interrelations among the outcomes and covariates that “*might* be worth further investigation.

      • I agree, there is definitely a strong potential for abuse of this sort of thing (I’ve seen it), and you have to be careful to advise investigators appropriately. It amounts to an observational/retrospective study, even though the data derives from randomized trials, and you have to ensure investigators report it way. But it provides a means of investigating metabolic and other processes and helps to identify new research directions. It does require that you put some deep thought into what data to use (probably just the control group) and how best to model it.

        • > put some deep thought into what data to use (probably just the control group) and how best to model it.
          Agree, that is the key (and why most of it should just be binned and preserved).

Leave a Reply to Raghu Parthasarathy Cancel reply

Your email address will not be published. Required fields are marked *