“Non-disclosure is not just an unfortunate, but unfixable, accident. A methodology can be disclosed at any time.”

Mike Spagat writes about researchers who refuse to share their data or even describe how they conducted their studies:

Non-disclosure is not just an unfortunate, but unfixable, accident. A methodology can be disclosed at any time. It’s not like you’re up there on a quiz show and the $50,000 question comes up:

How did you conduct your survey?

Inexplicably you hem and haw and reply:

I’m not telling you

You lose the $50,000 and are kicking yourself for years. Why didn’t you just answer the question rather than panicking?

Well put. At any time—even now, four years after the above-linked post—researchers can share data and experimental protocols. They can even admit they’ve made mistakes.

Heck, it’s not even too late for David Brooks to post some corrections. He ain’t ever gonna do it, but nothing’s stopping him from doing so.

As Mark Twain apparently never said, “It is better to keep one’s mouth shut and be thought a fool than to open it and remove all doubt.”

In this case, the people in question have already opened their mouths and removed all doubt—but it’s not, and will never, be too late for them to open their mouth one more time and correct the record.

P.S. More context from Spagat here.

9 thoughts on ““Non-disclosure is not just an unfortunate, but unfixable, accident. A methodology can be disclosed at any time.”

  1. well, the general audience and “market” for scientific-research doesn’t seem to demand stringent details of methodology & data.

    Professional journals, academia, and the media are flooded with such “unsupported” research. It sells well as is.

    For the mass of average researchers, why risk focused criticism by handing your potential critics, buyers, employers, and competitors all the details of your work ?

    Incentives and disincentives matter in markets and society.

      • We can start by acknowledging that journals’ most valued (but not most important) services are to certify research quality and to credential researchers, then decouple that function from publicizing science. I guarantee you that taking away the burden on publications to sell the authors themselves–and remember that editors and peer reviewers are researchers, too–will remove a major incentive for undercutting science.

    • Hi and yes, of course, I agree and journals should really insist on this.

      Just to be clear, I was just trying to make the point that not non-disclosing methodology (and data) is a shortcoming that could be fixed at any time following the principle of “better late than never”.

      (I don’t think we’re actually disagreeing about anything)

  2. Unfortunately, contrary to the drive for open science, the biggest growth industry in health care is “data privacy.” In the past several years, more and more administrative restrictions are being brought to bear that restrict the ability to share health data, and lock it down with data use agreements. Whereas at one time data exchange could be accomplished by a few quick emails between investigators, now institutions have a staff of lawyers devoted to imposing as many restrictions as possible.

    There is some pushback against this tendency. The National Institutes of Health requires grant applications to include plans for sharing of data and other resources–though “we can’t do that, our sources won’t let us” is still an acceptable response to that. And journals are beginning to ask authors to provide their data to reviewers, or to the readers–but again, they readily take no for an answer.

    It’s a difficult problem in health care. There are legitimate concerns: some of the information we deal with is sensitive, and people can be grievously harmed by its disclosure to parties with adversarial interests. Various approaches to removing identifying information are in use, but it is generally acknowledged that with determination and adequate resources, a party who wants to re-identify a data set probably can do so.

    It will be interesting to see how this all plays out, and what the standard practices will be five or ten years from now.

    • Sorry–a couple of errors in the above.

      The NIH requirement for a data sharing plan applies only to certain types of grants, but the most important ones, large clinical trials, are included in that.

      Similarly, not all biomedical journals are asking for data; in fact, at present, it’s only a small minority that are, although many of the “high impact” journals are doing this.

    • “but it is generally acknowledged that with determination and adequate resources, a party who wants to re-identify a data set probably can do so.”

      Seriously? Is that much / that specific data really necessary most of the time? I don’t know, I’m not involved in medical research but it seems like that doesn’t need to be possible for the vast majority of medical studies and would be a concern mostly for less common health conditions.

      I wonder if people are grabbing piles of data that they don’t really need to do a particular study with the idea that they’ll use it for exploration in the future or even just p-hack it to get a publishable result.

Leave a Reply

Your email address will not be published. Required fields are marked *