Skip to content
 

As always, I think the best solution is not for researchers to just report on some preregistered claim, but rather for them to display the entire multiverse of possible relevant results.

I happened to receive these two emails in the same day.

Russ Lyons pointed to this news article by Jocelyn Kaiser, “Major medical journals don’t follow their own rules for reporting results from clinical trials,” and Kevin Lewis pointed to this research article by Kevin Murphy and Herman Aguinis, “HARKing: How Badly Can Cherry-Picking and Question Trolling Produce Bias in Published Results?”

Both articles made good points. I just wanted to change the focus slightly, to move away from the researchers’ agency and to recognize the problem of passive selection, which is again why I like to speak of forking paths rather than p-hacking.

As always, I think the best solution is not for researchers to just report on some preregistered claim, but rather for them to display the entire multiverse of possible relevant results.

12 Comments

  1. Jonathan (another one) says:

    We live in the age of incredibly cheap cloud data. Why not just have suitably anonymized databases of every medical experiment (inputs and outputs) available to anyone interested in downloading and doing their own analysis? Why should medical experimental data have lower availability to independent researchers than baseball data? (I have an answer, but it doesn’t reflect well on the producers of original medical research.)

    • Anoneuoid says:

      suitably anonymized databases of every medical experiment

      I am personally concerned about privacy, but I see many people now with always on recording devices in their homes sending everything it hears to some corporation, going everywhere with active tracking devices in their pockets, etc. I really think the vast majority of people participating in a clinical trial would agree to let the data be shared out of altruism with a minimum of anonymization. Just leave the names out.

      On the other hand, I experienced that the US government doesn’t even want raw data about rodents to leave the premises. They imagined a risk that it could get compromised by a foreign power (China) who would use it against the US.

      • Joe Nadeau says:

        Can you expand on the rodent data aspect? What do you mean ‘premises’? Lab, institution, US? Lots of raw and analyzed rodent data is freely available at many many websites. So I am curious about your experience….

        • Anoneuoid says:

          I worked at a federal government facility and they tried to prevent me from taking my data home on an external hard drive, using that as an excuse.

        • Anoneuoid says:

          I should also add that these fears were especially ridiculous in light of the lax infosec procedures in place.

          Once I forgot my password and discovered all that was required for a reset was to call a number found on every mousepad and tell them the name of someone who worked there. Then there were the ridiculous password constraints and constant password cycling that actually made their network less secure.

  2. Brent Hutto says:

    The model you seem to be describing is where a “researcher” is basically someone given a government contract to run experiments and collect data. Not someone responsible for the entire chain of theory formulation, study design, experimental trial, data collection and finally data interpretation.

    When described like that it’s easy to see what a hard sell that would be to investigators vested in the current system. If I were such a person I’d be happy to form my theories, design my studies, sit back and let someone do all the hard graft, then analyze and interpret the resulting data as it is released.

    Running the experiments and, especially, collecting the data would seem a relatively thankless task.

    I also always wonder about just where, precisely, the division of labor falls between “data collection”, “data cleaning and QC” and “data analysis”. In my experience, if given the actual rawest of raw data from someone else’s experiment I’d need some extremely extensive documentation of every contact between research subjects and the research team.

    It’s not infrequent when dealing with free-living individuals recruited for a study to have an awful lot of going back to resolve missing or erroneous data. That sort of QC has to be done as soon as possible after each bit of data is collected. But then there’s a level of trust required from anyone analyzing that data at a later time. Either trust or, as I say, voluminous documentation.

    So I think “raw data” in complex, multi-year, population-based studies is never going to be totally raw. It will have a certain level of cleaning and QC applied as the study plods along. A feedback path allowing an eventual analyst to interrogate the details of that process is required in my opinion.

    Unless of course one wants to take a purist approach where every notation made on a chart, every box checked by a participant, every number recorded by a measurement device is walled off at the moment it is recorded and no cleaning or QC is allowed.

    • Anonymous says:

      Quote from above: “If I were such a person I’d be happy to form my theories, design my studies, sit back and let someone do all the hard graft, then analyze and interpret the resulting data as it is released.”

      Isn’t this basically what happens already: “resarch assistants” performing the actual data collection while the “professor” sits back in his chair and comes up with “brilliant” ideas and “theories”? It was also a big part of social psychologist Diederik Stapel’s fraud, where (if i am not mistaken) he gave people data-sets (that he manipulated or fabricated).

      Aside from that: i think the possible division of the many tasks a researcher can have, and researchers not being “responsible for then entire chain of theory formulation, study design, experimental trial, data collection and finally data interpretation.” is very “bad” for science.

      For instance, i think that people in Psychology might be relatively “bad” at statistics, and that this may have contributed a lot to the mess that field has gotten into. I also think that this line of reasoning can perhaps be extended to include things like logical reasoning, which i think influences many things in science (i.e. hypothesis formulation, theory formulation, writing, debating, etc.).

      I reason division of labour/tasks in science sets up a system where (in the long term) nobody really knows what’s going on, and where there is no form of being able to verify things. Additionally, i reason that such a system is conducive for potential abuse, manipulation, control, and bureacracy.

      It’s therefore (from a scientific point of view) incomprehensible to me that i think some sections of academia are trying to set this up as we speak: i believe it’s being called (or should i say “sold”?) under the name of “collaboration” and “crowdsourcing”.

      • Brent Hutto says:

        We may somewhat be talking across each other as I have no experience with the types of small-scale stuff they publish in Psychology journals. And very limited experience with anything that could remotely be imagined as amenable to “crowdsourcing”.

        • Anonymous says:

          Quote from above: “We may somewhat be talking across each other as I have no experience with the types of small-scale stuff they publish in Psychology journals. And very limited experience with anything that could remotely be imagined as amenable to “crowdsourcing””

          I was talking/thinking about Psychology indeed. I do however think this does not matter much in light of the points i am trying to make.

          I was also thinking and talking about things from a more general perspective. The 1st part of your comment has been on my mind a lot recently, and that’s why i commented on it.

  3. RoyT says:

    I am sympathetic to the criticism that some authors fail to report primary outcome / analysis of a clinical trial in their first publication. However, in most trials I have been involved with, it would be impossible to report in detail all of the secondary outcomes in a single manuscript. Protocols have large lists of secondary analyses. Maybe there is a way to display them graphically as Andrew suggests. Finally, it isn’t uncommon for reviewers to suggest useful additional analyses that weren’t thought of beforehand. These should also be reported. It doesn’t make sense to me to argue with a referee that you can’t present the results of a suggested analysis simply because it is post-hoc.

  4. Michael Nelson says:

    Imagine if architects all believed their designs are so complex, and the construction process so dynamic, that they can’t possibly create a blueprint that others could follow, even in retrospect. Every architect would have to double as a construction site foreman, directly overseeing everything from masonry to plumbing, and then would have a hell of a time convincing a building inspector that the finished building was up to code. This is the system within which social scientists are trained and ultimately work. The only major difference is that we are each other’s inspectors, so we get nervous about applying too stringent of standards on others lest we receive the same treatment.

    Real-life architects draft blueprints, “preregister” them with the city/county, then write up practical instructions for the contractor, whose builders’ and technicians’ work is documented and inspected at key points. In principle, anyone can go and build the same building or check that the original building is sound. So it can be done. The amazing thing about science is that the building materials are data and the real estate is software, roughly speaking, so any scientist with those resources can reconstruct any study, or variations on that study. But this is scary, too, because giving up control of those resources means another scientist may construct a better study, or find flaws in yours. And they can beat you to the next grant, the next publication, etc.

    So I guess my point is: the incentives for sharing data are noble, but the disincentives are personal. Science would be better if everyone shared their data, but that really is asking a lot from anyone trained in this system and dependent on it. Architects would keep their methods hidden, too, if they were allowed. So changing behavior on any kind of meaningful scale will require changing these incentives–hopefully something that open journals and changing attitudes will help accomplish.

Leave a Reply