Move along, nothing to see here

[cat picture]

I don’t really want to go into details on this one as our paper is under review at a journal, but the short story is that my colleagues and I have conducted replications of a high-profile psychology study. Not all our replications had results that made sense, and so from a perfectly reasonable Bayesian perspective we chose the more reasonable outcomes to include in our meta-analyses.

Anyway, this is no big deal and our paper will be peer reviewed in any case. But I felt the need to bring it up here because some methodological terrorists have been going on social media bugging me to post all my data. I’m sorry but it doesn’t work that way. We have IRB rules, subject confidentiality to consider, and also I don’t like the precedent. What would happen if every reputable lab had to share data with bloggers, the rabble on twitter, etc? It would end up like that PACE trial, where all sorts of unqualified outsiders started second-guessing serious researchers. This is just bad news. We have serious science to conduct here. I wish the snipers would spend a little more time doing science themselves and a little less time criticizing. As the great Satoshi Kanazawa put it—or was it Stewart Lee?—those who can, do research. Those who can’t, blog.

We’ll now return to our usually scheduled programming.

31 thoughts on “Move along, nothing to see here

  1. > “We have … rules… We have serious science to conduct here. ”

    “Hell, there ain’t no rules here. We’re tryin’ to accomplish somethin’.”

    –Thomas Edison

    And what currently prevents the rabble from second-guessing serious researchers, with or without their research data ?

  2. I couldn’t agree more. When I was an editor of an APA journal, I can’t tell you how many disputes arose over requests that authors share their data. Basically, the decision hinged on a) whether the data were proprietary, b) whether informed consent was explicit about the possibility of the data being shared with other researchers, and c) the purpose(s) for which the data were being requested.

    The last point was especially important. I always encouraged the two parties to put in writing what the recipients of the shared data could do with it. APA was fairly explicit that the two main reasons for requesting data were narrowly defined as a) replication of the authors’ original analyses, and/or b) a different analyses directed at the authors’ original hypotheses. If publication after a new, alternative analyses were possible, once again, I urged the two parties to put in writing authorship considerations.

    Often, it was fairly clear that the party requesting the data either didn’t have specific plans for what they would do with the data, or they didn’t want to collect their own data that would have served as a potential replication/extension of the originally published study.

    Occasionally, the party requesting the data wanted me (in my role as editor) to compel the authors to release their data if the authors voiced legitimate concerns about doing so. Data sharing is certainly an important part of the scientific enterprise, but as editor, I had no authority to compel data sharing. I could certainly encourage it if I thought that the request to share was reasonable and the purpose narrowly defined.

    Sharing data with every party that makes such a request without having a clear agreement between the two parties as to what can and cannot be done with the shared data is a prescription for trouble down the line. The default response probably should be to share, but authors do have rights and responsibilities vis-à-vis their data, and those should be respected.

  3. Why not anonymize the data and release it? At the institution that I work at we used to release a public use data set of our survey data annually, but this got to be a real PITA so we stopped. But other than the inconvenience, I don’t see why it can’t be done, especially if you take money to defray the costs of processing.

  4. Asking for “the purpose(s) for which the data were being requested” is a prescription for trouble down the line. Instead of pretending we live in a perfect world, let’s compare the dangers associated with releasing data to researchers that are either incompetent or malicious with the dangers associated with having editors or authors possessing veto power over such requests. I don’t think it is productive or wise to require those requesting the data to have purposes approved by authors and editors. Too much room for bad practices to evolve.

    Wouldn’t it be simpler to finally have granting authorities and academic promotion and tenure committees decide for themselves the merits of research that claims to invalidate or raise suspicions about previously published work? We seem to go to great lengths to “protect” careers when the easiest protection is to do our jobs and judge both the quality of published work as well as the quality of critiques of that work.

  5. Ta, Andrew: I was thinking all the April’s fools this year were horrible but this makes all the hassle of having such a day worth it. Also, as a person who likes kitties, I applaud your habit of finding a matching kitty pic for your posts most of the time now-a-days.

      • Aw, thanks. Nice to see that other people (I am now assuming a lot of things about that +1) appreciate Andrew’s sense of humour. I just wish these +1’s would actually add up to something! Or do I? It’d be silly. But whatever, maybe it’d help at the gates of the heaven if one could say to have so and so many +1’s from Gelman’s blog. Ah, jesus, how many would I need to counter all the evil I’ve done… but now I’m just rambling, all I really wanted to say was that my happiness was increased by your approval of my comment. That counts right now, regardless of what might happen at hypothetical gates of hypothetical heaven.

        • You will need to explain why you broke the first rule of April 1st – never remind anyone what the date is until after 12:00!

          Good luck ;-)

        • Never heard of that rule — but maybe kitty’s in a different time zone where 12:00 was past at 11:18 EDT?

        • I wasn’t aware of that rule, so there is no explanation: I just was wrong. On the other hand (when will I learn to shut up?!) I’m from a country in which the time was already more than that. But that doesn’t change the fact that I did not know that rule, and obviously I just was wrong.

          Apropos: I’ve read your comments here (on Andrew’s blog) and you seem like a really cogent person and I wish you continue to comment. I’m just a kitty, and I don’t always know what to say, and I apparently broke some rules. Well, there it is!

  6. I came here wondering, “What did Andrew post on this particular day?” I got a good laugh (and then another one when I followed the link in the post).

    Here’s my favorite quote from the latter:

    “He thought he had one final Bayesian trick up his sleeve: By hiring a skilled hypnotist he manipulated his priors, his own beliefs (!) in Power Posing. But even with these inhumane levels of disbelief, the posterior always indicated beyond a doubt: Power Pose prevailed. It was almost like the data were trying to tell him something – but Dr. Gelman had forgotten how to listen to evidence a long time ago.”

    Oh, and I hope the “kitty from an alternative bubble of reality” keeps on commenting. It’s great that this blog has inspired kitties (around the world, apparently) to speak for themselves.

    • My kitties are showing increasing interest in scientific publication. OK, the Siamese throws up every few weeks, but this is just practice. I don’t know which target journal he has in mind.

  7. “Those who can’t, blog.”

    No. I decided to stop playing the publication game in the 1990s. In some fields you can do your own research and publish without getting the blessing of the magazine industry and all the rubbish that goes with it. It’s got to start somewhere, hasn’t it?

    This person said those who can’t, blog:

    https://sciencepolice2010.wordpress.com/2016/02/11/jingmai-oconnor-values-peer-review-and-palaeo-phds-and-demonstrates-their-uselessness/

    She got a lot of other stuff wrong too.

Leave a Reply to Leo Cancel reply

Your email address will not be published. Required fields are marked *