Meta-meta-science studies

August Wartin asks:

Are you are familiar with any (economic) literature that attempts to model academia or the labor market for researchers (or similar), incorporating stuff like e.g. publication bias, researcher degrees of freedom, the garden of forking paths etcetera (and that perhaps also discusses possible proposals/mechanisms to mitigate these problems)? And perhaps you might know any empirical (economic) literature evaluating the effect of some policy measures to mitigate these problems?

I send this along to Paul Smaldino who wrote a paper, The Natural Selection of Bad Science, a few years ago, and who has more papers on the topics on his website. Smaldino added:

There are an increasing number of models of these processes. Here’s a recent paper I like that I think is a nice, simple model.

This article, by George Akerlof and Pascal Michaillat, seems similar to another paper by those authors that we discussed here a few years ago.

Now that there’s a whole subfield of meta-science studies where researchers construct and compute theoretical models of the scientific process, the next step is meta-meta-science, where we have models of the incentive and communication structure of meta-science. And then meta-meta-meta-science . . .

Also I’m interested in any answers to the question that Wartin asked in the last sentence of his above-quoted email.

12 thoughts on “Meta-meta-science studies

  1. I can think of quite a few things that the literature says will be extremely effective at mitigating poor research practices:

    1) Researchers should keep their desks and offices in a clean and tidy state, lest they be tempted into a life of scientific perfidy.
    2) Researchers should avoid eating lunch at work, lest they be primed to think about “forks” and then “forking paths”.
    3) The results of every eighth analysis should be presented in a red font, to force researchers to attend to the number of analyses run and therefore limit their analysis intake.
    4) Researchers should try to wait until they are 29, 39, 49, 59, 69, 79, 89, 99, etc. years old before taking on particularly ambitious projects, since they are certain to be more productive in those years of their lives.
    5) Researchers should keep bald eagles, American flags, busts of George Washington, and other patriotic paraphernalia in their offices to keep them constantly reminded of the noble principles they should uphold. In the event that these are not available or, even worse, that the researcher lives outside the United States, they should just stick as many smiley face stickers on the wall as they can. Every little bit helps!
    6) Always be sure to consult with a Cornell undergrad prior to undertaking any research endeavor. No need to engage in questionable research practices if you know ahead of time what the results will be!
    7) If you happen to be a woman who does research, if you are single, you should avoid doing research while on your period, because during that time you will be too liberal in your acceptance of data pruning, alternative hypotheses, and alternative analyses. If, on the other hand, you are married, you should *only* do research while on your period, since this is when you will be most conservative and adhere to the rules of good science. If you are a woman who cannot easily be classified into “married” or “unmarried” (e.g., in a domestic partnership, widowed, divorced, in a long-term relationship, etc.) or are post-menopausal, you should probably just avoid doing research altogether since you cannot predict how your body will cause you to act. Can’t be too careful!

    • > Anytime the researcher is closely involved in the design of the data gathering process, or even worse its implementation, there will be abundant scope for taking actions to increase expected r-squared and thereby bias results.

      This is from the second — is this a popular stance with anyone these days? Seemed kinda the opposite of what I’d expect.

      • Please explain what you mean by “taking actions to increase r-squared and thereby bias results.” Are you certain that you understand what “r-squared” or “bias” actually means technically? Please be careful using terms that have very specific technical meanings and definitions.

        • I’m not the author of that quote. That’s from the second paper (The ‘>’ was me indicating it was a quote). I was more asking about the first part — the idea that if someone is involved in the data collection and the analysis that leads to mistakes (I think we’re assuming the person is being honest).

          The alternative (separating these things between multiple people and setting up an interface) doesn’t seem obviously less mistake-riddled though. I’m curious if this is a common belief that I just missed out on or something.

          But back to those terms, selecting on outcomes is something you can do on causal things and it can make your r-squared look better and then you get selection bias in your causal effect estimates. Given the author is in economics, I assume that’s the concern.

        • Ben —
          I’m sorry, but I have no idea what you mean by this: “But back to those terms, selecting on outcomes is something you can do on causal things and it can make your r-squared look better and then you get selection bias in your causal effect estimates. Given the author is in economics, I assume that’s the concern.”

        • I think the Regression and Other Stories example is, say you want to know the effect of an education program on test scores 2 years in the future. So you do your randomized trial and your regression is just y ~ t where t is whether or not someone got the treatment.

          Then you decide to measure test scores after one year cuz why not and you do the regression y ~ s + t. Presumably your fit will get better cuz you’ve measured the intermediate test outcome after 1 year, but if s does help the prediction, it changes the coefficient attached to t and messes up your interpretation of the treatment effect. But if that isn’t clear (and I’m not sure it is — I don’t exactly know this stuff super well), the RAOS causal section has it.

          Searching for intermediate outcome in the blog gives: https://statmodeling.stat.columbia.edu/2006/04/28/amusing_example/

          There’s probably also stuff if you search for pre-treatment vs post-treatment (adjusting for post-treatment being the bad thing).

          https://statmodeling.stat.columbia.edu/2009/02/02/mostly_harmless/ has a section explaining it (and it does it in linear algebra terms).

          There’s also mediating outcomes which I think is in this theme of things.

  2. There’s an old joke about an economist talking to a colleague about some personal decision he has to make (I forget what it was — perhaps having to do with getting married). His colleague says, “Why don’t you do what you tell your students to do — list all the pros and cons, and base your decision on the expected utility.” The first economists says, “C’mon. This is serious.”
    [Does anyone have the exact citation to this?]

  3. The Berkeley Institute for Transparency in the Social Sciences has some relevant papers, here’s one that seems to fit:

    The Impact of Data Sharing on Article Citations: This study estimates the effect of data sharing on the citations of academic articles, using journal policies as a natural experiment.

    https://www.bitss.org/projects/data-sharing-and-citations-causal-evidence/

    Also a shoutout to the work Eva Vivalt (+ co.) has been doing really interesting stuff in this world for a minute now:

    How Do Policy-Makers Update Their Beliefs?: We present results from experiments run in collaboration with the World Bank and Inter-American Development Bank on how policy-makers, researchers, and development practitioners update in response to results from academic studies.

    http://evavivalt.com/wp-content/uploads/How-Do-Policymakers-Update.pdf

Leave a Reply to Ben Cancel reply

Your email address will not be published. Required fields are marked *