Skip to content

The replication crisis and the political process

Jackson Monroe writes:

I thought you might be interested in an article [by Dan McLaughlin] in NRO that discusses the replication crisis as part of a broadside against all public health research and social science. It seemed as though the author might be twisting the nature of the replication crisis toward his partisan ends, but I was curious as to your thoughts.

From the linked article:

The social-science problem is that “public health” studies — like that NRA-convention study — can be highly subjective and ungoverned by the rigors of hard sciences that seek to test a hypothesis with results that can be replicated by other researchers. Indeed, the social sciences in general today suffer from a systemic “replication crisis,” a bias toward publishing only results that support the researcher’s hypothesis, and chronic problems with errors remaining uncorrected.

NRO is the website of the National Review, a conservative magazine, and the NRA (National Rifle Association) convention study is something we discussed in this space recently.

I think McLaughlin is probably correct that studies that seek to advance a political agenda are likely to have serious methodological problems. I’ve seen this in both the left and the right, and I don’t really know what to do about it, except to hope that all important research has active opposition. By “opposition,” I mean, ideally, honest opposition, and I don’t mean gridlock. It’s just good if serious claims are evaluated seriously, and not just automatically believed because they are considered to be on the side of the angels.


  1. Dale Lehman says:

    On the NRA study – thanks to the data received by D Kane, I’ve looked at the data the authors used in their results (not the raw data, which they won’t release, but the manipulated data that they based their results on) and it is highly suspicious. One year out of the 9 they aggregated is driving the results – 2012 – and if the data is to be believed, there were twice as many medical claims during the 7 weekends that year than any of the other 8 years. That’s twice as many medical claims nationwide among the insured population. When I inquired about that anomaly, the lead author said they had noticed that as well but didn’t have an explanation. Without the raw data to investigate, I have to say I find the study completely without merit – based on lack of public release, unbelievable results, and suspicious aggregated data.

    To relate this to this blog post – and I’ve said this numerous times. The best protection against research being subverted to a political agenda is for decision makers to announce that any “studies” that might influence policy will be disregarded if the data is not made publicly available. [since this comes close to the recent discussion about potential EPA policy to require all studies to be based on publicly available data, I might want to make an exception for analyses conducted by public agencies – provided that they are open to a public input process, which they presumably are] Think tanks and academic researchers are getting away with sloppy (possibly intentionally so) research that satisfies clients who either pay directly or indirectly for the opinions this research supports. The best way to prevent this is to reduce the payoff from that research by rendering it ineffective. At least, I think that is easier than trying to convince researchers that they shouldn’t do these things. The fact that regulators and legislators appear to have no interest in taking this simple step makes me believe they really like “science” that is politically motivated. It is a sorry state of affairs.

    • Politicians love using “science” to promote their hobbyhorse, and they love to cut off science that they don’t like. It’s politics as usual.

      Recently there was an announcement that Ontario CA was canceling their trial of UBI (a topic I am very interested in). There was not really much about why, such as it not being effective or harming people or anything, it was basically just “we’re stopping this”. Of course it occurred right after the majority party switched in a recent election. My own suspicion is that UBI ultimately frees people to be independent of government oversight, and reduces the need for government programs, therefore it doesn’t meet the requirements of the politicians: control and favors to sell. So I see this as an attempt to subvert a scientific experiment because it might not help the politicians in power.

      • Corey says:

        Conservative movers and shakers in Ontario are neither for nor against the independence and freedom-from-oversight of people receiving benefits from the government — they just want to stop giving tax money to the poors, period. I anticipate Common Sense Revolution redux — spending on social assistance, health, education, and infrastructure will be slashed in favour of a nice fat tax cut. (To be fair, the Harris government put Telehealth Ontario in place and it’s a great program.)

        • If UBI turned out to be a terribly efficient and effective form of welfare that actually caused GDP to increase so that the average real income post UBI was actually higher than pre-UBI and higher even than the “zero redistribution” lower-tax reality… it wouldn’t support the “let’s get rid of all redistribution” ideology. I think this is the real reason they canned the experiment. If you’re against giving say drug X to people because of say moral reasons related to religious ideas, any experiment showing that taking drug X improves people’s health and lives and is obviously a good thing… would not be good for your purposes.

          The thing that I don’t get about the “don’t give tax money to the poor people” attitude is how it’s clearly not optimal for anyone. Sure taxes are lower, but generally if you could allocate the tax money efficiently, there are always going to be people for whom spending the tax money is obviously better for society. For example a small group of mentally ill people who will otherwise wind up in ERs several times a year for visits costing $250k/yr on average. And then the list goes on: kids and parents with abusive partners, veterans with PTSD, disabled victims of violent crime, whatever, its easy to find subsets where some kind of welfare clearly is going to help lower crime rates, prevent wasted resources in other places, or lead people to be able to hold jobs they otherwise wouldn’t, whatever… thereby improving these people’s lives far more in some abstract sense than the marginal cost of the taxes to everyone else, this is particularly true when real income *goes up* for everyone due to less destruction of real wealth.

          It’s often easy to argue that the optimal redistribution quantity is less than whatever we’re currently doing, but it’s totally trivial to argue that the optimum is higher than 0. The beautiful thing about UBI is that it accomplishes a lot with tremendous efficiency, it doesn’t require bureaucrats to figure out how to allocate funds into an efficient set of services without any real significant knowledge of who needs what and how badly. In other words it applies distributed information-gathering market forces to welfare rather than inefficient politburo central planning, which is the real stupidity of traditional welfare.

          From the article I linked above:

          “it remains controversial, even among libertarians. Some libertarians are firmly against the idea, arguing that it is as unjust as any other form of wealth redistribution. Others say a UBI would be less intrusive and more cost-effective than a traditional welfare state, and therefore would be a step toward smaller government”

          It seems stupid to me to think that we’re ever going to get anywhere near “zero redistribution” and in fact it’s kinda obvious that zero isn’t the optimum in terms of justice (how just is it when crime rates increase due to desperation among the poor?). Given the obvious reality of the situation that some “Platonic ideal of zero redistribution” is neither practically ideal nor achievable, people who are generally for reduced government waste and increased individual autonomy and increased economic / GDP growth would be against UBI seems so stupid.

          Of course, the number of people who actually hold principled views on things like liberty, freedom, economic welfare etc is a pretty small fraction of the population, and it seems a particularly small fraction of modern politicians.

          • The other thing that is wrong with the focus on “don’t give money to the poor” is that “money” just doesn’t matter, it’s *real wealth* that matters, and if by implementing UBI we can increase the rate of real wealth production and thereby decrease the prices of goods and services, then we could easily wind up *more wealthy* through time even though we take home maybe fewer dollars than we otherwise would have. This fact seems to be lost an all but a tiny percentage of people. Eliminating the wealth-destructive effects of bad welfare policy is the main thing IMHO.

        • jrkrideau says:

          @ Cory
          I tend to agree. I, also, don’t think that we can discount a “Calvanistic” idea that the poor are poor do to their own moral deficiencies. Therefore the govenment should not help the dross of society

          And it may just be that Doug Ford likes kicking helpless people. I see him as a totally despicable person.

    • Terry says:

      About the doubling of claims: could it be due to timing issues? Perhaps a lot of claims are filed at the beginning of the year because people get new policies that start January first.

      Presumably they should be tracking when the injuries occurred, not when the claims were filed. Are they using injury dates or filing dates?

  2. Garnett says:

    Certain industries have a track record of sowing doubt about public health research that puts pressure on them. I was wondering if they would marshall the “replication crisis” as another point of doubt.

    • anonimus says:

      +1. That is the biggest problem.

    • Thanatos Savehn says:

      Ah yes, the coffee industry – peddlers of poison, merchants of misery, dealers of death – trying to trick you into skepticism with their sneaky, misleading questions about methods and inference.

      By the way, ever wonder why you don’t hear about verdicts in smoking cases? Ever wonder why there are no lawyer ads for smoking plaintiffs? Ever wonder why the smokers wound up outside in the rain while the lawyers, states and manufacturers wound up in bed together sharing a smoke after some three way intimacy? If you don’t, you should – if you’re brave enough.

      • I’ll bite. I’d assume it’s because the tobacco industry got some kind of laws passed limiting their liability in exchange for a couple lines of warning on their product or whatever.

        I’m in favor of warnings on products though, however I really think they need careful thought in the wording. For example “smoking cigarettes causes lung cancer, heart attacks, and strokes” is a good warning (very specific), and “smoking cigarettes is bad for your health” is a bad warning (not specific enough)

        The CA warning “this product contains chemicals known to the state of california to cause cancer, birth defects, and other reproductive harm…” or whatever is in the end a terrible warning because *everything* “contains chemicals”. I mean analytical chemists can find a single lead atom in a liter of water if you give them enough money, and you might as well assume every liter of water is going to contain a single lead atom or more… so while it’s true that lead is bad for you, without a requirement for a *substantive harm from the actual contents and when used in an actual type of usage* just a warning that things “contain chemicals” winds up everywhere. I’ve gotten that warning included with a pen I bought, it’s on the fish counter at the grocery store routinely, it’s included with my internet router… whatever

        I may have gotten a little off track ;-)

  3. My long interest has been the sociology of expertise, by chance actually. It began in my effort to aid my father’s academic career. Although not successful in that effort, I did manage to witness how international relations how knowledge base was being forged. And this much I can say, understanding the sociology is indispensable to it. That is understanding the temperaments, learning dispositions, and relationships among experts. In terms of the statistics crises, I learned a good amount from observing the dynamics among experts on Justify Your Alpha, Lakens et al. There is no guarantee that personal biases will be put aside, if even that is possible.

    In short, I believe academics themselves have brought on the doubt in the integrity of research. In this I agree with John Ioannidis. A positive step would be in redefining the incentives structures for research. To be sure John Ioannidis’ family background [two physician/math trained professors give him keen insight into the role of academia in relation to private industry and government funded efforts.

    Then of course the litigation as it pertains to biomedical enterprises is an even more complex matrix of evaluating research results.

  4. Re: Opposition

    Of course there is merit in cultivating opposition. The problem is that it devolves, eventually, into adversarial brownie point seeking debates. Devil’s advocacy has contributed to it. I disagreed with Robert Nozick and Martha Nussbaum’s endorsements of it. I believe Robert Jervis has been less enthusiastic of its use in debates, as I. I am sure that in some cases, its use may bring up some good insights. But it violates Occam’s Razor in most of the debates I’ve come across. I think, in part, the extent of our resort to litigation is one primary reason for how debates in the public sphere fare.

    I think some individuals are just simply better at communicating their opposition and do not get all bent out of shape when confronted with disagreement. Temperament matters.

    Moreover we should be more granular as to what we mean by replication. Steven Goodman, Daniele Fanelli, and John Ioannidis have written a very good article, distinguishing three types of replication. It helps in framing the broader crisis of scientific research theory and practices

    • Anonymous says:

      “Re: Opposition

      Of course there is merit in cultivating opposition. The problem is that it devolves, eventually, into adversarial brownie point seeking debates. Devil’s advocacy has contributed to it. I disagreed with Robert Nozick and Martha Nussbaum’s endorsements of it. I believe Robert Jervis has been less enthusiastic of its use in debates, as I. I am sure that in some cases, its use may bring up some good insights. But it violates Occam’s Razor in most of the debates I’ve come across”

      Yes ! (if i am understanding you correctly).

      I have been annoyed with so-called “debates” (also in science) where folks seem to me to think it is useful, and valid, to “let everyone give their side of things”, and claim that this is “nuanced”, and a useful way to have a debate.

      I came across this wiki page that, i think , captures (parts of) what i find annoying: “false balance”

      Perhaps when you are trying to be “fair”, “balanced”, and “nuanced” you are in fact not really being any of these things at all…

  5. Kyle C says:

    I have a nice comment stuck in moderation.

Leave a Reply