“Merchants of doubt” operating within organized science

I came across this post from 2018, where Dorothy Bishop wrote:

In Merchants of Doubt, Eric Conway and Naomi Oreskes describe how raising doubt can be used as an effective weapon against inconvenient science. On topics such as the effects of tobacco on health, climate change and causes of acid rain, it has been possible to delay or curb action to tackle problems by simply emphasising the lack of scientific consensus. . . .

The parallels with Merchants of Doubt occurred to me as I re-read the critique by Gilbert et al of the classic paper by the Open Science Collaboration (OSC) on ‘Estimating the reproducibility of psychological science’. I was prompted to do so because we were discussing the OSC paper in a journal club* and inevitably the question arose as to whether we needed to worry about reproducibility, in the light of the remarkable claim by Gilbert et al: ‘We show that OSC’s article contains three major statistical errors and, when corrected, provides no evidence of a replication crisis. Indeed, the evidence is also consistent with the opposite conclusion — that the reproducibility of psychological science is quite high and, in fact, statistically indistinguishable from 100%.’

That article by Gilbert et al. is indeed absolute crap and was definitively refuted by Brian Nosek and Elizabeth Gilbert (no relation to the author of Gilbert et al.) back when it came out in 2016; see more here.

In 2018 Bishop did a literature search and saw that the crappy paper by Gilbert et al. was indeed cited a bunch of times; she writes:

The strong impression was that the authors of these papers lacked either the appetite or the ability to engage with the detailed arguments in the critique, but had a sense that there was a debate and felt that they should flag this up. That’s when I started to think about Merchants of Doubt: whether intentionally or not, Gilbert et al had created an atmosphere of uncertainty to suggest there is no consensus on whether or not psychology has a reproducibility problem – people are left thinking that it’s all very complicated and depends on arguments that are only of interest to statisticians. This makes it easier for those who are reluctant to take action to deal with the issue.

The news is not all bad, though:

Fortunately, it looks as if Gilbert et al’s critique has been less successful than might have been expected, given the eminence of the authors. This may in part be because the arguments in favour of change are founded not just on demonstrations such as the OSC project, but also on logical analyses of statistical practices and publication biases that have been known about for years . . . social media allows a rapid evaluation of claims and counter-claims that hitherto was not possible when debate was restricted to and controlled by journals. The publication this week of three more big replication studies just heaps on further empirical evidence that we have a problem that needs addressing. Those who are saying ‘nothing to see here, move along’ cannot retain any credibility.

Let’s not forget, though: back when the replication crisis was blowing up, the merchants of doubt were spraying gallons and gallons of squid ink into the scientific literature. And those were just the active participants in the game; as Alexey Guzey and I discuss, a big chunk of the leadership of academic psychology has just looked away at non-replication, research misconduct, and even outright fraud.

It goes like this:

1. Lots of people are doing research using flawed methods, resulting in some literatures that are scientific dead ends.

2. Some of this bad work gets lots of publicity, which leads to scrutiny, which leads to the widespread realization of methodological problems, along with various specific examples of flawed work and failed replications.

3. There’s concern, not just about a few bad apples, but about entire subfields.

4. The merchants of doubt come in and argue that the replication rate in psychology is “statistically indistinguishable from 100%.”

5. That claim is ridiculous, but it creates just enough plausible doubt for the powers-that-be to continue business as usual.

I’m guessing that Gilbert et al. are sincere in their doubts, in that they’re true believers that their work is serving the world and that Bishop and the rest of us are just a bunch of haters; also they (Gilbert et al.) are in that methodological uncanny valley in which they know just enough statistics to think of themselves as experts but not enough to know that they don’t know what they’re talking about. That doesn’t really matter, though, as long as they can play the useful role of providing leaders in their field with covering fire, an excuse to deny the problems. As Guzey and I wrote, quantitative analysis when used unscrupulously can serve as a sort of squid ink that hides the holes in scientific reasoning, and it is the role of statisticians (and, more generally, quantitative researchers) to be bothered by this when it happens.

55 thoughts on ““Merchants of doubt” operating within organized science

  1. Couldn’t you just as well call those casting doubt on the psych/cancer literature by running the replication projects “merchants of doubt”?

    Actually, science is *all* about doubt and distrust, always has been (nullius in verba). It is the “merchants of certainty” (selling a type of religious belief) that we need to worry about.

    And we know there is no way to prove a theory/explanation due to affirming the consequent, nor disprove one due to Quine-Duhem thesis. Ie, because negating the conjunction (Theory + Auxilliary assumptions) only tells us *at least one* assumption is false.

    So there is never certainty in science, only different relative probabilities as calculated with Bayes rule. Anytime further evidence accumulates or a new theory/explanation is thought up then the equation changes.

    • Anoneuoid:

      The phrase “merchants of doubt” does not refer to people who have legitimate reasons for doubting a scientific claim. It refers to people who have an interest in disputing a legitimate scientific claim and who manufacture spurious reasons for doubt. Gilbert et al. are an example: they have an interest in disputing legitimate results of poor replicability of claims in psychology, so they manufacture bad reasons for distrust in the replication studies. The arguments of Gilbert et al. are ridiculous (see linked post), but by introducing these bad arguments into the discourse, Gilbert et al. can make everyone else’s life a bit more difficult. They’re trading in part of their scientific reputations in order to prop up their failed theories about the world. In the above post, I draw an analogy to the cigarette-company-funded researchers who traded in part of their scientific reputations to dispute the smoking-cancer link and the evidence that smoking is addictive.

      Also, I agree with your last paragraph; see my article with Shalizi for why. Figure 1 of that paper displays the conventional inductive philosophy, which is what you’re talking about in your comment, and which we generally think is inappropriate for making judgments about scientific theories.

      • From skimming the paper it seems to be addressing some strawman that Mayo, etc bring up. I’m not really clear on the context of the arguments you were responding to.

        But what I am talking about in my post is *not* inductive reasoning. It is adbuctive* (guessing an explanation for observations) then deducing the consequences of that explanation and comparing those predictions to new data.

        *Anything goes for the abductive step (praying, ouija boards, defaults like “no correlation”, using analagous or simlar situations), although some methods are more fruitful/efficient than others. So inductive reasoning is an optional part of the process.

        As for illegitimate or fraudulant research, that is always a possibile explanation. I don’t see the benefit in spending time worrying about the exact motivations involved.

  2. Not a fan of the framing or rhetoric. ‘Merchants’ should be derogatory and doubt is a very good thing in science, and at the boundaries of knowledge.
    In any case, those creating doubt to protect their own self-interest aren’t behaving in a mercenary fashion, for example, so the association with a commercial motive doesn’t seem a useful accusation.

    • If you don’t like the framing, read the book. I learned the basic physics of global warming in 1971. For a decade or so, people could hope that negative feedbacks in the climate system would temper the warming, but after that the scientific consensus has been clear. The disinformation campaign described in the book is a big part of the reason that we are only now starting to take it seriously.

      • “For a decade or so, people could hope that negative feedbacks in the climate system would temper the warming”

        So if we keep pumping out CO2 the warming just increases until the oceans boil away? This is one of the best examples of where global warming theory has gone beyond self-correcting. The climate is a highly stable system, staying within 1% of baseline over hundreds of millions of years when measured in degrees Kelvin. Proposing that there is a positive feedback in a highly stable system is an extraordinary claim that requires extraordinary evidence. It would have to flip at some threshold – an extremely complex relationship – and we haven’t the slightest idea how that would work. Capacitance of the oceans makes it far more complicated yet.

        Not much has worked out. The literature on arctic amplification is so scattershot that I had to go back and find out where the idea came from in the first place. As near as I can tell, it began as an artifact in the first generation of GCMs, and then gradually had theory filled in around it (melting sea ice! methane! etc.).

        Climate science remains in late infancy and will only advance to being a mature field when system stability – an understanding of how NEGATIVE feedback prevents runaway excursions – can be incorporated in models.

        Here is Richard Betts:

        “One thing that strikes me about the scientific literature on “tipping points” is that there are a lot of review papers like this that end up citing the same studies and each other – indeed, my colleagues and I wrote one a while ago. There is a great deal of interesting, insightful research going on using theoretical methods and calculations with large approximations. However, we have yet to see an equivalent level of research in the highly-complex Earth System Models […]”

        • No one is talking about “the oceans boiling away” (it happened on Venus of course). The issue relates to warming that makes things increaingly difficult for our societies to adjust.

          The climate is only a “stable” system because the solar output is relatively constant on the 10-100,000 timescale. It’s ludicrous to suggest that by using a Kelvin scale, extraordinarily dangerous heating (say 3 oC warming) can be magically dismissed. It’s like suggesting that iif you have a body temperature of 3 oC above normal that there’s nothing to worry about because it’s “within 1% of baseline”! It’s a sleight of hand that employs the sort of disreputable graphing that’s been pointed out before on this site (i.e. you can pretend that nothing much is happening over a meaningful temperature range of perhaps 10 oC-40 oC by expanding the y axis to 0K – 350K).

          In any case what’s your evidence that Earth surface temperature has stayed “within 1% of baseline over hundreds of millions of years”? It’s varied by much more that that over the last 500 million years ago. There have been several massive extinction events during that period and mostly these are asociated with massively enhanced greenhouse effects. Temperature rises were much larger than the “within 1% of baseline”

          I also think you’re misusing the term “positive feedback”, I suspect using it in the sense that’s it’s used in engineering. But it’s very well known there are a number of positive feedbacks in the climate syste associated with enhanced greenhouse gas levels. Enhanced CO2 results in warming of the atmosphere which as a result enhances levels of atmospheric water vapour, a greenhouse gas. That’s a positive feedback. Enhanced warming results in loss of sea and land ice with a reduction in albedo so that more solar radiation is absorbed in the Earth system. That’s a positive feedback.

          Feedbacks in the climate system don’t equate to runaway positive feedback in the way that you seem to be misconstruing…although it’s possible that they could (a la Venus). No one considers the latter as a meaningful scenario. John Williams who you’re responding to is lileky referring to potential negative feedbacks (e.g. cloud cover responses that temper the effects of enhanced greenhouse warming) that have been studied in depth and found not to be particularly significant. The only really negative feedback is the temperature-dependent weathering that reduces atmospheric CO2 but which acts on the multi-1000 year timescale.

          bottom line – when people discuss the massive dangers of greatly enhanced greenhouse gas levels and positive feedbacks they are not talking about “runaway excursions” because “positive feedback” in climate research is not equivalent to “positive feedback” in, for example, engineering. Other than the very long time scale weathering, it’s not obvious that there are negative feedbacks in the climate system as the massive temperature rises and mass extinctions of the last 500 million years show us. However even these horrific events don’t involve “runaway excursions” which is pretty much a red herring.

        • The climate is only a “stable” system because the solar output is relatively constant on the 10-100,000 timescale… It’s like suggesting that iif you have a body temperature of 3 oC above normal that there’s nothing to worry about because it’s “within 1% of baseline”!

          I don’t follow this argument. The human body *is* a stable system due to negative feedbacks. In terms of environmental temperature (analogous to solar output here) there is a range of 20-30 C it can adapt to. That doesn’t mean the body can’t still transition to a different stable state (eg, chronic illness or death).*


          I just don’t see how you concluded the first sentence, and it seems a priori implausible. Can you name any complex system besides climate that lacks negative feedbacks? Either natural or manufactured.

          * It is probably better to think of it in terms of chaos theory and attractors, where there are fluctuations around relatively few stable states and many intermediate (unstable) temporary states.

        • Anoneuoid – I agree, the human body is a stable system since it has homeostatic mechanisms involving negative feedback that are part of a “control” system with sensors and responses (e.g. sweating/shivering) that are effective within a range of ambient temperatures like you said. You could even say that above the level of basic physiology there is a level of control that allows accommodation to a larger range of environmental temperature (putting on/taking off clothes; turning up/down the heating etc.)

          Is there evidence that the Earth has such control systems? I don’t think so. There are elements that contribute to surface temperature stability (rather constant solar irradiation; a massive ocean that damps surface temperature variation, absorption of CO2 by the oceans and plants and on longer timescales by sequestration by animal shells and weathering) – but these aren’t like animal body temperature homeostasis.

          Your “fluctuations around relatively few stable states” is pretty good IMO and better than “homeostasis” as a description of the climate system. e.g. ice age cycles show that with redistribution of the spatial patterns of a relatively constant solar irradiation, and including albedo and CO2 positive feedbacks, the Earth’s temperature can transition between different states around which the temperature fluctuates (better maybe to call “positive feedbacks” “amplifiers” to avoid confusion with system control usage).

          The climate system is complex but it’s also bounded by relatively simple input/output energy flows. It’s how the bounded energy distributes itself spatially within the climate system that makes things complex (e.g. to model, as I understand it).

        • Can any of the bourgeoisie experts here tell me: Won’t we soon find out the truth about the strength of the effect of CO2-emissions? What I read online; temperature should increase by about 0.1 degree Celsius every second to third year. Sea level will (possibly) rise about 30 centimeters by 2050 (NASA), which means about 1.5 cm a year! Shouldn’t that by possible to measure accurately pretty soon?

          By my house I have a boat which is attached to a buoy which are attached to the sea floor by a rope (which was cut to be about the same length as the sea depth). This buoy should for example be under water by 2030-2035 (only about 10-15 cm is above the sea surface), it certainly does not look like that is going to happen (it was put out in the early 80s).

          Maybe I am a victim of the merchants of doubt (I doubt it).

        • Savoy –

          The projections you speak of (the aren’t predictions, BTW) are based on averages. Do it seems to me you should expect variation around those averages in different locations (perhaps due to local impact of land or water use, or other influences directly related to climate chamge). Ajd so if I’m right should avoid looking at one potential outcome of the projections in one location and then reverse engineering to judge the accuracy of the projections (especially over relatively short time frames, – even on a decadal scale – as the increase in global temperatures is not projected to be monotonic)

        • Is there evidence that the Earth has such control systems? I don’t think so.

          The evidence is the existence of the climate itself. It seems entirely implausible to me that such a complex system can exist without negative feedbacks.

          I can think of no analagous examples, and that is why your attempt using the human body doesn’t work. So you seem to be assuming something totally unprecedented by default, when the “missing feedback” is easily explained via lack of knowledge.

          Can you name any other complex system that persists without negative feedbacks? Even relatively simple devices like my phone will slow down, turn down brightness, etc if it starts overheating.

          Also, why is it possible to calculate the relative (to Earth) surface temperatures of Venus and Titan knowing only the mass and distance to sun (with zero free parameters)? That is 100% of objects with troposphere and surface, and is evidence of a common mechanism for equilibrium/stability.


        • Comparing the homeostasis of the temperature of the human body to the stability of the temperature of the planet has always seemed a bit, I dunno, maybe teleological is the right word.

          The human body has evolved “negative feedbacks,” as it were, to maintain a certain temperature range, because humans die if they don’t (although I’ll acknowledge that thinking of death as a stable state is an interesting twist). This, evolution comes into play.

          I’m not sure it makes much sense to think of planets as evolving in that way. I mean sure, those planets that don’t maintain stability in some ways don’t exactly live to tell the tale – and thus there may be a kind of survivorship bias – but certainly there are planets that have “survived” without maintaining the kind of climatic stability that we have (thus far) on this planet.

          And further, regarding Matt’s whole “we haven’t changed much so far, how could we changed a lot in the future thesis,” (and aside from the “tipping points = oceans burning up strawman,” as Chris points out), seems to me the whole point of the projections of “tipping points” is that we’re creating a condition that has never before occurred; as I understand it, while co2 levels have been high before, at least in theory they’ve never been juiced like this to rise so quickly – adding a new and anomolous condition

        • There’s a guy who posts a lot about his earth-shattering theory that thunderstorms prove the “homeostasis” of the planet, in other words as adjustments to maintain climatic stability.

          I dunno. We think of human responses to external conditions as “adjustments.” Does it make sense to think that planets “adjust? “

        • Chris wrote:

          ““positive feedback” in climate research is not equivalent to “positive feedback” in, for example, engineering”

          OK. Wow. I have no way of responding to that. Could you please explain what you mean by that and elucidate the differences, because I am genuinely trying to understand what you were thinking when you wrote that.

        • Anoneuoid – you linked to a page describing the use of the Stefan Boltzmann (SB) equation. Isn’t that sufficient to address your points?

          First, what’s the requirement for negative feedbacks? The solar energy reaching the earth’s surface is known/can be measured, and if the earth temperature is to remain at equilibrium it must radiate an equivalent amount of long wave (LW) infrared (IR) energy to space. So the earth, and all the planets, are sitting at their orbital position bathed in solar energy and will come to some equilibrium temperature that can be calculated by the SB equation – i.e. fluctuations around an equilibrium without negative feedbacks.

          It’s a little more complicated since in a planet with an atmosphere that contains greenhouse gases, the long wave IR isn’t emitted from the surface since its path to space is impeded by the greenhouse gases. So on average, LWIR emission occurs from higher up in the atmosphere and since it gets colder as you go up the system warms up, right down to the surface, to bring the height of emission to the temperature given by the SB eq, that allows loss of energy through LWIR radiation to equal incoming solar energy. That’s the greenhouse effect. Given a constant solar energy that’s a stable system and still doesn’t require negative feedbacks – note btw that LWIR emission to space isn’t considered a feedback.

          Another analogy: Take a pan of water and put it on a hot plate at a very, very low heat. The water will warm up and eventually come to equilibrium at a temperature where the warming from the stove is balanced by loss of heat through convection and radiation. Now put a lid on the pan. The water will warm to a new equilibrium temperature – that’s a little like the greenhouse effect. No negative feedbacks required to keep the system at a stable equilibrium in either case.

          Let’s say we consider what’s going on in the pan as “the climate”. We can try to simulate that by parameterizing all the water molecules and how they move and interact and transfer thermal energy and so on. It’s complex, but if our model is good we should find that although there may be temperature gradients and local fluctuations and so on, the system neither gains or loses energy even in the absence of negative feedbacks – like the climate, in whatever manner the energy is distributed by local fluctuations, the overall system neither gains nor loses energy since the total energy is bounded by the balance of net incoming (from the stove/sun) and outgoing (from radiation-convection/LWIR emission) which sets the overall energy of the system.

        • Matt – sure, no problem.

          “Positive feedback” can have different meanings. For example the meaning of the term in electronic engineering: e.g. https://en.wikipedia.org/wiki/Feedback#Positive_feedback

          is quite different from that in climate science. So the increased concentration of water vapour that results from CO2-induced atmospheric warming is considered a “positive feedback”.

          It’s perhaps more descriptive to describe the enhanced water vapour in response to CO2-induced temperature increases as an “amplification” rather than a “positive feedback”. That can be useful in cases where someone may understand the term “positive feedback” to signify a tendency towards a “runaway” effect. I wondered whether you were misconstruing the term as used in climate science when, for example, you said:

          “Proposing that there is a positive feedback in a highly stable system is an extraordinary claim that requires extraordinary evidence”

          But there unquestionably are “positive feedbacks” in the climate system, as indicated in the water vapour/CO2 effect above. They result in a shift in the atmospheric temperature to a new equilibrium but they don’t signify a tendency towards some “runaway” consequences.

        • Joshua: Thanks for replying! I would imagine that extreme sea level rise should be pretty easy to test empirically (in <15 years), given the global projections/predictions. Even with a lot of noise, the signal should be pretty strong.
          I agree that imagining one counterfactual at one particular spot may be idiotic. But I live in a part of the world where sea-level rise is predicted (projected) to be large (10 % higher than the global average).

          Andrew: thanks for that plain speaking advice :)

        • Chris wrote:

          “But there unquestionably are “positive feedbacks” in the climate system, as indicated in the water vapour/CO2 effect above. They result in a shift in the atmospheric temperature to a new equilibrium but they don’t signify a tendency towards some “runaway” consequences.”

          It’s a positive feedback but it doesn’t tend towards “runaway consequences” because why? A new equilibrium? We are not making progress here. Let’s try this:

          At the highest level of extraction, we can imagine that a system – a phenomenon that has input and output and displays any of various traits of stability – can be fully described by a discrete set of control equations. There is no reason to think that the climate of the earth is not a system by this definition.

          At the next lower level of abstraction, the climate can be represented as a semi-stable input-output (I/O) control mass in which the “equilibrium” global temperature is determined by the net difference between the input and output fluxes of thermal energy. This is the thermal budget, and that budget is the basis of our GCMs because why wouldn’t it be? There is no other way. While the physics work better with a control mass, it is easier to imagine the earth as a control volume with an invisible membrane that contains all of the gases that have any influence on climate. This outer boundary, conveniently referred to as the ToA (Top of Atmosphere), would be the place to measure flux.

          At the next lower level of abstraction, we see that greenhouse gases create a capacitance inside our control volume, an ability to store and release energy in the atmosphere in a way that changes the average global temperature over periods of time that are important to humans. Meanwhile the ocean has a much higher capacitance than the atmosphere and so changes much more slowly. This looks like a complicated control equation between temperature and numerous different mechanisms of change, even if we accept %CO2 as the most important one. The control equation for the water cycle – which after all dominates the mid-infrared – must be much more complicated. And then there would be a control equation between CO2 and the water cycle.

          Climate science has made tremendous progress in the last thirty years. But the original claim in this thread, that an increase in %CO2 creates a positive feedback to increasing average global temperature (or at least that is how I interpreted it), delves into an area where we are still trying to understand basic relationships. You can’t just make up stuff about equilibriums and feedbacks when you really don’t have anything more than a vague idea of what these control equations look like. That is the same point that Richard Betts was trying to convey.

    • Ben:

      It’s “merchants” because they had a commercial, not just an ideological, agenda. And, as I wrote in my comment above, there are different kinds of doubt. I distinguish between legitimate reasons for doubting a scientific claim, as compared to spurious reasons for doubt that people make up in order to muddy the waters. The cigarette company example was an extreme case because their own internal research showed that smoking caused cancer but they still raised fake doubts. The Gilbert et al. example is different in that I think they are clueless true believers—but they still have an interest in muddying the waters regarding replication research.

  3. > statistically indistinguishable from 100%.”

    Ok, that’s nuts.

    But that said, I have seen some analyses that question the magnitude of the replication “crisis, ” and I think that “crisis” is largely undefined in this context, and it needs to be. Just throwing “replication crisis” around does potentially add to the credulous skepticism phenomenon you spoke of downstairs.

    • I offer this reference and would welcome feedback from people who can understand the statistics:


      This paper, I think, tries to place the “crisis” into a fuller context.

      Our results should reassure scientists that the scientific enterprise is not in jeopardy, that our understanding of bias in science is improving and that efforts to improve scientific reliability are addressing the right priorities.

      • The problem is people are incentivized to “get results”, then once said results are published it is much easier for later research to be published when it agrees rather than disagrees with the literature.

        There is no way a meta-analysis of *published* results can address this. The problem is the studies that are not published, or even funded, to begin with.

        • Another aspect is what gets included as a replication.

          Say a drug is reported to lower blood sugar in male mice, then the next study uses female mice (it is not novel to repeat the experiment w/ males).

          The result will be considered replicated if results are similar, but not even a replication attempt if results are different.

          Just do direct replications, there is no shortcut.

  4. I think this post shows how “accept uncertainty” can go wrong. Assuming that Andrew is right that the Gilbert et al. paper is junk (I haven’t read it but I’ll take Andrew’s word here), uncertainty is being overstated. But since it’s often the case that uncertainty is understated, outside observers may tend to give Gilbert et al. more credence than deserved. Point being that pushing non-experts to by default increase doubt of scientific claims can have harmful effects, especially if the pendulum swings too far (btw I’m not saying Andrew has advocated for this). I don’t have a solution for this or anything, just a cost to consider for those who do advocate for this position.

  5. Anoneuoid –

    > The problem is people are incentivized to “get results”, then once said results are published it is much easier for later research to be published when it agrees rather than disagrees with the literature.

    If you apply that logic across the board, then you’re going to throw out all the babies with all the bathwater. Everyone is incentivized to be right, to find the truth, to be smart, to be successful, inside and outside of the science publishing process. We’re all hard-wired for confirmation bias.

    Sure, you’re talking about a kind of external magnifier for confirmation bias – leading to exacerbating the problem.

    But what’s the alternative? You can’t find anything to build understanding on if you eliminate all information that’s tainted bt confirmation bias.

    For all the caterwauling about the “replication crisis,” or “perverse incentives” in science, where’s the evidence that we’re worse off in balance? I don’t think we are. And as real as the problems are, where’s the evidence that they’re getting worse in a relative sense. In an absolute sense, we have more bad science than we used to. But we have more science. Is the percentage that’s bad growing? Maybe – but I think there’s a clear trajectory of science contributing to general welfare.

    Binary thinking is sub-optimal.

    • The “alternative” is for direct replications to be standard along with deducing otherwise surprising predictions from your explanations for the data. Ie, use science.

      • direct replications are good but an inefficient way of ensuring science is done properly. Registered Reports have many advantages, including getting feedback from reviewers before a study is done

        • Once you can show registered reports have a replication rate over 50% (yes, that is the very low standard I hope for), then it may be worth considering. I highly doubt any heuristic can replace actually running the replication studies though. Until then, it should not be used as a replacement since there is no evidence it works.

        • I agree with you that direct replications are good (in an ideal world) but inefficient. Also they tend to be done anyway (not completely directly) if a subject is considered important enough. But “not completely directly” I mean in the sense that if someone determines the structure of a Covid protein (say) and another group is interested in identifying drug interactions, they are likely to redo the structure determination perhaps as cocrystals with their drugs of interest. The problems I have with the direct replications are: who decides what’s replicated, and who does the replication, and in any case what does it mean if the study isn’t replicated? It seems more efficient simply to allow the normal course of science to proceed. If something is important it will be followed up and issues found if they exist, if something doesn’t seem important it won’t be and that’s fine.

          Registered Reports are excellent for clinical trials. Would they work for the normal course of science? In effect a grant application is a Registered Report where one’s proposed study is assesed for importance (according to the context of the funding body) and is critiqued by reviewers. I can envisage all sorts of issues with using the idea of Registered Reports for scientific research in general

      • Anoneuoid –

        Are you comparing direct replications to conceptual replications to argue for the former in comparison to the latter? Are you saying that the percentage of conceptual replications is too high relative to direct replications? If so, what is the ratio and why should it be shifted? Seems to me that direct replications would have an inherent problem of spurious results due to sharing confounds.

  6. Tangentially related to this thread (fraud in science, criticism thereof, the response thereto, and pushback from critic (YAY!)) are two letters and an Editor’s Note in the 26 August 2022 issue of Science. The response from Science is gloriously scathing.

    Concerning the senior article of a paper with fraud by the second author Science writes: “She acknowledges the concerns about image manipulation that our story raises but downplays their importance and assumes no responsibility as Lesne’s mentor and senior author of dubious studies.”

    Like I said: “YAY!”.

    • Wow, thanks for the tip.

      The other letter says “By 2008, 2 years after the Lesné paper, Alzheimer’s researchers determined that the findings supporting Ab*56’s role in cognitive decline could not be replicated. Several, including one of us (D.S.), tried (2, 3). Although unaware of malfeasance, the field policed itself by not confirming these findings.”

      Science responds: “[Yet] over the past 5 years, [you] continued to cite it…”

  7. So the “merchants of doubt” are now doubting the doubting of scientific results. And now one has to doubt, the doubting of the doubting of scientific results. Talk about being meta and kafkaesque

  8. Anoneuoid – you linked to a page describing the use of the Stefan Boltzmann (SB) equation. Isn’t that sufficient to address your points?

    Not at all. The problem is the equation does not include constituents of the atmosphere or albedo, which vary greatly across the three objects.

    So how can it work unless either those factors are not very important or somehow adjust to cancel each other out?

    Or it is a coincidence. Maybe, and more data doesn’t appear likely in the near future, but as coincidences accumulate the probability you’ve got a degenerating research programme in your hands increases.

    Either way, that is just supplemental to the main problem: 100% of known complex systems rely on negative feedbacks to persist. So that should be the default assumption, rather than something totally unprecedented.

    • Anoneuoid – ok I we may have to agree not so much to differ but to decide that it’s too much effort to keep pursuing this.

      But I expect that the Stefan-Boltzmann equation works in your example because it can accurately calculate the planetary temperature as measured at the top of the atmosphere (TOA). That’s because by definition outgoing radiation from the top of the atmosphere must balance incoming solar energy. So it would be the temperature measured by a temperature sensor pointed at the planet from space.

      The next step (e.g. determining surface temperature which is what we’re interested in on our pesky home planet) requires knowledge of albedo, atmospheric composition etc. We can use the SB equation to determine what earth’s surface temperature would be without an atmosphere and thus determine the magnitude of the greenhouse effect.

      • The problem is we *can* calculate the surface temperatures without including albedo, etc. With *zero* free parameters, and derived from basic physics. That is the reality. According to your theory that should be impossible.

        But yes, it is pointless to keep repeating the same thing.

        • OK I see what they’ve done. They’ve used the Stefan-Boltzmann eq. to calculate the blackbody temperature of an atmosphereless planet (effectively the temperature at the top of the atmosphere) but then they’ve effectively “put back” the atmosphere by introducing a term for the lapse rate (how the temperature varies with height in the atmosphere as a result of having a greenhouse effect). i.e.

          “The second term of the equation comes from “correcting” the effective/reference temperature to the temperature at the surface. In this case, the reference on earth is being taken at sea level, where (T,P)=(T⊕,1 atm). Here is where the lapse rate condition is important:”

          This is quite tedious btw. It’s a typical blog argumentation, bordering on trolling, where someone links to someones blog and demands that the other person has to read the damned thing and explains some thing that is already completely understood by the relevant scientific community. So I’m going to stop here.

        • OK I see what they’ve done.

          Unfortunately, you clearly still do not. It is fascinating how people can write so many paragraphs without comprehending the most basic point of what they are responding to (“healthy vaccinee effect” was another one that was apparently too confusing).

          I never saw that until covid. If this is the “long covid” people are talking about, we are in trouble.

        • Anoneuoid –

          > “healthy vaccinee effect” was another one…

          It’s noteworthy how often you appeal to your own authority to defend equating opinion and fact.

    • I think you guys are talking past each other when it comes to what ‘positive feedback’ means. I think Anon is using it in the engineering sense and Chris is using it in the sense of ‘amplification’, which is how climate scientists use it. The climate has negative feedback in Anon’s sense but not in Chris’s sense.

      If we stabilize the CO2 in the atmosphere at, say, 450 ppm, the climate will eventually warm up and establish a new quasi-equilibrium that is consistent with that condition. I think both of you agree with that, so I think you are arguing about words rather than concepts.

      This comment will my only attempt to mediate what I see to be a well-intentioned but ultimately silly argument between you.

      • Phil, I sent a link about surface temperature. The title mentions surface temperature, the content is all about surface temperature, and my posts here were all about surface temperature. Then *after writing a dozen paragraphs on the topic*, he claimed the link was not about surface temperature.

        That is not talking past each other. And the other day it was claimed on here that if one group has lower baseline mortality rate than the other, we can’t account for that without knowing exactly why the rate is lower (the CDC’s use of the word “healthy” to mean “lower rate of death” was too much to handle).

        And in another case, it was argued that if you have an event that happens 37 times in one group vs 33 times in the other, we can say *nothing* about the relative rates. As if the uncertainty interval extends to infinity when you calculate a “non-significant” p-value.

        I do believe these are attempts to engage in productive conversation, but marred by an inability to recognize the most basic facts at issue.

        Like if I claimed the title of the blog does not contain the word “Statistical”, you would wonder if I was having a stroke or something. That is not something a rational mind could argue.

        Btw, I think the new OECD data comes out this week. If all cause mortality did indeed start dropping substantially this spring, that will be very interesting.

        • Anoneuoid —

          I asked you the following question just about exactly one year ago:

          > Have you seen any evidence of increased non-covid related mortality among vaccinated as compared to non-vaccinated?


          You didn’t answer then. I’m guessing you won’t answer now, but the question still stands.

          And no, just hand waving at a “healthy vaccinee effect” (particularly without control for other significant, potentially causal variables such as differing behaviors, speculation based on flu vaccines, etc.) won’t really answer the question.

        • Folks, this thread is starting to turn into “garbage time” again, as it typically does when Anoneuoid is involved. Andrew asked us to be cognizant when this occurs and nip the back-and-forth in the bud.
          The best arguments to be made aren’t necessarily associated with the the last argument (or individuals trying to make the last argument) in a back-and-forth (which, as is often the case, is unfortunate for Anoneuoid).

      • Phil wrote:

        “I think you guys are talking past each other when it comes to what ‘positive feedback’ means. I think Anon is using it in the engineering sense and Chris is using it in the sense of ‘amplification’, which is how climate scientists use it. The climate has negative feedback in Anon’s sense but not in Chris’s sense.”

        Positive and negative feedback are terms used for specific phenomena that can be represented mathematically. Climate scientists are using the term to refer to phenomena that do not involve feedback at all, and so are using the term wrong. We don’t get to do what we want with the meaning of words that are already endowed with full conceptual and mathematical meaning. I think lots of folks here would be unhappy if I made up my own definitions of well-established statistical terms.

        Chris isn’t referring to amplification either, at least not in any simple way. The climate interactions that Chris is thinking of are “knock-on effects” because temperature, which is a derived value from stored thermal energy, is a different variable than CO2. Using the terms feedback and amplification would both mean that %CO2 ramps faster than it otherwise would, while climate scientists are really referring to knock-on effects that “amplify” the temperature response. It’s a rhetorical shell game and it is unnecessary and annoying.

        • I, too, get frustrated with poorly chosen wording. For instance, “significant”, whose meaning in statistics is very different from its meaning in plain English, but just near enough to be confusing to many people.

          That said, I haven’t yet been annoyed with the way climate scientists use “feedback.” An increase of CO2 in the atmosphere increases the mean air temperature. Warmer air can hold more water, so we end up with more water vapor too, which leads to an ‘amplification’ compared to CO2 alone. It does not bother me to call this ‘amplification’ or even ‘feedback’. But I do understand why it bothers other people.

      • Yes, thanks Phil, it is silly which is why I dropped out of the “argument” a couple of days ago. But not before writing a post that explains exactly what you did about the fact that “positive feedback” has different meanings in different disciplines and that in climate science it is used to describe an amplification of a temperature response.


        It’s a little weird that anyone could (i) consider that there’s something suspect about a science because a term is used in a different manner than how it’s used in their own discipline, and (ii) not even bother to find out about how the term is being used in the discipline under discussion especially when it’s been explained on a couple of occasions.

        as the man said “The Word Is Not The Thing”!

        This is actually a very common occurrence in science. Right now I’m preparing a revision of a paper where a referee has asked that we define how we’re using the term “scaffold” (we’re using it to describe structurally invariant parts of proteins that support the more mobile elements involved in function). The referee points out that the term “scaffold” has another meaning (there are “scaffold proteins” that have particular roles in cell biology) and that we should clarify our meaning…which we have.

        Since we’re generally trying to communicate (even in blog posts!) we should make at least a rudimentary effort to address potential semantic confusion (unless, of course, we prefer argumentation to communication!)

  9. It’s good to see the progress being made toward acknowledging the really serious institutional change needed in science. Looks like Andrew is engaging this task.
    Psychology may be worse than others, but even more rigorous fields also have a problem. Perhaps the biggest problem in my view is the use of the literature as a personal or technical marketing tool. This leads to a lot of selection bias of positive or good looking results even when there is a wide range of possible results or outcomes.

    Things got really crazy during the pandemic in medical sciences. John Ioannidis had a really good piece in Tablet magazine on this. It turns out that there is a huge industry that uses “science” for political ends. This requires a cartoon version of science in which science is almost omniscient. But the reality is that complex nonlinear systems almost always defy quantitative prediction.


Leave a Reply

Your email address will not be published.