Are statistical nitpickers (e.g., Kaiser Fung and me) getting the way of progress or even serving the forces of evil?

As Ira Glass says, today we have a theme and some variations on this theme.

Statistical nitpickers: Do they cause more harm than good?

I’d like to think we cause more good than harm, but today I want to consider the counter-argument, that, even when we are correct on the technical merits, we statisticians should just pipe down and not interfere with the publicity machine.

Part 1: “How many years do we lose to the air we breathe?”

We recently had a discussion about a Washington Post feature story, “How many years do we lose to the air we breathe?”, which featured the claim, attributed to the University of Chicago’s Energy Policy Institute, that “the average person on Earth would live 2.6 years longer if the air contained none of the deadliest type of pollution.”

I was disturbed because this number seemed to be coming from a discredited analysis from a few years ago, and then Kaiser Fung went into the report and seemed to find the same problem.

The next question is, Why should we care? I mean this question seriously, not cynically. Air pollution is bad, it undoubtedly causes health problems, and it would be good to reduce it. At some point the costs of pollution reduction are greater than the benefits, but I’m pretty sure that in many parts of the world there are some relatively cheap remediations—and even if pollution reduction only saved, on average, 0.26 years of life rather than 2.6 years, much could be done. So, from that standpoint, the “2.6 years” claim is just putting a number on something that we already know.

Part 2: Should we think of early childhood intervention studies as benevolent propaganda?

It’s similar to that iffy claim (see section 2.1 here) that early childhood intervention increases adult earnings by 25% or 42%. Early childhood intervention is good, right? So we can think of this sort of quantitative study—even if flawed, and even if its authors refuse to address these flaws—as a form of benevolent propaganda, in which the qualitative conclusion is more important than its quantitative correctness.

Part 3: Instant power and the utilitarian argument for not being a party pooper

Another example is that study that claimed, “That a person can, by assuming two simple 1-min poses, embody power and instantly become more powerful has real-world, actionable implications,” even though it had no evidence of anyone actually becoming more powerful. Even after the claims in that paper failed to replicate and were retracted by its first author, critics of that study (including Kaiser Fung and me) were characterized as “social media hate mobs” by superstar linguist Steven Pinker. (I guess he won’t be providing any free legal advice to us anytime soon.) Setting aside the whole “hate mobs” thing—I’m just glad he didn’t call us “methodological terrorists”—I think we can extract a positive argument from Pinker’s message, and that goes as follows: Even if “power pose” doesn’t actually work, and even if the published research from 2010 that kicked off that line of study was, if standard practice for its time, in retrospect too noisy and sloppy to give any useful information (again, see here), and even if later attempts to salvage the claim were fatally flawed, still, there’s the potential for some ideas along this line which could be both scientifically interesting and beneficial to people.

In short: by being critical of this admittedly-flawed work, Kaiser and I are dissuading others from pursuing this line of research, and that could have a human cost. In short: why do we have to be such pedants, why focus on the negative, why not just say nice things?

My quick answers are: 1. Opportunity cost, and 2. Division of labor. Opportunity cost is that if researchers are making errors in their design, data collection, analysis, or presentation of results, then this represents resources that could be better used, either by studying these topics more effectively or by studying something else. Division of labor is that Kaiser and I are statisticians so this is something we can do to help.

But that’s just my take on it. In reply, Pinker could argue that social psychology has the potential to benefit the lives of millions and so we should say more nice things about it. He’s implicitly making the utilitarian argument for not being a party pooper.

Part 4: Is it irresponsible to talk about an error, if it distracts the news media from a political priority?

Here’s another example. A year or so ago, someone criticized me online for “scoffing at the Case/Deaton finding about U.S. life expectancy.” Actually I never scoffed at Case/Deaton; I merely pointed out that their claim of declining life expectancy went away after age adjustment, but the real point is that my correspondent’s criticism was not the “scoffing,” if any, but rather that the stagnant-life-expectancy problem was important, and it was important that Case and Deaton raised the alarm, and my pointing out errors in their analysis had a negative social effect by decreasing the sense of alarm about these problems. My correspondent argued that it was fine that I sent my corrections to Case and Deaton and that I published my criticisms in a journal, but not that I posted the criticisms on a blog or that I spoke with journalists who contacted me in response to those blog posts. In those conversations with journalists, I emphasized that I was not disagreeing with Case and Deaton’s main point of stagnant life expectancy, but I did also talk about their technical mistake.

In short: the argument is that statistical nitpickers cause social harm by decreasing public confidence in a published claim. The argument is that even if the published claim has errors, we should either shut up about the errors or talk about them very quietly. (For example, my correspondent did not mind that Case and Deaton spoke to the news media, but he didn’t like that I did so, because he felt that their message had social value, and my criticism would diminish that impact.)

My collaborators and I don’t feel that way—we believe our nitpicking has value, both in a direct sense of giving the public and policymakers a clearer view of what the science currently says, and indirectly in motivating future research and journalists to be more careful about their claims—but it’s important to recognize that this other view is out there, the view that the social value of certain claims is important enough that any critics should speak very softly so as not to diminish their prestige. It’s a utilitarian argument with which I disagree, but it’s a position you can take.

82 thoughts on “Are statistical nitpickers (e.g., Kaiser Fung and me) getting the way of progress or even serving the forces of evil?

  1. Statistics shouldn’t even be that important to your conclusions anyway. It should be a minor aspect that gives you an idea of “are these results consistent with the predictions of my theory?”. There’s always going to be some assumption that is off or some source of systematic error so the exact calculation should be taken as an estimate anyway.

    Also, these people pushing their bizarro science or even fraud as science are destroying a pillar of western civilization, and possibly even dooming the human species. The discovery of cheap energy in the form of coal and oil has yielded a great opportunity, but there is no guarantee that will last. If it is squandered on power pose, etc and other feel good BS (and I don’t just mean in the form of research) there may not be another chance.

    • > Statistics shouldn’t even be that important to your conclusions anyway.

      There’s a great quote for that:

      > “If your experiment needs statistics, you ought to have done a better experiment.” – Ernest Rutherford

      • “If your experiment needs statistics, you ought to have done a better experiment.” – Ernest Rutherford

        That’s hilarious! I’ve been thinking the exact same thing! :)

        But it’s even worse if you
        a) have an error so large you don’t want to talk about it; or
        b) can’t even calculate a realistic error

      • One of my physics professors was fond of saying “if you need a statistics book to analyze your data, then you can take the data and the book and throw both in the trash”.

        Perhaps he was afraid of the fact, oft observed, that once a field comes to rely heavily on statistics, progress shortly stops.

      • Ever since I caught the measurement error bug here I’ve been thinking a lot about it (in my shallow but earnest way). And it seems to me that Rutherford’s comment is exactly wrong when applied to the context of what we usually converse about here – living things and the systems they create. When measuring some physical constant variation is a bug and not a feature of the experiment whereas in living things variation is a, and perhaps the, feature of the experiment. In the former statistics tell you how good is your experiment whereas in the latter statistics tell you how poor is your understanding of your subject; which seems pretty important to me.

        • “whereas in the latter statistics tell you how poor is your understanding of your subject; ”

          I submit that you’ve hit the nail squarely and perfectly on the head.

          To my sensibility, these papers that are problematical from a statistical standpoint, are largely papers that are problematic from an intellectual standpoint as well. Even if power posing, walking slower when you see an old guy, being happier when you are forced into a smile by chewing a pencil were true, they’d be boring and irrelevant to understanding what makes humans and human intelligence so kewl and wonderful*.

          Andrew has gotten close to saying the above when he talks about small effect sizes and interfering effects (they can’t all be significant at the same time and turn into meaningless mush when they co-occur).

          *: I was a Cog.Sci. end of AI type (all-but-thesis from Roger Schank) once upon a time, before I realized I wasn’t finding anything to say for myself.

  2. Your regular readers mostly know this already, but the problem has to come out of education and incentives. Scientists used to be quiet folk that pondered their data in discussions in the common room over tea, then at some obscure symposium, then at some conference, then maybe in a report, then eventually it got written up for a journal. After five years or so of lengthy reviews, it might come out. I’m not saying everything was great, I’m just saying that it has gotten a lot faster. And if it’s not high-impact, then prepare to be replaced with someone less scrupulous. That happens in academia, commerce, governments and NGOs alike. Some jurisdictions (like the UK, praise be) still have some regulatory structures that keep the government number-crunchers independent and protected. In science class, you hear about the romantic heroes who overcame the odds and saved the world single-handedly, and you learn that’s how to behave. If you aim in early career to be fast, and flashy, you’re going to land on your ass. A lot. It seems obvious in skateboarding, why not in science? Maybe because you don’t see the wipeouts quite so publicly. In which case… we do a great service to expose them. I stand by this kind of nitpicking, even though I can’t do it so much any more as an ex-academic.

    But what’s wrong with being a methodological terrorist? What would Chuck D do?

    • Robert said,

      “If you aim in early career to be fast, and flashy, you’re going to land on your ass. A lot. It seems obvious in skateboarding, why not in science? Maybe because you don’t see the wipeouts quite so publicly. In which case… we do a great service to expose them. ”

      Well put!

    • “In science class, you hear about the romantic heroes who overcame the odds and saved the world single-handedly, and you learn that’s how to behave”

      Science’s credibility, created by “quiet folk that pondered their data in discussions in the common room over tea”, has been ripped off by political movements. It’s really a wonder that more scientists aren’t complaining.

    • is this also having a $$$ impact?

      To the extent that “save the world / social justice” memes in science are marketing ploys manifest through journalism, they may also be contributing to a decline in the perceived value of journalism. Why would you pay for a newspaper that’s full of junk (as many health claims made in the newspaper turn out to be)?

      I recognize the web is the main issue in the decline of journalism, but OTOH, on-line subscriptions are really cheap but still very few(?) people buy them.

  3. I really like the clear admission that there are counter arguments, but I’m skeptical that they are the best arguments that can be made.

    I’d add the following:

    1) It’s unclear that you couldn’t have similar impacts by conducting these criticisms not-in-public, by first contacting the author and explaining what the issue is and how it should be corrected, then escalating to the journal / journalist’s ombudsman / university research ethics board, etc., and avoid going public unless unavoidable – without many of the downsides above. (You dismiss this in part 4, but don’t “steelman” the argument being made, which I think is the better approach if you’re actually looking to consider the best arguments against your side.)

    2) We should be fighting to stop the public from taking preliminary research seriously at all, not arguing that specific studies are wrong. Until replicated independently and standing up to strong attempts to disprove the claims, the paper is not “science” as in accepted facts, it’s just the process of science, mostly about preliminary hypothesizing and tentative evidence. The problem with attacking flawed studies is that it sends the message non-flawed studies are fine to cite uncritically.

    3) Scientists that spend time pushing against specific studies would be better off spending their time doing research well, and pushing for more fundamental reforms to journals, such as publication of data and code, (or at a minimum, depositing it with the journal,) independent boards that govern journal retractions, instead of leaving it up to the journal itself.

  4. “benevolent propaganda” is a great phrase.

    “Our nitpicking has value” is even better. We should make t-shirts!

    My collaborators and I don’t feel that way . . . but it’s important to recognize that this other view is out there, the view that the social value of certain claims is important enough that any critics should speak very softly so as not to diminish their prestige

    I sometimes get the sense that at least some of your collaborators feel this way when it comes to certain views, with regard to, say, HBD and criticisms of the consensus on global warming. But maybe I am being unfair . . .

    • “I sometimes get the sense that at least some of your collaborators feel this way when it comes to certain views, with regard to, say, HBD and criticisms of the consensus on global warming. But maybe I am being unfair . . .”

      I think it is a fair point. None of the nitpickers are going to chime in on how ridiculous Michael Mann’s hockey stick was in terms of statistics. So there is some sort of power dynamic involved.

      I certainly have no problem with public criticism of shoddy science. I sometimes feel that statistical approaches form a sort of an iron cage, in that logic structures that show the limitations of statistical approaches are not considered to be important arguments because there are no numbers attached.

    • Maybe I’m old-fashioned, but I’d prefer a t-shirt that says, “The devil is in the details” — it’s merit is that it may have prophylactic effect.

  5. “In short: the argument is that statistical nitpickers cause social harm by decreasing public confidence in a published claim. The argument is that even if the published claim has errors, we should either shut up about the errors or talk about them very quietly.”

    Huh? Is this really even a counter argument? If so, then that’s scary. In other words: stop telling the truth that our conclusions are flawed so that we can deceive the unwitting public for their own benefit, because we know what’s best for everyone.

    Hey, is this science or social engineering?

    • > is this science or social engineering?

      As has come up in a couple discussions around here, I think (and Martha has articulated this point better than I) that the perspective of people who think scientific nitpicking is bad is that the “science” is really just a rhetorical framing device that they can use to promulgate what they *know* to be the “real truth”. So to them, the nitpicking is bad not so much because it undermines their claims—because those claims were never really built on science anyway. Instead, nitpicking is bad because it is a *distraction* from the “real truth”.

      It’s like if you write a beautiful poem but there’s a minor typo on line 5; you’d understandably get frustrated if people focused on the typo because it distracts from the beauty of the poem and, besides, we all knew what the word was supposed to be anyway.

      So given that this seems to be the perspective that the nitpickers are going against, I think it is fair to say that nitpickers, even if they are not impeding progress or serving in the Legions of Evil, might also be missing the point somewhat. In other words, it is worth thinking about how We (as the Forces of Good) can effectively argue against the underlying “science-as-rhetorical-gloss” perspective?

      • it is worth thinking about how We (as the Forces of Good) can effectively argue against the underlying “science-as-rhetorical-gloss” perspective?

        Yes, I really like this phrase “science-as-rhetorical-gloss” because there’s a lot of that going around these days and it really hits the nail on the head as to the problem.

  6. People are making strong claims about the world that are demonstrably false across all of human endeavors, and then when you point it out, you let people call that “Nitpicking?”

    Cambridge dictionary defines nitpicking as “giving too much attention to details that are not important, especially as a way of criticizing”

    At the moment hardly any real science is getting done in many many fields. Go to google image search and look for “worst lice infestation ever” and ask me if those people would think “nitpicking” is “not important”.

    • Daniel:

      The lice analogy is interesting! Literal nitpicking takes a lot of effort and often doesn’t work—just miss one or two eggs and in a week you can be back where you started from. I assume that the better way to control lice infestations is through a population approach. Nitpicking is a necessary part of the solution, but nitpicking alone won’t work so well: even if you do manage to keep your kids’ heads clean, if other parents aren’t doing their jobs, you’re likely to get the infestation back soon enough.

      So the solutions are:
      1. Just accept the lice, or wait it out until lice season ends.
      2. Population control, which includes intensive nitpicking during occasional outbreaks.
      3. Shave your kids’ heads.

      By analogy, here are the three corresponding solutions for science:
      1. Just accept the level of bad science, or wait it out until fads such as embodied cognition, air rage, etc., go away.
      2. Work toward zero tolerance of bad science, throughout the institutions of science (from Science/Nature/Cell all the way to Psychological Science and NPR), which includes intensive nitpicking during occasional outbreaks.
      3. Only accept results from preregistered designs.

      • Literal nitpicking is slow, the better method is permethrin or shaving heads. The better analogy to shaving heads than “preregistered designs” is a scorched earth policy against fake science. Lawsuits, random audits, and jail time for defrauding the federal government for example.

        Unfortunately by the time you’re in that position there’s really no hope for the patient. Academia wouldn’t survive the treatment, and the people in academia who are devoted to real science would be collateral damage. Perhaps a couple of lifetimes of intensive nitpicking and a hope for my eventual grandchildren’s future career possibilities is all that remains.

        As “formereconomist” says below “The bigger problem is that bad science being trumpeted has discredited expertise in the eyes of the people”

        This is as it should be, many “experts” have no credibility. It’s easy to highlight the worst of it, Wansink and Hauser and soforth, but the bigger problem is a disseminated epithelial cancer covering the surface of the whole endeavor, people who want to do well can’t survive because of the toxic blanket all around them.

        My wife used to think I was being melodramatic, and then a guy came to her university and gave a talk recently, he’d published lots of stuff in Science, Nature, Cell, etc over the last few years, gotten big grants, ran a big lab, and it was clear as she listened to his talk and asked questions afterwards that he didn’t have even the most basic knowledge of the background biological facts in the area he was talking about. My wife was flabbergasted, her colleagues were livid… It was an “I told you so” moment that I wish I didn’t have.

        Even when a patient is dying of cancer, the cancer is a small fraction of the mass of their body, but it kills them nevertheless. Fake science is like that more than its like lice.

        • I like the sound of “Lawsuits, random audits, and jail time for defrauding the federal government”. The analogy I draw is tax and company accounts. Exactly these forms of censure apply to small business owners (like me) with regard to tax. It doesn’t discourage us much from running businesses. Sure, some people might avoid self-employment because of it, but we’re probably better off without those cowboys. Let’s apply the same to scientists, who can hurt many more people than tax dodgers. If I feel nervous, I can hire an accountant. If they feel nervous, they can hire a statistician. But we need to make them nervous first.

        • The big problem is defining “bad science” in a way that works properly. The tax code is pretty much bogus. I personally advocate a flat tax + universal basic income (which is actually a *more* “progressive” system than the current “progressive” tax, and yet could be implemented in less than a US Letter size page of tax code). But I think it would be hard to create a “science code” that was worth a damn.

          In principle I like the idea. In practice it’s probably better to have some NGO organization that exists to audit scientists, kind of a Consumer Reports of science… just out the crappy stuff people are doing as unreliable junk in big public reports

        • Lets not forget also giving high ratings to “real” science, it’s not all about nitpicking. People who take the extra effort to make their data publicly downloadable, describe the data collection effectively, publish models that explain their ideas, support the work of others who come up with alternative models, etc should get praise too.

        • You don’t need a UBI, just inflation adjust the income tax brackets so they only apply to the top 5% (as it was when introduced in 1913). No one making under ~$200k per year should even need to file taxes. That has the same end result.

        • Well, that’d be better than what we’ve got perhaps, but UBI + flat tax has many many other benefits, including being an effective substitute for government run benefits programs for people who make no or very little money. If you take away taxes from a person making say 50% of the cost of food and shelter in their region, they still can’t live on their income. They should be able to just-barely squeak by on a well designed UBI + low income. Those with disabling medical conditions who can’t work at all could receive a different UBI so they can get by too all without the govt running any benefits, no means testing, no perverse greater than 100% effective tax rates causing poverty traps, etc etc.

        • Bringing it back to the original point: complicated prescriptive codes always have lots and lots of loopholes, and always come to be “regulatory captured” by the people who have the most resources to put towards their special interest.

          The same would be true of a code which allowed you to sue and audit scientists on the basis of some list of “requirements”. The better method is to have an organization that makes its bread and butter from being right and being transparent and honest about what it’s doing. A “consumer reports” sort of thing, which randomly selects studies to “audit” and then provides a detailed analysis of what was paid for, what was done, what data came out of it, what the conclusions were, what alternative conclusions might be, how cost effective this research was, what the follow on research might be, who stands to benefit, who stands to lose, etc etc.

          each situation will be unique and you couldn’t write a bureaucratically enforceable “science code” that was worth a damn.

        • One benefit to having everyone pay taxes even if a very small amount is that everyone feels like they are in the same boat and everyone is contributing. If only five percent pay taxes you get the double whammy of exacerbating class war while making 95% feel worthless and parasitic.

          Many relatively poor people give to charity. They don’t want to feel poor.

        • If you take away taxes from a person making say 50% of the cost of food and shelter in their region, they still can’t live on their income.

          What percent of the population is in this position? Lets say $8 per hr for 40 hrs a week. That is $1280 a month. You can live fine in a nice place for $400 a month (with roommates), spend $100 a week on food, and still have $480 a month left over.

          As long as you didn’t make dumb decisions like take out student loans for a non-marketable degree or have kids without being able to afford it you should be fine in the US. Now, there are also the mentally ill who literally cannot have a job. Giving them a UBI is not the right solution, because they are not going to make good decisions with that money.

          a different UBI

          Once you have different UBIs the bureaucracy is back. The point would be to give it with no questions asked.

        • One benefit to having everyone pay taxes even if a very small amount is that everyone feels like they are in the same boat and everyone is contributing.

          Yes, it would be quite rude to people to not tax them. That would be offensive.

        • The solution for the mentally ill is to appoint a family member or friend as conservator or trustee to oversee the funds. this is far better than the govt doing the decision making.

          some level of bureaucracy is inevitable. a level at which we assess people who claim disabling physical and mental conditions and add additional funding on that basis is way way better than for example running special hospitals or offering people special care options and hiring special government workers to deliver food for housebound people and etc etc.

          but yes the baseline level of UBI should be automatic

        • The solution for the mentally ill is to appoint a family member or friend as conservator or trustee to oversee the funds. this is far better than the govt doing the decision making.

          And when they drive away all their family and friends (for the same reason they can’t hold a job)? But I was more thinking these are the people you think are “making say 50% of the cost of food and shelter in their region”. Is the UBI supposed to be designed primarily for people who really can’t work?

        • It’s not designed primarily for people who can’t work, but it can do double duty for them. There are many more people with physical ailments than severely mentally ill people. my sister is a psych person so I’m aware of the difficulties. Severe mentally ill people are a very difficult issue. Currently they are extremely expensive as they tend to wind up in the ER at enormous costs. It can easily be 300k a year or whatnot. Even appointing professional case managers as conservators to manage UBI funds would be cheaper than what we do today. A case manager could handle say 5 patients at cost of $100k/yr

          But I think this is a tangent. the fact that UBI is an idea that could replace welfare for all but the most seriously mentally ill is a distraction. What it really does is stabilize people and keep them from needing specialized services.

          I have the ACS data so I could look and answer your question, but it’s something like 25-30 percent of the population that has Zero income from wages

        • There are many people who would lose 30 or 40k worth of benefits if they got a job. When I was a child my mother refused raises multiple times because getting them would have made her ineligible for child care and she would have had to quit her job. The biggest reason for UBI is it eliminates this kind of perverse structure which affects just about any family making less than about the median household income which is around 55k. So basically something like half the countries families with children are trapped in a perverse system that holds them on the edge of poverty with a big “energy gap” they have to leap to get ahead. If you can make about 75k with a single earner your family escapes, but that’s substantially more than median household income. and most households above that have two earners.

          Fertility rate is at an all time low during my lifetime in the US, there’s a lot of reasons but financial traps are a big part.

        • There are many people who would lose 30 or 40k worth of benefits if they got a job

          Well, those “benefits” are far more than enough for someone to live what I would frankly consider a more than comfortable life in my experience, even extravagant. Perhaps what is happening is that people are either taking on responsibilities they should not under our current economic system and/or giving their money way to various scams like health insurance, etc.

        • :) something you wrote a while back got me thinking about it. Maybe it is a good idea even if it’s not likely to happen. It certainly would reduce the cost of government. I still think though that there are real economic benefits from certain kinds of tax incentives, like corporate tax breaks for research and personal tax breaks for some things.

        • One of the virtues of thinking about ideas like this is it provides us with a good point of reference.

          Note that it’s still possible to tax certain consumption, like CO2 taxes or etc. This is the inverse of “tax breaks” and its far far easier to administer special taxes on the purchase of things than it is to offer tax breaks. It’s very similar to the difference between “mail in rebates” which are like tax breaks, and “in store markdowns” which are basically overhead free for decision makers.

        • Yes, so based on those graphs for example, what we have right now is *effectively* an almost flat tax already, around 28% or so… If we just made an actually 28% flat tax, plus say a 10% sales tax, and funded a UBI while eliminating much of the most complicated to administer federal welfare programs, such as food stamps, EIC, housing programs, etc we could dramatically lower the complexity of the federal government while dramatically improving outcomes for most people…

        • To complete my domestic policy agenda, we create a medical billing clearinghouse, similar to the Federal Reserve bank’s checking clearinghouse. All bills go to the clearinghouse identified by unique subscriber ID (I’d argue it should be a UUID rather than SSN but whatever), then private insurers download this data weekly and pay their component, and provide a single monthly bill to their subscribers to pay the coinsurance/copays.

          Finally, we provide a “stop loss” insurance policy from the federal govt for all citizens and permanent residents. We calculate the mean cost of a single hospitalization in each given year, and the federal government pays 100% of the costs for any person past this stop loss point. Anyone who breaks the stop loss barrier 3 years in a row has 100% of their medical costs covered until they drop below that stop loss point for 3 years in a row, making administering the seriously chronically ill very simple, and removing their complexity from the private insurance pricing.

          Private insurance becomes just coverage between 0 and the stop loss costs, with all subscribers guaranteed to have less than 100% utilization through time (due to the chronic ill rule above). These days mean cost of hospitalization is something like $10000 or thereabouts. So we’re talking about private insurance covering up to and no more than $10,000 / person / yr. Since only about 8% of people have hospitalizations in a given year, and hospitalizations are the largest cause of medical expenses, we should be able to have private health care coverage for everyone for a cost on the order of $80/mo/person. Even if my calculation is off by a factor of 2 it’s still only $160/mo or $640 for a family of 4 at the upper end… that’s not that far from just my family’s portion of our health insurance, not including the substantial contribution from my wife’s employer.

          So, yeah, if someone wants to appoint me god for a day, I’ll solve the US’s domestic problems in about a half a day and 10 pages of paper to write the regulations, and the net result will be better for literally *everyone* including the ultra poor, middle class, and super-rich because we’ll eliminate some tremendous quantity of overhead

          sigh…

        • Think of the savings just from the 1.2 million “accountants and auditors” currently earning a mean wage of $78k/yr. That’s 94 billion dollars a year to pay people to comply with complex regulations. Certainly we could cut that in half with a flat tax scheme. So we’re talking maybe a $50 B/yr savings right there alone.

          Then there’s something like 150000 people employed in medical billing earning something like $40k/yr on average (that doesn’t sound right at all, but it’s what BLS says). So that’s $6B/yr that we’d have to be able to cut in half easily…

          I’m sure a bigger savings will come from cutting the staff of medical insurance companies easily in half… Apparently there are something like 3 million insurance company employees and if you assume say a typical wage of $50,000 that’s $150B/yr so another 50 or 70B savings is likely.

          Put it all together and just the savings from not having people do stupid things that don’t need to get done in the taxation and medical insurance and billing and associated fields would probably cover about 1/3 of the US military budget

  7. The bigger problem is that bad science being trumpeted has discredited expertise in the eyes of the people. When people see outlandish claims given the endorsement of “it’s scientific, you can’t disagree with the experts,” they start to lose faith in the whole system of science, experts and research. How many people disbelieve climate change because they see errors in other fields?

    • Climate science is not as easy to ridicule because publications in the field are not as accessible as for example power pose papers, but it is past time for some methodological terrorists to take a look at the field.

      • Yes. My bet is that there’s an enormous amount of data fudging there. But it ain’t in the direction you think. I’d guess that the blokes doing the work realize that things are incredibly bad, but no one would believe them if they said how bad they really are. So they fudge the numbers down.

        With the Greenland ice sheet (2.0 kilometers thick by an area of 1.7 million square kilometers) melting faster (or getting qualitatively weirder) every time someone looks, and airline travel going up 10% a year, every year, for all but 2 of the last 20 or so years, I’m expecting to see major climate change caused disruptions in my lifetime, even though I’m already “elderly”. As arctic snow and ice go away, more solar heating occurs. If the Siberian tundra melts, tons and tons of methane will be released. Americans can’t live without a car (I’ve only owned a car for 1/67 of my life). I could go on. Basically, we make lemmings look rational.

        • What evidence made you come to your conclusion rationally? I assume it was not from journalists or pop science writers, since we all know how well they understand science.

          Seriously though, I would appreciate it if you can point me to a well done systematic review on this.

        • That shows a rise of ~10 cm over 40 years (0.25 cm/yr). If we just use the most naive guess possible it would be that the rise continues at the same rate on average it has been since the LGM:

          According to Blue Marble 3000 (a video by the Zurich University of Applied Sciences), the average global temperature around 19,000 BC (about 21,000 years ago) was 9.0 °C (48.2 °F).[7] This is about 6.0 °C (10.8°F) colder than the 2013-2017 average.

          According to the United States Geological Survey (USGS), permanent summer ice covered about 8% of Earth’s surface and 25% of the land area during the last glacial maximum.[8] The USGS also states that sea level was about 125 meters (410 feet) lower than in present times (2012).[8]

          https://en.wikipedia.org/wiki/Last_Glacial_Maximum

          So we would use a rate of 12.5k cm over 21k years (0.6 cm/yr), and guess a rise of 24 cm since 1970 (vs the 10 cm observed). So sea levels are rising at about half the historical rate…

          And we know of vast land areas that flooded due to this rise:
          https://en.wikipedia.org/wiki/Doggerland
          https://en.wikipedia.org/wiki/Sundaland

          So the climate change is ongoing and we should expect much worse than what the IPCC predicts, but I don’t see any reason their solution (reducing CO2 emissions) would do anything to prevent this.

        • And even rounding up to 1 cm/yr (1 m/century) isn’t very threatening. Eg, last century people needed to raise the levees all along the Mississippi 3+ meters just due to other problems:

          At St. Louis, the 1993 Mississippi flood peaked at a stage of 49 feet. In ~1927, the same volume of water at St. Louis would have reached only 39 feet. In the 66 years between the 1927 and 1993 floods, the cumulative changes to the Mississippi channel – including continued levee building – increased flood stage by 10 feet. At Chester, Illinois, the difference is 16 feet. In short, levees increase flood heights.

          http://all-geo.org/highlyallochthonous/2011/05/levees-and-the-illusion-of-flood-control/

          A rise of ~300 cm in 66 years works out to about 4.5 cm per year, over 10x what the IPCC is so worried about… Basically, all this climate change fuss is about rounding errors to what we should already be planning for, and historically far worse problems have been dealt with largely successfully (using less advanced technology and more expensive energy).

        • Reading Science (e.g. articles on Greenland) and being horrified by the insane amount of fossil fuel we burn. The idea that this is a minor problem that can be fixed by reducing consumption a few percent a year is clearly nuts, especially in the context where almost no one is actually reducing energy consumption and/or carbon emissions. (There was some good news for a bit there when China recognized that they had a problem with air pollution in Beijing and claimed to be cutting back on new coal, but the latest news is that they’re actually completing several major coal projects that they said they had cancelled. Japan is burning coal instead of nuclear now that the populace is largely convinced that nuclear is a bad idea (the nominally left-wing newspaper here has anti-nuclear (explicitly or in effect) articles almost every day. But doesn’t mention that they’re burning coal*, since LNG is too expensive by the time it gets here. (An MIT grad friend here is working on a project to pipe Siberian NG to Japan without liquifying it, which would get Japan back to being merely somewhat worse that it was before Fukushima rather than much worse.))

          Replacing a reasonably efficient gasoline car with an electric car costs similar amounts of energy burned in making the car than would be saved by running it on solar, but when you run it, you run it on electricity that’s 40% coal.

          *: Japan talks about “clean coal”, which is a joke itself, but only has one experimental plant.

        • David said,
          “Replacing a reasonably efficient gasoline car with an electric car costs similar amounts of energy burned in making the car than would be saved by running it on solar, but when you run it, you run it on electricity that’s 40% coal.”

          Yes, if you run it on electricity from the grid — but if you run it on electricity from your own solar collectors (as my sister does), that’s another matter.

        • Well, yes. But the energy cost of making that car was steep*. Wind and solar installations aren’t carbon-free to create either. I think nuclear is critical, since it’s the a base power source that’s zero emissions, but it’s not completely zero emissions if you count the cost of building the plants in the first place (also, decommissioning isn’t zero emissions). Which is why we should, at the minimum, run all existing nuclear plants as long as possible. (Nuclear is especially important in Japan, since the sun goes away for weeks at a time here (the spring rainy season) and there’s nothing like the US Midwest wind resources.)

          Basically, we’re not doing anywhere near enough things that actually make a difference. We need to drive 10% fewer miles every year, we need to fly 10% fewer miles every year. That means someone has to actually drive less, someone has to go to fewer conferences, someone has to see their family less often. But no one’s doing that. No one’s saying “I don’t need a car, I’m going to live without one.”

          *: Sure, it’s the same as making a non-electric car and if you “have to” own a car, it might as well be electric. But the bottom line is that the only thing that will actually help is to be part of this years’ 10% of car owners who figure out how to live without a car. The funny thing is, the private car is an insane idea: they’re dangerous (second only to opioids nowadays), expensive, and a horrific sink of time compared to public transportation.

        • Yes. My bet is that there’s an enormous amount of data fudging there. But it ain’t in the direction you think. I’d guess that the blokes doing the work realize that things are incredibly bad, but no one would believe them if they said how bad they really are. So they fudge the numbers down.

          .

          Really?

          The National Climate Assessment seems to consistently do the opposite, i.e., exaggerate the numbers upwards. The NCA recently released a set of charts and graphs “explaining” how bad climate change was. Each chart and graph had a different, cherry-picked time period that made things look as bad as possible.

          See: https://www.powerlineblog.com/archives/2019/09/exposing-junk-climate-science.php

          Watch the linked 12-minute video if you can. I don’t see any evidence they are fudging the numbers downward.

          Caveat: the narrator in the video does a couple of things that struck me as incorrect.

  8. You write on a blog, which is not normally considered “social media”. Social media includes Facebook, Twitter, Instagram, Tumblr, etc. You yourself have noted the shortcomings of twitter vs blogs. One of the common complaints about, say twitter, is that the “social” aspect makes mobbing more likely and people are particularly activated by negativity.

  9. As usual, very perceptive. The problem is getting worse, and will get much worse – because reviewing science is not scalable but producing science is becoming much more scalable. In data science/Big Data, with all the tools coming on line, the cloud services, deepfakes, and so on, it takes little effort to generate “science”. Critiquing published work requires time and effort – how can we keep up with the pace? How do we recruit more people to become critics?

    Years ago, I re-visualized a chart used by climate scientists to draw attention to the methane problem. Some were very upset because supposedly by criticizing the chart, I was abetting the “enemy”. But I thought I was trying to help, since I believe the improved chart would tell their story to more people.

  10. Why be a stickler for accuracy, transparency and honesty about how data ought to be interpreted? I think there are two reasons in addition to the two adduced by AG:

    1. The ideal of science as a mode of human inquiry that distinguishes what we know very well and can rely upon from other things we might envision or believe in is important beyond reckoning. Science is obviously only one mode of learning, and it has definite shortcomings, but by privileging knowledge that passes stringent tests it is a uniquely progressive realm of activity. Today’s science is nearly always better than yesterday’s because of this—something you can’t say about today’s art, romance or political practice. It’s fine for people to speculate on power poses or whatever else interests them, but they should not offer it as science unless it is painstaking in its attention to reliability. There will always be reasons for some people to try to appropriate the imprimatur of science for ideas they are invested in and want to promulgate, and so there will always be a need for gatekeepers to try to block unjustified appropriation.

    2. As AG says, research (including statistics) is hard. We are all of us in a learning mode no matter how much we already know. Informed critique, if it is presented civilly, is a benefit to anyone who wants to do their work better. And the beneficiary of the critique is not just its immediate target, who may be too invested to take it in, but also all the researchers working on related topics or using related methods. I suspect a lot of work in social psych these days is more reliable because of the well-publicized criticisms targeting a handful of egregious cases.

  11. Aren’t they saying ‘don’t nitpick studies that I agree with’? If they mean ‘don’t nitpick studies I disagree with’, that would be interesting, but I have trouble believing anyone means ‘accept the other side’s bad arguments’. If they think like people usually do, meaning they want you to nitpick things they disagree with, that’s defensible on a human level but it’s not science and they shouldn’t pretend it is. If they mean reduce standards and let the marketplace sort it out, then maybe one can argue they have a point. It’s a weird point but at least it’s an argument to say marginal work, misleading work, crappy work should be accepted by whomever wants to accept it, even by scientists whose jobs and ethics require them not to do so. Maybe then you can stratify into popular media and science, but that requires over-riding the ethical standards science imposes at the scientist level. I note that because media standards don’t really exist (except when idealized or in self-puffery), so you could stratify by ethics. That would be weird.

    • Jonathan:

      I think what you’re saying is part of it but not the whole story. Some of the anti-nitpickers’ position is pure proceduralism: if a paper has been published in a peer-reviewed journal then it can’t be criticized except perhaps by personal friends of the authors. But another part of it, I think, is that they really believe their theories, and they don’t want to recognize that even if a theory has virtues, it can still be associated with bad experiments. For example Steven Pinker is a fan of evolutionary psychology, so he has sympathy for anything that can be vaguely tied to evolutionary psychology, including some experimental claims that don’t replicate and which, for statistical reasons, should not be trusted in the first place. And then there’s his natural inclination to support science that comes with the Harvard brand. Put these together, and he’ll think of nitpicking as missing the point: after all, if these theories are true, who cares about the details of the data?

      • First, let me say that I am in favor of so-called “nit-picking”. It is truly shocking to see how poorly people understand science/statistics, and as a result, how much poor research gets published, cited, and used as a guide for policy, treatment, etc…

        That said, it seems to me that many “anti-nitpickers” are assuming only type M errors, and no type S errors.

        I have to say that *IF* this were true (and I don’t believe it is), I’d have some sympathy for their argument, unless the type M errors were *HUGE*. In my area of work, we are always just hoping to be in the right magnitude ballpark, and avoid sign errors. Maybe I’m setting the bar too low, but I’m generally happy with studies that (I hope) achieve that.

        • 100% agree. When people use the term “directionally correct”, they are explicitly assuming no type S error… but in most cases, when “directionally correct” is used, it’s because the data are barely informative, not that they have proven confidence in the direction!

  12. Air pollution is bad, it undoubtedly causes health problems, and it would be good to reduce it. … even if pollution reduction only saved, on average, 0.26 years of life rather than 2.6 years, much could be done.

    This raises some questions.

    (1) Should everyone inflate their results by a factor of 10? If some do and some don’t, won’t we get our priorities wrong?

    (2) If we already “undoubtedly know” it is worthwhile to do something about problem x, why are we doing more studies?

    (3) How do we know it is worthwhile to do something about x? How do we know we aren’t relying on faulty studies by scientists who manufacture bogus results using the logic quoted above. Perhaps x isn’t really a problem at all. If 10 people are lying about problem x, each will believe their lie is “good” because, after all, “there is a consensus” among the other 9 liars.

    (4) Should we also suppress results that contradict what we “undoubtedly know”? The logic above says yes. So … if we are inflating confirmatory results and suppressing contradictory results, isn’t that like … really bad?

  13. All good points in the article and comments.

    Taxpayers fund academics to produce:
    – rubbish
    – minute incremental improvements on existing work (if you want to keep your academic job you have to publish lots of papers)
    – occasional quite useful stuff
    – very occasionally ground breaking work

    Imagine the howls of agony if a government cut the funding going to academia by 80%. Imagine the insults that would be hurled. Being familiar with a few fields I don’t imagine this would cause any detriment to world progress.

    If individuals, companies, activist organizations.. want to fund useless science they should. Without reducing government funding the outpouring of rubbish and useless studies will continue. And so will the spruikers.

  14. The criticisms of your debunking of bogus studies, once the public at large hears about them, can only convince people to pay less attention to any “scientific” study, since it will become obvious that many scientists have a political agenda, and even those who don’t might have a career price to pay when they publish politically inconvenient things.

    But maybe it is a good thing that the public find out. People in the science cult loudly proclaim that “science” is something that everybody should accept without question. But in reality they are insisting that *scientists’* proclamations should be accepted without question.

  15. Much of the argument breaks down when the discussion turns (or arguably should turn) on magnitude issues. Eg vaping is superficially similar to smoking, but doesn’t have a remotely similar risk profile. Being unable to get an accurate discussion on that is a problem for something that so many people seem to want to do.

Leave a Reply to Robert Grant Cancel reply

Your email address will not be published. Required fields are marked *