Skip to content
 

The status-reversal heuristic

Awhile ago we came up with the time-reversal heuristic, which was a reaction to the common situation that there’s a noisy study, followed by an unsuccessful replication, but all sorts of people want to take the original claim as the baseline and construct high walls to make it difficult to move away from that claim. The time-reversal heuristic is to imagine the two studies in reverse order: First a large and careful study that finds nothing of interest, then a small noisy replication whose authors fish around in the data and find an unexpected statistically significant result. The idea is to remove the “research incumbency effect” and to consider each study on its own merits.

Recently we discussed something similar, a status-reversal heuristic:

Sometimes I think the world would be a better place if, every time an economist or a journalist saw a published claim by an economist, they were to be told that the research had been performed by a sociologist, or an anthropologist. This would induce in them an appropriate level of skepticism.

Similarly with medical research: Suppose that every time a doctor or a journalist saw a published claim by a M.D., they were told that the research had been done by a nurse, or a social worker. Again, then maybe they’d be appropriately skeptical.

Oooh, I like this game. Here’s another one: Every time you see a paper by a Harvard professor, mentally change the affiliation to State U. Then you’ll be appropriately skeptical.

And, every time you see a paper endorsed by a member of the National Academy of Sciences . . . ummm, I guess that one’s ok, we already know not to believe it!

Arguments against the status-reversal heuristic, and responses to those arguments

I can think of two arguments against. (Maybe you can think of more—that’s what the comments section is for, to explain to me how wrong I am!)

1. It’s kind of weird to see me advocating this status-reversal heuristic, as I have all sorts of status: the Ph.D., the Ivy League faculty position, the access to national media, etc. I’ve even published in the Proceedings of the National Academy of Sciences!

In response, all I can say is: Sure, the status-reversal argument can apply to me too. Feel free to evaluate this post, and other things I write, as if I had no pre-existing status. I think this post, and what we have on this blog more generally, holds up fine under the status-reversal argument.

2. The status-reversal heuristic is anti-Bayesian. Status provides relevant prior information. Sure, some Ivy League professors are blowhards at best and frauds at worst, but lots of us have done good work—indeed, that’s what can get us to the Ivies in the first place. Similarly, if economists have more default credibility than sociologists, maybe there’s a reason for that.

I have two responses here. First, sometimes I don’t think status adds any information at all. For example, if the topic is medical policy, I don’t see why we should expect an M.D. to have a more informed opinion than a nurse or a social worker. Indeed, the M.D. could well be less informed, if he or she has been trained to ignore the opinions of non-M.D.’s. Second, even in areas where status is correlated with expertise, I’m guessing that people have overweighted the prior information associated with that status. So removing that weighting can be a step forward. Third, status can be abused. Here I’m thinking of journals such as New England Journal of Medicine and Lancet that will publish bad research that pushes a political agenda, or PNAS, which publishes, bad research that pushes a particularly set of scientific theories.

My main argument here is the second one I just gave: Even in areas where status is correlated with expertise, I’m guessing that people have overweighted the prior information associated with that status. So removing that weighting can be a step forward. Perhaps this could be studied empirically in some way. I’m not quite sure how, but maybe there’s a way using prediction markets? We could ask Anna Dreber. She doesn’t teach at Harvard, but she’s an economist and she’s coauthored with some famous people, so there’s that.

41 Comments

  1. pwyll says:

    This sounds very related: https://twitter.com/gravity_levity/status/1183952383728062464

    “The question that most intrigued me was whether this “prestige bias” persists in time, or whether it eventually dissipates. Would the way people evaluate me (and others who get to bask in the reflected glow of prestige) eventually settle onto something close to the truth? Or do people base their evaluations heavily on past evaluations, leading to a runaway bias effect in which a “prestigious class” of people looks more and more impressive as times goes on, because of one biased evaluation compounding on another? So I came up with a way of modeling the process of iterated evaluation, using the language of Bayesian inference.”

  2. Matt Pettis says:

    This reminds me of successful authors, like Stephen King and JK Rowling, who have published under pseudonyms. I believe the main reason Rowling did so was to have the pseudonymous work assessed without the baggage of her prior work. She did not want her crime novel judged based on her famous fantasy works. These cases seem to have different environmental pressures from science: in science, you want to leverage your status for more status and hopefully more funding. For some authors, after a certain success, they do not need to add to their success, but want craft-honing or more freedom to diversify that their status prevents.

    http://mentalfloss.com/article/90873/10-authors-who-write-under-different-pen-names

  3. David in NYC says:

    “Status provides relevant prior information. Sure, some Ivy League professors are blowhards at best and frauds at worst….”

    Name the frauds. TIA.

  4. Terry says:

    Some good points.

    Let me suggest a closely-related and simpler line of reasoning. The status of *every* expert should be knocked down a few pegs because we are too quick to believe research results … all research results. Economists should be treated like anthropologists, and anthropologists should be treated like reasonably bright household pets.

    So its not really a reversal, its an additional helping of skepticism applied across the board.

    (You kind of say this near the end of your post, so this might just be a clearer statement of that point. On the other hand, it seems wrong when the post says the weighting on the priors is wrong, rather, it seems that the prior itself is demonstrably wrong.)

  5. Dale Lehman says:

    This is a tricky issue. On one hand, I don’t teach at an Ivy League college so I’m all for rejecting the status awarded to those that do. On the other hand, I have a PhD, and if I am not accorded any sort of deference to my expertise, I feel under-appreciated (by those with “lesser” background). I think we all have feelings such as that. If we put aside our emotional responses, then I like the advice of pretending the “expert” comes from a different background or affiliation – that extra dose of skepticism is warranted, given the abuses of privilege that we have all witnessed. But it doesn’t have to be all or nothing. My attitude is that our credentials should be recognized and given some benefit of the doubt. It is then our responsibility to earn the respect that it confers.

    In other words, if someone teaches at Harvard, or has a medical degree, or is an economist, etc. I am willing to say we should accord them some privilege – the right to be heard as a potential expert or the right to have their research considered as useful. But, once accorded that privilege, it is their responsibility to earn continued respect. If they were to adopt the attitude that criticism from lesser mortals (due to teaching at a State U, for example) is to be discounted, then I think they are well on the way to un-earning that benefit of the doubt. Once they get the stage, their behavior should show humility, gratitude, and they should not behave as an elitist.

    I think this fits with a Bayesian approach. The prior is affected by the pedigree. But I’d put a lot of weight on the new information that their analysis provides – the prior does not provide unearned belief in the validity of their analysis, it only provides the opportunity for them to try to earn respect for their analyses.

    • Maybe it’s just me but I think “teaches at Harvard” is basically going to reduce my own confidence that someone knows what they’re doing. Maximum confidence probably comes from someone who teaches at a second tier type school. Such and such state university of tiny county is likely not the place for top quality researchers, and Harvard or Yale or Stanford are likely the places for too many self-promoting blowhards… UCDavis, UMich, U Georgia, Georgia Tech, Cal Poly SLO, UT Austin, U Washington, U Oregon, U Nebraska or whatever those are places where people are likely to go if they care a lot more about doing good work than they do about getting flashy recognition.

      It’s a kind of inverted U shaped curve… starting at the far left, increasing “prestige” increases reliability, until it hits a peak, and then it falls off as you get towards the top. But I recognize that’s an insider type view, not the typical person-on-the-street view.

      • Chris c says:

        I’m an insider, having been at U Washington, U Michigan, Yale, U Florida. I’ve found the the
        Yale faculty to be excellent. I don’t suppose it’s true in all disciplines.

      • Martha (Smith) says:

        My impression is that there are indeed a lot of people at UT Austin who “care a lot more about doing good work than they do about getting flashy recognition”, but there are also some who care about getting flashy recognition and have low standards (e.g., “TTWWADI”) for what constitutes good work.

      • It’s all about probability, of course at every location there are probably some good researchers and some real flash seekers. But knowing only that someone got to a top tier place in the last 20 yrs you have to assign higher probability to flash seeking. That might not be the case for older faculty…

        Its just like having gotten to 85 years old you have to assume a person takes reasonable care of their health and doesn’t excessively take risks like driving too fast or shooting up heroin… survivorship bias is real, and more intense at the higher tiers.

        • Jeff says:

          There’s probably a large group in the middle: good researchers who are doing competent work that is given undue attention because of a prestigious affiliation. Since prestige and success build upon themselves, I imagine it would be easy for these people to be fooled by their own success into believing that their expertise is broader than it is. (Think Gwyneth Paltrow, Howard Schultz, or Tom Brady.)

          None of this contradicts what you’ve said about probability and the inverted U curve–that all seems right–but it maybe shifts focus from the individual people to the work and how it’s received. I think the heuristic would be less compelling if it just filtered out the “real flash seekers.”

      • Terry says:

        Maximum confidence probably comes from someone who teaches at a second tier type school. Such and such state university of tiny county is likely not the place for top quality researchers, and Harvard or Yale or Stanford are likely the places for too many self-promoting blowhards… UCDavis, UMich, U Georgia, Georgia Tech, Cal Poly SLO, UT Austin, U Washington, U Oregon, U Nebraska or whatever those are places where people are likely to go if they care a lot more about doing good work than they do about getting flashy recognition.

        Disagree.

        I’ve found quality to be pretty monotonic. Middle-tier schools seem to be places where people who did a modest amount of good work once upon a time settle into a sinecure and become administrators with maybe some consulting on the side.

        So I agree with Chris C.

        • You bring up an interesting aspect, which is time-dependence. The longer you are in a place with somewhat lesser reputation, the longer you are wading through high water trying to get support for your research. Most likely early in a career is different from later in a career. People who are mostly interested in doing good work, who are lucky enough to get into high status jobs probably extend that work for longer periods than people at lower status locations who have more resistance due to their lower status.

    • Terry says:

      On the other hand, I have a PhD, and if I am not accorded any sort of deference to my expertise, I feel under-appreciated (by those with “lesser” background).

      Have to be careful here.

      The original post is about skepticism towards new research findings by high-level experts. A lot of skepticism is due because it is just hard to find new and important things.

      But, you are talking about the ignorance of non-trained people versus the basic knowledge of a trained academic. Here, there expert knows far more than the ignorati about things that are well-established scientifically. (Think people asking for antibiotics for the flue.) In that case, expertise is extremely valuable.

      • Dale Lehman says:

        I don’t think you can have it both ways – and that was my point. I’d like to feel justified in being skeptical of those “high-level experts” while at the same time believing my “expertise is extremely valuable.” And, I do feel that way – but I think it is difficult to maintain these two positions legitimately. Not impossible, but it is a bit too convenient to hold both of these. Similarly, I would love to just agree with Daniel’s position – most of my career has been spent near the peak of his “inverted U shaped curve” but that also smacks of confirmation bias.

    • jim says:

      Doesn’t obtaining funding and publishing the results constitutes sufficient recognition for one’s education and position?

      Once the work is done and published it’s researcher skill and knowledge that matter, which don’t directly correlate with credentials.

      We should have skepticism about every scientific paper, no matter the writers, because science is complex and even the most intelligent and thorough researchers can:

      1) overlook implicit / hidden assumptions;
      2) follow lines of reasoning that turn out to be wrong;
      3) misunderstand complex issues;
      4) make calculation and data handling errors
      5) become beguiled by popular attention and/or otherwise lose focus on hard-headed science;
      6) become cornered by political positions;

      I’m sure there are others…

      • Martha (Smith) says:

        I’m not sure what your first two paragraphs are trying to say, but I agree with the rest of your comment.

        • jim says:

          “I’m not sure what your first two paragraphs are trying to say”

          Both are answering to the sentiment Dale Lehman’s expressed, which is I think a widely held sentiment in academia and research, and to some degree justifiably so:

          “On the other hand, I have a PhD, and if I am not accorded any sort of deference to my expertise, I feel under-appreciated”.

          BUT…there is an answer to that view (P1): one might say that an academic’s position, which allows h/her to obtain funding, perform research and thus frame the conservation, constitutes significant if not sufficient deference to h/his expertise.

          AND..so P2: once you start doing the actual work, your position no longer matters. It’s the quality of the work that counts.

  6. Garnett says:

    A few years ago I (and others) published a series of papers about the effects of Sunday alcohol sales on motor vehicle accidents. The alcohol industry hired several economists to publicly critique our work. I always wondered if the decision to hire economists was because they assume that economists could be more easily bought off, or if there was a perception that the public is more inclined to believe economists.

  7. NaN says:

    I re-wrote this because it was too agressive and I don’t mean it to be. I hope I’ve toned it down enough to make the point without exciting rancour. I may have failed, i’m sure someone will tell me. I kinda want it to be understood. Maybe disagreed with but undertstood at least.

    “Status provides relevant prior information.”

    Unreliable at best, I would say for starters. Then go on to say it’s not even especially useful when you have vastly better idicators, indicators contained in the work itself.

    Greatly more important is to consider even if you disagree with that is “at what cost” are you obtaining this “relevant information”.

    You’ve encoded the status quo in your model and with that prejudice against the outsiders in any insider-outsider analysis of it. The lack of women in a field – that’s useful information as to whether women are capable in that field so let’s encode it the model of their capability.

    Are you comfortable with that because I am really not. It strikes me as the same fault that has bigots claiming “it’s not bigotry, there’s a reason for it.” And I don’t think it’s much of an improvement if the “insiders” are selected criteria other than the quality of work being produced even if it isn’t the obviously and egregiously hideous ones of race, gender, sexuality, attractiveness, phrenology …

    I am emphatically accusing nobody of bigotry. I want to emphasise that.

    Good work coming out of Ivy Leauge institutions is good work regardless of its origin. IMHO this is a prior we don’t need and we’re actually a lot better off in a number of dimensions without. Failing to exclude it and all others like it makes me think dark thoughts about bayesian analysis.

    This is a university with high status that is over 700 years old:
    https://en.wikipedia.org/wiki/University_of_Coimbra

    This is a Nobel Prize winning professor from that university:
    https://en.wikipedia.org/wiki/Ant%C3%B3nio_Egas_Moniz

    It really doesn’t get much higher status than that in academia. Do we need to go on?

    • Andrew says:

      Nan:

      Perhaps this was not clear, but my above post is in agreement with your stated position. My proposal to use the status-reversal heuristic is intended to subtract status from the equation and push people toward evaluating work directly. When I wrote, “Status provides relevant prior information,” this was an argument against the status-reversal heuristic, because I think it’s useful when proposing an idea to also explore the arguments against it. I do think that status provides relevant prior information, but (a) this prior information can easily be overwhelmed by data information if, for example, you carefully read an article rather than just looking at the name and impact factor of the journal where it was published, and (b) status-based reasoning has moral hazards, as for example the prestigious journals that abuse their positions.

      • NaN says:

        I didn’t and still don’t see anything you’ve said here about using status as conveying relevant information in your prior as having a considerable status-quo supporting cost (or benefit ymmv). *That* is my position. In addition to that, as I made clear, status is pretty much useless at best in any prior anyway. The latter you seem to agree with. Great. But yeah, this:

        “2. The status-reversal heuristic is anti-Bayesian. Status provides relevant prior information.”

        That rubs me the wrong way for all the reasons I listed of which I didn’t see you mention any, perhaps you did elsewhere? Do you see it? Disagree if you want but please see that for what it is. Women lack status in many fields as just one example of many. Put that as “useful information” in our priors? I’m saying “hell no”. I’m saying that’s garbage. In fact I’m going a little further than that…

        I wonder if you think influential papers, that in fact turn out to be total nonsense, are more or less likely to be published by those with high status.

        P(total nonsense and ifluential | high status) == P(total nonsense and influential | lesser status)

        I listed one of the worst offenders I know in Antonio Moniz. High status propelled brain maiming. “Zero adverse affects from lobotomy” the man said in his nobel acceptance speech, I’m told. Maybe you can match that with some academic work that were low status at time of publication that became as influential and also as thoroughly damaging to people? I haven’t been able to yet myself. “Crossing over with John Edward” perhaps? Comparing the lists in some way might instructive as to whether status reversal is a useful risk mitigation technique for just this kind of thing. Regardless, maybe this conversation has gone as far as it usefully can.

        • Andrew says:

          Nan:

          As noted in the title of my above post, I think the use of status in scientific reasoning causes all sorts of problems. It could well be better if status could be removed from the equation entirely. Given that this is typically impossible, I proposed the status-reversal heuristic as a first shot of bias adjustment. I accompanied my proposal with the best arguments I could think of from the other side. I don’t find those arguments convincing but I wanted to lay them out there.

          Regarding status providing relevant prior information: I think it provides some information. But, as you say, giving credit for status while not accounting for selection bias can create lots of problems. Here’s what I wrote in my above post: “First, sometimes I don’t think status adds any information at all. . . . Second, even in areas where status is correlated with expertise, I’m guessing that people have overweighted the prior information associated with that status.” I guess we could add that the prior information can go in the opposite direction because of selection bias.

  8. John Richters says:

    Meehl’s excellent “Credentialed persons, credentialed knowledge” piece is relevant here.

    Meehl, P E. (1997). Credentialed persons, credentialed knowledge. Clinical Psychology: Science and Practice , 4, 91-98.
    http://meehl.umn.edu/sites/meehl.dl.umn.edu/files/168credentialedpersons.pdf

  9. Peter Dorman says:

    The main argument regarding status reversal is interesting, and I have nothing to add to it. Related to it, however, is this: sometimes status is deliberately invoked to lend credence to a claim, and being an appeal to authority, it should be seen as a negative indicator. The most common case is the book, often in self-help or on the border between science and pop spirituality or politics, that proclaims its author as “so and so, PhD”. Not even a doctorate in a specific, relevant field, just some sort of PhD. Immediately I find myself doubting the contents. Or writeups of research in general interest outlets that play up the name university where the work was done: “the Harvard study”, etc. (For some reason, no one ever referred to work I’ve done as “the Evergreen study”.)

    What I’m saying is that academic status is a warning sign when it is consciously invoked to lend credence to someone’s work. The broader situation, in which the relative status is simply there and induces credibility because of *our* priors/biases is a tougher nut.

  10. Jonathan (another one) says:

    Joseph Henrich, in The Secret of Our Success, argues that we are genetically prone to emulate the teachings of the prestigious. It is by following those with prestige that cultural evolution works, even as it leads us down the path of basketball players making millions of dollars endorsing breakfast cereals. The benefits (on average) greatly exceed the costs, or so he argues. That of course doesn’t tell you what to do in particular circumstances, but as a general rule, the status-reversal heuristic is a bad idea, or at least would have been throughout much of history.

    • Eh, I think it depends a lot on the utility function. If your goal is to be a comfortable upper middle class economically secure person who is respected by a community and “marries well” and has lots of children, then sure.

      On the other hand, if your goal is to understand celestial mechanics, then you should follow Galileo and Copernicus both of whom were ostracized by the high status church, Galileo to the point of being put under house arrest for the remainder of his life and forced to “recant”

      • Jonathan (another one) says:

        But for every Galileo and Copernicus there are N (N=1,000? 10,000?) would-be Galileos whose pioneering spirits led them to outright error (whom we call “cranks”), or who never discovered anything (wasting a lot of effort and time) or to higher truths that they were unable to persuade the world the truth of (the lost prophets.)

        I’m not fully sold on Henrich’s book (though it contains a lot of interesting stuff I never knew before and have no reason to doubt) but the notion that society progresses more through prestige imitation than through inspired reasoning of a few visionaries (and of course it does both) is really well supported. The book is a severe comeuppance to those of us who felt (and feel) that reasoning got us where we are.

  11. Adrian says:

    Well, having an economics background this post hit back home. But I also think that there is no reason as to why priors couldn’t be gradually updated in such a way as to make the heuristic unnecessary.

    As transparency becomes the norm (however slow this process may be), I think using priors will be less necessary and thus the updating of them will be quicker – the priors are necessary at least in part because getting the data for checking what’s up used to be a lot harder in the past. If this information were not provided, I don’t see why wouldn’t a prior of believing higher-ranked fields or schools of origin be all that irrational.

    Of course, this is a great development – making it easy to look at the actual work to judge it and thereby de-emphasizing rank is a good thing, given how uninformative the latter might be in practice. It would probably improve quality across the board, including among those who extract rents based on others’ perception of their quality.

    In other words, this long overdue trend is just making it cheaper to observe actual research quality in a signaling game a la Spence, but of course these perceptions (or prior distributions) change slowly – but they do and will change, particularly in a world in which there is a replication crisis in science.

Leave a Reply