John “not Jaws” Williams writes:
Here is another example of how hard it is to get erroneous publications corrected, this time from the climatology literature, and how poorly peer review can work.
From the linked article, by Gavin Schmidt:
Back in March 2022, Nicola Scafetta published a short paper in Geophysical Research Letters (GRL) . . . We (me, Gareth Jones and John Kennedy) wrote a note up within a couple of days pointing out how wrongheaded the reasoning was and how the results did not stand up to scrutiny. . . .
After some back and forth on how exactly this would work (including updating the GRL website to accept comments), we reformatted our note as a comment, and submitted it formally on December 12, 2022. We were assured from the editor-in-chief and publications manager that this would be a ‘streamlined’ and ‘timely’ review process. With respect to our comment, that appeared to be the case: It was reviewed, received minor comments, was resubmitted, and accepted on January 28, 2023. But there it sat for 7 months! . . .
The issue was that the GRL editors wanted to have both the comment and a reply appear together. However, the reply had to pass peer review as well, and that seems to have been a bit of a bottleneck. But while the reply wasn’t being accepted, our comment sat in limbo. Indeed, the situation inadvertently gives the criticized author(s) an effective delaying tactic since, as long as a reply is promised but not delivered, the comment doesn’t see the light of day. . . .
All in all, it took 17 months, two separate processes, and dozens of emails, who knows how much internal deliberation, for an official comment to get into the journal pointing issues that were obvious immediately the paper came out. . . .
The odd thing about how long this has taken is that the substance of the comment was produced extremely quickly (a few days) because the errors in the original paper were both commonplace and easily demonstrated. The time, instead, has been entirely taken up by the process itself. . . .
Schmidt also asks a good question:
Why bother? . . . Why do we need to correct the scientific record in formal ways when we have abundant blogs, PubPeer, and social media, to get the message out?
His answer:
Since journals remain extremely reluctant to point to third party commentary on their published papers, going through the journals’ own process seems like it’s the only way to get a comment or criticism noticed by the people who are reading the original article.
Good point. I’m glad that there are people like Schmidt and his collaborators who go to the trouble to correct the public record. I do this from time to time, but mostly I don’t like the stress of dealing with the journals so I’ll just post things here.
My reaction
This story did not surprise me. I’ve heard it a million times, and it’s often happened to me, which is why I once wrote an article called It’s too hard to publish criticisms and obtain data for replication.
Journal editors mostly hate to go back and revise anything. They’re doing volunteer work, and they’re usually in it because they want to publish new and exciting work. Replications, corrections, etc., that’s all seen as boooooring.
With journals, it’s all about the wedding, never about the marriage.
It’s not just journals. Two weeks ago a Washington consulting firm (really one individual with lots of political connections) put out a “report” that was full of errors and quite misleading. The headline was picked up by the national press (Forbes, CBS, NY Times, etc.). I was familiar with the data and quickly wrote a note describing the problems with the report. I sent it to the person that produced the report, the CBS author, and one or two other contacts I could find (I couldn’t find an email address for the NY Times reporter, which I believe Andrew has mentioned as a problem in the past). I got no responses. I finally found one online publication that was receptive to my point and quickly published my comment as an op ed.
Peer reviewed publications are slow and not very receptive. The media is quick – too quick, so they rarely look back (it’s old news). In both cases, it is hard to correct the record or even invite substantive discussion. The level of discourse seems to be degrading rapidly.
Dale:
Can you give a link to this story?
Here is the original report: https://www.theheagroup.com/blog/ensuring-a-living-wage-through-higher-education. The Press page will link to some of the stories in the media. Here is my op ed: https://www.diverseeducation.com/opinion/article/15665839/must-headlines-deceive.
That’s a great op-ed.
I don’t understand why the mainstream media haven’t run pieces that exactly contradict their previous ones. They do t have to do them as retractions, they could just report your findings as a new story. The news cycle is short, this could be a new story.
I fail to understand how this could be actually meaningful to begin with, rather than just sensationalistic. The only meaningful comparison, it seems to me, would necessarily have to have at least some measure of adjustment for the various attributes of the students (as you basically pointed out in your op-ed).
I’m increasingly unsettled by the general level of analysis I see on topics that overlap with politics and ideology. And as bad as it might be in “mainstream media” (whatever that means), it’s just far worse on Twitter and the podcast-o-sphere, and other social media.
Maybe it’s getting worse, or maybe I’m just getting old.
I provided the links but they appear stuck in moderation for the time being.
Dale:
You did a great job detailing out the numbers. I agree that training programs (“beauty college”, auto repair, phlebotomy) don’t qualify as “education” and should not be treated as “college”; and that comparing US states to Puerto Rico is bogus.
Just the same I don’t think it’s appropriate to exclude 2-year degree institutions, public or private, from the analysis, nor to exclude “tribal” schools. Both public community colleges and private and/or technical colleges are selling themselves as “college” – and they are selling their degree programs as a ticket to higher earnings. I suspect most students in these programs come from families where no one has more than HS education so: a) to them these programs are real “college” b) it’s important to inform them that these programs aren’t getting a return for their money.
This is a great problem to show how statistics can be misleading no matter how you cut them ideologically. The reason the media lapped this up so fast is that it’s a common belief in one of the lefty media’s core constituencies – poor youth – that education isn’t delivering on its promise of higher earnings. That’s one reason you see these movements for free community college in so many areas.
Joshua:
I’m unsettled by almost all the statistical analysis I see. Most statistical analysis is garbage – but even when the analysis is half-way decent, the results are wildly overhyped. But the left got what it wanted: statistical analysis has been democratized. Thousands of Righteous Dons have been unleashed and have raced out to heroically joust windmills. Anyone can download R, Python, SQL, or even just excel and make a “discovery” and plot it up in oh-so-beautiful graphs and pedal it to the meme-hungry media.
And we haven’t even peaked yet! Claude, Bard and GPT have only just come on the scene! Now you barely have to be literate to conjure up a bunch of sweet-looking BS graphics and memes: “Claude, can you find some statistics that make [my political enemies ] look like criminal(s) and email them to some credulous people in the media? Thanks buddy!”
Dale, to further illustrate the point about complexity of hte issue:
As we discussed previously your point about Puerto Rico could just as well apply to many places in the US: while CA now has a $20hr min for fast food workers, many states still have the Federal min wage of $7.25. It’s concievable that a college graduate in some areas could be making less than the min wage in CA – and still doing reasonably well for themselves by local standards. So as I’ve suggested several times, national average wages are meaningless. One can’t hope to find any real relationships between wages, wealth, or people’s personal financial conditions without controlling for location and local cost of living – in addition to other factors tht influence wages.
Regarding the types of schools, the appropriate approach to such an analysis would be to group the schools by type and region, rather than lump them all together as “college education”, or exclude some types of schools because they are deemed to not be “college” education.
Chipmunk –
I await the day that one of your comments doesn’t contain a gratuitous shot at “the left,” or “the media” or some other vaguely assembled Frankenstein of “others” you’ve stitched together.
+1
Joshua:
Thanks for addressing the weaknesses of my primary argument. You did a great job with that.
Chipmunk –
Write a comment with an actual argument rather than just primarily trying to own libz, and I’ll be happy to respond on point
Once again we get a he said/he said story about science that completely ignores the science. Does anyone here even care about what Scafetta claimed and what Schmidt refuted?
Because if you do, you should read the science, not the breathless journalism. Start with what Schmidt wrote:
https://www.realclimate.org/index.php/archives/2022/03/issues-and-errors-in-a-new-scafetta-paper/
It’s just really weak sauce. Scafetta used ensemble means instead of every single member individually, something that the consensus climate scientists have been doing for decades to smooth the most extreme runs out of the picture. Now suddenly the extreme runs are worth their weight in gold.
All the refutations are in the same vein. If Schmidt wants to make a point, the data is always as precise as he needs it to be, but to refute a skeptic the broadest error bars and the most extreme outlier model runs suddenly become essential to our understanding.
Well, yes, by all means read Gavin’s post. Then you can decide for yourself whether it is weak sauce.
I’ve given up on caring what Scafetta claims, as he’s published paper after paper of complete rubbish without ever (so far as I know) admitting error or correcting himself. I do encourage anyone curious enough to read the primary documents, as the characterization here by “Matt Skaggs” is misleading to the point of outright dishonesty.
Not really on topic, but I am wondering if readers can help answer the question “Which journal has the most editors and associate editors?”. This came up in a promotion committee I sat on recently at a respected institution.
At first this example listing 400+ editors seemed like a contender: https://www.frontiersin.org/journals/nuclear-engineering/editors
But no, here is an example from the same publishers with 5,600+ “editors” listed:
https://www.frontiersin.org/journals/energy-research/editors
These journals also helpfully have a button labeled “Join our editorial board” on their webpages.
I was once sent an invitation to join the editorial board of a Frontiers journal, which I accepted. Not only have I never been sent a paper to edit, I’ve never even been sent a paper to review.
5,684 editors (accessed March 7, 5:32 PM CT)
5,333 total articles published (accessed March 7, 5:33 PM CT)
With an impact factor of 3.4, the per editor impact factor would be
3.4/5684 = 0.0005981703
If we pay each editor 100 Euros for each impact factor point generated, they should pay their editors 0.05981703 Euros each. In other words, each editor has contributed their 5 cents worth, quite literally.
Frontiers in Psychology is also quite a joke. I once re-analyzed the data from a paper edited by mainstream psycholinguists, and the central claims of the paper didn’t have any support in the data provided. I asked the editors (whom I know personally) to retract the paper but nothing happened.; they just asked the authors if they want to retract the paper, and they replied that they “prefer not to retract”, LOL. Someone in the comments above said that most statistical analyses are garbage—this is certainly very close to the truth in my own experience within psycholinguistics.
Shravan:
Don’t forget: according to a certain prominent scholar and Ted talk performer, each citation is worth $100,000. Now you’re talking real money!
> To be specific, the observed temperature record can be thought of as consisting of a climatological trend, internal variability with a mean of zero, plus structural uncertainty related to how well the observational estimate matches the real world:
> ΔT_obs = ΔT_clim + ΔT_int + ΔT_err
> with T_clim assumed to be constant by definition over each decade,
What does it mean that the trend term is assumed to be constant by definition?
> and so σ²_obs = σ²_int + σ²_err
Is it so? Even if there was no “internal variability” and no “structural uncertainty” if there any “climatological trend” it seems that there will be some variance in the sample.
“The scientific record” is more full of junk than the bible. The idea these Augean stables can be meaningfully corrected is very funny.
I guess if we cut out every result that has never been replicated and every theory making predictions too vague to discriminate from methodological error, then maybe those 1/10,000 papers could deserve the respect this phrase implies.
And thats fine, science is not about trusting some authoritative record. It is about distrusting everything and everyone, especially yourself.
Looking aside the irony of YOU saying that distrusting yourself is a fundamental principle of science, I think your appeal to distrust is also problematic: Trust and distrust actually are both strange approaches to “science,” imo.
Its the origjnal motto of the royal society:
https://en.wikipedia.org/wiki/Nullius_in_verba
And I do live by this, not really believing anyone or anything. There are only relative probabilities of various tentative premises, and approximations thereof.
Read more carefully if you think otherwise. I think may be impossible for those who heavily rely on authority/consensus heuristics to comprehend the thought process of those who don’t. Like trying to have a mental model of a dolphin or something. Its just too foreign.
Anoneuioid –
> And I do live by this, not really believing anyone or anything.
From my observations, that doesn’t apply consistently in your comments. You have way too much trust in your own lack of bias, in my view. This is an example of what I’m referring to:
https://statmodeling.stat.columbia.edu/2024/03/05/how-large-is-that-treatment-effect-really-my-talk-at-the-nyu-economics-seminar-thurs-7-mar/#comment-2335823
> There are only relative probabilities of various tentative premises, and approximations thereof.
I’ll try to explain again, as it seems you missed my point. I agree that it’s about evaluating probabilities. I’m merely saying that “distrust” is as flawed as “trust.” Neither is relevant to the process of evaluating probabilities and a devotion to “mistrust” is just as problematic as a devotion to “trust.” They’re both reflections of bias, and they both reflect a blind spot to bias. Again, look at the comment I linked which talks of the problematic dismissal of bias in your comment.
> I think may be impossible for those who heavily rely on authority/consensus heuristics to comprehend the thought process of those who don’t. Like trying to have a mental model of a dolphin or something. Its just too foreign.
Perfect. In your condescension you’re not seeing your biases. Read Roxana’s comment again – she gets to the heart of it.
That’s a weird interpretation of what the founders of the Royal Society meant. Here’s what the Royal Society say [ https://royalsociety.org/about-us/who-we-are/history/ ]:
Science is not about doubting/distrusting everything – what a pointless and negative approach to advancing knowledge that would be. It’s about evidence (“facts determined by experiment”).
The scientific advances I can think of arise from scientists acknowledging (not doubting!) work of others, even if provisionally, and being interested enough to test the implications of the evidence. Happily scientists don’t spend their time “distrusting everything and everyone” and that’s been the case since the inception of the Royal Society who’s members were progressive from the start.
There’s a difference between “Don’t take anyone’s word for it” and “distrust everyone.”
That’s binary thinkinf. The one doesn’t (necessarily) follow from the other, and it’s intersting that Anoneuoid seems to often demonstrate falling into the logical trap of of thinking it does.
Don’t take anyone’s word for it can just don’t have blind trust. It doesn’t mean that you can’t have some measure of trust in anyone, particularly if they’ve established a track record of careful treatment of evidence.
Anoneuoid’s logic is particularly problematic when what you often get is “Don’t take anyone’s word for it, but do some Google searches and don’t tkskdnrjr time to develop domain relevant experience and expertise, and then think you know better then everyone else because you’re such a genius who’s interested in the unbiased truth and everyone else is a dope or corrupt.”
This is what I think Dale is responding to – where people take fairly banal truisms and axiom like” don’t take anyone’s word for it, or “Don’t appeal to authority” to justify the facile thinking we see so ubiquitously (particularly online) where so many isolate into self-sealing epistemic solar systems, where they can dig into unfalsifiable thinking. Anyone who disagree with them is necessarily corrupt and impure.
So what is the role of independent replication and comparing predictions to observation?
That is how you establish the “facts”, then test the premises and logical soundness. And this is currently systematically discouraged. Instead we have peer review, and testing a default null hypothesis no one believes.
Its how you reduce the need to rely on trust. Omniscient Jones (who the religious authorities pretended to have access to) doesn’t need to trust anyone. Humans do, it is an inconvenience stemming from limited time/resources.
In the end trust is just a computationally cheap proxy for doing things right. If you can get away with it that is great! But you need to recognize it for what it is: a heuristic.
Anoneuoid –
That is how you establish the “facts”, then test the premises and logical soundness. And this is currently systematically discouraged. Instead we have peer review, and testing a default null hypothesis no one believes.
Sorry, but that argument just really annoys me. I have a hard time just letting it go.
Again, you create a false binary, mixed with a straw man. Very few people, let alone many people (anyone?) here, believes in blind trust of authorities, or blind trust in peer review.
People employ useful heuristics – leveraging expertise as a way to assess the probabilities of various interpretations of the evidence. It’s a flawed method. All models are wrong, but some are useful.
> In the end trust is just a computationally cheap proxy for doing things right. If you can get away with it that is great! But you need to recognize it for what it is: a heuristic.
Aside from the straw man assumption that “trust” equals blind trust, no one here doesn’t recognize that they’re using a heuristic.
Take you, for example. You have given me many reasons to not have blind trust in your reasoning. This was on example that stuck out – where you were clearly wrong about something (your bet about the rate of Covid deaths) and then misleadingly (don’t know if it was intentional or not) minimize your error:
https://statmodeling.stat.columbia.edu/2021/09/16/wanna-bet-a-covid-19-example/#comment-2025148
Another example was when you distorted (unintentional or otherwise) what Walensky said about Covid vaccines preventing infection.
But that doesn’t mean that I “distrust” anything, let alone everything that you say. You’ve written things that I’ve found useful and interesting. As an example, your the first person I saw who explained why IM Covid vaccines wouldn’t likely be very effective in preventing infection, at least for very long.
So just because I don’t have blind trust in what you say, it doesn’t mean that I distrust what you say. It just means that some things you’ve said have led me to discount the probabilities of what you say being reliable, and some things that you’ve said have let me to upgrade those probabilities.
That’s not to say that no one ever underestimates the probability that the heuristics they use are flawed. People do that all the time. But you really ought to climb off your high horse once in a while and (1) stop flattening the landscape and demeaning others to interact with false binaries and cartoonish straw men so as to confirm your own biases and, (2) recognize that you’re just as inclined to over-rely on your heuristics as anyone else.
Don’t you see Joshua? Real science is pristine and noble, and performed by people of real character who are able to see far beyond the reaches of the erring and addled multitudes.
These ubermenschen do not even have to stand on the shoulders of giants, for they are giants themselves; skeptical of all that came before them, yet able to reproduce all of humanity’s collective knowledge from first principles or by diligent application of The Scientific Method^{TM} and with the foresight to avoid incomplete or flawed datasets.
They learned to doubt their middle school science teachers from the minute they made an oversimplified statement about the different phases of matter. As a result, they started reading about density functional theory before become disillusioned with the local density approximation in their sophomore year. The summer before college, they had one last fling with received wisdom and attempted a solution of the black hole information problem, but became disgusted at the potential absence of the interaction picture in QFT — a scenario they encountered without ever hearing the term “Haag’s theorem”.
And of course, unlike you or I Joshua, these true people of science, continually in doubt and re-deriving themselves from scratch before lunch every weekday, are able to scan with ease multiple blog posts about the failures of the unthinking hoi-polloi and the shortcomings of their futile quest for knowledge, and post several thoughtful (and not at all repetitive) critiques on how such attempts are doomed from the start.
We mere mortals can only tug our forelocks, genuflect and kow-tow (for if nothing else, we are quite flexible and spineless) to this wisdom, and accept that unlike Descartes, we cannot even think, let alone doubt.
This is exactly the opposite! Its messy as hell and no one can be trusted at all.
I really think its like the dolphin thing, trust is so important to your worldview that it is incommensurable with this concept. The priorities are too foriegn.
Ie, the number of believers *is* the measure of success. This works great for navigating social issues but is a heuristic that totally fails for science.
In fact, people trusting you is annoying. You want them to bother distrusting your results/arguments because thats how you find the hidden assumptions and errors.
Anon:
The trust thing is tricky.
On one level, I agree with you 100% that I don’t want people trusting me. I want people to send me waves of skepticism. Stress testing not happy talk.
On a different level, I do appreciate some trust. It’s not helpful if people think I’m lying or cheating. First, I don’t like it because such statements are false and annoying. Second, a focus on these nonexistent problems makes it harder to get to the interesting places where I get things wrong.
As Dan Davies discusses in his book a high-trust society enables risks of fraud while at the same time reducing local transaction costs and making life more pleasant all around. When people look at my work, I’d like there to be very little trust on what I’m doing—I can make mistakes at all levels, including silly math errors, garbled code, leaps in logic, all sorts of things—and this will all go smoother when operating on a basis of trust regarding my general motives and workflow. Not 100% trust—hey, maybe I really am super-corrupt, maybe so corrupt that I don’t even realize it myself!—but enough trust that critics can focus on the places where my errors really occur.
But, yeah, I agree with your general point that good scientists make mistakes at all levels, and it’s been disastrous when researchers whose work is criticized try to turn everything into some sort of moral debate. They to save their leaden scientific claims from sinking by tying them to them to the raft of their personal and professional reputation—but then if they’re not careful they’ll sink the whole enterprise, reputation included.
Roxana –
> Real science is pristine and noble, and performed by people of real character who are able to see far beyond the reaches of the erring and addled multitudes.
I think you nailed it. I’ve been watching this phenomenon for quite a while (specifically in the climate-o-sphere), where self-styled “skeptics” note corruption and perverse incentives in “others” (which, of course, is always unfalsifiable) while not recognizing that by definition biases are built in to their own analytical processes as well. This notion of “true” or “pure” science is almost always a part of the rhetoric, as seen with Aneuoid’s comments here.
Andrew –
On one level, I agree with you 100% that I don’t want people trusting me. I want people to send me waves of skepticism. Stress testing not happy talk.
On a different level, I do appreciate some trust. It’s not helpful if people think I’m lying or cheating. First, I don’t like it because such statements are false and annoying. Second, a focus on these nonexistent problems makes it harder to get to the interesting places where I get things wrong.
I think that framing this around “trust’ is a problem. I don’t “trust” what you write. I have a mental concept of your reliability, with a kind of confidence interval based on the experience of reading what you’ve written and testing it against evidence. I also use other factors, like how willing you seem to be to recognize error and to be accountable and to course correct in one fashion or another. Of course, I also mix in an understanding that my observations will be biased, and that your previous work is no guarantee that anything you might say next won’t be of a different character than what you’ve said before.
The topic of trust is timely and goes well beyond “science.” I’m not going to say anything about the degree of trust or distrust appropriate for the practice of science. But I think some degree of trust is essential for civil society and probably for human life more generally. Much of what we do depends on some degree of trust – that our vote will not be altered, that the tip we leave will not be stolen by another client, that the work my surgeon did was done competently, that the data I received was really derived from the stated source, and so on. Each of these should contain a degree of skepticism – blind trust is foolish, at least based on my life experience. However, total distrust is incapacitating.
One of the biggest tragedies of recent history (in my opinion) is the breakdown of trust. Some of that is necessary – “trusted” sources based on pedigrees, number of publications, citations, etc. deserve to be taken down a peg (or two). But we are rapidly reaching a point where trust based on civility and reason has broken down – replaced by blind tribal trust – to the point where it endangers our well-being. The deterioration has seemed so rapid to me that I fear it cannot be overcome. Paraphrasing the words of the recent rejoinder to the State of the Union speech, I am fearful (though not of migrants or inflation).
My comments seem to be getting stuck in moderation (this one didn’t have links, so I am starting to “distrust”).
I think trust is a concept with much broader implications than just science. Much of what we used to “trust” – pedigrees, publications, citations – is subject to increasing skepticism and that is welcome. But we are losing sight of the critical importance that trust plays in our lives. Without trust, life would become unbearable and human existence might even become threatened. We need to trust many things, big and little – that the tip we leave will not be stolen by other patrons, that the vegetables in the market have not been tainted with poison, that our credit card numbers are not stolen by cashiers, that the gas we put in our cars is not laced with antifreeze, ….. Some degree of skepticism is appropriate – but blind distrust is just as bad a blind trust.
What I feel has happened – and relatively quickly – is that trust based on civility and reason has rapidly dissolved, only to be replaced by blind tribal trust. It isn’t clear to me how/if we can restore what is essential for our well-being. Paraphrasing the words of the recent response to the State of the Union address: I am fearful.
Yea, you need to deal with responses from the “peanut gallery”. But this is a price well worth paying. Those who blindly trust give misleading feedback (or none at all) since their naive agreement makes you more confident even though they never really checked your work beyond a superficial level. It is a version of the file-drawer problem.
Anoneuoid –
I’m going to keep at it.
> In fact, people trusting you is annoying. You want them to bother distrusting your results/arguments because thats how you find the hidden assumptions and errors.
I’m trying to figure out if that’s merely wrong, merely bizarre, or some measure of both.
You much more efficiently find out hidden assumptions and errors through a good faith exchange with people with whom you share some level of trust. The most informative discussions I have are where I can share with people in good faith, where we work together to help each other interrogate our own biases. Bad faith discussions rooted in distrust can, almost by happenstance, produce insight but it’s a far more wasteful process because you need to weed through all the crap to find what’s of be benefit. In distrustful exchanges all most no one is open to change, and is far more likely to just double-down and be defensive and avoid accountability for errors.
However, I do think I’m beginning to better understand why your comments sometimes have the attributes they have.
I’ve lost the thread here, but Anon says, regarding trust:
“Humans do, it is an inconvenience stemming from limited time/resources.
In the end trust is just a computationally cheap proxy for doing things right. If you can get away with it that is great! But you need to recognize it for what it is: a heuristic.”
I think this is a very interesting perspective. On a practical level I might agree. But I think this is very unfortunate thinking. I have no background in anthropology – other than working with some very good anthropologists (and I’ll refrain from the joke about the price of a pound of anthropologists’ brains or their GPAs – don’t worry if you don’t get those obscure references). But I believe anthropologists have an appreciation for the value of trust. It is not just a convenience. It is part of the richness of human experience. On a personal level, the fact that there are a few (and very few) people that I truly trust is something I treasure. It enriches my life and not (solely) because it is computationally cheap. If it is merely a heuristic, then I suppose technology might replace it some day. Imagine a world where there is no trust – because it is not required. Would this be a world you would want to live in? Some might (my economist brethren might find it idyllic), but I would not.
Sure, everything is far easier and cheaper when you can trust yourself and others. It very much *is* something to be treasured. Too bad that doesn’t work for science.
And anyone who has actually collected data, etc knows how easy it is to bias the results or otherwise make mistakes. All of them know this, even if they are honestly (rather than manipulatively) about “trust the science”, deep down they know it.
It is not just easier and cheaper – it is more fulfilling, enriching, and satisfying. It is more difficult for science, but “doesn’t work” is not the way I’d put it. It is easy to bias results and even easier to make mistakes. There is a difference – trusting people to not mistakes is a big ask – the world is complex and everybody makes mistakes at times. Trusting someone to do their best and not intentionally bias results is more feasible. There are a few people I trust in that respect, but it is not a null set. For you, it apparently is and that is your loss.
“Easier” and “cheaper” is transactional terminology. Those are real considerations, but not the only ones. That is why I mention technology – perhaps technology could reduce verification costs sufficiently that trust would no longer be needed. Would that be a gain, with no offsetting costs. I don’t see it that way. Trust is a property of human relationships. My relationships don’t solely reduce costs – they also (and I’d like to think primarily) enrich my experience. In the same way, Walmart reduces lots of costs and those are benefits. But when my neighborhood pharmacy or hardware store disappears, there is something lost. It isn’t just nostalgia – it is the richness of a human connection that had value beyond the transaction.
If you don’t have these experiences and only see trust in terms of cost and ease, then I’m sorry for your loss. And I don’t see why science is any different in these respects (I suspect there is some sociology of science that could be debated here – does science exist apart from human experience? I don’t think this is the place to engage in this philosophical debate).
Why do you think people crave food, avoid human waste, and so on? The standard explanation is natural selection. Ie, that you find things fulfilling, etc because it is adaptive (ie, makes surviving cheaper and easier) Same for love and anything else you feel.
Rather than me being some kind of a “poor soul” missing out on these experiences, it seems we have the same ones but you believe in some version of creationism instead of natural selection?
If you are going to invoke nonstandard premises like that it sure is gunna confuse everyone if you fail to mention them.
I have no idea where that came from. It’s about as irrelevant as that response to the State of the Union speech. Natural selection does not equate to cheaper and easier. Rather, the latter may be a sufficient condition for the former, but I’m not convinced it is necessary. Humans do things that are not always cheaper or easier – and I view these as part of natural selection.
The last word is yours – as I know it always will be, so go ahead.
Anoneuoid –
Sorry, but the stuff you say just annoys me sometimes.
> The standard explanation is natural selection. Ie, that you find things fulfilling, etc because it is adaptive (ie, makes surviving cheaper and easier) Same for love and anything else you feel.
This Just-So reasoning about evolution and natural selection strikes me as simplistic and facile.
Sure, the thinking about efficiency and avoiding cost has some merit, imo, but evolution is certainly largely, and perhaps more largely, shaped by factors such accidents and mutations and deletions and rearrangements, of the sort that prompt dramatic adaptations.
This tendency to reverse engineer “evolution” in support of one’s own views about issues such as “trust” strikes me as stunningly vulnerable to confirmation bias. Your notuon about what is or isn’t more efficient is just completely speculative, not even remotely empirical.
Kind of like making assumptions about very complex society-wide phenomena based on anecdotal convenience sampling among your group of acquaintances. If you get my drift.
Dale, I think there’s two different kinds of trust you’re referring to. I may trust you to make a sincere effort to understand me, but I don’t “trust” you to automatically actually know what I’m saying. Similarly, I may trust you to make a sincere effort to understand what a certain dataset tells us about some economic question, but I don’t necessarily trust you to get the right answer. That is, whatever you say I don’t “trust” that it’s correct, only that you didn’t try to lie to me or yourself or others.
The trusting a person to make a certain level of effort is a very valuable kind of trust, but trusting someone to have the correct answer is not such a valuable kind of trust.
Many people say “whatever the journal article I just read says about the economy is true, I trust it to be true because the authors are smarter/more knowledgeable than me and the peer reviewers are too, and they have all verified that the conclusion is correct.”
I think that’s a terrible heuristic, and a decade or more on this blog has convinced me that incorrect scientific conclusions are RAMPANT in the literature. Nevertheless, many people I know who have no science background at all use that kind of heuristic all the time.
Daniel –
> Many people say “whatever the journal article I just read says about the economy is true, I trust it to be true because the authors are smarter/more knowledgeable than me and the peer reviewers are too, and they have all verified that the conclusion is correct.”
Who is it that says that? What do you mean by “many?”
I could be wrong but I think that’s a straw man except maybe in pretty rare cases.
I think that most people say something on the order of “What I just read in that highly ranked journal has been vetted by people with highly relevant domain expertise. In that sense it is likely to have solid value. However, I don’t assume all the conclusions are necessarily 100% accurate.”
Even if you think that statement wrongly evaluates the probabilities, it still doesn’t mean the level of” trust” that you’re foisting on people is accurate.
The way I look at it is like this. Even if most peer-reviewed articles can’t be replicated. Even if a higher percentage are just “wrong.” That still means the probability of them having value is higher than most the crap I read elsewhere. The standard of saying that peer-reviewed articles are likely “wrong” is the the wrong standard, imo.
I think the standard that should be used is not whether a peer reviewed article is surely correct, but whether having undergone peer review is likely to increase the probability that an analysis will be of some value over just reading some random analysis on the interwebs or somewhere else. Perhaps you’d disagree, but as a general response I’d say “yes.”
Joshua, I might be exaggerating but only slightly. Do you not know people with zero science background? Like say 70 or 80 year old people who got a bachelors in art history or something in the 1960’s and read articles about health issues like nutrition, or education issues about elementary school teaching or something about the effect of daylight savings time on obesity or whatever in the newspaper and they think this latest news which is definitely accompanied by a reference to an article in Lancet or BMJ or whatever is definitely the latest and greatest information about the reality of whatever XYZ it’s about?
And it’s not just elderly people, I mention them not because of age but more because of the changes in education since they were in college. But, I’ve heard middle aged school administrators say things like “the science PROVES that starting school later in the day has massive effect on student learning” or whatever, maybe it’s something about “growth mindset” or how pre-K massively improves lifetime income or whatever. There were plenty of people weaponizing science articles over COVID questions throughout the pandemic. Have you forgotten that already?
The primary purpose of science articles by the public is to be wielded as a kind of spiked mace to clobber anyone who dares disagree with them.
I’m hardly even exaggerating. It’s a major pathology. Enough that people write protest signs and march on Washington DC about wanting “evidence based policies after peer review” or whatever.
The worst part is even scientists I know sometimes point at stuff and claim that it means X when in fact after looking carefully at the analysis even though the authors claim X it’s really consistent with the opposite… I’ve got an example right here on the blog from a few months ago or so https://statmodeling.stat.columbia.edu/2023/09/02/latest-observational-study-shows-moderate-drinking-associated-with-a-very-slightly-lower-mortality-rate/
It’s particularly bad when it’s some kind of long-running or combined meta-review or whatever claiming to integrate multiple sources of information from the historical record in science to come to the one-true-fact that …
People lap this stuff up. I see it all the time. If you don’t it’s because you’re hanging out with a less gullible or less political crowd, or maybe I just see it because everyone I know who gets one of these maces to the face comes to me to figure out whether it was made of nerf or steel. It’s almost always nerf.
Daniel –
I’m not sure how we resolve this difference in view. I Googled a bit to look for information on how the general public views peer review and it seems there isn’t much.
The closest I’ve seen is the polling in the issue of “trust” in scientists and scientific institutions. In line with what Dale discusses, that polling always bugs me because “trust” there has such a nebulous meaning…but…
While in one sense the polling shows that general level of trust there is reasonably high (especially compared to something like the press or politicians), it also shows that the trust is not absolute. And perhaps most relevant, relatively large %’s think that scientists are quite capable of altering their findings out of personal bias or self-interest.
You say:
Do you not know people with zero science background? Like say 70 or 80 year old people who got a bachelors in art history or something in the 1960’s and read articles about health issues like nutrition, or education issues about elementary school teaching or something about the effect of daylight savings time on obesity or whatever in the newspaper and they think this latest news which is definitely accompanied by a reference to an article in Lancet or BMJ or whatever is definitely the latest and greatest information about the reality of whatever XYZ it’s about?
I think it’s mixed. Speaking in global terms, I think a lot of people kind of think the information is definitive but also kinda think that these kinds of reports are often wrong and have often been wrong in the past and they have actually quite little absolute confidence in what they read in the media.
So what conclusion can be drawn from all of that?
As an example, quite a few people I know eat low fat yogurt or whatever because they have been conditioned over the years to think that dietary fat affects cholesterol levels (and body fat). They’ve no doubt read many stories recently that call those results into question. But they still eat low fat yogurt. So if a person has blind trust in articles that report peer-reviewed science one way or the other, then what are they going to do? I don’t see clear evidence that supports the confident assessment that you’ve made. I think people make these decisions as shown by revealed preferences out of a complex mix of conflicting information about issues like the reliability of peer-reviewed science.
And it’s not just elderly people, I mention them not because of age but more because of the changes in education since they were in college. But, I’ve heard middle aged school administrators say things like “the science PROVES that starting school later in the day has massive effect on student learning” or whatever, maybe it’s something about “growth mindset” or how pre-K massively improves lifetime income or whatever. There were plenty of people weaponizing science articles over COVID questions throughout the pandemic. Have you forgotten that already?
All true, no doubt. Certainly, I haven’t forgotten how people weaponize peer-reviewed literature in order to reinforce their ideological biases (covid being the perfect example because it’s so high stakes and in a sense shocking that people will weaponize evidence for ideological purposes – shocking although not surprising).
But I don’t think that translates to blind trust in peer review as being causal – because people filter their trust in specific peer-reviewed science through their ideological lens. So what’s casual isn’t the blind trust. The blind trust, to the extent it exists in a consistent way (I think it’s often not consistent) is more an outcome than a driver. That doesn’t mean that some level of blind trust doesn’t exist but I do think it means that it’s more a by-product and that blaming blind trust for the problems kind of misses what’s more important – which is the causal role of ideology-based and identity-based cognition.
The primary purpose of science articles by the public is to be wielded as a kind of spiked mace to clobber anyone who dares disagree with them.
Sorry – that just feels too categorical to match up with reality for me. Of course that’s a factor, but to describe it as the *primary* purpose strikes me as hyperbolic.
I’m not saying that what you’re describing isn’t at issue – but using your observations from this blog, or people you encounter in your life, or what the signs people carrying in Washington say, just don’t make the case for me for the degree of confidence you express in such strongly categorical determinations. Describing “trust” in peer-review, in the general public, seems too complicated to me to rely on such methods.
Daniel –
As it happens I just ran across this remarkable article – that is very timely given our convo.
https://www.nytimes.com/2024/03/05/magazine/younger-dryas-impact-hypothesis-comet.html?smid=url-share
Joshua. I’ll agree that these things are very contextual.
1) A person has some question that they honestly want the answer to, goes to google etc and searches up articles on that question, finds a meta review that definitively states that there’s no evidence that alcohol has any benefit or there’s strong evidence that diet sodas are good/bad for you or that giving free preschool will increase lifetime earnings by 20% or whatever… the cited paper agrees with their initial assumption… the person has no formal scientific training… I think many many people in this situation will have fairly blind trust. Furthermore, I think the main reason non-science-oriented people go and look these things up is to make some argument in some context, whether it be to their doctor, a school board, or their friends and family, so they’re “weaponizing” this information in some sense… they’re using it to win some argument.
2) A person is on the other side of that above equation? Someone else cites some article to them to try to convince them, and the person’s base belief is opposing or not convinced of the general idea in the first place… this person will have more skepticism for sure.
I think the average non-science person doesn’t bother looking up science articles except in cases where they want to affect policy or argumentation… which is why I say the primary purpose is to weaponize.
those who have some science background tend to look up these things with more open questioning and integrative outlook. Although this is an increasing group of people as more people have science educations, it’s still a relatively small group compared to the overall population.
Unfortunately I can’t access the NYT article you mention.
I’m not sure how we got sidetracked onto trust from a post about unwillingness to correct mistakes. But I think debating trust is mostly a waste of time. I never liked those political polls that ask people how much they trust a candidate – what in the world does the question mean? Daniel says there are 2 types of trust, Joshua says trust is multidimensional. It is. If you ask me who I trust to always be correct – I’m with Anon (and everyone else, I expect) – nobody, least of all myself. So, anybody who says they trust someone to always be correct is clearly delusional, uneducated, or expressing something else (blind tribal association).
If you ask me who I trust to try their best and not lie or deceive me, then my list is very small and highly valued. For me, there are absolutely no politicians (alive or dead) on my list. Does this type of trust play an evolutionary role? I think it probably does, although I don’t know if there is definitive research on that. It may eventually play an important evolutionary role (we have yet to find out). Does this type of trust reduce costs? I think it does. Is that all that trust is, a cost-reducing phenomenon? I don’t believe so, but perhaps Anon does.
So, I am agreeing that when it comes to research, I trust nobody and nothing to always be correct. Score one for Anon. Does that mean I totally distrust everybody and everything when it comes to research? No, my distrust is not total – this is not a binary question. Score one for Joshua. I trust some research more than others – it is a continuum. And, for me, trust must be earned. Returning (only slightly) to the original post by Andrew, willingness to correct mistakes is one way to earn my trust (and conversely).
Dale –
> I think it probably does, although I don’t know if there is definitive research on that. It may eventually play an important evolutionary role (we have yet to find out). Does this type of trust reduce costs? I think it does. Is that all that trust is, a cost-reducing phenomenon?
I think there’s much “common sense” reasoning about the mechanisms of evolution, even from people who claim expertise like Jordan Peterson or Bret Weinstein, that is just facile.
The idea of applying an “evolutionary lens” to complex psychological phenomena such trust and tribal behaviors from a cost/benefit angle seems highly fraught to me. As an example, at some level heuristics that lead to in-group alignment may well have an efficiency benefit but at another level they could be very costly, or at another level cross-group collaboration or cooperation can be beneficial also.
I am VERY skeptical of people who claim they’ve sussed out the evolutionary cost/benefit payoff ratio of complex social behaviors, specially since almost always their interpretationsins up with their ideological orientation.
FWIW.
I agree that evolutionary arguments are fraught with problems. What exists today has survived evolutionary forces – at least until now. Does that mean they will survive what comes? Who knows? Ex post, we can always make an evolutionary argument, but it is tautological. Ex ante is mostly speculation. I think there are major issues with trying to apply evolutionary arguments to social phenomena. An ecologist once told me: “man does not operate by natural selection.” I think that must not be true, but there are many things that don’t appear to fit (as with much of our medical practice). I think that was an ecologist’s way of getting out of dealing with these thorny issues. So, what can we say about trust? Some forms of trust seem to favor human survival and reproduction while others do not. I think we agree that this avenue of thought is not likely to shed light on the reasons for, and dangers of, different forms of trust.
The evolutionary explanation is that you have these feelings of “fulfillment” and so on because it was advantageous for your ancestors to feel that. It could be 2000 years from now “Dale Lehbug” gets those same feelings from the queen proboscising his fat sack to nutritionize the eggsack.
But there are other premises you can adopt instead. You would be in good company. It reminds me of Kepler’s speculations when the moons of Jupiter were first reported by Galileo:
https://gwern.net/doc/science/1965-kepler-keplersconversationwithgalileossiderealmessenger.pdf
Basically moons exists to cause awe in planetary occupants so they learn more about the universe. That type of thinking is out of fashion today, but doesn’t mean it is wrong.
Anon
Yes, that evolutionary argument could be correct. Or not. How can we tell? Eventually, we should find out, but I don’t see any way to tell whether the fulfilling feeling of trust today is simply evolution or something else. If you define it as evolution, then the reasoning becomes circular – everything we observe today must be evolutionary adaptation by definition – although we know some things will turn out to not be traits that last. Again, how can we tell the difference? But I’m willing to believe that trust is important for human survival, not simply because it lowers costs, but because we need it to avoid killing each other.
Sure, there is a long history of people saying “evolution” seems unfalsifiable in principle. Here is Paul Meehl on it:
https://meehl.umn.edu/sites/meehl.umn.edu/files/files/125_fossils.pdf
Basically if you compare two theories bayes rule says:
p(H[0] | E) = p(H[0])*p(E | H[0]) / [ p(H[0])*p(E | H[0]) + p(H[1])*p(E | H[1]) ]
Where H[0:n] are the hypotheses/theories and E is the evidence.
Then if both are vague enough to explain anything we are saying:
p(E | H[0]) ~ p(E | H[1])
Those terms cancel out so you get:
p(H[0] | E) = p(H[0]) / [ p(H[0]) + p(H[1]) ]
There is nothing wrong with that. All I was saying is when posting on this blog others are going to assume by default you are working with H[0], and you should make a point to mention if its some other theory H[n].
The solution is to come up with a theory that makes more precise predictions. Then:
p(E | H[0]) !~ p(E | H[1])
Another aspect is some have proposed logic should be based only on self-proving premises:
https://en.wikipedia.org/wiki/Consequentia_mirabilis
Whether that is possible remains an unsolved problem.
@Dale Maybe the analogy I would make — and I will admit to both being a relatively new reader of this blog (so I’m sure there have been many posts and comments debating the central point of what I have to say), and a neophyte when it comes to Bayesian analysis — is to having informative priors and updating them based on experience and new information.
Suppose an issue arises in the sphere of public discussion — perhaps the efficacy of a new medication, or some new meteorological phenomena that is predicted to cause some disruption. Having no expertise in either of these issues, how and where should I calibrate my beliefs regarding the issue?
Very broadly speaking, I would say one should start off by consciously asking, “Does this contradict anything that I feel very sure about the world, based on my knowledge and experience, and if so, how great is the contradiction?” I think this is the base level of skepticism with which most of us approach the world: sensational headline and lede? “That can’t be right …”
If the issue at hand isn’t some earth-shattering revelation, or if we recognize that our priors include rules of thumb which have large loopholes, I think we should then move with the consensus view of experts on the issue — if such a view exists, and if there are such experts. That’s what “trust” means to me in this context — anchoring our estimates of credences based on the collective opinion of people who have had more experience in such matters.
Of course, it is very often the case that there is no consensus view, nor many reputable experts on a particular issue. With the caveat again that the issue is not some world-upending information, we should be more agnostic, and the distribution of our credences on the matter should be more uniform/flat.
What I find sometimes is that there are some people who either
(a) believe they know so much about the world that they feel justified in dismissing almost all new information which contradict their previously formed priors (what I’d call “maximum arrogance”),
(b) think that the present state of all scientific endeavor is so compromised by scandal and fraud that almost all new information should be heavily discounted (what I’d call “maximum defeatism”), or
(c) feel that one should always approach every issue with a totally non-informative prior *and* the only allowable updates must be those which they can verify by themselves (what I’d call “maximum distrust”).
There’s probably some overlap between these categories etc., and of course a lot of the posts on this blog make a good case for adopting viewpoint (b). Maybe we all have subfields in which we take the stance described in (a). But (c) — leaving aside the issue of non-informative priors in more specific settings, and focusing on their use as a general, epistemic standpoint — seems generally unhelpful and not how most people approach the world. Though I agree that if one were aiming to, say, try to systematize/re-organize a self-contained area of research which is largely full of heuristics and vague principles, this is the approach one would take.