The other day we talked about checking survey representativeness by looking at canary variables:
Like the canary in the coal mine, a canary variable is something with a known distribution that was not adjusted for in your model. Looking at the estimated distribution of the canary variable, and then comparing to external knowledge, is a way of checking your sampling procedure. It’s not an infallible check—–our sample, or your adjusted sample, can be representative for one variable but not another—but it’s something you can do.
Then I noticed another reference, from 2014:
What you’d want to do [when you see a problem] is not just say, Hey, mistakes happen! but rather to treat these errors as information, as model checks, as canaries in the coal mine and use them to improve your procedure. Sort of like what I did when someone pointed out problems in my election maps.
Canaries all around us
When you notice a mistake, something that seemed to fit your understanding but turned out to be wrong, don’t memory-hole it; engage with it. I got soooo frustrated with David Brooks, or the Nudgelords (further explanation here), or the Freakonomics team or, at a more technical level, the Fivethirtyeight team, when they don’t wrestle with their mistakes.
Dudes! A mistake is a golden opportunity, a chance to learn. You don’t get these every day—or maybe you do! To throw away such opportunities . . . it’s like leaving the proverbial $20 bill on the table.
When Matthew Walker or Malcolm Gladwell get caught out on their errors and they bob and weave and avoid confronting the problem, then I don’t get frustrated in the same way. Their entire brand is based on simplifying the evidence. Similarly with Brian Wansink: there was no there there. If he were to admit error, there’s be nothing left.
But David Brooks, Nudge, Freakonomics, Fivethirtyeight . . . they’re all about explanation, understanding, and synthesis. Sure, it would be a short-term hit to their reputations to admit they got fooled by bad statistical analyses (on the topic of Jews, lunch, beauty, and correlated forecasts, respectively) that happened to aligned with their ideological or intellectual preconceptions, but longer-term, they could do so much better. C’mon, guys! There’s more to life than celebrity, isn’t there? Try to remember what got you interested in writing about social science in the first place.
Moving from “admit your mistakes” to accepting the inevitability of mistakes and their fractal nature
I wonder whether part of this is the implicit dichotomy of “admit when you’re wrong.” We’re all wrong all the time, but when we frame “being wrong” as something that stands out, something that needs to be admitted, maybe that makes it more difficult for us to miss all the micro-errors that we make. If we could get in the habit of recognizing all the mistakes we make every day, all the false starts and blind alleys and wild goose chases that are absolutely necessary in any field of inquiry, then maybe it would be less of a big deal to face up to mistakes we make that are pointed out to us by others.
Mistakes are routine. We should be able to admit them forthrightly without even needing to swallow hard and face up to them, as it were. For example, Nate Silver recently wrote, “The perfect world is one in which the media is both more willing to admit mistakes—and properly frame provisional reporting as provisional and uncertain—and the public is more tolerant of mistakes. We’re not living that world.” Which I agree with, and it applies to Nate too. Maybe we need to go even one step further and not think of a mistake as something that needs to be “admitted,” but just something that happens when we are working on complicated problems, whether they be problems of straight-up journalism (with reports coming from different sources), statistical modeling (relying on assumptions that are inevitably wrong in various ways), or assessment of evidence more generally (at some point you end up with pieces of information that are pointing in different directions).
“A mistake is a golden opportunity, a chance to learn. You don’t get these every day.”
Actually, I get these multiple times every day. One thing that may partially account for this attitude against making mistakes is our educational system. Mistakes are penalized – even if we tell students that they learn from mistakes, in the end we give them worse grades when they make mistakes. I am a believer in incentives, and the incentives we provide are perverse and exhortations about learning from mistakes don’t overcome the very direct effect of the incentives we provide through testing and grading.
I also remember many years of teaching undergraduates where they were merciless if I made a mistake. I used to think it was great role-modeling to make an error and then figure out how to learn from it. But when I’d do that in class, it would come back to bite me. I’ve read many teaching evaluations that complained that even the instructor couldn’t get it right (graduate students have been more open-minded in my experience). Against these backdrops of penalties for making mistakes, it is hard to generate enthusiasm for this golden opportunity.
Dale:
Agreed. Coincidentally, after posting the above but before seeing your comment I added a whole new section to the post making exactly this point.
Dale:
In sports, when people don’t score high they work harder and try to score more. But according to you this doesn’t work in academics? Or should we stop scoring in sports to make people better at it?
I have *literally never* had the experience of being treated badly because I admited a mistake in either academia or my working life. I sometimes had lower scores in academics becasue I made mistakes. My academic experience is completely the opposite: faculty and instructors are happy to help you figure out a problem. Same with work but even moreso: supervisors *really* appreciate it when you recognize an error and notify them immediately – especially if it’s something that will go up the chain, because it gives them the chance to do the same with their supervisors.
**WAY** more common experience: I was right about something but others insisted I was wrong; or I made a suggestion that was derided when I made it but later adopted.
You claim that “the problem” in education is that people are having to suffer the injustice of scores on their math homework. Yet you ignore that “score” model of educaiton has been **highly** successful for literally hundreds of years – going all the way back to the New England colonists in the early 1700s – making the US one of the most educated countries in the world.
Why did it work right up until now?
Here you on one thread with this idea that 300 years of education success should be overturned because you think it suddenly doesn’t work; and on another thread you’re wondering why people seem to have “extreme” opinions?
Chipmunk:
I, like you, have never had the experience of being treated badly because I admitted a mistake in either academia or my working life. That’s one reason I’ve found it sooo frustrating that David Brooks, Nudge, Freakonomics, Fivethirtyeight almost never admit or confront their mistakes and get so bristly when anyone suggests they might have made a mistake.
One theory I have is that all the above-mentioned people get lots of uninformed stupid criticism to go along with the legit criticism.
So you get the sad spectacle of Nate Silver resisting even some very clear narrow technical criticisms of his forecasting methods—criticisms that he should welcome in that they could only help him do better in the future—because he’s in a crouch, busy brushing aside stupid criticisms coming at him from the left and right. It’s frustrating, especially when he says the media should be “both more willing to admit mistakes” . . . without seeming to realize that he is part of those media.
Andrew –
With all due respect, it’s probably a mistake to generalize as if you’re a representative sample with regard to how you you’ve been treated but also with regard to the attitude you being towards addressing the notion of “failure.”
With regard to education, seems to me what’s most relevant is what the outcomes are now on this issue and what could be done to improve them.
In my experience, it’s VERY difficult to encourage students to embrace a constructive attitude towards “failure,” and towards what failure does and doesn’t really look like.
As someone who has spent a great deal of time inside a very wide variety of educational institutions, and as an interested observer of how our educational paradigm regarding “failure” overlaps with broader society outside those institutions, I’m not surprised in the least by those examples you reference.
I should note I wasn’t suggesting that you are generalizing from your own experiences, although it prolly came across that way.
I suspect that the majority of commenters here, more or less by definition, may have difficulty understanding why for many, many students, a “mistake” turns into a feeling of inadequacy.
Readership at this site is a very self-selected group. Again, more or less by definition, a group of people who have NOT taken “failure” in an academic context as a sign of inadequacy AND (not coincidentally) a group of people who have experienced far, far less “failure” in an academic context over per year of academic lifetime.
So the interesting questions (fie me at least) are, of course, what has led to this group being an unrepresentative sample (speaking generally of course) and, how can the educational experiences for students who are more representstive be modified so as to increase the rate at which students exiting our educational institutions, look at academic “failure” as an opportunity to learn and not as an indication of inadequacy.
Of course, necessarily, the answers to those questions will be different across the different levels of our educational system, as the answers are necessarily tied to developmental factors.
chipumunk
True to form, I totally anticipated this response from you. Further, I don’t appreciate you putting words in my mouth. I am not advocating giving up on grading people – I use grades myself. But it is a bit of a quandary – if we penalize people for making mistakes, then we shouldn’t expect them to welcome mistakes as a learning opportunity. It is a delicate balance to use grades as necessary feedback without it undermining the learning experience.
I won’t bother with the rest of your rant. It doesn’t address what I said or think and I’m tired of you using every opportunity to tell me how wonderful the world is.
I agree that owning up to and learning from mistakes in a teaching context is fraught. You’re pushing up against the rest of the culture that students have been exposed to, plus the incentives you mention. I don’t know how effective it’s been, but I would make a point at the beginning of the term to include a short statement something like this: “Schools are special institutions created by society where mistakes don’t cause much damage. Mistakes in the rest of the world cause bridges to collapse, patients to die, companies to go bankrupt, etc. In schools, mistakes don’t have any of these consequences. You probably have a certain number of mistakes you’re going to make in life. Make as many of them as possible in this class, as long as they’re honest mistakes and you can learn from them.”
This is a really good point. Do you mind if I quote you in future syllabi?
It’s all yours. No need to reference.
Dale –
> One thing that may partially account for this attitude against making mistakes is our educational system. Mistakes are penalized – even if we tell students that they learn from mistakes, in the end we give them worse grades when they make mistakes.
Yes. Precisely. This is meaureable by a frequent outcome where students look at education as a passive process:
“You tell me, oh teacher, what I should do and then you judge me on whether or not I followed your instructions well, and tell me fje end result of your judgment. Or better yet, judge me not so much on whether I followed your instructions well, but how well I followed them compared to the other students – ’cause that’s what’s most important.”
measurable as well as meaureable.
The irony is this draws from the nudge crowd, but why not simply tell students they get credit for what they did correctly? You don’t lost points for mistakes, you simply earn points for correct work. I don’t think grades per se are a problem at all, but how we teach has been degraded by lessening our focus on learning and thinking, and in some cases for not caring about mistakes.
Spoiler: more anecdotal data.
My first “theme” (essay) for my required Freshman English course came back all marked up with a grade of C+. I learned some things not to do and got an A- on next week’s theme, and A’s after that.
In my senior year I took a graduate course in Probability and Statistics. In my first test I got 50 out of 100. In the next one I got 100.
On the other hand for my third and last required phys ed class I took Bowling. Started with an average of 103 for the first week and ended with an average of 144. Got a C for the course. I think I might have gotten a B except I started throwing a big hook (the lanes were old and pretty dry) and with the instructor watching threw one that curved so much it left all but the 7 pin on the extreme left. She instructed me to aim between the two rightmost marks for the second shot and straighten my swing a little, and it went right into the pocket and got all remaining 9 pins for a spare. But on the next pin I threw the big hook again and got another lone 7 pin. She muttered “and on a spare!” I had not learned.
Maybe there should be a required course “Learning To Learn” or something. With examples of a lot people who failed and learned from it. In Einstein’s Zurich Notebook he tried a few things that didn’t work for GR before Riemann’s equations. He is the author of my favorite quote: “All mathematicians make mistakes; good mathematicians find them.” (Could be generalized to all thinkers and find or acknowledge them.)
Solzhenitsyn had one of his characters in “The First Circle” say something inspiring on the subject. Something like, “Don’t be discouraged if the way is difficult. If you are digging and your shovel hits something hard, maybe it is the iron strap around the treasure chest!”
I’ve written something about how we might formalise thinking about certain types of errors in our research by drawing an analogy between risk-of-bias assessments for internal validity in medical research and the need for similar assessments for external validity in descriptive inference/generalisation of causal inferences. See https://arxiv.org/abs/2308.11458
(Feedback welcome, as I’m sure there are numerous errors and I can do some useful learning from them too!)
The idea is to force people to think about these and present potential problems clearly, hopefully learning something about the reliability of conclusions in the process.
I noticed after posting that preprint that survey sampling researchers have suggested some overlapping things as well: https://academic.oup.com/pnasnexus/article/2/3/pgad049/7091615
(I will add this to the next version).
Following up on a reply from a few days ago, why can’t we just say (science not having been worked out in advance), all knowledge is uncertain and subject to refinement.
The >kinds< of uncertainty are legion – error, lies, approximation, confidence. And we may tolerate certain kinds of errors. Anecdote – I've met quite a few scientists in discipline X who will say that the reporting on X in newsmagazine N is all nonsense, but they do have good articles on discipline Y (my area). One can eventually draw the conclusion newsmagazine N is all rubbish, but how many newsmagazines can truly report on developments in law, science, economics with great and equal facility.
I'd go so far as to say the difference between academic journals and newsmagazines is (in part) the degree of tolerance we have for their errors. We want our daily news, and the latest download of an operating system, and tolerate a certain amount of bugginess. Less so for scholarship, but with the corporatization of research …
Not my idea, again an adaptation from Henry Kyburg, Jr.'s work.
In the vein of admitting mistakes, I had been thinking I was pretty smart for coming up with the idea of using a canary variable as an external check on poststratification procedures a bit over a year ago, but now I think it extremely likely I just read it on your blog or in your regression book! Darn.
I like to use the presidential approval rating, as what I am measuring is often a similar attitude that can fluctuate and is related to political leanings etc., and presidential approval is measured very frequently by lots of different organisations, so there are always timely estimates to compare to. Although, this admittedly is not so helpful if all the other polling orgs are themselves making big mistakes in their representativeness corrections
Andrew’s statement,
“Dudes! A mistake is a golden opportunity, a chance to learn”
leaves this paranoiac shaking his head while in the fetal position. Philosophically, the statement has much to recommend it. Practically speaking, a mistake can end a career, a marriage or even, life itself. Perhaps all of the above simultaneously. I personally have over eight decades of mistakes and have never, ever felt that I was better off for the experience. And that goes for slipping on the ice in February, 2014 on the way to the gym and thus ruining the rotator cuff on my right arm.
So to speak, “wrestling with my mistakes,” would be another mistake, especially since I did in the rotator cuff on my left arm in 2018 by merely getting out of bed on a docked barge in Amsterdam. With no functioning rotator cuffs, learning is, so to speak, out of reach.
“Practically speaking, a mistake can end a career, a marriage or even, life itself. Perhaps all of the above simultaneously. ”
Right. So learning to avoid them by understanding things ahead of time is a really good idea, and the penalties for making them in school is a lot less destructive than the penalties that occur in real life.
“I personally have over eight decades of mistakes and have never, ever felt that I was better off for the experience”
Think about that poor bastard Prigozhin: he made a big mistake by attacking Putin.
You make a great point: feeling bad about mistakes isn’t caused by the scoring of math homework in the school system! It’s a society-wide phenomenon. It reflects the unfortunate reality that serious mistakes can have very serious consequences.
Perhaps then a mistake ‘can’ be a golden opportunity to learn. Some mistakes are fatal, and for non-fatal but terrible mistakes it can feel like nothing good can come of them. But in general, there often is something to learn, which can prevent the chance of more serious or consequential mistakes. I crashed a motorcycle on holiday after just a day or two of arriving there and it easily could have been fatal, though as it happened it was just a lot of bandages, pain, and a ruined holiday. But I have learned from what led up to the crash and how I felt before it happened to recognise things in the future, and have thought about what I would do differently in all sorts of ways in the future that – if I find myself on a motorcycle again – will reduce the chance of similar or more consequential mistakes from happening.
I have a pet theory, more consistent with the never back down mentality than the deluge of too many critiques that you dismiss them all outright. People (academics included) have a very difficult time distinguishing legitimate critiques from ad-hominem attacks.
https://andrewpwheeler.com/2021/05/18/academia-and-the-culture-of-critiquing/
Andy:
Interesting. I particularly like your point, “University professors don’t really have a supervisor like normal jobs, we each evaluate our peers research through various mechanisms.” This applies to freelance journalists such as Freakonomics/Nudge/Fivethirtyeight as well as to columnists such as David Brooks. OK, I guess Brooks does have a boss but my guess is that the boss takes a hands-off supervisory role, in the same way that university deans don’t usually concern themselves with the labors of their faculty.
Considering the case of Nate Silver, for example: As a freelance journalist, Nate answers to the market, not to any boss. On one hand, this should make him particularly grateful for outside evaluation, as this is pretty much the only feedback he gets; paradoxically, though, such feedback sends him in a defensive crouch so strong that he can’t process it. As discussed in the final section of my above post, I think a key problem here is that he thinks of a mistake as something to be “admitted” rather than a natural thing that happens every day. As an experienced statistician, I know that just about any model has flaws, and a big part of applied work is to find such flaws. Given Nate’s analytical experience and expertise, I was surprised to find that he couldn’t handle the idea that his forecasting method might have such flaws, and I was saddened to see him resist facing his errors even when I graphed them so clearly. Yes, the never-back-down attitude is part of this, and perhaps such attitudes derive in part from a lack of realization that mistakes of varying severity are occurring all the time, and that’s how it should be. A world without everyday mistakes would be a world without everyday experimentation.
I think the market is not especially well informed that all models are going to contain some kind of error or be erroneous when stretched to certain applications. To that extent, Nate may be thinking – and is probably correct – that ‘admitting’ or confirming such a mistake will lead to a decrease in his prestige and the attention paid to him in the marketplace. Most people like these models because there is the idea that they have some special insights and ‘get things right’ in a way that other experts do not. To find out that they have errors just like anything else would likely decrease the demand for them
Jamie:
Indeed, I think that’s Nate’s point when he writes, “The perfect world is one in which the media is both more willing to admit mistakes—and properly frame provisional reporting as provisional and uncertain—and the public is more tolerant of mistakes. We’re not living that world.” I think he’s agreeing with you.
The question then arises: What does Nate think about his own mistakes?
It’s possible that he recognizes his errors and is just following the implicit advice of this comment below and keeping quiet about them. But my guess is that he honestly doesn’t realize that he’s actually made these mistakes. I suspect that he’s trapped in a schema under which his forecasting models are just fine, they’re calibrated, the critics don’t know what they’re talking about, etc. And that, in turn, could be a product of his implicit attitude that mistakes are relatively rare, that they’re a big deal, something that needs to be “admitted,” etc.
Andrew, that makes sense. I would not be surprised if the thought process is something more like ‘the model is fine and calibrated, and these so called errors are minor things pointed out only by pedantic people – I know nothing is perfect, this is an imperfection maybe, but not a mistake. And I certainly can’t refer to it as a mistake and scramble to correct it, because mistakes are bad and I will lose credibility’. But, perhaps behind the scenes and for some thing in the future he starts looking for how to ‘improve’ it by ‘correcting’ the model to not be vulnerable to these critiques, if that is possible without a total overhaul.
However, I never read what pushback was received or how he responded to your critique – nor what has happened with the models since. I don’t think he works with 538 anymore?
Jamie:
I’m not sure. Nate hasn’t been responding to my emails lately. Or maybe his email has changed? And, yes, now that he’s no longer doing Fivethirtyeight.com, maybe he’s moving away from election forecasting and poll aggregation. At this point, lots of other people are doing those things so it could make sense for Nate to move to something more interesting.
I have a lot of respect for Nate and his work. As I’ve said, everyone, including me, makes errors all the time, so the fact that I found some errors in his forecast doesn’t make it horrible. But I have a feeling that he found my criticisms to be annoying, which could be related to the fact that he gets so much criticism from so many directions that it’s overwhelmed him, to the extent that he hasn’t developed a way of processing informed criticism that could actually be useful to him. Again, it’s a shame, even if understandable.
I also watch/listen to a lot of 538 stuff and enjoy it a lot more than much other news/polling coverage, and have found it a bit less interesting since his (apparent) departure – I’m not sure if he’s left or just taken more of a back seat recently.
I completely agree it’s a shame that we can become resistant to acknowledging mistakes and especially when they are opportunities to grow. It may be a slight case of being a victim of his own success – his stuff is probably a lot better than much of what is out there, but by acknowledging such mistakes or intricacies he may be unfairly penalised in the public eye relative to other ‘models’ or projections that are simply worse but not under such scrutiny. I feel a microcosm of this every time I’m required to defend and fully explain all the minor details of a Bayesian model, which I would not get if I did a clearly inappropriate but familiar t-test! He probably also comes under a lot of unjustified or ill-considered critiques that you could end up just seeing even valuable ones as a nuisance
Jamie:
Yes, the Fivethirtyeight team was replaced and I think the new team (headed by Elliott Morris, my collaborator on the 2020 Economist presidential election forecast) is more politics-focused so probably less fun. Nate’s a lot of fun and I kinda wish he’d retire from political/media punditry and instead focus more on data analytics. That said, I’m sure many people think that about me too!
Perhaps he is getting more involved in poker now instead – his all time earnings are almost 1mil from it, and very healthy earnings in 2023 alone: https://pokerdb.thehendonmob.com/player.php?a=r&n=283302
Well, this blog is very fun and goes beyond just political punditry. I’d enjoy watching you at the poker world series though if you’d ever expand your horizons :-D
> Nate’s a lot of fun and I kinda wish he’d retire from political/media punditry and instead focus more on data analytics.
Gets my vote. I really liked his election stuff at 538 (in particular his approach on uncertainty, which I see as being closely linked to dealing with “mistakes”). I’ve been rather surprised to see how vacuous I find his political punditry on Twitter. It feels somewhat incongruous.
> That said, I’m sure many people think that about me too!
I wonder how bad you’d be on Twittee. It’s probably not irrelevant that you avoid Twitter.
Jamie:
I was in a Thursday-night poker game in grad school. I was ok but there are people who are much better than me. I’d get destroyed in Vegas for sure.
Indeed, one could argue that one of the attributes that makes me a good scientist is transparency, and this trait is a negative in poker!
Andrew: haha fair enough – you can’t win ’em all!
Joshua: I don’t have a twitter account so am thankfully spared seeing people’s behavior on the platform most of the time!
In contrast to other’s takes, such as Chipmunk, my own take is that it’s important to make a lot of mistakes and as quickly as possible. Basically when you start a project to discover something about the world you immediately start doing things like grabbing data, graphing it, and theorizing about what it means. Your early theories are always going to be wrong. The PURPOSE of the work you’re doing is to *make mistakes* and discover them. As you go along you refine the ideas enough that the naive mistakes aren’t there anymore, but there’s still stuff that’s wrong pretty much all the time. The purpose of the project is to stamp down on the wrong bits until they’re small enough that you don’t care about that scale of wrongness anymore.
Navier Stokes equations are *wrong* the real description is closer to molecular dynamics… But it doesn’t matter for designing a sailboat. It does matter for designing micro-fluidics.
Models of how to teach math to 8th graders or Gerrymandering or economic effects of monetary policy are for sure wrong… but the wrongness there is likely to be a lot bigger and more important than the wrongness of Navier-Stokes for sailboat makers. But you still look to stamp down on the parts that are so wrong it leads you dramatically astray. You can’t do that until you have something to work with.
The fear of being wrong is also the fear of *starting a project*. Because the start of a project is always wrong. And if you pretend it isn’t then you make even bigger mistakes. There are people whose whole career in science is built on carrying out **required mistakes** of using NHST to design their research methods. Making these mistakes **benefits them** but harms society.
So, no, we shouldn’t punish mistakes in education by recording some permanent record of them. What matters is not how well you did on the 3rd quiz of the semester, or your first year calculus class or whatever, but what did you actually become able to do by the end of the course, or in the follow-on course, or as a professional. Giving people feedback is important, but recording and averaging the degree of mistakes they made at different stages in education is not itself beneficial. It does in fact lead to a lot of bad risk-management where people choose not to engage with “harder” material because of the “permanent record” of failure. My friend with a PhD in molecular biology never took the “hard” calculus classes or Physics classes in college and settled for “calculus for bio majors” similar to my wife. They’d both have benefited from more physics and more calculus, but they didn’t go there because of the “signaling effect” of what would probably have been mediocre grades. That’s not a good way to design education systems.
A couple things:
1. It’s perfectly possible to both learn from a mistake and publicly deny it. People do it all the time, pretending a mistake didn’t happen in public but correcting it in private and future correspondence.
https://pubs.aip.org/physicstoday/article/58/9/43/399405/Einstein-Versus-the-Physical-Review-A-great
If you’re really determined, you can both convince yourself that it wasn’t a mistake and still modify your future behavior. You can have your cake and eat it too! Of course, it’s not golden all around; the cost is poisoning public discourse and potentially confusing others who might otherwise learn from your mistake.
2. On the subject of education, when I was in high school, the trouble wasn’t scoring or correction, but rather, as Daniel pointed out, the proliferation of permanent records for everything that factor into everything, combined with the increasing competitiveness of certain desirable positions. The accepted GPA and standardized test scores for everything is increasing over time. The number of mistakes you can make on your path to be, say, a doctor, is shrinking and shrinking. The perverse incentives follow; the cheating, the game-playing, the credit-begging, the regrade requests, it all follows from this risk calculus.
I encourage anyone interested to ask the prospective doctors between the ages from 16-22 about their strategy. It’s a fucking nightmare. Take AP psych because it’s an easy A that won’t bring down your unweighted GPA but will boost your weighted GPA, skip lunch to fit in more classes, do the honor society because it involves few responsibilities but counts as a resume line item, apply to this undergrad college because it has higher grade inflation than this other one, don’t do a mathy major to keep your undergrad GPA high, don’t take the MCAT more than once because med schools see all your scoring attempts…
https://forums.studentdoctor.net/forums/confidential-expert-advice.268/
Bask in the misery
Doctors have higher or at least similar suicide rates as combat veterans being treated for PTSD also. Imagine doing all that shit for decades to find out you hate your job, it’s all run by MBAs and you are stuck with $400k in debt.
When a politician admits to a mistake, he gets hammered endlessly by the press. Articles can appear neutral by frequently bringing up his admitted mistake.
I suppose there is not much to add to the blog post and the thoughtful comments. This is deep stuff, to be sure, and harder to live up to than talk about. Normatively, I know that I have to acknowledge my mistakes and learn from them. In practice, some mistakes are really hard to acknowledge. I think the mistakes that are harder to acknowledge are the ones we
(1) think we should not make, because we define ourselves by our expertise (I am so good, I do not make mistakes in this thing.), or
(2) the consequences are so serious that we immediately become defensive (XYZ led to this bad, bad consequence, I am innocent!)
While this blog post and the comments are more philosophical, I know that this is also a topic in psychology, in the area that deals with coping mechanisms.
Move from “I should not make mistakes in this field because I’m an expert” to “I make mistakes in this field, and then when I find them I fix them, and *that’s what makes me an expert*”
Another one of my favorite quotes, attributed to Niels Bohr, is “An expert is someone who has already made every possible mistake in a narrow field.”
The point I tried to get across was, easier said than done. A large part of the problem is not that people are unwilling to correct their mistakes – each of the commenters on this blog post agreed in some way that this is what they do when they make personal mistakes. The bigger problem, I think, is recognising the mistake as a mistake. Nullo actore nullus iudex (If there is no accuser, there is no judge). In statistical language, I would rephrase my point (1) as a strong prior belief in one’s own abilities not to make a mistake in XYZ field; a prior so strong that in the face of undisputable evidence of error, the posterior does not change enough to sway one’s judgement that there is a problem that would need fixing. The problem, then, is to find these overly strong priors/ beliefs about ourselves and the world, and replace them with better ones. Uncovering these false beliefs that we hold is a lifelong exercise if one chooses to embark on this journey. And it takes a lot of the unpopular virtue of humility to succeed on this quest.
My point is just that it is not easy to acknowledge mistakes, because it requires more than just a change in attitude towards mistakes: It takes no less than soul-searching.
In the spirit of bias confirmation, I loved this post. Whenever we have a mistake in our department, one that we catch before there’s a bad outcome, I always make sure we take it seriously and treat it as if there WAS a bad outcome. I tell the team this is a “good mistake” in that we get to run the drills and safeguard our process. Mistakes ignored are crises invited.