“My love she speaks softly, She knows there’s no success like failure, And that failure’s no success at all.” — Bob Dylan
I had an idea related to psychology and science reform, but this time it has nothing to do with replicability of psychology research. It has to do with learning from failure. The idea came up in a discussion I was having with a colleague about the bad behavior of a prominent researcher. My colleague suggested that this guy’s previous successes were bad for him, in that they made him feel invincible. This is an interesting correlation/causation story: if in the past I have made bold guesses and were correct, I’m tempted to think that I can continue to make bold guesses and be correct. Indeed, I can start guessing bolder and bolder, only stopping when my face slams into the wall that is reality. But in social science, a theory can stay aloft forever (reminiscent of the saying that markets can stay irrational longer than you can stay solvent). It struck me that what this researcher (not my colleague, but the other guy we were discussing) has really missed here is an opportunity to learn from failure.
I then started to introspect and it seems that failure has been good for my own intellectual development. I’ve been telling people for years I proved a false theorem. But then someone burst my bubble by pointing out that if it’s false, it’s not actually a theorem. It’s more accurate to say that I published a false proof of an incorrect statement. Anyway, being wrong in that way was been good for me, just making me realize something like that can happen. Similarly, it was good for me that I lost 40% of my life savings in the 2008 crash. In summer 2008 I thought the stock market looked iffy and I thought we should really move our money to something safer, but I didn’t get around to it. Had I done so, I’d’ve been super smug about it, one of those party bores who goes around saying things like, “Well, I’m no economist, but even to me it was obvious that something was iffy with the stock market. I’m not saying the people who lost their money deserved it, exactly, but they were kind of greedy to be staying in at that point, no? Etc etc.” Losing a ton of money saved me from that attitude, which would’ve been both obnoxious and incorrect, as I would’ve attributed to perspicacity what would have been more accurately attributed to luck! Also it wasn’t anything close to 40% if you count anticipated future income as part of my life savings, and it indeed should count. So I learned a lot from my failure, and what I lost from it wasn’t even that much. Another example of scientists learning from failure is the famous “50 shades of gray” experiment of Nosek et al.
At first I was thinking that Nosek et al. and I have been lucky to have such failures from which we could learn so much, but then I realized that just about everybody has failures; it’s just that not everybody learns from the experience. The first step of learning from failure is to admit that you’ve failed, and people often don’t want to do that. Even prominent researchers regularly have failures—but they just don’t take the opportunity to learn from them, and maybe part of this is that scientists aren’t really trained to learn from failures. I mean, sure, there are some famous stories like Thomas Edison trying hundreds of different filaments until he got the light bulb to work, or the double helix dudes going through lots of different models for DNA before they figured it out—but these aren’t really stories of failure, exactly. They’re more stories of persistence leading to success. Actual failure—not just setbacks but actual dead-ends, bad ideas, active wastes of time and effort, etc.—they happen to all of us.
So I was thinking that maybe there’s something to be said here about the importance for a scientist to be receptive to failure. There’s a lot of talk about how scientists should be receptive to success: intuition favors the prepared mind, if penicillin shows up in your lab you should be prepared to learn from it, etc. But maybe not about how we should be prepared for failure and prepared to learn from it. It strikes me that many prominent researchers have failed to learn from their failures. Sometimes this makes sense: if you’re a scientist who’s famous for one thing and it’s wrong, maybe it makes career sense to stonewall it. But there are many cases of prominent scientists who’ve had long careers, and hence the occasional failure, but did not learn from them. And that’s too bad. But maybe part of the problem is they had no template for learning from failure.
I mentioned this idea to a social worker who said there’s a whole literature in clinical psychology on “resilience,” which apparently is not just about being able to recover from setbacks but also involves learning from failure. Would it be possible to adapt this idea to science and science reform? This seems pretty important to me. I’d not thought of this concept at all until recently, but it’s already much studied, at least for people’s personal lives.
I sent the above thoughts to psychology researcher Francis Tuerlinckx (famous as the co-creator of type M and type S errors), and he replied with the above musical lyric and the following reactions from Peter Kuppens:
There is indeed a lot of work on resilience, big name is George Bonanno. A lot of that work is about resilience after some kind of trauma (lost work, partner died etc), but I’m not aware of work around failures in work or academia. There are people who keep a CV with failures, apparently that goes back to this article by Melanie Stefan.
But then Simine Vazire said:
I think there are valid concerns about the validity of Bonanno’s conclusions, and work on post-traumatic growth and resilience more generally. See, e.g., work by Frank Infurna and Eranda Jayawickreme, like the stuff cited in this paper or this paper.
Even setting this dispute aside I think there’s more to be said on the topic of learning from failure. I’m not quite sure what, but I think it should be a more clearly defined part of our education as well as our workflow.
I agree with all you have said. But I kept thinking about Wasnink’s mea culpa. He suffered a variety of failures in his career (losing his graduate funding, being jobless, most recently his retractions and resignation) and he describes (somewhat eloquently) how he learned from his failures. I think he is sincere in that – but I might object to what he learned. So, perhaps it is not just learning from failures, but also learning what to learn from failures.
Dale:
Sure, with Wansink, Hauser, etc., it seems like the lesson they learned from failure is that it was a bummer that they got caught! It’s my impression that, to the extent they have scientific goals beyond being famous and important, they have certain hypotheses they believe are true, and they feel that the point of experimental science is to prove the truth of what they already know. From that perspective, all the criticisms of their research methods and data handling are irrelevant because they already know that their theories are right. They haven’t reached the stage of Nosek et al. of recognizing that they could actually wrong in their theorizing. Perhaps we should chalk this up to a failure of science education, that a scientist can go through his entire education, training, and career without being able to entertain the possibility that he might be wrong.
To put it another way, I don’t think that Wansink, Hauser, etc., learned from failure, in the sense that they never recognized that there were “failures” beyond that they got caught. The first step to learning from failure is to recognize that the failure happened.
This brings to mind how mathematicians often work: We use the term “conjecture” for something we suspect is true, but it hasn’t yet been proven. And we don’t just try to prove it; we also look for “counterexamples” — examples that have the “If” part of the conjecture, but not the “then” part. This approach works well — it doesn’t just reduce the “junk” in the literature; it turns out that often trying to find a counterexample for a conjecture can give insight into how to prove the conjecture– or sometimes just modify it to have slightly different hypotheses.
+1. I think one of the best academic examples of learning from failure is Frege’s response to Russell’s discovery of a paradox in Frege’s system, and I think that is because mathematicians/logicians view negative results as just as important as positive results. Perhaps, all sciences should cultivate that view, teaching the negative results, and how those failures lead to new research programs and theories. The Michelson-Morley experiment is one such famous example.
Interesting thoughts and certainly consistent with the idea that the world model our brains build only ever work so far — until we hit a wall and realize that the model was too limited and needs to be revised. Or we choose to ignore it at our own peril.
But philosophy aside, in motivation science the ability to confront failures and keep at it until you succeed is called achievement motivation. No, I’m not talking about the stuff that the tons of self-report measures out there assess. I am talking about what McClelland and colleagues originally conceived of and measured. And their construct is not about being better than the next person or always doing more than what’s required. Their construct is about the affective response to failure. For the achievement-motivated person, difficulty and failure are Pavlovian cues signaling that the kick of mastery may just be around the corner; hence she or he keeps trying until she or he succeeds. This in turn reinforces the pattern. This is not necessarily a conscious decison; it’s more like an emotional reflex: if you fail, dig in. There’s actually a lot of research that shows that achievement-motivated people are not particularly motivated as long as everything appears to be going well. It’s only when the signal of difficulty or failure comes along that they will become motivated. If they’re in science does the need to achieve help them homing in on the truth rather than becoming blase after a bunch of successes? This would require some in-depth research. But I would think so, because it’s the nature of the need to achieve to get bored by lack of challenge. It may also be instructive to think this through from the other end, from low need for achievement: those are the people who typically shun failure and difficulty, presumably because they have not learned to associate it with the subsequent sweet smell of success, provided that they put in the necessary effort. Hence, I would expect them to revel in success as long as it lasts, yes, but shut their eyes when they encounter failure. They have not learned to make the vital connection between the one and the other.
Huge learning moment for humans, not merely for scientists. Unfortunately, “learning from failure” is nearly always framed as an encouragement to persist: “if at first you don’t succeed…” and other similar tropes. The much more valuable lesson is that ignoring failure leads to increased self-belief: “if I think something, it is, obviously, a good idea” and the inevitable corollary, “counterpoints are attacks requiring me to hold my beliefs more strongly.” Failing to learn the one leads to less success; failing to learn the other leads to institutionalized slavery, as one obvious example.
Two other points and I’ll shut up.
Talking heads and academics (and many others) are often punished for even acknowledging failure and richly rewarded precisely for not learning from it. What do we think will happen?
Resistance to learning from failure is of course a natural child of fearing it in the first place. Insidious secondary effect: fearing failure leads to limiting success. If I try something new, I might fail, so better stay in my lane and crank out non-insightful bullshit. And so the industry of unhelpful chronic succeeders…
I think if it not so much as “learning from failure,” as not personalizing success or failure.
Saying that “I will learn from failure,” as essentially meaning “I am a person who learns from failure,” as compared to people who don’t learn from failure, it seems to me, mostly perpetuates the problematic mindset. Resliance, imo, largely means not personalizing failure, which actually goes hand in hand with not personalizing success – which is a tough nut to crack.
I think this is less about categorizing people into different groups based on how they deal with failure, so much as recognizing that there are sliding scales that effect us all, in different ways in different contexts. We are all dopamine producing and assimilating machines.
Isn’t this somewhat what competitive sports is about? A loss is a clear “failure”, and if you are not willing to go over and figure out why you lost, you rarely get better. Sometimes it is just that the opponent is better physically or athletically, but often it is correctable things you can improve on. This said from having coached for 30 years. You always need to learn from your losses.
+1. You always hear from sports broadcasters like a particular star “hates to fail” and that hating to fail is the key to success. But it’s not the steely determination not to fail that keeps people from failing. It’s learning from failure. The steely determination is what allows you to learn from failure.
Second, there are a lot of people who don’t like Megan McCardle (I’m not among them) but her book The Up Side of Down, which is about the essentiality of learning from failure, is really good (and with blurbs from Tyler Cowen and Tim Harford, how could it not be?)
Jonathan (ao) –
If you can get to it, I’d appreciate a reponse from previous thread.
> But it’s not the steely determination not to fail that keeps people from failing. It’s learning from failure.
I think there’s another distinction. Some people are less deterred or discouraged by failure than others. Some are more driven to succeed than others.
With sports, I think that’s essentially what is meant by steely determination not to fail. I think of LeBron, or Steph, and their determination to always add to their game, and their incredible commitment to conditioning (which is interesting to compare to someone like Bird who wasn’t really ever particularly “in shape”).
Of course it’s a mistake to think we can really understand athletes’ personalities merely from observing them play or from reporting.
But as a 6ers fan, Ben Simmons is an interesting counter-example.
Plays extremely hard and aggressively in some ways when on the court. But often is judged as having a fear of failure as negatively shaping his game in very significant ways. Hasn’t improved his game at all over four years. Still has that same chicken-wing, no knee bend form on foul shots despite being bad by historic standards.
I think there are two levels of agnosticism: (1) Learning counterfactual outcome is hard. It is because of the “fundamental attribution error”: We can but we seldom adjust for confounders when it comes to one’s own life. (2) Making causal inference on never-taker or always-taker is impossible or irrelevant. Pedantically speaking, “previous successes made him feel invincible” should be a wrong conclusion to draw in either view. View 2 is also interesting in providing another potential benefit of failure: failure (or a uniform distribution of success and failure at early stage) provides a better estimation of treatment effects, and better treatment effects leads to better Bayeisan optimization and experimental design in the longer run or the larger population.
One reason to learn from. Failure is just that the base rate of failures is much bigger than the base rate of successes (since there’s many more ways to miss the mark than hit it).
unless you move the mark to where you hit
(all too common among researchers)
Shifting our gaze from that of the individual there has been some wonderful work done on learning from failure at policy and organizational scales.
Dorner’s book “The Logic Of Failure: Recognizing And Avoiding Error In Complex Situations” from 1997 is a great read.
Bovens, M. & ‘t Hart, P. (2016) Revisiting the study of policy failures. Journal of European Public Policy, along with their earlier work Bovens, M. and ‘t Hart, P. (1996) Understanding Policy Fiascoes, New Brunswick, NJ: Transaction are really insightful.
so, uh, should i get out of the market now?
Yes.
Either it’s a good decision, or it’s a bad decision you can learn from. Win-win. ;-)
Andrew – the issue you raise is methodological. In statistical analysis there is ample use of back and forth processes which are driven by “failures”: The model does not fit, the simulation does not converge, the results do not match with first principles…
Moreover, published papers appear carved in stone. As you have repeatable demonstrated, there is room in revisiting publications, update and correct them.
Statistical methodology should explicitly account for this.
In teaching statistics one often used “successes”, i.e. data that can be analyzed with a reasonable regression model and, in fact, students do not learn a lot from this. Focusing on unstructured problems as a starting point and experiencing the back and forth steps mentioned above would probably teach much more, and more effectively.
Teaching statistics should explicitly account for this.
+1. Textbooks are the worst as they predominantly contain “success” stories. It is much more fun for students (and I can attest your teaching evaluations will be higher) if you use examples where the analysis being learned “works.” Data sets where the analysis is likely to lead to failures, where nothing definitive can really be said, just don’t go over well in the classroom. I’ve resisted the temptation to feed students successful analyses, but I can tell you it is not popular.
Perhaps, textbooks should include more history of science rather than treating scientific/mathematical as if it had descended from God.
Steve,
That would be fine, but my recollection from reading textbooks as a student was that when they did history of science, it was pretty much straight-up great man theory, so you’d hear about the brilliance of this or that discoverer. That’s one reason I didn’t put history of science in my own books: I was reacting against the annoying version of it I remembered from textbooks of the past.
When I did maths tutoring, when I saw a student make a mistake, I learned to not correct them, but instead to wait, and if necessary, help them splt something is wrong, find the error, and correct it. (Something most people learn when they take up computer programming.)
The experience of being able to transform a failure into a success by themselves helped students recover their confidence who thought of themselves as mathematical failures.
For a complete “learning from failure” arc, you need positive reinforcement in the end. If you leave that out, it won’t work as well, if at all.
Dear all,
this entry reminds me of Schulz, K. (2010) Being Wrong. Adventures in the Margin of Error. Portobello Books.
It was, imho, a good read.
Relatedly, how do we teach from our failures to our students? Failure is very much experienced as a within-person process, and how do we share that (how do we go from within- to between- here?) so that we truly capitalize on experience/knowledge? It seems to me that failure is a great source of knowledge but it goes mainly in the tacit knowledge category which is hard to transmit formally.
One example of an anti-fragile set of operating procedures to take inspiration from is the NTSB, specifically regarding how they learn from every failure in airflight. NASA has a similar program. On a less institutional level, it is common when coding web applications to set up a “test suite” that ensures that bugs are only seen *once* and any new bugs are not repeats of older issues. Atul Gawande makes a similar point in the The Checklist Manifesto. The thing these all have in common is ensuring that any learnings are made concrete.
To the extent that accurate science is “mission critical”… well, the question then becomes how might we set up something like one of the aforementioned programs for science?
Regarding learning from failure and the frequent inability of even very smart people to do it, there is a great quote from the economist John Kenneth Galbraith:
“Faced with the choice between changing one’s mind and proving there is no need to do so, almost everyone gets busy on the proof.”
Contrarian view here.
Let’s start by separating what needs to be learned from how it can be learned.
Given a set of things to learn, failure is a great opportunity to learn. It makes us stop, rethink, check, and question ourselves.
Where I’m saying the same thing as Andrew: failure helps us learn, and the lack of failure makes it harder to learn certain lessons. That’s why we give grad students opportunities to fail and sometimes less guidance than they think they’ll need.
I’m also saying something completely different: we should be learning the hard lessons regardless of failure. Yes, we can all improve the ways we learn from failure, including milking even smaller failures for their lessons. But wouldn’t it be great if we were all more aware of our flaws, regardless of whether they fly in our face?