The checklist manifesto and beyond

A few years ago, I was motivated to write about the intervention and the checklist: two paradigms for improvement, after reading Atul Gawande’s classic checklist manifesto. My angle was that in statistics we are trained to think about interventions (and studying their causal effects), but intervention is not the only paradigm.

Gaurav Sood sends along the following contextualization of the checklist issue:

We fail because we don’t know or because we don’t execute on what we know (Gorovitz and MacIntyre). Of the things that we don’t know are things that no else knows either—they are beyond humanity’s reach for now. Ignore those for now. This leaves us with things that “we” know but the practitioner doesn’t.

Practitioners do not know because the education system has failed them, because they don’t care to learn, or because the production of new knowledge outpaces their capacity to learn. Given that, you can reduce ignorance by (a) increasing the length of training, (b) improving the quality of training, (c) setting up continued education, (d) incentivizing knowledge acquisition, and (e) reducing the burden of how much to know by creating specializations, etc. On creating specialties, Gawande has a great example: “there are pediatric anesthesiologists, cardiac anesthesiologists, obstetric anesthesiologists, neurosurgical anesthesiologists, …”

Ignorance, however, ought not to damn the practitioner to error. If you know that you don’t know, you can learn. Ignorance, thus, is not a sufficient condition for failure. But ignorance of ignorance is. To fix overconfidence, leading people through provocative, personalized examples may prove useful.

Ignorance and ignorance about ignorance, however, are not the only reason we fail. We also fail because we don’t execute on what we know. Practitioners fail to apply what they know because they are distracted, lazy, have limited attention and memory, etc. To solve these issues, we can (a) reduce distractions, (b) provide memory aids, (c) automate tasks, (d) train people on the importance of thoroughness, (e) incentivize thoroughness, etc.

Checklists are one way to work toward two inter-related aims: educating people about the necessary steps needed to make a decision and aiding memory. But awareness of steps is not enough. To incentivize people to follow the steps, you need to develop processes to hold people accountable. Audits are one way to do that. Meetings set up at appropriate times during which people go through the list is another way.

Sood also shares some interesting notes he prepared on Gawande’s classic checklist manifesto. Here are some excerpts from Sood’s notes:

Portions of the book suggest that this is less about checklists and about engineering processes that reduce errors. Any process can be called a checklist—you do X followed by Y followed by Z—but that is stretching it.

. . .

Problem Statement: How do you make sure that people know and are following the process correctly?
To improve outcomes—study routine failures and how you would amend the process to improve outcomes.

. . .

Americans today undergo an average of seven operations in their lifetime, with surgeons performing more than fifty million operations annually—the amount of harm remains substantial.

. . .

“On average, the study reported, it took doctors seventeen years to adopt the new treatments for at least half of American patients.”

. . .

Concern w/ Some Checklist Implementations And Some Solutions

– Incentives for following checklists may be weak. How do you get people to follow?
– social pressure — checklist publicly marked as in a surgery
communicate clearly the issues and evidence on the efficacy of checklists
– get people to own the checklists—put their name, get their ideas on it — induce accountability
– checks of whether the stuff was followed and incentives and rewards based on that.
– People may stop using their brain and just follow the checklist
-how to put in checklists that clarify that brain cells are imp. and incentivize that.

– Train people to use checklists

18 thoughts on “The checklist manifesto and beyond

    • I don’t think that book review accurately reflected the book, at least as I recall it (and I did read the book). The critique seemed to involve examples of bad checklists. My reading of Gawande was that checklists are extremely important and useful – if applied in the right contexts and done well. Many checklists are done poorly and misapplied. I did not read Gawande as advocating checklists for all circumstances, but I think the reviewer was attacking that straw man (I wonder when we will get to straw women).

      • Checklists have had great empirical success in most high stakes, rapid decision making domains for decades. e.g. Airline pilots work with a thick binder full of checklists for almost every eventuality.

        Chemical plant control rooms and those are full of checklists too. Similarly for nuclear plants I think. Most large equipment (e.g. turbines, generators etc.) also come with checklists made by the manufacturer for troubleshooting a large set of problems.

  1. We’ve been working on a checklist and pedigree table for assessing the uncertainty of our models. It’s a work in progress and comments are welcome.
    http://opensource.nibr.com/xgx/Resources/Uncertainty_Assessment_Pedigree_Table.pdf

    Key elements in the check list are reminders to assess uncertainty with regards to the data the model is based on, the underlying structural model itself, and the underlying assumptions of the models.

      • +1 very cool.

        It made me think that I confuse “best I can do” with “best possible” when I’m thinking about stuff I’ve done.

        Cause if someone asks you about something you’ve done, you’re sort of answering on a 1 parameter scale and that’s somehow unsatisfactory. It either sucks compared to some things, or is good compared to some things, or is average, … It’s misleading all around.

        I like the idea of having a few checklist items to go through and say “It sucks here, but it’s good here, and it’s alright here” and not having to ad lib that checklist on the spot.

  2. Greetings Andy,

    Can you recommend an article or book that lists out the criteria? What has been most helpful to me is when scientists have discussed their work informally. Then I at least can speculate on the biases more easily. That is to say they are more amenable to answering questions raised about their reasoning. I favor collaboration across different disciplines as well toward assessing the quality of data and model.

    Publication bias can blind us.

  3. Speaking of checklists, I get the feeling that most contracts and legislation is currently written as a big verbal block of hard to read text always cross referencing other parts by number etc when, in fact, structurally it ought to be like a nested block of if then else loops. Or even better a graph.

    Wonder if there are any efforts to represent things in this way. Does the lack of this derive from the fact that most lawyers do not come from a math / sciences background?

    • Good questions. A “nested block of if then else loops. Or even better a graph.” does sound much better to me than “a big verbal block of hard to read text always cross referencing other parts by number etc”. I wonder if this is connected with preference/inclination for “linear” thinking vs preference/inclination for “spatial” thinking.

      • > preference/inclination for “linear” thinking vs preference/inclination for “spatial” thinking.

        I’ve noticed this distinction in scientific writing as well (here referring to journal articles). Speaking just for myself, I read very spatially, jumping around between points of interest in an article in order to construct my own understanding of the material rather than linearly read from start to finish. I feel like I get a lot more out it, though I can’t say if it’s the “spatial” aspect or the active aspect that helps.

        I also tend to write very “spatially” in that I use lots of subheadings, cross-referencing, and appendices and fully expect readers to skip over and around these sections in a nonlinear way according to their interest. But of course lots of people—including many with whom I’ve collaborated!—don’t think about writing or reading this way and have a more typical “narrative” approach, so it’s a challenge to try and satisfy both types of writers/readers.

    • I don’t think the goal of most contracts and legislation is to be clear and helpful. It’s the opposite: obfuscation and needless complexity that begat more fees.

  4. “Americans today undergo an average of seven operations in their lifetime”

    What counts as an “operation” in this statistic? For example, does it include having a skin tag removed by freezing? Routine colonoscopy? (or only if polyps or something else unusual are found and removed?)

  5. It seems the checklist issue would be context specific.

    Frankly, I was astonished checklists were not already a thing in healthcare when they became a thing — given the potentially severe cost of errors. In my experience, in research and data analysis people who make lists and check them off are more the norm rather than the exception. We are, on average, a fastidious lot relative to population norms. What I believe the greater danger for the social sciences is “– People may stop using their brain and just follow the checklist”. This must surely be how the NHST problem began to spread across areas of research and disciplines of study.

    Recipes are a great way to begin and a great way to disseminate new, improved, or scarcely used methods. They are a great way to begin learning about a method you don’t yet understand. When they become a dogmatically required check mark, they will inevitably become a barrier to innovation (NHST). In some sense this is simply a part of how humans typically behave in groups. We learn from others’ examples, we teach others based on what we’ve learned and what we’ve managed to devise on our own, and we argue about which processes lead to the best outcomes.

    Read cookbooks. Learn to cook. Share your own recipes. Argue about which is better and how that can be determined. Wake up and do it all again. Not a bad way to make a living, if you can.

  6. I have a family member in a hospital right now. Not COVID, scary to be in hospitals now.

    The “care team” is at least four people. If you talk to them individually, each gives ~half the story; and for each ~half their story is original, and half is bits and pieces of the others’ stories.

    A perfect situation for a checklist.

Leave a Reply to binx ballin' Cancel reply

Your email address will not be published. Required fields are marked *