Storytelling as predictive model checking

[cat picture]

I finally got around to reading Adam Begley’s biography of John Updike, and it was excellent. I’ll have more on that in a future post, but for now I just went to share the point, which I’d not known before, that almost all of Updike’s characters and even the descriptions and events in many of his stories derived from particular people he’d known and places he’d been. Having read different stories by Updike in no particular order and at different times in my own life, I’d not put that together.

Today’s post is not about Updike at all, though, but rather about a completely different style of writing, which we also see in many forms, which is storytelling as exploration, in which the author starts with a character or scenario that might be drawn from life or even “ripped from the headlines” and then uses this as a starting point to explore what might happen next. The writing is a way to map out possibilities in a way that follows some narrative or other structural logic.

There are different ways of doing this as a storyteller. You can start from the situation and work from there in a sort of random walk, or maybe I should say an autoregressive process in that the story will typically drift back to reality or to some baseline measure. (I’m reminded of my point from several years ago that the best and most classic alternative history stories have the feature in common that, in these stories, our “real world” always ends up as the deeper truth.) Or you can set up an intricate plan that links individuals to social history, as was done so brilliantly in The Rotter’s Club.

I got some insight into all this recently when reading a posthumous collection of Donald Westlake’s nonfiction writing. (That’s the book where I encountered this list, which observant readers may have noticed I’ve been using for new post titles.) Somewhere in this book, I don’t remember where, it comes out that Westlake did not plot his novels ahead of time; instead he’d just start out with an idea and then go from there, seeing where the story led him. This was a surprise to me because Westlake’s novels have such great plots, I’d’ve thought this would’ve required careful planning. Upon reflection, though, the plot-as-you-go-along scenario seemed more plausible. At the purely practical level, the guy had tons and tons of experience: he’d skied down all the runs before so he could find his way without a map. And at a more theoretical level—and that’s why I’m bringing all this up here—one could say, with reason, that the development of a story is a working-out of possibilities, and that’s why it makes sense that authors can be surprised at how their own stories come out.

Starting at the beginning and going from there: This can be a surprisingly effective strategy, especially if you’ve done it a few times before, and if you have a bit of structure. Structure can work in a direct way: from page 1, Richard Stark knows that Parker’s gonna get out alive by the end, so it’s just a matter of figuring out how he gets there (I just reread Slayground the other day, which was fun but it got me sad to realize that I no longer have enough days forthcoming in my life to reread all the books on my shelf). Or structure can work indirectly, as in Westlake’s novel Memory (posthumously published but written in the early 1960s), which brilliantly works against various expectations of how the story will develop and resolve. In either case, though, if you start with confidence that you’ll get through it and you have the technical tools, you can do it.

(Indeed, the thoughts that led to the present post arose indirectly from the following email I received the other day from Zach Horne, a man I’ve never met. Horne wrote:

I regularly read your blog and have recently started using Stan. One thing that you’ve brought up in the discussion of nhst is the idea that hypothesis testing itself is problematic. However, because I am an experimental psychologist, one thing I do (or I think I’m doing anyway) is conduct experiments with the aim of testing some hypothesis or another. Given that I am starting to use Stan and moving away from nhst, how would you recommend that experimentalists like myself discuss their findings since hypothesis testing itself may be problematic?

My reply was that this is a great question and I will blog it. I’m looking forward to my reply because I’m curious about the answer to this one too. Like Donald Westlake, I’ll start at the beginning, go from there, and see where the story ends up.)

Anyway, to return to the main thread:

If storytelling is the working out of possible conclusions following narrative logic applied to some initial scenario, then this can be seen as a predictive endeavor, in the statistical sense. Or as Bayesian reasoning: not in the canonical sense of inference about parameters or models conditional on data, but Bayesian inference for predictive quantities conditional on a model which in this case is unstated but is implicitly coded in what I was calling “narrative logic.”

In statistics, one reason we make predictions is to do predictive checks, to elucidate the implications of a model, in particular what it says (probabilistically) regarding observable outcomes, which can then be checked with existing or new data.

To put it in storytelling terms, if you tell a story and it leads to a nonsensical conclusion, this implies there’s something wrong with your narrative logic or with your initial scenario.

In one of my articles with Thomas Basbøll, we discuss the idea of stories as predictive checks, there focusing on the idea that good stories are anomalous and immutable. Anomalousness is relevant in that we learn from stories to the extent they force us to grapple with the unexpected, and immutability is important so that surprising aspects of reality cannot be explained away in trivial fashions. (That’s where copyist Karl Weick went wrong: by repeatedly changing his story to suit his audiences, he removed the immutability that could’ve allowed the story to help him learn about flaws in his understanding of reality.)

P.S. Tyler Cowen illustrates the general point in this recent post.

9 thoughts on “Storytelling as predictive model checking

  1. This post resonates as well as hitting on what I’ve been thinking about Wansink’s work. I was appalled both by some of his research and especially his reactions to the errors that were found. I in no way want to excuse errors, poor methodology, poor advice to researchers, etc. But I also went back and read through his blog posts and the reactions to his posts and his reactions to those reactions. I am somewhat more sympathetic to his view than before. Part of this may be that I share some of his experiences – losing financial support in a PhD program, needing to find a new advisor, not getting tenure at a couple of schools, etc. The difference is that he then hit upon “success.” And it seems to be based on telling good stories.

    Yes, we can criticize and be distressed about much of his work and reactions. At first, I thought his reactions were so bizarre that they were machine-generated. However, after reading them again, I think he is serious and is listening. He reaches different conclusions about the meaning of the criticisms. He has been successful and none of the criticism appears to have shaken that belief. So, he accepts much of it and says he will do better. But he appears to think he is doing fine.

    This is where the storytelling comes in. I think he is doing something along the lines you have described above. The difference – and unfortunate thing – is that he needs to couch his stories in terms of standard statistical practice, making a mockery of it in the process. Why isn’t it enough to run his experiments with a small number of undergraduates, and if something interesting appears, conduct a somewhat larger study to see if it goes anywhere. It can be a good story. It can even lead to serious research. But it seems that the requirement that his under-powered and ill – thought out studies lead him to do poor work. He may not care because it is the idea that matters to him, not the statistical or scientific integrity. Yes, that matters to me, but I think his work may have its place. As Andrew has said in a number of other posts, explorations are fine, just don’t pretend that statistical significance is a useful measure to employ on such stories. Perhaps academia has left no room for good stories to be “publishable” without the veneer of “scientific” validity.

    • Dale:

      I had similar feelings as yours regarding Wansink’s post—at first. But then I learned that he did nearly the exact same thing five years ago in response to earlier criticism. Which makes me concerned that his responses are part of a public relations strategy and not a sincere as they might seem based on their wording.

      • Andrew: The earlier exchange mentioned by Tim Smits was not with Brian Wansink himself but with Mitsuru Shimizu, who was a post doc in Wansink’s lab for about 5 years. Of course, Shimizu may have been responding to Smits under Wansink’s direction, but we don’t know that.

    • Ironically, academic publishing incentivizes this story telling bit.

      To the extent that in a brutal system of tenure etc. people learn to indulge in storytelling at all costs, even bending the truth.

  2. Interesting, I could have used what as at the bottom of page 16, top of 17 of http://www.stat.columbia.edu/~gelman/research/published/storytelling.pdf here http://statmodeling.stat.columbia.edu/2017/01/11/the-prior-fully-comprehended-last-put-first-checked-the-least/

    In that exercise one did not want a representative draw from the prior but rather a challenging draw – so the set to be generated was “designed” to better test the prior.

  3. “almost all of Updike’s characters and even the descriptions and events in many of his stories derived from particular people he’d known and places he’d been”

    If you value your privacy, it’s toughing being related to a great novelist.

    Many top novelists are ruthless about exploiting their loved ones’ private stories for their books. General readers probably won’t figure out who the stories are about, but the loved one’s friends will.

Leave a Reply to Carol Cancel reply

Your email address will not be published. Required fields are marked *