Embracing Variation and Accepting Uncertainty (my talk this Wed/Tues at a symposium on communicating uncertainty)

I’ll be speaking (virtually) at this conference in Australia on Wed 1 July (actually Tues 30 June in our time zone here):

Embracing Variation and Accepting Uncertainty

It is said that your most important collaborator is yourself in 6 months. Perhaps the best way to improve our communication of data uncertainty to others is to learn how to better communicate to ourselves. What does it mean to say to future-you: “I don’t know”? Or, even more challenging, “I know a little but I’m not completely sure”? We will discuss in the context of applications in drug testing, election forecasting, and the evaluation of scientific research.

P.S. The session was fun. It was a remote conference so there was a chat window with questions and answers. Here are the questions that were addressed to me, and my replies:

Q: US-based data journalist here. In reporting racial justice, what stastitical concepts do you suggest we use when exploring disparate proportions (for example percent of Black Americans in the population versus percent killed by police)? And where do we need to highlight uncertainty? Keep in mind our general audience.

A: I like the recent paper by Gaebler et al., Deconstructing Claims of Post-Treatment Bias in Observational Studies of Discrimination: https://5harad.com/papers/post-treatment-bias.pdf

Q: I am currently building Bayesian based COVID-19 case predictions. I was wondering, in your opinion , what type of uncertain variables (such as asymptomatic cases) could we consider in Bayesian prediction of COVID-19 r nought numbers?

A: It’s hard or me to answer such a general question. But, yes, you’d like to have latent variables such as the number of asymptomatic cases (and how this varies over time) in your model.

Q: How can we best integrate non-statistical expertise with statistical evidence to bridge the gap between data and conclusions?

A: If you can track down the assumptions that are involved in the statistical model, then you can bring in non-statistical expertise to evaluate and perturb these assumptions.

Q: Do you think there’s anything that can be done unless you actually “build a model” of the unknown process? (eg wrestler v boxer). Or is it better to go Decision Theoretic?

A: It’s not either/or. You can build a model–or, at least, consider what information it would take for you to build a model–and in the meantime you can use decision analysis to translate your uncertainties into decision evaluations.

Q: In the resources industry, there are work-culture incentives to overstate certainty, and penalise caution. I think this operates elsewhere too. Any comments on implications of this?

A: Yes, we sometimes talk about the “auto mechanic” incentive. The auto mechanic has an incentive to tell you he knows what’s wrong with your car. If he acts uncertain, you might go to a different mechanic.

Q: You mentioned story telling as a soft skill when explaining stats to a general audience. What are some of the ways to engage the audience without getting too technical?

A: I guess the key thing is to contribute in your expertise. If you have technical expertise and people are talking with you, then I think they will want to hear it.

When it comes to communicating uncertainty, I like Gerd Gigerenzer’s idea of using “natural frequencies.” For example, instead of saying 3% are exposed, you say: “Consider an auditorium with 1000 people. 30 would be exposed.”

Q: Asking from a foundation of ignorance…is it more likely that a model that predicts a less dramatic result (e.g. 50/50) more likely to be wrong by chance than a model that predicts a less “likely” distribution of results (e.g. 90/10). I.e. if a model that predicts something unexpected represents the real data well, is that model more likely to be theoretically correct than a model that predicts something less unexpected. (Feel free to ignore if this question doesn’t make sense – or would be better asked in a different session).

A: Any prediction can be wrong. One question is what are you going to do with it. Is a prediction just for fun, or will it affect decisions? What of your decisions might be different if you are told that Biden’s win probability is 90% rather than 50%?

Q: I am interested in iterative near-term forecasts of biodiversity. In this approach the model captures the process and also quatnifies the uncertainty in making the forecast, then new observations are used to update each forecast.

Do you see any issues with iterative near-term forecasts for the statistical concepts of uncertainty?

A: It’s good to consider iterative forecasts. Iterative forecasts must satisfy certain mathematical conditions–the so-called martingale property. Recognizing these constraints can help you find incoherence in your forecasts.

It’s fun having a stream of questions. It becomes like a game, to answer them as fast as they come in.

4 thoughts on “Embracing Variation and Accepting Uncertainty (my talk this Wed/Tues at a symposium on communicating uncertainty)

  1. ‘Embracing uncertainty’ sounds like the defeatist serenity prayer of a statistician going through a mid-life crisis. The purpose of empirical research is to reduce uncertainty. If that’s not your goal then give up now.

    • Sentinel:

      Read the above title carefully. I never said anything about embracing uncertainty. I do want to reduce uncertainty, but I recognize that I can’t reduce it to zero, and I think we should accept the uncertainty that remains.

    • I think that everyone can agree that we should try to reduce uncertainty when we can.

      But the issue is about the common practices of pretending that uncertainty doesn’t exist, or pretending that we are more certain about things than we can reasonably be. In other words, we shouldn’t sweep uncertainty under the rug not that we should take things we already know and make them uncertain.

Leave a Reply

Your email address will not be published. Required fields are marked *