This is Jessica. Yesterday I had the chance to speak at a public session of the President’s Council of Advisors on Science and Technology. The topic was communicating science to the public. The task put to the speakers was to provide concrete recommendations in the form of organization X should do Y by time Z. 

You can watch the session here, it’s the second half. I’m not sure if I succeeded in being as specific as they wanted, but here’s a summary of what I talked about: 

My premise, following points previously made by Chuck Manski, is that conventional certitude – the practice of presenting point estimates as if they are ground truth because that’s how it’s always been done, or that’s what’s expected from consumers – pervades government reporting of data-driven estimates. The Census reports population estimates, including for the entire country, to the single digit. CBO, BLS, BEA report estimates of the federal budget, unemployment rates and counts, GDP, without acknowledging uncertainty or acknowledging only some forms (e.g., sampling error) and burying the information about those away from the top level estimates. The CDC has been reporting total and new infections and deaths throughout the pandemic as point estimates, despite widespread scientific acknowledgment that the data was crap early on. Etc.

There are various documented reasons why scientists and other experts are wary of expressing uncertainty, the most obvious of which is that they think that the average person won’t know what to do with uncertainty. These days, where we see belief in science itself becoming politicized, wariness about conveying uncertainty may also stem from concerns that admitting to any fallibility or possible error in scientific forecasts can be dangerous because it might be weaponized.

But this kind of thinking ignores the basic contract that needs to be in place for public trust in government estimates to last. At the very least, members of the public should be able to expect that the government will be honest about how much they know. Even if a person does a poor job of translating from a distribution of possible outcomes to a decision strategy in a given situation (i.e., they don’t know what to do with the uncertainty as feared) it’s still better to have expressed it. Doing so establishes the basic requirements for individual accountability and reduces the chances that the forecaster will be blamed (see e.g., Susan Joslyn’s work).  

I used the Census Bureau debacle over the new DAS to illustrate what can happen when the veneer of conventional certitude is challenged. The Census has been noising data for years, but then computer scientists come up with a better set of approaches based in differential privacy which involve adding calibrated noise to the data. So the Census updates their pipeline to accord with the state-of-the-art. There are some differences (e.g., block counts are no longer invariant), but by and large the public and other stakeholder reactions are much more drastic than any available evidence that the new system is substantially reducing the usefulness of the data — most analyses have suggested differences are fairly nuanced. What did change is that suddenly the idea that Census data is precise, or that we don’t have to consider possible error when we consult it, was challenged. This shouldn’t have been a revelation, but somehow has been shocking enough to enough people that the legitimacy of Census data is now being questioned. 

So my high level advice was, quantify uncertainty wherever possible and report it with all point estimates. But step two is the need to be strategic about how you communicate it. Verbal phrasing like “masks can help stop the spread” is obviously better than saying “masks stop the spread,” but we shouldn’t assume that verbal expressions are the only way to express uncertainty. There are lots of ways to communicate uncertainty that acknowledge the need to make information engaging and concrete and of varying levels of resolution while also anticipating that people will try hard to ignore it, including:

-Visualizing it to capture attention, and because what we visualize implies what we think is important. Relegating uncertainty information to liner notes or linked spreadsheets while putting unadorned point estimates on the main page tells the public we are pretty sure we can’t be wrong, which works great until we’re wrong of course.

-Use frequency framing including icon arrays for visualizing base rates and test error rates at the same time (e.g., this) and sets of icon arrays for relative risks (e.g., this). Use frequency formats for continuous variables (quantile dotplots, hypothetical outcome plots) in place of error bars or text intervals, which tend to produce biases. 

-Use sets of scenarios or narratives or anecdotes with information about how representative they are. E.g., if you want to communicate how effects of a new health or climate intervention or law might play out differently based on circumstances, precede descriptions of the scenarios with language like “Here’s something we expect to see a lot,” “here’s something we expect to see sometimes”, “here’s something that could happen on rare occasions, but which is worth considering because of the high stakes.”

-Tailor information to different needs and levels of attention (many agencies like CDC do this already), but in doing so, integrate uncertainty information at all levels (which no one seems to be doing well), not just in the detailed reports that are hidden behind multiple clicks. And, expect people to be trying to suppress the uncertainty at all levels. I used election forecasting and the progression of FiveThirtyEight’s top level forecast displays between 2016 and 2020 as an example. In 2016 we got text probabilities, which many probably rounded for lack of a better idea of what to do with them. In 2020, we got a grid of colored maps proportional to the forecast’s prediction of Biden’s chances of winning, and a sentence saying he was favored to win. It’s easy to ignore the uncertainty in the former, hard not to internalize it in the latter. 

-Explicitly acknowledge transitory uncertainty (another term Manski has used). Many government agencies make revisions to estimates over time (e.g., the BEA regularly revising GDP estimates), and of course scientists revise their estimates about climate, health outcomes, etc. across papers over time. It makes no sense to report point estimates when we know revisions are coming. A simple starting place when a modeling approach is relatively established is to assume the revision process is stationary and use past data to estimate how much an estimate might be revised in the future; this uncertainty can also be propagated forward even after observed data has come in. Bank of England fan charts are a great example. 

Additionally many agencies such as CBO and BEA have the necessary information to calculate the error rate of their past forecasts, but don’t represent it, or will report it separately, using units that are not easy to judge among non-experts (e.g., here). Amanda Cox’s Budget Forecasts, Compared to Reality chart (adaption here) is a very simple way of expressing past prediction error, all you have to understand is that the light blue lines are the guesses and the thick line is what was observed. This kind of chart should come standard with any reporting of new projects for an established model. More forthright communication about how estimates or recommendations have changed over time could also be useful for that matter, to signal to the public that the government is aware that evidence changes. 

When transitory uncertainty is hard to quantify exactly, such as early on in the pandemic where many were struggling with what assumptions were appropriate in trying to estimate the amount of bias in early covid infection rates, labeling data with qualitative quality scores like low, medium and high based on expert guidance could help people adjust their sense of confidence in their decision strategy to the quality of the data that its based on. You could even color code the estimates with familiar stoplight colors to visually suggest that some come with a warning of poor input data quality. 

-Label partial expressions of uncertainty (i.e., of risk) incomplete. On the occasions when quantified uncertainty is reported, it represents uncertainty only in the narrow “small world” sense defined by the assumptions of the model. For example, BLS reports sampling error, but not non-sampling errors like non-response. In instances like epidemiological models used to project outcomes under certain scenarios, like the SEIR models behind covid policy, major classes of outcomes are ignored, like behavioral or economic responses. The problem is that when most people see a set of predictions or a chart and they even have error intervals, it’s easy for them to assume that they’re seeing a complete expression of uncertainty. We should be labeling forecasts from models in a way that conveys that they are exploratory, like “results of hypothetical experiments.” 

I ended it by going back to the Census example I started with, and suggesting (as others have) that the Census Bureau transition to releasing the noisy measurements file in the future, rather than only the post-processed data meant to keep up the appearance of precision. There’s a perspective from which negative counts, and counts that don’t aggregate perfectly over different areal units, could be a feature in the sense of normalizing public expectation that no data-driven estimates are perfect, rather than a bug that we need to shield the average person from. I guess this view could be controversial. 

The other talks (by Arthur Lupia, Consuelo Wilkins, and Kathleen Hall Jamieson) were quite interesting. Hall Jamieson had a number of specific ideas on how vaccine communication could be improved, including simple verbal changes like community immunity instead of herd immunity, and not calling it the Vaccine Adverse Event Reporting System which implies that a causal link between the vaccine and whatever happened exists. Wilkins made points about the need to understand the reasons different communities might lack trust and what their priorities are (especially marginalized communities whose concerns about engaging with the government are more likely to concern issues like privacy and profit incentives behind the research). This resonated with me as I think the value of the kinds of methods that make up user-centered design, which is the de facto approach to designing software interfaces, are often overlooked or just not as widely familiar as they should be (ie empathizing with the audience goes a long way). Also having thought about how one could realistically get public buy-in to the new Census DAS has made clear to me that trust has to be built through community leaders first (imagine trying to explain differential privacy concisely to the average person; probably not going to work!) Lupia talked about the need for science communication to separate the recommendations being made (which are based in values that can be contentious depending on politics or ideology) from the evidence, and suggested a template approach to any recs that would separate the two.