Peter Dorman writes:
This piece by Robinson Meyer and Alexis Madrigal on the inadequacy of Covid data is useful but frustrating. I think they could have dispensed with the self-puffery, rhetoric and sweeping generalizations and been more detailed about data issues. Nevertheless the core point is one that you and others have stressed, that too much attention is given to analysis when measurement itself is often the biggest problem.
There is a secondary question the authors never raise: why people in leadership positions were so acquiescent to inaccurate and misleading data production. No doubt the culture of “data-driven” decision-making promulgated by MBA, MPA, MPH and similar programs contributed, but I think there is also an incentive problem. Such people are judged on the basis of data delivered to peers and constituents, however flawed. If you “flatten the curve,” even if the curve itself is illusory, you get rewarded. This is speculation of course.
I agree with Dorman on all points. The linked article is definitely worth reading. Key quote:
Before March 2020, the country had no shortage of pandemic-preparation plans. Many stressed the importance of data-driven decision making. Yet these plans largely assumed that detailed and reliable data would simply . . . exist. They were less concerned with how those data would actually be made.
They do they say some things that don’t completely make sense, like “Data are just a bunch of qualitative conclusions arranged in a countable way.” Huh? Also, I wish they’d point to other data summaries such as Carnegie Mellon’s Covidcast (discussed here). Their main point, though, that data are important and don’t come by themselves, is super important.
The only thing that puzzles me is the idea that the government should be so bad at this. The Bureau of Labor Statistics is 136 years old! And then there was all the data collection in the New Deal period, and the World War II mobilization. I guess that data got a bad rap during the Vietnam War, back when government officials promised the country they could run the war as efficiently as they’d run Ford Motor Company. Still, the Federal data system is huge, and they have a lot of competent, serious professionals all over. It didn’t seem so outlandish to assume that the CDC was on top of this sort of thing.
I feel like Meyer and Madrigal, or somebody, needs to write a follow-up article on how this all went wrong. They write, “Perhaps no official or expert wants to believe that the United States could struggle at something as seemingly basic as collecting statistics about a national emergency,” and they talk about federalism—but federalism is an issue with just about every government statistic. I remain baffled as to what has been happening here.
Maybe we should look at it another way by comparing to familiar economic statistics. We all know that inflation and unemployment measures are imperfect: inflation depends on what’s in the basket, and there’s also this weird thing where inflation is supposed to be a bad thing but it’s also supposed to be a healthy thing if “property values” are going up. The unemployment numbers don’t include people who are looking for work. And much of the time it seems that the stock market is used as a measure of the economy, which is really weird (no matter what that Stasi guy says). So, my point is that even in the world of economic statistics, there are difficulties of interpretation, and opinion leaders will often grab on to numbers without thinking clearly about them. This fits in with Meyer and Madrigal’s point that, not only does it take important work to collect, compile and clean data, it also takes work to put data in context.
Measurement, baby, measurement.