As if we needed another example of lying with statistics and not issuing a correction: bike-share injuries

This post is by Phil Price

A Washington Post article says “In the first study of its kind, researchers from Washington State University and elsewhere found  a 14 percent greater risk of head injuries to cyclists associated with cities that have bike share programs. In fact, when they compared raw head injury data for cyclists in five cities before and after they added bike share programs, the researchers found a 7.8 percent increase in the number of head injuries to cyclists.”

Actually that’s not even an example of “how to lie with statistics”, it’s simply an example of “how to lie”: As noted on StreetsBlog, data published in the study show that “In the cities that implemented bike-share…all injuries declined 28 percent, from 757 to 545. Head injuries declined 14 percent, from 319 to 273 per year. And moderate to severe head injuries also declined from 162 to 119. Meanwhile, in the control cities that do not have bike-share, all injuries increased slightly from 932 to 953 per year — 6 percent.”  There’s a nice table on Streetsblog, taken from the study(make sure you read the caption).

So the number of head injuries declined by 14 percent, and the Washington Post reporter — Lenny Bernstein, for those of you keeping score at home — says they went up 7.8%.  That’s a pretty big mistake! How did it happen?  Well, the number of head injuries went down, but the number of injuries that were not head injuries went down even more, so the proportion of head injuries that were head injuries went up.
According to StreetsBlog, University of British Columbia public health professor Kay Ann Teschke “attempted to notify Bernstein of the problem with the article in the comments of the story, and he was initially dismissive. He has since admitted in the comments that she is right, but had not adjusted his piece substantially at the time we published this post.” (I don’t see that exchange in the comments, although I do see that other commenters have pointed out the error).

To be fair to Bernstein, it looks like he may have gotten his bad information straight from the researchers who did the study: The University of Washington’s Health Sciences NewsBeat also says “Risk of head injury among cyclists increased 14 percent after implementation of bike-share programs in several major cities”. It’s hard to fault Bernstein for getting the story wrong if he was just repeating errors that were in a press release approved by one of the study’s authors!

But how do Bernstein, the Washington Post, the study’s author at University of Washington (Janessa Graves), and the University of Washington justify their failure to correct this misinformation?  It’s a major error, and it’s not that hard to edit a web page to insert a correction or retraction.

[Note added June 18: When I posted this I also emailed Bernstein and the UW Health Sciences Newsbeat to give them a heads-up and invite comment. Newsbeat has changed the story to make it clear that the proportion of injuries that are head injuries increased in the bike share cities. They do not note that the number of head injuries decreased, and it looks like they forgot to correct the headline so it’s still wrong. At least they acknowledged the problem and did something, although I daresay most readers of that page will still be misled. But it’s no longer flat wrong. Except the headline.]

Of course, even simply retracting the story is a missed opportunity: the real story here is that injuries went down in bike share cities in spite of the fact that there were more people riding. That’s a surprise!  As a bike commuter, I know that it has long been argued that biking becomes safer per biker-mile as more people ride, because drivers become more alert to the likely presence of bikes. But I would not have expected that the decrease in risk per mile would more than counteract the number of miles ridden, such that the number of injuries would go down. Or, of course, maybe that’s not what happened, maybe there were other changes that were coincident with the introduction of bike share programs, that decreased risk in the bike share cities but not the control cities.

This sort of thing — by which I mean mis-reporting of scientific results in general — is just so, so frustrating and demoralizing to me. If people think bike share programs substantially increase the risk of injury, that belief has consequences. It affects the amount of public support for such programs (and for biking in general) as well as affecting individuals’ decisions about whether or not to use those programs themselves. To see these stories get twisted around, and to see journalists refuse to correct them…grrrrr.

This post is by Phil Price
[Andrew, please add “Ethics” and “Journalism” categories to this blog]

Average predictive comparisons in R: David Chudzicki writes a package!

Screen Shot 2014-06-15 at 5.49.41 PM

Here it is:

An R Package for Understanding Arbitrary Complex Models

As complex models become widely used, it’s more important than ever to have ways of understanding them. Even when a model is built primarily for prediction (rather than primarily as an aid to understanding), we still need to know what it’s telling us. For each input to the model, we should be able to answer questions like these:

On average, much does the output increase per unit increase in input?
How does the influence of the input vary? (Across its range? In combination with other inputs?)
How much difference does variation in this input make to the predictions? (Relative to other inputs?)
For example, if our model were a linear regression with no interactions and no transformations of the inputs, the (1) would be answered by our regression coefficients, (2) would be answered “It doesn’t vary”, and (3) would be a little harder but not too bad. All of these questions are much harder for more complicated models.

This R package is a collection of tools that are meant to help answer these questions for arbitrary complicated models. One advantage of the fact that they work work equally well for any model is that they can be used to compare models.

The key feature of the approach here is that we try to properly account for relationships between the various inputs.

Chudzicki provides some more context in an email:

There’s a demonstration of the package in a logistic regression example with simulated data that’s simple enough to understand everything going on, and in another example with a real data set where the model used is a random forest.

I haven’t solved any of the open issues about the level of smoothing etc. so the user still needs to think about that, unfortunately. Maybe using something like BART to sample from u|v will be a good way around that later.

I’m very interested in working with users and possible collaborators to make the package do what they need.

I haven’t tried out Chudzicki’s package but I love the underlying idea, and I think this sort of implementation is important, for two reasons:

1. Once it’s out there, people can use it in applied work. And once a method gets used for real, we learn a lot more about it.

2. As Chudzicki discusses, the method of average predictive comparisons is not fully specified. There are some loose ends. Writing a program puts one face to face with such problems, as they have to be confronted in some way. In an article you can just B.S. your way thorough, but a computer program is more rigorous. You have to make the decisions.

Hurricanes/himmicanes extra: Again with the problematic nature of the scientific publication process

Jeremy Freese has the story.

To me, the sad thing is not that people who don’t understand statistics are doing research. After all, statistics is hard, and to require statistical understanding of all quantitative researchers would be impossible to enforce in any case. Indeed, if anything, one of the goals of the statistical profession is to develop tools such as regression analysis that can be used well even by people who don’t know what they are doing.

And the sad thing is not that the Proceedings of the National Academy of Science publishes weak work. After all, there’s a reason that PNAS is known as one of “the tabloids.”

To me, the sad thing is that certain researchers, after getting well-informed and open criticism, respond with defensive foolishness rather than recognizing that maybe—just maybe—some other people know more about statistics than they do.

Incompetence is not so bad—all of us are incompetent at times—but, as scientists, we should try to recognize the limitations of our competence.

It’s too bad because the himmicanes people always had an easy out: they could just say that their claims are purely speculative, that they’re drawing attention to a potentially important issue, that yes they did make some statistics errors but that’s not surprising given that they’re not statistics experts, and that they recommend that others follow up on this very important work. They could respond to the points of Freese and others that their claims are not convincing given huge variation and small sample size, by agreeing, and saying that they regret that their work was so highly publicized but that they think the topic is important. Their defensiveness is completely inappropriate.

I blame society

To return to one of our general themes of the past year or so, I think a key problem here is the discrete nature of the scientific publication process. The authors of the himmicanes/hurricanes paper appear to have the attitude, so natural among those of us who do science for a living, that peer-reviewed publication is a plateau or resting place: the idea is that acceptance in a journal—especially a highly selective journal such as PNAS—is difficult and is a major accomplishment, and that should be enough. Hence the irritation at carpers such as Freese who won’t let well enough alone. From the authors’ point of view, they’ve already done the work and had it approved, and it seems a bit like double jeopardy to get their work criticized in this way.

I understand this attitude, but I don’t have to like it.

P.S. Above image from classic Repo Man clip here.