Amy Cohen writes:
A surgeon showed me the “report card” his hospital received about his surgical group. The figure below shows what the report card looks like. I am very curious to hear what you think about the “deciles of the odds ratio” approach to evaluate and rank hospitals used by the American College of Surgeons.
I replied: I don’t know enough about the substantive context to say too much here. But, speaking generally, I am skeptical of those very high values for some of those odds ratios. I suspect that much of what is plotted in this graph is just noise. But I respect the goals underlying the plot, and I like that it displays a lot of information efficiently.
And then she wrote:
My concern with the decile approach to identifying opportunities for improvement is that it violates a principle of quality improvement: chasing noise is fruitless or worse, because it leads to tampering with a system in a way that can make it worse. In the paper, the authors admit that the 95% CIs don’t yield many signals, so they added highest decile of the ORs as signals, too. In other words, they superimposed a ranking system that encourages the hospitals to chase the noise!
That’s not good.