Benford’s Law is fun; it also is the source of endless confusion. Math is hard!

This came up recently when Kevin Lewis pointed to this paper, which states:

We use Benford’s law to examine the non-random elements of health care costs. We find that as health care expenditures increase, the conformity to the expected distribution of naturally occurring numbers worsens, indicating a tendency towards inefficient treatment. Government insurers follow Benford’s law better than private insurers indicating more efficient treatment. . . .

This sounded interesting, but I’m pretty sure they’re doing it wrong, because they try to evaluate the fit to Benford’s law within each “price bucket” ($100-999, $1000-9999, $10,000-99,999, and $100,000-999,999). Based on my understanding of the processes underlying Benford-like behavior, you wouldn’t necessarily expect the pattern to occur within each bin in that way.

Here’s an example of how things go wrong. The authors write:

We also follow Drake and Nigrini (2000) by calculating the mean of absolute deviations (MAD) to use as a way to assess conformity to the expected distribution. . . . We find that at the MAD for the first bucket of charges (0.010) shows a marginally acceptable conformity to Benford’s law. However, for the second (0.023), third (0.049), and fourth (0.092) buckets the MAD is greater than 0.012 indicating nonconformity. As expected, the MAD increases with the level of total-charges. . . . An additional possible explanation for this finding is hospital pricing strategies . . .

OK, here’s the problem. Here are the data:

This should really be a graph, but let’s not worry about that right here.

Let’s focus on the fourth bucket, because that’s where the discrepancy is highest. You see what’s happening, right? In that fourth bucket, we’re up there in the tail of the distribution, the tail is dropping fast, so, yeah, there are very charges over $200,000. That doesn’t mean anyone’s cheating in their billing; it’s just what you’d expect to see in the tail of the distribution. Benford’s law applies when the underlying numbers come from a distribution with have a wide dynamic range, and by binning in this way you’re destroying that.

I’d say I’m surprised this got published in a legitimate journal, but, you know, the problem with peer review is the peers. Everyone’s doing the best they can, Benford’s law is a bright shiny object, it gets misused just like linear regression gets misused, just like logistic regression gets used, just like hypothesis testing gets misused and misused and misused. The Benford example is just a little bit more interesting because the math confusion is something a bit less familiar than the usual statistical mistakes we see every day. Hence why I bothered with this post.