Unconvincing (to me) use of Benford’s law to demonstrate election fraud in Iran

Benford’s law is an amusing mathematical pattern in which the first digits of randomly sampled numbers tend to have a distribution in which 1 is the most common first digit, followed by 2, then 3, and so forth. It’s the distribution of digits that arises from numbers that are sampled uniformly on a logarithmic scale.

In our Teaching Statistics book, Deb and I describe a classroom demonstration where we show how Benford’s law applies to street addresses sampled randomly from the telephone book. In a more serious vein, Walter Mebane has written about the application of Benford’s law to vote counts.

In the past several days, a few people have asked me about applying these ideas to the recent Iranian election. Today, someone pointed me to an article by Boudewijn Roukema, which states:

The results of the 2009 Iranian presidential election presented by the Iranian Ministry of the Interior (MOI) are analysed based on Benford’s Law and an empirical variant of Benford’s Law. The null hypothesis that the vote count distributions satisfy these distributions is rejected at a significance of p < 0.007, based on the presence of 41 vote counts for candidate K that start with the digit 7, compared to an expected 21.2-22 occurrences expected for the null hypothesis. A less significant anomaly suggested by Benford's Law could be interpreted as an overestimate of candidate A's total vote count by several million votes. Possible signs of further anomalies are that the logarithmic vote count distributions of A, R, and K are positively skewed by 4.6, 5.8, and 2.5 standard errors in the skewness respectively, i.e. they are inconsistent with a log-normal distribution with p ` 4 × 10−6, 7 × 10−9, and 1.2 × 10−2 respectively. M's distribution is not significantly skewed.

I don’t buy it. First off, the whole first-digit-of-7 thing seems irrelevant to me. Second, the sample size is huge, so a p-value of 0.007 isn’t so impressive. After all, we wouldn’t expect the model to really be true with actual votes. It’s just a model! Finally, I don’t see why we should be expecting distributions to be lognormal.

Maybe there’s something I’m missing here, but that’s my quick take. This is not to say that I think the election was fair, or rigged, or whatever–I have absolutely zero knowledge on that matter–just that I don’t find this analysis convincing of anything. I will say, though, that Roukema deserves credit for presenting the analysis clearly.

P.S. In response to comments: let me emphasize that I’m not saying that I think nothing funny was going on in the election. As I wrote, I’m commenting on the statistics, I don’t know the facts on the ground. To move my comments in a more constructive direction (I hope), let me pull out this useful comment from Roukema’s article: “One possible method to test whether this is just an odd fluke would be
to check the validity of the vote counts for candidate K in the voting areas
where the official number of votes for K starts with the digit 7.” Further investigation could be a good thing here.

I did not find Roukema’s argument convincing; that does not mean that I consider it a bad thing that the article was written. The article is a first draft of an analysis; it might end up leading to nothing, or it might be unconvincing as it stands now but lead to some important breakthroughs. We can see what further analysis turns up. Again, my verdict is not a Yes or a No, it’s an “I’m not convinced.”

10 thoughts on “Unconvincing (to me) use of Benford’s law to demonstrate election fraud in Iran

  1. I was pointed to this last night, so I'm happy to see you've looked at it too.

    One criticism: you don't present the graphs! The 7 issue sticks right out (but the relevance is unclear), and the whole thing about the 1's and 2's looks like chance being lumpy.

    Oh, and I'm glad you were confused by the log-normal thing too. I wondered if I was missing some piece of election statistics lore.

  2. Great and useful comment, thanks Andrew! (I almost always enjoy your posts… and after reading the bizarre vitriol on 538, thought I'd give you a thumbs up…)

  3. Eduardo: Good point. I'd been lazily thinking of the total number of votes rather than the number of vote totals. That's one reason I added the P.S. above, agreeing that it could make sense to look into the suspected anomalies in more detail.

  4. Also, for Benford to apply, even approximately, don't the numbers need to span several orders of magnitude? If I'm looking at the first digits of heights of adults as measured in inches, Benford just isn't going to work. Do they do so here?

  5. Dear Prof.Gelman,
    I am an iranian, and as you may know there is great conflict about this election in my country nowadays which turns into a bloody street war some days.
    It is so important for me to know if scientific analysis like Benford's law shows honesty or fraud in this election.

    Hope your scientific analysis can help us to end the conflict.

    I will read your respond here.
    Best Regards

  6. Mehdi,

    I have no idea what's going on in Iran. I did not find the above-noted Benford's law analysis to be convincing–it struck me as a naive, if well-intentioned, application of mathematical ideas to elections. The article by Mebane (see comments above) is much more on topic.

  7. What's unfortunately forgottin in these analyses of numbers is a simple fact: that the Iranian election was rigged can be proven from physical reality and need not be considered from the viewpoint of mathematics.

    It is a fact, just like the spherical shape of the Earth is a fact, that massive election has taken place. Why so? Consult Iran's political demography. You'll find your answer there with much higher certainty than you could expect from the application of any statistical criterion to the rigged numbers.

Comments are closed.