Alex Scacco and Bernd Beber follow up on their analysis of the Iran election data:
After we wrote our op-ed using the province-level data, we’ve now also done some preliminary tests with the county-level data. In the latter dataset, the last digits don’t appear fraudulent. Why might we find suspicious last digits at the province level, while, at the same time, Walter Mebane and Boudewijn Roukema find evidence that first and second digits are fishy at the county level?
We can only speculate about what happened behind closed doors, but here is a scenario of top-down fraud that is consistent with the patterns found in the quantitative analyses mentioned above:
As votes began coming in on Friday night, the clerics began to think they might lose. Someone in the Ministry of the Interior then fabricated the province level results. The pattern in the last digits suggests to us that some of those vote totals were made up wholesale (why bother fudging last digits? It’s an inefficient way to shift votes around).
That still left the task of making sure the numbers in the county-level spreadsheet added up to the province-level vote counts. (And we checked — the county numbers do in fact add up to the province numbers.) A province contains 12 counties on average, so it may have been necessary to shift votes in several counties per province in order to get the leading digits to line up. This could explain possibly suspicious patterns in the first or second digit. But once the leading digits match, it’s easy to get the last and second-to-last digit to match by changing just one county per province, which isn’t enough to be picked up by our tests.
An example might make this clearer. Suppose we make up vote counts for some province, but they differ from the sum of county vote counts in the Ministry’s spreadsheet. Let’s say we happened to give one candidate 124,561 more votes than he has in the spreadsheet and another candidate that many fewer votes. There are ten counties, and no more than 100,000 votes to go around in each county. So we’ll shift 30,000 votes in one county, 20,000 in another, and so on, but we have to change first and perhaps second digits in more than one county until we have a rough match with our province-level result. But to get the numbers to add up in the trailing digits, we can change just a single county.
Again, this is just speculation. But type of centralized fraud in this story is consistent with qualitative accounts suggesting proceedings were generally clean on the ground on election-day itself, and with the strange behavior of the Ministry of the Interior the following day (for example, releasing the results early).
Our digit-based tests can’t tell us whether Ahmedinejad actually won the election. Even without manipulation, he might very well still have won. But we feel reasonably confident that the province-level numbers were tampered with.
I'm only guessing here… but I suspect there are an awful lot of statisticians, quants and interested parties trawling through the Iranian election results. Whilst the more sophisticated will apply a bonferroni correction or similar, do we also need some kind of 'global' bonferroni? (I'm reminded of the argument in Ioannidis – why most published research findings are false).
This is more a question out of interest than a claim that the Iranian election was not tampered with. (Stats aside, I tend to think it was just from a cynical 'realpolitik' perspective – it could have been and so it almost certainly was).
Another explanation for the 1st and 2nd digit discrepencies is that the sizes of the chunks of votes used here don't follow a power-law distribution, and Benford's law shouldn't be expected to hold. A quick glance at a histogram of county sizes is enough to realize that this analysis doesn't fly. Can the authors quantify "reasonably confident?"
I am not an expert in statsics. The ideas behind Scacco and Beber 's paper and Benford law seems contradictory to me. One says the last digit should have uniform distribution, the other says digit 1 should appear much more than other digits(30% of time). Could you help me understand what is the difference between claims?
Jack: Benford's law states that the first digit in a number chosen randomly from a power-law distribution will be biased towards lower numbers. There's also a smaller pattern expected for the second digit. Beyond that, digits in numbers pulled from almost any distribution of real-life data are going to be completely random — a uniform distribution.
The pattern in the first digit is pretty easy to understand: you have a decreasing probability of higher numbers in a power-law distribution; as such, there are more 1,XYZ numbers than 2,XYZ numbers, etc.
The claims aren't different; they're two totally different things. I don't see why it's at all reasonable to expect Benford's law to hold for this data (although I understand Membane has looked at more finely grained data than in his first analysis), but its reasonable to expect the last and second-to-last digits to be random for elections returns with four or more digits.
Note that I'm not a stats expert, either.
Zach: So you mean if we assume the numbers are from power law distribution, the Benford law should hold. Whereas if we assume them to be from uniform distribution, Scacco and Beber's claims would hold? So both claims cannot be true at the same time. Is it right?
Look at the set of numbers used here (number of votes for each candidate in a given county or town or whatever). If these numbers can be described by a power-law distribution, you'll see the follow:
1. 1st and 2nd digits follow Benford's law
2. Remaining digits are uniformly distributed
If the numbers are not power-law distributed, 1 will not hold any more (and probably won't be uniformly distributed either for something like an election), but 2 almost certainly will in the absence of fraud. Scacco and Beber's previous work proved that the last digits will be uniformly distributed for any distribution you'd expect in a fair election. Looking at the last digits avoids the problem of having to assume things about the underlying distribution.
Membane has recently updated his analysis to look at Benford's law results for individual ballot boxes instead of counties. It's more likely that the ballot box numbers are power-law distributed, but I suspect they still won't be. I'm not familiar with his earlier work justifying this sort of analysis, but it seems to be widely respected.
Thanks a lot Zach. I finally understood your point. One is working on the first (or the 2 first) digits, while the other is working on the last digits. I was not careful enough and thought they were both working on the last digits.
You got it; also, looking into it more, it appears that Dr. Membane has looked into a lot of election data and found that Benford's law generally holds for the *second* digit in elections even if it doesn't hold for the first digit. His test is for the second digit; Roukema's work looks at the first digit, though, so it probably deserves a good bit of skepticism.