David Radwin writes:
I am seeking a statistic measuring an estimate’s reliability or stability as an alternative to the coefficient of variation (CV), also known as the relative standard error. The CV is the standard error of an estimate (proportion, mean, regression coefficient, etc.) divided by the estimate itself, usually expressed as a percentage. For example, if a survey finds 15% unemployment with a 6% standard error, the CV is .06/.15 = .4 = 40%.
Some US government agencies flag or suppress as unreliable any estimate with a CV over a certain threshold such as 30% or 50%. But this standard can be arbitrary (for example, 85% employment would have a much lower CV of .06/.85 = 7%), and the CV has other drawbacks I won’t elaborate here. I don’t need an evaluation of the wisdom of using the CV or anything else for measuring an estimate’s stability, but one of my projects calls for such a measure and I would like to find a better alternative.
Can you or your blog readers suggest a different measure of reliability?
My reply: If you are stuck here, go back to first principles. If you just need a measure, you can supply both the estimate and the standard error. But it sounds like you are looking for a rule of some sort? Maybe it would help to try to quantify the gains and losses from classifying an estimate as “stable.” Then you could do a decision analysis. I’m not saying that the formal decision analysis should decide your answer but it could give you some intuition about what various proposed procedures are doing.
Based on my experiences, I think you could make general progress by constructing a solution to your specific problem.
Any other thoughts?
“I don’t need an evaluation of the wisdom of using the CV or anything else for measuring an estimate’s stability…”
(I’m going to egregiously ignore the above-quoted line.) It would help to know what kind of quantity is being estimated. For example, roughly speaking, the coefficient of variation is useful for always-positive (or always-negative) quantities — especially things that look like scale parameters. The standard deviation is useful for quantities that take a value on the real line — especially things that look like location parameters.
For a percentage like employment/unemployment, I think a useful thing to do would be to transform it to an “odds” analogue via Odds = P/1-P, where P would ordinarily stand for “probability” but here would stand for “percentage”. Odds are always-positive, so I’d expect the CV to be a useful measure of reliability for them…
i thought that the coefficient of variation was the standard deviation divided by the mean?
Jimmy, you are correct. The statistic to which I referred is really a “relative standard error.” But “coefficient of variation” is equivalent for the purpose of this question and seems to be more commonly used. Thank you for clarifying.
Andrew and Corey,
Thank you for your responses.
To clarify a bit, my question is not about finding a better way to judge the quality of the estimates, so I don’t need a decision rule. Rather, the publication standards for my project flag as unreliable/unstable any estimate (regardless of the type of estimate, what is being estimated, or even whether it might be zero) with a CV exceeds 30% and suppress any estimate where the CV exceeds 50%, period. Despite what you or I might think about them, they are the standards that govern this publication.
Some estimates I produced fall into the CV > 50% category and ordinarily would be suppressed, but perhaps I could obtain an exception if there were an alternative statistic showing that the estimates were indeed reliable/stable.
To Corey’s question, I am reporting percentiles, percentages, and means, so this alternative statistic (should it exist) would ideally apply to all three.
David,
Perhaps this is outdated by now (the paper is form 2002), but the RSE<30% criteria is/was not uniform for all federal surveys. Some health surveys use (or used, if this is outdated) different criteria for common and uncommon events.
see: http://www.cdc.gov/nchs/data/statnt/statnt24.pdf
Can you go to a different journal?
Dan
Dan,
Thank you for the reference. I can’t switch venues, but perhaps the diversity of standards used by the agencies described in this paper might support an exception to the policy.