Ivan Oransky points us to this amusing retraction of a meta-analysis. The problem: “Standard errors were used instead of standard deviations when using data from one of the studies”!
Actually, I saw something similar happen in a consulting case once. The other side had a report with estimates and standard errors . . . the standard errors were suspiciously low . . . I could see that the numbers were wrong right away, but it took me a couple hours to figure out that what they’d done was to divide by sqrt(N) rather than sqrt(n)—that is, they used the population size rather than the sample size when computing their standard errors.
As Bob Carpenter might say, it doesn’t help that statistics uses such confusing jargon. Standard deviation, standard error, variance, bla bla bla.
But what really amused me about this Retraction Watch article was the this quote at the end:
As Ingram Olkin stated years ago, “Doing a meta-analysis is easy . . . Doing one well is hard.”
Whenever I see the name Ingram Olkin, I think of this story from the cigarette funding archives:
Much of the cancer-denial work was done after the 1964 Surgeon General’s report. For example,
The statistician George L. Saiger from Columbia University received [Council for Tobacco Research] Special Project funds “to seek to reduce the correlation of smoking and diseases by introduction of additional variables”; he also was paid $10,873 in 1966 to testify before Congress, denying the cigarette-cancer link.
. . .
Ingram Olkin, chairman of Stanford’s Department of Statistics, received $12,000 to do a similar job (SP-82) on the Framingham Heart Study . . . Lorillard’s chief of research okayed Olkin’s contract, commenting that he was to be funded using “considerations other than practical scientific merit.”
So maybe doing a meta-analysis badly is hard, too!
“Standard errors were used instead of standard deviations when using data from one of the studies”
This remark puzzled me, at first. Standard errors are, after all, standard deviations. The question comes down to this. Standard deviations of what?
Indeed, the standard deviation of measurements is often called “the standard deviation” and the standard deviation of a statistical estimate based on combining multiple samples (such as an average, or a median, or a trimmed mean or whatever) is called the standard error of that estimator. But it’s just the sampling standard deviation of the quantity you get by taking a sample of N values and computing the estimate.
Bayesian analyses allow you to be more explicit in an easy way:
“the posterior standard deviation of the parameter mu” or “the posterior standard deviation of the predictive quantity y”
the term “standard error” refers usually to the “sampling standard deviation of the estimator f(X)” which is a frequency based concept, as it’s built on the idea of repeated sampling.
Thanks, Daniel. :)
Also, for Bob Carpenter: “Doing a meta-analysis is easy . . . Doing one well is hard.”
Especially if you have to drill really deep or through a lot of hard stone.
It’s not helped by software like R which reports the estimate for the standard deviation of the residuals in a regression model as the “Residual standard error”. Other references (including textbooks, of course) call it the “Standard error of the estimate”, or other variations on this totally incorrect terminology.
At least that other textbook regression term, the “Coefficient of determination” (what real statisticians call “R-squared”) is just stupid rather than being actively misleading.
> “Standard errors were used instead of standard deviations when using data from one of the studies”!
This happened repeatedly in the 1990s, so one of those things that regularly recurs.
Disappointing about the tobacco incident but he always tried to bring meta-analysis to the attention statisticians which until recently was chronically ignored.
He had a very different take on meta-analysis that I could not grasp for a long time.
When I did I sent the email below (no response).
From: Keith ORourke
To: “[email protected]”
Sent: Monday, December 12, 2011, 03:53:43 p.m. CST
Subject: Your Ottawa talk last week
Ingram:
I had to leave your talk early but I really enjoyed it!
When I first read your book (after I had published the L’Abbe paper) I noticed that you were addressing something other than “essentially similar repeated experiments” that I had taken as my task in the randomised clinical trial research area where the diagnosis, treatment and outcome (usually mortality) seemed identical. Because of that, I continued with the Fisher/Cochran/Yates approach based on repeated agricultural trials.
Not so sure now about the “identicalness” in RCTs now, but I did not realize that you where taking meta-analysis as meaning where this “identicalness” was hopeless wishing and for instance trying to get defensible likelihoods was silly.
Now your approach makes much more senses to me – thanks.
By the way, currently working through using functions of p-values at different time points and for different sub-groups in the setting of multi-level models to get a sensitivty analysis (or sensibility analysis) given how complex those models get. At some point I should check if someone else has already done this.
Cheers
Keith
Maybe the authors were just using the exciting new definition of standard deviation found here: https://vixra.org/pdf/1711.0326v1.pdf
I think we seldom pay attention to the difference between “objective” research and “advocacy” research.
Let’s illustrate this by a simple marketing research example. (1) I might be asked to compare the relative strength of our brand versus the leading competitor on a number of dimensions. That’s “objective” research.
Or, (2) I might be asked to prepare a list of our brand’s strengths and advantages versus a competitor, for inclusion in advertising or sales materials or PR. [or, you might be asked to provide reasons why the competing brand’s claims are questionable.] This is also research, and I’d already done #1 I would be perusing those results for this task.
There’s nothing particularly evil about either side, provided both you and your audience realize which position you are taking.
That’s a problem with academics getting funded by industry. In the press, this is most likely presented as “a study by University of Whatever researchers”, not “a study funded by the GoodGuy Institute, mostly funded by the DirtyAir Company” — a study not likely to see the light of day if the results are in the wrong direction (nor is the researcher likely to get another grant).
Yes, this is a spectrum, but this post is already long enough.
Doing a meta-analysis is easy . . . Doing one well is hard. Yup. that’s what I say tell people about ‘qualitaitve anlysis’