Paul Alper writes:
I do not believe your blog has ever dealt with the following phenomenon which might be called “(widening) moving the goalposts.” Drug companies and the medical world at large often create powerful drugs and procedures for people who are far (many standard deviations) from the norm (mean) and via randomized clinical trials, the relevant authorities approve. But there aren’t enough of those people to be truly profitable so the next step is to ask for approval to prescribe the same for people who aren’t that far (fewer standard deviations) from the norm. Or, just move the norm (center) so as to pick up a much larger number of patients. Afflictions include hypertension, cholesterol, overweight, osteoporosis.
The result is what is often called “the worried well,” who receive little or no benefit but suffer harms from the treatment.
H. Gilbert Welch has written extensively on this “goalpost” issue. He is the author of
He also has spoken on NPR
I wrote this for Chance News regarding his book:
See his pages 64 and 66 which shows graphs which illustrate how diagnoses increase markedly while deaths are flatlined. The Guardian’s Ben Goldacre often writes about the goalpost issue.
Maybe because health care is so important, folks think it is being well managed :-(
Similarly, they might also think that what is called evidence based medicine is itself evidence based.
Some recent interesting comments on this issue here http://www.parliament.uk/documents/commons-committees/science-technology/Clinical%20trials%20combined.pdf
The stuff by Senn and Chalmers (starting pages 37 and 57) make very clear arguments that getting “evidence” from non-regulated sources seems currently hopeless (and only regulators actually get evidence that can be taken as credible and complete).
I am sure more regulation written by people who know nothing about statistics or science will help. Is there not an inverse correlation between government funding of science and quality?
The moon landings or the decryption techniques developed by the NSA look like instances of government funded science with high quality. But maybe you’re right. Could you perhaps give one or to references for your claim?
I was thinking of the formation of NIH corresponding to the rise of nill-null-NHST-based science. As NIH funding spread from pyschology/psychiatry to other fields NHST spread with it. I am speculating, and was hoping someone else would pop up with a paper/book. I have not had the time to research this myself.
Also, most of my work has been funded from the government. Including Stan! I wrote my books on my own time but many of the ideas in them come from government-funded work.
None of this speaks directly to the question of correlation, though, as we’d need to compare to non-government funded science. And of course there’s the difficulty of figuring out what counts as “government funding.” I’m pretty sure we’d have to take all of Neumann and Ulam’s later work as government funding, but what about Newton? Laplace? Neyman worked at the University of Warsaw and the University of California—do they count as government institutions? R. A. Fisher worked at Rothamsted Experimental Station starting in 1919. From Wikipedia: “In 1902 Daniel Hall moved from Wye College to become director. Hall took a lower salary to join an establishment lacking money, staff, and direction. Hall decided that Rothamsted needed to specialise and that it needed new sources of finance. He was eventually successful in obtaining state support for agricultural research.”
Andrew, there is also the question of how to define “scientific quality”.
I’m not sure but the early day examples of Govt. funded research were of a different flavor I think: Researchers were identified & employed on pretty broad grants or goals. Govt. was picking a good researcher & then giving him a broad mandate (e.g. improve farming practice, increase wheat yields etc.)
Now the Govt. micromanages, and tries to pick specific tiny projects, goals etc. I suspect this transition gives rise to some of the more undesirable features of Govt. Funded Research.
Interesting point. Certainly it doesn’t seem that Neyman’s government-funded jobs at Warsaw and California, or Fisher’s job at Rothamsted, were tied to particular narrow pieces of research.
Another change is that a lot of Govt. funding is now disbursed too much by committees of peers. I think this is happening more & more in the post WW-II Vannevar Bush era. To some extent it is natural: The merits / technicalities of a Large Hadron proposal are best evaluated by particle physicists.
But I think this has made Funding too inward looking & hierarchical. Maybe we should have more & better ways of outsiders having a say in what project deserves funding. Need not be total laymen, but say a Chemistry proposal judged be Chemical Engineers & Economists. This may seem bizarre or sacrilegious to some but I feel we are often funding projects that look silly to everyone except those researchers in a narrow field whose perspective is often distorted or prejudiced.
Perhaps a two tier system will help; like a judge-jury where technical experts offer inputs but a panel with wider perspective judges suitability & utility? The NSF / DoE grant cycle circus I saw was pretty hilarious: X submits a proposal in May that Y, Z, and Q judge; in October Y submits & X, Z & Q judge. And so on.
Agree that is likely to drive quality – especially when it’s on the basis of here is less money than you really need and if you don’t find anything interesting this time we may never fund you again and you might even lose your job and we won’t check on anything and your university will block most efforts by anyone to check on what you have done.
I was very fortunate in that the first group I worked with had mostly permanent long term funding, with a primary goal of training clinical researchers and discerning possible improvements from other’s research that could be adopted in the city’s teaching hospital. That makes so much more in one’s interest to do high quality work and enables it.
I think Senn was mostly concerned with making sure research is regularly checked on, especially all the raw data and efforts are in place _to make sure_ anyone (qualified) can check on what you have done.
Maybe, I should have included this link http://www.senns.demon.co.uk/Bad%20Karma%20pre%20publication.pdf
Speaking of EBM and government influence on science. I wonder if K? or anyone has any opinion on the way Daubert vs Dow 1993 has influenced the evidence based medicine movement. The courts since then have apparently given NHST a crucial role in determining what counts as evidence. This case occurred soon after Gordan Guyatt coined the term.
When I googled to find what paper I had worked Gordan on I got this.
Measuring Menial Capacity
M Silberfeld, GH Guyatt, K O’Rourke – ANNALS-ROYAL COLLEGE OF PHYSICIANS AND …, 1997
(somehow the t in mental got morphed into an i)
Some of the co-authors listed on Gordon’s paper, have been involved as expert witness and have discussed their experiences. To me, a mess and I’ll take Tukey’s advice “not to given advice on scientific[legal] process based on mere technical knowledge of statistics” or something like that.
But some links involving the last author on Gordan’s paper (Peter Tugwell) other’s might be interested in
Pingback: “Widening the goal posts” in medicine | Masters of Science