Michael Collins (here’s a link but I don’t know the original source) picked up on my article about statistics and the cigarette companies, which mentioned a consulting job of famed Stanford statistician Ingram Olkin, and noticed that Olkin was recently in the news as a coauthor of a report on organic food:
A widely publicized study claiming that there is no demonstrated difference in nutritional value between organically and conventionally grown foods just appeared in the Annals of Internal Medicine. Broad mainstream media coverage produced headlines like Stanford Scientists Cast Doubt on Advantages of Organic Meat and Produce. The media failed to mention one point that may be of major interest. . . . The article co-author with recognized expertise in meta-analysis, Ingram Olkin, applied for a grant from Council of Tobacco Research (CTR) in 1976. . . . Olkin applied to the CTR to conduct a project on the statistical methods used in the Framingham Heart Study, the landmark project linking cigarette smoking with increased risk of heart disease. . . .
Olkin’s history of being paid to work for the tobacco companies is being used to discredit his current work on organic vegetables.
Is this fair? I don’t know. On one sense, it seems pretty rough to slam a guy based some past consulting job. I’d be pretty annoyed if that were to happen to me.
On the other hand, there’s a symmetry to the accusation. When they hired Olkin, the cigarette companies were, to some extent, buying his reputation as a leading applied statistician. So maybe it’s appropriate for the inference to go the other way: being funded by cigarette companies based on “considerations other than practical scientific merit” can affect one’s reputation as a statistician. Olkin got the $12,000 but now he’s paying for it (a bit).
But this is not a deterministic relation; just ‘cos someone is a distinguished statistician, it doesn’t mean that his cigarette-funded report on the Framingham Heart Study is to be trusted; and just ‘cos someone took cigarette $ to write that report, it doesn’t mean that his other work can’t be trusted.
P.S. Just to be clear: This post is not intended to be an attack on Olkin. Rather, it’s an exploration of the ways in which people use statistical inference in judging someone’s work.
As you implied into the your Chance article (or as I inferred from it), an ethical issue for statisticians is being hired to focus their study on the healthy leaves on some trees, ignoring the fact that the forest is diseased. In such a situation my response would be to turn down the assignment, recognizing that others (who may be more naive or hungrier) may have a different response.
Sorry for the lousy proofreading in the first few words. SwiftKey is great, but requires constant vigilance!
If your professional reputation could be damaged by working for the wrong company, there is an incentive to decide before seeing the data what your conclusion will be. If your conclusion will be unpopular, you shouldn’t take the job. What an awful situation.
Statisticians should be able to work for unpopular clients and not risk their reputation as long as their work meets professional muster, just as lawyers can defend guilty clients and maintain their reputation.
If the reputation of statisticians was as low as lawyers, then just like lawyers, they wouldn’t need to worry about it going any lower.
John, I agree about meeting “professional muster”.
But the analogy to being a lawyer doesn’t seem right; as a lawyer, its recognized that your duty is to make the best case possible for the client; you are there to help the client. In contrast, statisticians should interpret data accurately, whether this helps the client or not.
A statistician with an established record of bending data to the will of the client should be mistrusted.
We need to be sure that our criticism is for “bending data to the will of the client” and not for choosing a distasteful client. If someone bends the data for The Society to Promote Motherhood and Apple Pie, that deserves our ire as well.
Agree. Also need to avoid criticizing someone who consulted for the Evil Multinational Puppy-Kicking Corporation but didn’t bend the data to suit.
A statistician bending data is equivalent to a lawyer perpetrating a fraud on a court. According to the rules such as lawyer can get sanctioned, or in theory lose his license. The reality is different. I caught a lawyer and a judge engaging in an ex parte communication, which is strictly forbidden. This is where one lawyer contacts the judge and has a private communication unknown to the other side. I caught them with a smoking gun: the hearing transcript had both the judge and the lawyer confessing. Despite my complaint to the bar, nothing happened to the lawyer. Nothing happened to the judge either. Yes the judicial system is corrupt in California.
+1
John:
I agree, but there’s still the “it goes both ways” argument. I have funding from the Defense Department. If someone wants to judge me for that, it’s only fair: I did take the money, after all.
Shorter OP: ad hominem arguments, although logically fallacious, nevertheless do capture Bayesian evidence.
Corey:
Yes, exactly.
Why does working as a consultant for a tobacco company discredit you if your work is valid? Shouldn’t we judge the work on its own merits?
How about lawyers? They often represent organized crime figures, child molesters, rapists, traitors, spies, other other loathsome creatures. How come the lawyer doesn’t get tainted by his client? Of course if the lawyer perpetrates a fraud on the court, he does get tainted, but otherwise not. There are exceptions of course. Gerry Spence represented Randy Weaver, and according to Spence, he got flak from some lawyers for that. The FBI murdered Weaver’s wife and child, and tried to frame him. Spence got a not guilty on the criminal charge against Weaver, and in a civil suit against the DOJ got civil damages. He also got flak for representing Imelda Marcos, where he also got a not guilty. On the other hand, a lawyer will likely get no flak from representing the likes of a vicious murderer like Stanley “Tookie” Williams. It seems our political class likes some kinds of criminals, but not others.
It seems to me that if a statistician does good honest work for a client, he should not be made to suffer because some people don’t like the client. Incidentally what is the risk of cancer form smoking? As best as I can tell from reading the literature, it’s about 10% for more than 20 years of smoking a pack a day. How does that compare to other risky activities such as hang gliding, motorcycles etc. Can statisticians consult for these companies without fear of reprisal from the ankle biters out there?
Zarkov:
1. I agree, except that there’s no evidence that the statistician in this example was doing “good honest work.” Recall this quote: “Lorillard’s chief of research okayed Olkin’s contract, commenting that he was to be funded using ‘considerations other than practical scientific merit.'”
2. I’m not going to get into a big argument about the addictiveness of tobacco; the subject has been much studied. I will merely say that I’ve never heard someone say that they want to quit hang gliding, skiing, motorcycles, etc., but they can’t do it.
3. Oddly enough, I’ve actually posted on lawyers who represent organized crime figures. To me, these lawyers are indeed tainted. And, just to keep some perspective, Ingram Olkin didn’t get fired or anything. Cigarette companies paid in part for his reputation; this is a piece of information that people can use in judging later work.
I’ve said it before and I’ll say it again. Those who disparage analysis based on the source of funding for that analysis misunderstand the process. Do tobacco companies want to buy a distinguished opinion? Absolutely. But the way that happens is that they canvass until they find someone who, after looking at the data, agrees with them. That requires them to occasionally file-drawer opinions that the data can’t sustain and to sometimes approach ten people before they find someone who agrees with them. But as A. Zarkov points out above, that makes the robustness of their conclusions and the ease with which the other side can rebut readily apparent.
The logical implication for a Bayesian is that the implication of paid-for experts is to consider even more carefully what it is they relied upon and to consider the contrary data and analysis. Fair enough. But characterizing the honestly-held convictions of paid consultants as espousing something they don’t really believe is wrong. As I told a client just this week when he wanted me to characterize the source of an opposing expert’s funding: “I’m all too willing to call him an idiot, but I’ll never call him corrupt wthout really good evidence.”
Jonathan:
My guess is that many of the recipients of cigarette funding don’t care much one way or other about the issues; it’s just some money they will take. Awhile back I gave a paid short course for Procter & Gamble. Do I agree, or disagree, with what they are doing? I have no idea. For all I know, P&G is doing horrible stuff all over the world. Or maybe they’re model corporate citizens. Pampers may be an environmental nightmare or they might be a great service to humanity (or maybe both!). I didn’t think much about this when taking the job.
I definitely agree with that. Not to get all Werner von Braun with you, if a company I feel to be doing the worst things on earth has a correct statistical position that they want me to espouse for some reason, I have exactly zero qualms about doing it. I’m giving my opinion about the thing I write my report on, not an explicit or implicit approval of anything else the company does or any notion of what my research will be used for.
1. Where is the quote from? What is the context? It seems to me that if you give a quote, you should tell us where it appeared, so we can see the context. Moreover, the quote doesn’t establish that Olkin’s work was not good or honest. Why does he carry the burden of proof? It seems to me that those who defame him carry that burden.
2. People quit smoking all the time. Yes it’s addictive, but not so addictive that it’s beyond most people.
3. So an organized crime figure should not get representation, or be forced to use a state supplied lawyer? How about people who sell him food, medicine etc. Are they tainted too? Do the doctors who treat organized crime figures also get tainted? When the state is trying to send you to the slammer, you need a lawyer more than a doctor. We have an adversarial system of jurisprudence, and as such having a champion is crucial. If you create a climate where some people get so demonized, they can’t get good representation because no lawyer wants to risk getting tainted, then you subvert our system of law.
Zarkov:
1. To find the quote, follow the links above.
2. I’ll leave this one to the researchers on addiction. I’ve known some people who’ve quit smoking, I’ve known others who’ve quit with difficulty, I’ve known others who couldn’t quit.
3. Nowhere did I say that “an organized crime figure should not get representation.” All I said was that to me, these lawyers are indeed tainted. But I don’t see why these lawyers should care that I have a lower opinion of them. That’s my problem, right? I think a lawyer should have the right to get paid to represent organized crime figures, and other people such as myself can feel free to use that information in forming our impressions of them.
P.S. Just to emphasize: I’m not saying that Olkin’s latest work should be in any way discredited because of the previous cigarette connection. I’m just saying that I can understand how such an inference can be drawn.
Andrew,
1. I will do that, thanks.
2. The number of people who engage in dangerous additive sports is small compared to addicted cigarette smokers. Nevertheless a number of people I know have confessed that they’re addicted to running, others to golf and tennis. For example the fellow in the office next to mine ran five miles every day, no matter what the weather. It wasn’t good for him, and he destroyed his knees. He was certainly addicted. But I don’t know anyone addicted to hang gliding because I don’t know any hang gliders. So I don’t think we can draw any conclusions about sports that are both additive and dangerous from our everyday experience.
3. You didn’t explain why these lawyers are tainted where other people who provide them with services aren’t. Of course you’re entitled to have an opinion about such lawyers. I’m pointing out the consequences should your opinion become widespread– demonized people won’t be able to get good representation. It’s a slippery slope. I’m simply urging you to think this through carefully.
Zarkov:
To me, it seems different because the lawyers are defending the criminal acts, whereas the groceries, doctors, etc., are just performing services. But I agree that I haven’t thought this through.
Andrew,
Lawyers don’t defend “criminal acts,” they defend clients accused of criminal acts. Generally a lawyer will attack the evidence against his client by impeaching witnesses, using expert testimony to attack forensics, or have evidence declared inadmissible. All that sure looks like services to me. Having a good champion in an adversarial process is crucial. Remember the Duke Lacrosse case? There we had a rogue prosecutor, and having a good lawyer was vital to combat the awesome power of the state against an individual.
You seem to think that the lawyer must approve of his client, or seeks justify the alleged crime. A good lawyer will always say,”I try cases, not causes.” Don’t confuse the lawyer with a cause.
The issue for me isn’t the evil of cigarettes, such as it might be. The issue is the longstanding track record of the tobacco companies, fully apparent in 1976 to those paying attention, of suborning the research process for their own profit. They covered up inconvenient results, misrepresented findings and used every method available to them to obstruct public understanding of the risks posed by their product. (Merchants of Doubt, Golden Holocaust, etc.) A conscientious researcher should steer clear of such stuff.
A tangential thought about addictiveness: is trying to divide activities into categories of “addictive” and “non-addictive” engaging in the sort of dichotomizing that Prof. Gelman eschews in a statistical modeling context? Some people are more prone to addiction than others, some activities are more addictive than others, and there’s an interaction (I might be personally predisposed to alcoholism but fine with moderating my gambling and video game activities). If we knew a whole lot more about dopamine receptors and how they were regulated by genetic background and personal experience, or if we had lots of observational/experimental data about how individuals responded to different stimuli, we might be able to quantify some of this …
Somewhat related, is this not also terrible advice:
http://lifehacker.com/5950930/learn-how-to-quickly-evaluate-the-worthiness-of-scientific-reports