## Cancer statistics: WTF?

This post is by Phil.

I know someone who was recently diagnosed with lung cancer and is trying to decide whether to get chemo or just let it run its course. What does she have to go on? A bunch of statistics that are barely useful. For example, its easy to find the average survival time for someone with her stage of this particular cancer. Let’s say that’s 12 months. Fine, but that’s an average over all ages, both sexes, and includes people who did and didn’t opt for chemo! Maybe the average time is 6 months if you eschew chemo and 18 months if you get it, and half the people do and half the people don’t? Or maybe 80% of victims get chemo, and those that don’t only average about 2 months.

The only other stat that seems to be widely available is “5-year survival rate.” Let’s say that’s 6%…that’s not so good. But if the victim is 80 years old they’re not so likely to live another 5 years anyway! Plus, again, the information that would be useful is the information that might change your behavior. If you have a 1% chance of living two years if you eschew chemo, and a 20% chance if you get it, then maybe you’d choose to get it even if you know you won’t make it 5 years.

What people should care about is conditional probabilities, not summary statistics. You don’t care what the average survival time is, averaged over all patients, instead you want to know it for people like you: your stage of cancer, your sex, your age, your physical condition.

I’m kind of shocked, and very disappointed, that there’s no website somewhere that lets you put in your age, sex, cancer type and stage, and maybe a few other relevant details, and get the statistical distribution of survival times if you do or don’t get chemo (or surgery, or radiation, or whatever). This lack was understandable in 1996 or even 2006 but it’s 2016 for crying out loud!

This post is by Phil.

1. MJT says:

I agree with the usefulness of such a plug and play delivery of conditional probabilities.

But where do you source these conditional probabilities from? Is it ethical to choose one model x methodology over competing ones in computing these “less biased” estimates?

I think federal statistical agencies face the same ‘methodological’ ethical-balancing act. They purposely aggregate publicized estimates and put the onus on the end-user to intuit these ‘less biased’ conditional probabilities.

• Phil says:

As Bill James says, the alternative to good statistics isn’t no statistics, it’s bad statistics. People have to make these choices somehow and the medical establishment should try to help with that.

If there are several models out there that give substantially different numbers then you can present them all, or present the one you think is best but mention that there are others that might be more accurate.

2. Chip Lynch says:

Hey Phil, I’m very sorry to hear about your friend’s diagnosis; I know how earth shaking that can be and I’ll hope for the best for both of you.

I just happened to work on a project with some undergraduates this past semester that does almost exactly what you describe; it was even specific to lung cancer… it’s unlikely to get past a school project or on the outside a short academic paper, but I literally have a draft paper open for editing while taking a break to read this post. We took some lung cancer data available from the NIH and performed some supervised machine learning techniques to do the sort of analysis you’re talking about; one of the students even built an R/Shiny front end to an ensemble of predictive models. I will tell you that it’s difficult to be very precise, and a doctor can still likely give some more custom advice, and I learned a lot about why there probably aren’t such tools in the wild. I wouldn’t feel comfortable just plugging in numbers just because we need some peer and ethical review to make sure we’re not spitting out nonsense, but I thought I’d share, since the work is so similar to what you’re discussing, and maybe we’ll get something like that out there soon.

FYI, the contact information on your website (that is linked to from your name at the top of this post) is down, but if you want to drop me a line, I’d be glad to discuss it to any extent I can. And again, I’m sorry you know someone suffering from this.

• Andrew says:

Chip:

I updated Phil’s link on the webpage.

• Martha says:

“a doctor can still likely give some more custom advice”

But different doctors amay give different advice.

• jd says:

I was looking for one of these a while ago and stumbled on this (non-working) one for pancreatic cancer from University of Iowa. I’ll post a link since it could be my browser that’s the problem: http://myweb.uiowa.edu/bjsmith/pancreas/

• Phil says:

Chip,
I’d be very interested in hearing about the project you describe and I’m sure others would too. If you’re not confident in the numbers that’s OK, maybe you (and your students) can do a brief write-up that describes why it’s a hard problem. I’ll speak for Andrew and say that if you write six or seven paragraphs he or I will put it here on the blog. It doesn’t have to be anything formal.

3. Fernando says:

@Phil

Totally agree. This happens all across medicine. It is a scandal.

• Innocent Bystander says:

This does indeed happen across all medicine.

I have early stage prostate cancer and the data I have access to as a basis for decisions about treatment is absolute garbage.

On top of which

* Most medical practitioners are statistically illiterate. Most endocrinologists cannot do a simple problem based on a medical test for a rare condition (most specialists were not within a factor of 3 of the right answer!), cannot even tell you what a P value means – 11% got both multiple choice questions right even with hints.

* Most medical researchers are not much better. Often you get “we found no significant effect from X so X has no effect” and worse. Also:

* Failure to publish useful information falling short of statistical significance. In effect a vast amount of information, collected at great expense, is lost. This would not be lost with competent Bayesian analysis.

* Massive biases in data collection (eg incidence of side effects from radical prostatectomy according to patients is *double* what doctors think).

Part of the problem is that the medical system selects people with tremendous physical and mental stamina who are good at memorizing things. Surgeons also need to have some manual dexterity. Note what’s missing here.

/rant

• Phil says:

I’m disappointed that this is the case for prostate cancer too, but I suppose it’s good to know that the problem is real, not just a false perception on my part.

Good luck.

• Moreno Klaus says:

If i am not mistaken, i think nobody ever did an RCT comparing all or even the main treatment options available, for prostate cancer. Even if they did, it would almost surely be underpowered to find an effect for every age group or cancer stage. It is just too expensive, it is also quite complex from an ethical point of view, and it would take maybe 20 years to get the final hazard ratio. By that time there would be new treatments which render the trial possibly useless… This of course is not an exclusive of prostate cancer, or cancer, it is a very general problem in medicine. So yes, it is still more art than science….

• The problem is thinking that the only way to get information about sub-groups is to test each subgroup. That’s more or less “model free” thinking and heavily influenced by the Null Hypothesis Testing paradigm.

How well would we do at predicting heating and cooling energy requirements for buildings if we decided that we needed an RCT for every type and size of building? Not very well.

More model building and less hiding or throwing out the data after you declare a “significant” finding would go a long way toward improving things. However, doctors are not mathematical model builders, and neither are (the vast majority of) biologists. And many mathematical model builders have virtually no biology background… so who will do this task?

I kid you not when I say that I would devote the rest of my life to this kind of thing if I could find a foundation or whatnot where I felt that I could make good progress… And I think there are probably plenty of other people who feel the same way, several on this blog even.

What’s needed is an organization that appreciates the need enough to pay for it, and which has both money and political clout to help change the “way it’s done”. An “outsider” organization, but one that the medical field isn’t going to immediately engage in battle with. It’s a big multi-faceted issue and the politics and cost is really more problematic than the technology.

• Karen Harker says:

I believe the data is out there – patient data has been collected for decades – it just needs to be made more available. Actually, Medicare data is available for research purposes and many epidemiological studies have been done to reveal patterns. The RCT is by far the least efficient method of answering the kinds of questions that Andrew and others are asking – what is best for me? It is heartening that data analysis seems to be gaining the interests of younger people – noting the increase in infographics and the popularity of Freakonomics.

4. Jennifer Brokaw says:

One of the first things that happens in an Oncologist’s office is discussion of the conditional probabilities of survival given course X vs. course Y. This seems to be the “special knowledge” of Oncologists. Very hard to find on the web. Of course, the patient hears none of it. Their minds are buzzing with the question: “Am I going to die? am I going to die? am I going to die? When? When will I die? WIll I suffer?” Nevertheless, when treatment course is discussed, the latest information about survival stats with conditional probabilities built in is always discussed. When we get more information about genetics, there will be even more conditional probabilities to factor in….tough stuff for a layperson to process “in the moment” is my main point (good to have a statistician friend accompany them to appointment!) Even better to have a website or graphic to come back to later when the emotion is not so overwhelming-Jennifer Brokaw, MD (not Oncology)

• It’s not so clear to me that the special knowledge of Oncologists isn’t *MAKING UP* conditional probabilities out of pretty meager oatmeal.

• Martha says:

Agreed. That’s what the recommendation given by my aunt’s oncologist seems to have been.

• Eric Rasmusen says:

Yes, the individual doctor has an idea– based on the patients he’s seen. Absurdly, the doctors don’t seem to get together and share their private data, even within a give hospital. Probably they worry about patient confidentiality, which is crazy since nobody needs to have their name attached, but might be legally problematic anyway. In fact, I suppose without advance human subject legal permission, it would be forbidden for two doctors to pool their experience.

• Martha says:

I’d guess that confidentiality isn’t as much an issue as that most physicians don’t have the habit or the inclination to share experience with other physicians. Possibly there is more of this in teaching hospitals.

5. Thomas Speidel says:

Understand the heated emotions, nonetheless, it’s hard to see a “scandal”.
In some of the most devastating cancers there is -unfortunately- not a lot of decision making from the patient perspective, although the physician would collaborate with the patient on issues such as quality of life, complementary therapies, palliative care.

Survival estimates may not find direct applicability to patient choices, but they do inform clinical research, protocols and guidelines and these, we hope, translate into improved outcomes. Survival estimates, population surveillance, cancer screening can also help with planning ahead (how many cancer care facilities? What screening programs? At what age? How many MRI, CAT machines?)

Prognostic models do exist and have existed for many decades. Here’s a pre-shiny example for head and neck:
https://www.msbi.nl/SV/Chart.aspx?model=Survival+model+Salivary+Gland+Malignancies

• Phil says:

That’s a great example of what I think should be available. Why aren’t there more tools like this?

Thomas,

666 patients across nine factors, one of which is age, median followup for 10 years on 20+ year old data.

6*4*2*4*2*2*2*4*4=25,000

Click on >80 and start adjusting the numbers towards lower staging etc, ACC glands etc.
80%+ 5 year survival on an >80yo?

OK…

6. Rahul says:

It’s disappointing yes, but it is hardly surprising. Such stats are rarely available for almost any medical condition, let alone cancer.

The root cause, I think, is the fact that our unit of analysis is a individual study. Ergo, all analysis gets done at the study level, the particular hypothesis under question is examined, conclusions written up and the raw data is essentially never again utilized.

Even meta analyses analyse what was published in the primary studies and not the raw data.

What we really need is good systems to store and analyse the raw data on demand to match it with the relevant clinical question.

It’s more complicated than that.
Just to get people started on it…

This post is by Phil.
“What people should care about is conditional probabilities, not summary statistics. You don’t care what the average survival time is, averaged over all patients, instead you want to know it for people like you: your stage of cancer, your sex, your age, your physical condition.”

How do we measure Mrs. Jones’ “risk”?
If we want to measure her height or weight, we get Mrs. Jones and measure her height and weight.

If we want to measure Mrs. Jones’ risk, we really can’t measure that in her.
We get a bunch of people “like her” and measure it in them, then we apply that number to her.

So what is “like her”?
Well maybe start grouping people in risk pools.
Male/female. Smoker/non-smoker. <50/50+. Staging of cancer etc…
We think these risk pools are causally related to the outcome but…

So we start getting smaller and smaller risk pools…with the hope that they become more and more "like her."
What are we going to take into account? Where does it stop? Can clinicians agree on what "like her" is, even if we could measure something like "physical condition."

If we just did the three dichotomous groups and four cancer stages you smart people can figure out what kinds of sample sizes we would need to find 20% differences in outcome. Keep in mind the event-rates of screening findings, for instance say breast cancer in screening mammography is…oh…say 1%.

Then there are the clinical trial issues already mentioned that usually don't really answer the question we want answered.

• Innocent Bystander says:

> If we just did the three dichotomous groups and four cancer stages you smart people can figure out what kinds of sample sizes we would need to find 20% differences in outcome. Keep in mind the event-rates of screening findings, for instance say breast cancer in screening mammography is…oh…say 1%.

This is true if you use frequentist statistics. Bayesian analysis in contrast will eke out whatever information is there. See eg Gelman “Bayesian Data Analysis” and/or Jaynes “Probability Theory The Logic of Science”.

• I second this. Medical “raw” data should be published in archives (anonymized as best as possible of course) so that we can combine information from multiple studies and so that people can undo the harm caused by bad analysis. Embracing uncertainty is a necessity in medicine and in many ways the frequentist notions are used to actually pretend that it doesn’t exist: (ie. we found a (statistically) significant difference, therefore the difference is (practically) significant, or we didn’t find any significant difference therefore the effect is zero).

• Martha says:

+1 to “Embracing uncertainty is a necessity in medicine”

• Anonymous says:

Unfortunately, most institutional review boards (IRBs) would make publishing anything close to the raw data difficult. The first (and perhaps only) directive of the IRB is to protect the patients. They do this through strict rules on what can be collected and how it can be used. Only the most limited data required for the study at hand can be collected. That means that much of the background medical information that would be useful (or necessary) for a multivariate analysis are not gathered or patients with any other confounding medical conditions are explicitly excluded. Therefore, if the data were collected, anatomized, and distributed, it would of limited utility for other analyses or to make a global risk model without quite sophisticated stats (which most physicians are definitely lacking.)

For example, the Collaborative Ocular Melanoma Study, examining intraocular melanomas, only compared two treatments depending on the size of the tumor. It excluded over 40% of patients who were attempted to be enrolled — anyone who was sick or on immunosuppressive medications. This was a huge eye cancer study, probably cost millions of dollars, and may be applicable to some, but not all, patients who have choroidal melanoma. For example, only 4% of the patients were older than 80yo.

A registry of some sort would be helpful, but they aren’t ideal either. Patients’ records would need to be anonymized to protect their privacy — therefore tracking patients over time if they change providers would be difficult or impossible. There is NO short-term benefit to doctors, clinics, or hospitals spending the time and money to register patients. In my own medical practice, I see 30+ patients a day (because there are so many patients who have waited to see me and because reimbursements are continually being cut so volume is the only way to pay the clinic staff). Adding an extra 3-5 minutes per patient or hiring extra staff to enter their information in a registry is not viable. Finally, the tests, exam, and interpretation that I get would not be identical to than exam by another provider (which could be handled by multilevel models, but no physician would understand that.)

Finally, it could probably be mentioned that ICD-10 is supposed to allow data-mining of patient records. Talk to any practicing physician, though, and they will tell you that medical coding and billing are inaccurate for any scientific purpose. We code so that we can justify insurance companies and Medicare to pay for the work that we do — accurate coding doesn’t help the patient sitting in front of me. In addition, the codes are often vague or worthless (see W61.11XA). Therefore, almost every doctor will code things differently making coding/billing not useful for actual science.

• Rahul says:

I think the ICD coding is a disaster from the point of view of research or analysis. It was meant for billing & that’s what it does best, but not even that it does very well.

• Eric Rasmusen says:

Wouldn’t it be very easy just to release all patient records to a registry, name replaced by a number based on the social security number? (this last only to track people across doctors— for a lot of purposes, like virulent cancer, total anonymity would be ok) Then a research could hire some grad students to extract the info he needs (e.g., age at diagnosis of prostate cancer, radiation yes/no, and age at death from prostate cancer). If there’s a lot of it, the task could be farmed out to a company in India.

• Phil says:

You’re sort of answering your own questions here. At any rate there’s nothing new or complicated about the concept of “risk factors.”

You take into account whatever information you have that is relevant. If you can estimate the effect of age and it’s not negligible, then take age into account. Ditto for sex, stage of cancer, choice of treatment, etc.

It’s complicated, far more complicated than even physicians appreciate. It’s impossible to have these conversations without some background. I follow this blog, and I haven’t seen these concepts discussed…maybe because they’re so obvious, but they may not be obvious to everyone.

Some starter questions are: “How was the cancer detected?” Was this a pure screening finding for the purpose of screening for lung cancer? Was it an incedentaloma discovered during imaging for another problem? Or was it because they were coughing up blood? Or something else? These all go to prior probability.

Screening findings have the potential to have tremendous bias as the clock starts at detection which gives the illusion of a longer survival time even if there isn’t one.

There is also evidence that screening detected cancers may be more likely to be indolent, or slow-moving and never cause a problem for the patient.

https://en.wikipedia.org/wiki/Length_time_bias

The clinical trials center mostly on “the benefits of screening” with screening detected cancers generally getting treatment, and looking at all-cause-mortality. The numbers aren’t encouraging with NNTs generally being quite high and trials reporting relative risk reductions (RR) and not absolute risk reductions (AR). Further, the harms of treatment and screenings are not disclosed.

But what we really want to know is in the screening detected cancer, are there better outcomes with treatment or not. Same with non-screening detected cancers.

The problem is the people that “opted” for this or that may have opted for this or that based on something, so it doesn’t inform decision-making.

https://en.wikipedia.org/wiki/Intention-to-treat_analysis

There generally isn’t equipoise, and the ethics of randomizing someone with a “known cancer”…well…would you want to be randomized to the Wwe discovered cancer in your lungs but you’ve been randomized to the ‘Let’s see what happens group. Good luck with that.’”?

The basic problem is one of disease vs. disease that matters vs. disease that matters that we can “fix” and improve outcome. Western medicine typically lumps everything in “all disease requires treatment.”

For an eye-opener, you can start with the Welch book:
http://www.amazon.com/Overdiagnosed-Making-People-Pursuit-Health/dp/0807021997

Dartmout and the BMJ have been hosting a conference on the topic for a number of years.

• Innocent Bystander says:

> the ethics of randomizing someone with a “known cancer”

This is true but there are many patients who do not have the recommended treatment. Eg prostate cancer at Gleason Score 6 the current consensus is no treatment (active surveillance). However I was just reading a PhD dissertation which found that half these people did in fact have some form of treatment.

Amazingly too there have been studies that did randomize the PC cases above! When the treatment has serious downsides (prostate removal can cause impotence, incontinence, even death, radiotherapy can cause fecal incontinence etc etc) and the benefits are not clear then it can be ethical to randomize, especially if you are offering people a benefit such as free treatment.

And there are techniques for dealing with non-random choices (Gelman “Bayesian Data Analysis” has a lot of detail). They are not perfect but this is not by any means a hopeless situation.

• Moreno Klaus says:

Actually, the % of people with Gleason 6 in the US getting Active Surveillance was only 10% in 2010, but it is already on its way to 50% in 2015. It simply takes time for both patients and doctors to get used to this paradigm change of no treatment, which is relatively new.

• Phil says:

Jkhademi, we can agree that it’s complicated. That doesn’t mean there should be no attempt to help patients take this sort of information into account!

8. Patrick says:

Here’s another calculator based on the SEER database that estimates the effect on survival of radiation therapy.

“The Surveillance, Epidemiology, and End Results (SEER) Program of the National Cancer Institute (NCI) is an authoritative repository of cancer statistics in the US. The SEER data attributes can be broadly classified as demographic attributes (e.g. age, gender, location), diagnosis attributes (e.g. primary site, histology, grade, tumor size), treatment attributes (e.g. surgical procedure, radiation therapy), and outcome attributes (e.g. survival time, cause of death), which makes the SEER data ideal for performing outcome analysis studies.”

http://skynet.ohsu.edu/nomograms/postrt/nsc-lung.html

A Nomogram for Predicting the Survival Benefit of Post-operative Radiotherapy for Patients with Non-small Cell Lung Cancer
Wang, S.J. et al.
International Journal of Radiation Oncology • Biology • Physics , Volume 72 , Issue 1 , S447

• Anoneuoid says:

SEER is pretty nice, but there are gotchas:

“The second caveat is that SEER areas are not entirely representative of the overall US population. SEER areas were selected to include a relatively large fraction of racial/ethnic minorities (refer to http://seer.cancer.gov/). SEER areas also overrepresent urban areas and higher income persons (16).”
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3203375/

The SEER population is also younger than the US population, so watch out for binning by age (a scourge of epidemiology imo).

• Phil says:

This is great, and is another good counterexample to my assumption or claim that this sort of tool doesn’t exist.

I guess some of these tools are out there, for some cancers, from some sources. Why on earth doesn’t NIH (or somebody) support the creation of these things for many cancers and other diseases?

• AND OTHER DISEASES! yes yes yes

• Anoneuoid says:

What makes you think that is accurate?

• Phil says:

I don’t know whether it’s accurate or not, although there are obvious reasons to believe it’s not wildly inaccurate. But its existence proves that there is not a regulatory hurdle or other major impediment to creating such tools.

• Anoneuoid says:

>”there are obvious reasons to believe it’s not wildly inaccurate”

There doesn’t seem to be a detailed description anywhere, but there is a link to a conference abstract at the bottom. There it says:
“A Cox proportional hazards (CPH) multivariate regression model was constructed using data from 51,423 patients with resected NSCLC diagnosed from 1988-2004 from the NCI’s Surveillance, Epidemiology, and End Results (SEER 17) database. Patient and tumor characteristics selected as covariates included age, gender, TNM stage, and histology”
http://dx.doi.org/10.1016/j.ijrobp.2008.06.1827

The biggest difference between treated vs not I’ve been able to get with that tool is 27 months median survival. I don’t see where they look at year of diagnosis. Did screening or diagnosis practices stay constant from 1988-2004? If not, perhaps the cancers were being detected ~2 years earlier as we get closer to 2004.

I don’t know how it is usually diagnosed, but CT scans are at least one method:
http://www.cancer.org/cancer/lungcancer-non-smallcell/detailedguide/non-small-cell-lung-cancer-detection

As we would expect, resolution/etc was improving throughout that 1988-2004 period, which should be related to earlier detection:
https://wiki.uiowa.edu/display/881886/CT+Historical+Time+Line

New radiotherapy protocols were also developed during that same time, it sounds safer so was probably used more often:
“While traditional radiation was previously more limited by technology for normal tissue sparing, modern 3D-CRT is able to deliver high-dose radiation to the tumor target areas while minimizing dose to surrounding tissues”
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3968556/

So if use of radiotherapy increased over the years and detection was occurring earlier, a large portion of the apparent treatment effect could be due to more treated patients later on (ie unrelated to the effectiveness of the treatment). I don’t know much about this in particular, just saying it is very easy to generate misinformation.

• Keith O'Rourke says:

Phil:

There was a real good resource for side effects for an adjutant cancer treatment in children that a clinical colleague and I found very helpful for a relative of mine almost 20 years ago. A fairly comprehensive registry had been set up and made accessible at least to clinicians. I was surprised but it is _just_ a matter of mobilizing resources (including dealing the ethics review barriers).

FDA’s min-Sentinel and now Sentinel seems to be better focused on benefit harms and the uncertainties of both.

It is a lot of work and requires a lot of resources…

9. Mark Danese says:

I agree about the importance of such data. I worked on something like this a few years ago (see http://www.diseasepath.com) as a demo. I only point this out to say that it is doable, but it is hard, requires a lot of data munging infrastructure, and has to be updated regularly across many cancers. We have actually been building infrastructure over the last 2 years to be able to do exactly this, but it is a long haul. It is complicated because therapy and comorbid conditions are not readily available from US Cancer registries in a systematic fashion. I mention this so you know it is not forgotten, nor is there a scandal. It is just hard.

• Phil says:

I can believe it’s hard to do it really right. I can’t believe it’s hard to do better than nothing.

But even if it’s hard, that’s OK, people do hard things all the time. It’s time for this to become one of them.

• Jacob says:

What you’re describing would be a tool used for diagnosing a disease, and would need to be regulated as such. That means going through all the approval processes the NIH requires for medical software. For a tool which will never make any money. I agree it would be great (provided doctors and patients understand the limitations), just weighing in.

• Phil says:

Uh, no, I guess I didn’t make things clear somehow. I’m not talking about a tool for diagnosing a disease.

The webpage Patrick points out, http://skynet.ohsu.edu/nomograms/postrt/nsc-lung.html is more or less what I’m looking for. Why don’t we have these for pretty much every cancer?

• Whether you would call it that is separate from whether the FDA and the courts etc would call it that. It’s definitely a question as to whether the person who put that link or others similar online at ohsu is or is not “practicing medicine without a license” etc.

In other words, there are a variety of potential legal issues here.

• Rahul says:

A legal issue never gets really resolved unless a test case goes to trial. Has anyone sued any of these existing tools yet?

• Anonymous says:

I’m not sure if people have gotten sued; but they have had the FDA come down on them hard. The 23andMe fiasco with direct-to-consumer diagnostic tests is a good example. I can’t imagine a large organization making a tool like this available with up-to-date information from a number of studies without a sound legal plan.

• Phil says:

Perhaps I should find out more about this; it seems ridiculous that you could be prevented from publishing estimates of survival rates. Free speech, and all that.

• debbie says:

well – there are several others
prostate cancer
MSKCC, NYC
https://www.mskcc.org/nomograms/prostate
MD Anderson – breast, colorectal, oesophageal
https://www.mdanderson.org/education-and-research/resources-for-professionals/clinical-tools-and-resources/clinical-calculators/index.html

but they are obviously incomplete and could be standardized

10. Shravan says:

It must be very distressing for your friend, Phil, to be in this situation. I have a very similar problem but definitely not with any of the urgency of your friend’s case. In about six years from now (even longer if between now and then German doctors engage in even more fraud regarding transplantation) I will have to decide whether to get a kidney transplant or stay on dialysis. The doctors always say, no question, get a transplant. But that kind of averaged out blanket statement is useless for an individual. Some of the people commenting to this post point out that one can’t really drill down to the individual case, partly because the cases will become more and more sparse as we condition on the specifics of the individual. But there must be a way to find a middle ground between gross average results and overly specific results for an individual.

It seems like a really important and intrinsically interesting research problem for statisticians working in medicine. If I wasn’t working full time in the completely useless (in terms of practical applications that save lives) area of psycholinguistics, I would work on this topic.

• Phil says:

Yes, exactly: we can never know your personal risk but we should be able to do better than averaging over all ages, all sexes, all heights and weights and levels of physical fitness.

11. Emile Decrespigny says:

@Phil

I was in a similar situation quite a few years ago. Then I came across Gould’s “The median isn’t the message”
Although this doesn’t solve the problem you and friend are facing it has been particular useful, reassuring and at least of some help for me personally.

Digging deeper I found the work of Gigerenzer on doctors diagnosing different forms of cancer/illnesses somehow interesting. This is from a purely professional background (I have a statiscal background in research) quite revealing but doesn’t solve the problem of information to make a kind of informed decision.
All the best!

12. Shelly C says:

Hi Phil. I am very sorry about your friend. I have been wondering the same thing, and I agree with Shravan that there must be a way to find a middle ground. I am glad that there are some like Mark D. who have started on this tough problem. In medical school, I remember memorizing the gross survival numbers and wondered where and how oncologists received their “special knowledge” to know what to say to their patients individually, particularly for treatments. I assume it’s based on previous studies (and personal experience), but can these studies which are often designed for a broad population (traditionally for white men) be applicable to the individual? Like Jhkademi said, what are characteristics that are meaningful to outcomes & representative of the individual at risk? Is it age, co-morbidity, tumour classification etc? Can someone’s behavior, lifestyle, and environmental factors also be meaningful and somehow captured as well? This is a tough problem, and a problem worth working on and finding a middle ground not to be an authority, but to provide a better guidance to patients.

When I read your post,I thought of Paul Kalinth & his Jan 2014 NYT’s article: http://www.nytimes.com/2014/01/25/opinion/sunday/how-long-have-i-got-left.html?_r=0 .(The recent article by his wife is heartwrenching btw). He talks about looking for an answer on his survival from his oncologist.Although he states that his oncologist does know his survival time frame, it is wide and uncertain. For me that’s equivalent of the oncologist not knowing, but no one knows.

• Phil says:

I’m very comfortable with the idea that the statistical distribution of survival times can be very long — “with this kind of cancer at this stage, some patients die in a few months while others live for years” can certainly happen. That NYT post says:
Be vague but accurate: “days to a few weeks,” “weeks to a few months,” “months to a few years,” “a few years to a decade or more.” We never cite detailed statistics, and usually advise against Googling survival numbers, assuming the average patient doesn’t possess a nuanced understanding of statistics.”

That is a really really good NYT piece, thank you for noting it. It touches on all the key themes here. I was struck by this paragraph: “Poring over studies, I kept trying to find the one that would tell me when my number would be up. The large general studies said that between 70 and 80 percent of lung cancer patients would die within two years. They did not allow for much hope. But then again, most of those patients were older and heavy smokers. Where was the study of nonsmoking 36-year-old neurosurgeons? Maybe my youth and health mattered? Or maybe my disease was found so late, had spread so far, and I was already so far gone that I was worse off than those 65-year-old smokers.”

I was also struck by the fact that the guy is a doctor, with access to other doctors, including experts in his disease…and they simply will not give him the information he is begging them for.

In his case, and perhaps in many cases, perhaps the information wouldn’t make a difference: if there’s one obvious treatment to try and there’s not a huge downside to trying it, you try it. That seems to be the case with this guy. But in a lot of cases there are important decisions to be made — the primary ones being whether to try treatment at all, and if so, which treatment — and it is, as another commenter put it, a scandal that patients can’t get information to help them with those decisions.

Thanks for passing this along.

13. Keith O'Rourke says:

Phil:

The advice my clinical epidemiology group was given by the director “If you are facing a serious medical problem, do not try and read the literature to sort it out yourself, find a physician you trust and do what they say.” Finding a physician you trust, even for accomplished clinical researchers is the over-whelming challenge.

In your situation, its not _you_ but I would never attempt to published clinically related material or especially give advice to a potential patient without working closely with a physician I trusted.

On the other issue of available information, this paper was incredibly hard to get published and though focused on a Canadian context, we were unable to publish it in a Canadian journal as according to one editor “it had no comparisons group nor tests” http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2634695/.

• Considering the large numbers of costly medical mistakes each year (400,000 per year dead from medical mistakes? millions with serious harm?) which if I remember correctly is a line of research that I actually learned about from you on this blog at one point, it seems a little shocking that you’d come out in favor of the “find a doctor you trust and do what they say” approach. It may really be that in the vast majority of situations no such doctor exists. What’s the backup plan?

Still, I acknowledge that reading the literature is likely to get you nowhere, it seems so so so hopelessly broken. The stuff that our favorite blog gadfly “Laplace” is always going on about can sometimes seem a bit over the top…. until you look at medicine where this **** matters a LOT.

• Keith O'Rourke says:

> What’s the backup plan?
Don’t get sick!

Seriously, it is like the alternative to growing old, there aren’t good ones.

Unfortunately, clinical context can be crucial and the understanding of that context variable in the community of clinicians.

And there is Tukey’s warning about not taking oneself to seriously only on the basis of mere technical knowledge.

• I totally get it that I’m not a doctor, but I also have found that the only way to cure a recent chronic illness where my doctors kept trying the same old thing that didn’t work was to wade into the very recent biology literature and figure out what was going on at the level of bacterial colony population dynamics and feedback systems (thank god some biologist had actually done some studies in 2013 or so!).

Sending a few articles to my doctors and then explaining what I found led uniformly to a response sort of like “oh wow that actually makes some good sense, I hope that works” One doctor actually suggested we write it up and publish it. The fact was that it did work. and I’m much better today than I was 6 months ago, but it would have taken maybe 20 years for medicine to finally come around to that kind of knowledge being “in the clinical practice”. I got lucky, my illness wasn’t actually life-threatening and the measurement technology had been developed by the rapid rise of biotech in the last 20 years, and the biology research and measurements had been done and published in the last few years.

The situation is similar to the treatment of Clostridium difficile by swallowing pills full of a healthy person’s poop. Any biologist would have told you that’s what you should do in 1965, but “fecal transplants” are like a 2010 or later concept.

Koala cubs will die unless they eat their mother’s poop (they’ll be unable to digest their food). True fact!

• All of which is basically to say that there are two sorts of situations

1) where there really isn’t much that is known about the illness which isn’t already in clinical practice

2) where things are known about the illness by biologists and researchers which are not commonly known by practicing doctors

I think one should always make an effort to figure out which situation one is in. Given the rapid progress in many fields of biology, the situation (2) can be much more common than is realized. Figuring this stuff out though can be difficult if you don’t have much research biology background.

So, maybe I’d modify your suggestion to “find a doctor you can trust, and a research biologist you can trust, and try to figure out what is known about your disease in the clinic as well as the biological background that might not yet be in the clinic”

And yes, Don’t get sick!

• Phil says:

“If you are facing a serious medical problem, do not try and read the literature to sort it out yourself, find a physician you trust and do what they say.”

I agree with Keith: Most people don’t know how to find a physician they trust in this way, and there are far more patients than can be handled by trustworthy physicians so most patients are not going to be treated by a physician who deserves trust.

There are hospitals, and even areas of the country, where Treatment A is favored over Treatment B even though Treatment B is better.

And more to the point, how is the doctor supposed to know the answer if they don’t have tools? If you said “there’s a website that is only available to doctors, where they can put in the patient’s age, sex, cancer type, cancer stage, and some other information, and they see the statistical distribution of outcomes for each different treatment” I might be unhappy about the tool only being available to doctors but I wouldn’t call it a scandal. But without such a tool, I don’t have any confidence that the physicians are making good choices.

• So, you agree with Keith that it’s hard, but disagree with Keith that it’s necessarily good advice?

In any case, I agree with you, that it’s hard to believe in doctors when evidently this kind of data just isn’t available, when basically, no-one knows what’s good or bad. It’s not like it’s their special secret handshake website it’s just not out there (Or mostly not out there). By the way I think if only doctors had access that WOULD be a scandal! especially since the funding is all federal.

The thing you find out is doctors are basically groping around in the dark in many fields. In a few fields some pharma company comes up with some research and develops a new miracle drug, and then the doctors can actually do something. It is very rare for any of these discoveries to be actually driven by medical research as opposed to say biological research (ie. by people who don’t treat patients). We don’t usually design massively successful new treatments in humans (where we’d need doctors), we figure things out in mice etc first and then move to humans afterwards. It’s not the only way, but it is the most common thing. The main area where doctors do research directly that can have big benefits seems to be in surgical technique.

This is why we have “evidence based medicine” as a movement, because some people who wanted to become doctors or be better doctors couldn’t really handle the fact that most medicine is completely made up without ever being checked against reality.

Days before he died of heart failure my grandfather’s cardiologist breezed into his hospital room and assured everyone that my grandfather would soon go home in a week or so after having an extremely expensive heart surgery at age 92… The family wanted to discuss hospice care given his overall health and she absolutely refused to discuss. She then went on vacation and had her friend do the surgery. From that point onward he declined for a few days and then died of heart failure in less than a week or so. This happened in Berkeley last year, your neck of the woods. I don’t feel like being cynical here so I won’t tell you what I really think, I’ll just say that decision making like that happens millions of times every day in medicine.

• Phil says:

Whoops, sorry, I meant I agree with YOU, Daniel! Most people can’t find a doctor they trust to really know what is good for them. I know I wouldn’t trust some randomly appointed oncologist.

Basically I think there’s a need for what I’m asking for — a website or tool that allows you to provide personalized inputs that are known to influence survival rates, so you can get a statistical distribution that is closer to reflecting your personal situation than just the average over all people everywhere — and it’s somewhere between unfortunate and scandalous that this doesn’t exist.

• Martha says:

I agree that finding a doctor you can trust is very difficult. My experience, though, is that specialists are often better than generalists. A couple of examples:

1: My PCP said, “That lump is probably just a swollen lymph node; there is a lymph node right there. Take these antibiotics and come back after two weeks. The swelling should go away.” I dutifully took the antibiotics and returned when there was no change after two weeks. SO the PCP sent me to a surgeon for a biopsy. The surgeon said there’s not lymph node there; he needed to remove the lump and send it to pathology to figure out what it was.” He gave a guess when he cut it out, but repeated that he needed to send to to pathology, that he was just guessing. Pathology identified it. So less trust in the PCP, but trust in the surgeon who was not overconfident.

2. Dentist says, “This X-ray shows that there is inflammation in the root of this tooth. You need a root canal,” and his assistant later says Dr. X is so good at spotting those things. The endodontist does a thorough exam and says there’s no inflammation there; no need for a root canal. Less trust for the overconfident regular dentist; more for the endodontist.

• Rahul says:

But isn’t that a feature more than a bug?

A specialist is a better trained at classifying a very specific set of patterns. A PCP sees everything under the sun.

A good PCP treats what he can and passes along the rest to the right kind of specialist. If a PCP was as good as the specialist we wouldn’t need the specialists, would we?

• Andrew says:

Rahul:

Sure, but you want the primary care physician to know what he or she doesn’t know. What you don’t want is a PCP who is overconfident, who pushes the panic button too quickly or, conversely, tells you not to worry when you should be worrying. I’ve seen both these things happen. Maybe PCP’s need to be trained to say they don’t know (although of course without always passing the buck, either, that’s the dilemma). Just to complicate the picture, I’ve also been told by a physician that patients often come in demanding some test that he (the doc) knows is unnecessary.

• Rahul says:

I think it needs an asymmetry of errors: Skew training towards false positives than false negatives.

If pressing the panic button means being sent to a specialist or ordering additional tests that’s not too bad.

Yes, to know that one doesn’t know is a crucial skill. Luckily in many interventions by the nature of the system, to “do” implies sending to a specialist anyways (for most treatments likely to cause inadvertent damage in case of a false positive) so there’s a natural safety in the hierarchical system.

e.g. If your PCP mistakenly thinks you have rabies he’s not going to try to treat it himself. But to miss rabies would be bad.

• Anonymous says:

Rahul said: “Luckily in many interventions by the nature of the system, to “do” implies sending to a specialist anyways (for most treatments likely to cause inadvertent damage in case of a false positive) so there’s a natural safety in the hierarchical system. “

My experience is that to “do’ often doesn’t imply sending to a specialist right away, but sending to a specialist only after the PCP has tried something — e.g., the antibiotics-first case above, and another case that comes to mind: A PCP tried freezing a nasal growth, requiring several attempts and producing side effects. He said cutting it off would require a local anesthetic that would be very painful to administer. I suggested an ENT. He at least agreed that might be a good idea. The ENT used a pair a scissors and no anesthetic. Took a very short time, didn’t hurt, got a clean removal, and no further occurrence.

• Anonymous says:

Rahul said: “Luckily in many interventions by the nature of the system, to “do” implies sending to a specialist anyways (for most treatments likely to cause inadvertent damage in case of a false positive) so there’s a natural safety in the hierarchical system. “

My experience is that to “do’ often does not imply sending to a specialist right away, but sending to a specialist only after the PCP has tried something — e.g., the antibiotics-first case above, and another case that comes to mind: A PCP tried freezing a nasal growth, requiring several attempts and producing side effects. He said cutting it off would require a local anesthetic that would be very painful to administer. I suggested an ENT. He at least agreed that might be a good idea. The ENT used a pair a scissors and no anesthetic. Took a very short time, didn’t hurt, got a clean removal, and no further occurrence.

• Shelly says:

I am a doctor (but switched to public health). I don’t believe in blindly going with whatever a doctor says, no matter how much you trust them. There can be some healthy skepticism:) A part of it is we’re all human, and we all make mistakes. In medicine, there is a movement (been going on for some years now actually) to double check each other. For example, if a doctor does a procedure, the nurse will have a checklist to make sure the doctor follows all the rules (example, washing hands).

The motto of trusting your doctor and going with whatever he/she says goes back to an old paternalistic model of this doctor-patient relationship where only the doctor knows best. However, there’s a lot of information (both good and bad), but there are a lot of smart people too who do a good job of discerning what is good.I don’t understand why in the healthcare world, this attitude still exists today. A patient will also probably spend more time researching the area of interest, maybe even more than the doctor has time to read about. Clinicians whom I admire, if you bring a study or something up, will explain how that study fits or does not fit into your situation.

Generalists and specialists have very different trainings. Specialists are “better” in the sense that they have more experience in that one area. Ask the specialists something outside their area, and they’re no longer “better.” Generalists do see everything and in a way are the coordinators, triagers, and long-term mangers. Usually, the expectation of generalists before sending a patient to specialist is that a workup is done.Sometimes it is making sure to obtain the necessary diagnostic tests that confirms the diagnosis. Other times, it is like what Martha stated- trying a treatment first that has failed. Generalists handle chronic conditions (heart disease, diabetes). Now if the disease becomes severe or out of the generalist’s comfort range, then a cardiologist or an endocrinologist can be very useful.

Re: statistical training- In medical school, it was only 1-2 weeks of biostats/epi learning for me. Even during residency with our once a month journal club, we tried to look critically at papers but it was limited to 1)does the study show significant results (ie look at the p value or CI) 2) what are some the biases that the paper presents? I did feel like it was on a very superficial level.Some doctors obtain further training like an MPH to receive more biostats training. However, training doesn’t mean they’ll necessarily be good at stats. A part of it is having the frame of mind and curiosity to critically analyze research that may or may not be read by clinicians.

• Anonymous says:

I would second the thought that we shouldn’t blindly go along with whatever the doctor says. Unless you’re a specialist in the area, I think that it would be very hard to judge how good a physician is at that topic. Having someone who is approachable, compassionate, and understands their limits is important, and likely is what patients gravitate towards; however, it doesn’t mean that they are any good. I’ve seen some doctors who are brilliant but very inter-personally very rough and others who are loved and competent but not great. I wouldn’t recommend trusting either blindly.

I agree with K?’s recommendations.

The difficulty is in the conversation.
Lacking ability to accurately asses a clinicians competence, attribute substitution is employed, using his confidence.

Taleb:

THE EXPERT PROBLEM, OR THE TRAGEDY OF THE EMPTY SUIT
So far we have not questioned the authority of the professionals involved but rather their ability to gauge the boundaries of their own knowledge. Epistemic arrogance does not preclude skills. A plumber will almost al- ways know more about plumbing than a stubborn essayist and mathematical trader. A hernia surgeon will rarely know less about hernias than a belly dancer. But their probabilities, on the other hand, will be off—and, this is the disturbing point, you may know much more on that score than the expert. No matter what anyone tells you, it is a good idea to question the error rate of an expert’s procedure. Do not question his proce- dure, only his confidence. (As someone who was burned by the medical establishment, I learned to be cautious, and I urge everyone to be: if you walk into a doctor’s office with a symptom, do not listen to his odds of its not being cancer.)

I will separate the two cases as follows. The mild case: arrogance in the presence of (some) competence, and the severe case: arrogance mixed with incompetence (the empty suit). There are some professions in which you know more than the experts, who are, alas, people for whose opinions you are paying—instead of them paying you to listen to them. Which ones?

Yep. Which ones?

14. City Walker says:

Kevin Drum has been writing about how hard it is to get any useful prognostic information from his oncologist. I know form my teaching that the medical profession is not made up of folks who are necessarily all that great at statistical thinking.

For instance:

“So how long should I keep taking it before we start to think about alternative treatments? I tried once again to dredge an opinion out of my doctor, and as usual I failed. So I have no idea.” http://www.motherjones.com/kevin-drum/2015/12/health-update

“Do I sound a little annoyed at my inability to ever get anything but happy talk from these folks? Yeah, I guess so. I understand that there’s not much point in getting bent out of shape about these results until I’ve been on the new meds long enough to get a truly reliable reading. I also understand that oncologists want to keep their patients from getting depressed. Still, I wish I had a little more visibility about what’s likely to happen over the next year or so.” http://www.motherjones.com/kevin-drum/2015/11/health-update

• Kyle C says:

Genuinely asking, not trolling — I assume that if my doctor won’t say, “Most people your age live as long with this disease as they would have without it,” but also won’t offer a timeline, it’s safe to assume that (1) the raw odds favor premature death but (2) there is so much variance in duration that the doctor thinks the confidence interval would just scare and depress me (e.g., 6 months to 12 years). Am I wrong? I expect that, when Kevin Drum eventually gets his answer, that is roughly what he will tell us he learned.

15. Phil. I agree wholeheartedly, and honestly I think there was no excuse for the crap that medicine calls statistics even in 1996 much less 2006 or 2016.

Consider that in 1996, VISA had massive database driven fraud detection algorithms with billions of rows of relational database data. Hedge funds had computer systems that would trade hundreds of billions or trillions of dollars of assets per year automatically. Large shipping firms had computer systems that would track the locations of billions of individual shipping containers and optimize the use of shipping resources. The airline industry would track hundreds of thousands of passengers and thousands of flights per day. Orbitz was getting itself set up in 1999.

Sure, data collection was a lot easier for them, but I think the real issue is that those groups needed this stuff to make money, and doctors (or at least some influential doctor organizations etc) are afraid that good data-based decision making will destroy their special pedestal in society. There are many societal issues regarding patient confidentiality and etc, but those issues have been around since before computers existed. Certainly by 1996 we could have had some foundational rules for data sharing in place, and refined them since then.

The National Cancer Institute was established IN NINTEEN THIRTY SEVEN!!! for god’s sake.

What other “science” based endeavor X has a special field called “evidence based X” ?

• Anonymous says:

At this point, evidence based medicine is more like NHST based medicine though… this can be worse than tradition, intuition, and common sense due to the active generation and perpetuation of novel misinfo that follows from NHST. At least old witch doctor type misinfo has survived the filter of time and natural selection, so what remains probably isn’t very harmful.

• Rahul says:

@Daniel

I think the whole paradigm of medical care needs a rethink. e.g. Given the huge number of symptoms, syndromes & lab values out there it’s hard to imagine how any one doctor’s brain can do an optimal job. 30 years ago we didn’t have a better benchmark so obviously a good physician was the best we could do. But is that the best we can do today?

Then comes the combinatoric explosion of treatment choices. Add to this the fact that any one doctor can only see so many variations during his training and hence his “training set” itself is severely limited in making the right diagnosis for novel conditions.

I think computer assistance at recognizing the patterns of a disease and also while choosing the optimal treatment has to become essential. Treatments must be chosen on patient specifics searched against a huge database of past outcomes.

I think the Doctor still has a vital role here because to make such a system efficient we will still need to use technical jargon and depend on a professional for eliciting the symptomatic information and also stuff gleaned from a medical exam.

But yes, we must more to vastly more powerful uses of IT into medical care to improve outcomes any more.

• I think doctors, or at least advocacy organizations for doctors (AMA etc) actively foil this. There’s really no reason you shouldn’t dictate your symptoms into your smartphone and come back with a treatment plan or a few followup questions and a treatment plan, the first few steps of which would often be something like: take ibuprofen at this elevated dose for 5 days, rest, apply ice, and take prescription cyclobenzaprine (a muscle relaxant) each night before bed at the following dosage, please select a pharmacy closest to you for this Rx to go to. If symptoms resolve then do nothing, otherwise follow up with an in-person doctors appointment in 7 days…

with the whole thing being essentially automated. I don’t personally believe in Rx only drugs. I think there should be “over the counter” drugs and “behind the counter” drugs and that’s it. Possibly with limitations on the number of pills dispensed. But even if you just let automated machines phone in Rx drugs it’d be an advance over what we’ve got.

On the other hand, it’d make doctors lives easier but reduce their power and influence in society, so the AMA will fight it tooth and nail. In most states a nurse practitioner can’t even do that stuff without setting up some kind of formal oversight arrangement with a physician.

• Shravan says:

Even ignoring the data protection and data sharing issues, my experience in the US (Columbus, OH) and Germany (Berlin) has been that clinics and hospitals that belong to the same establishment are unable to maintain the data in electronic form. In the Charite hospital in Berlin, the transplant center is on the 2nd floor and the eye clinic is on the first, in the same building. The first floor folks can’t access the voluminous data on a patient from the second floor, I was told this was because they keep forgetting the password. Instead, every time I have to go to the eye clinic, they ask me to tell them their medical history from the beginning, and they start writing it down. In Columbus (1997-2002) it was the same story. I had a primary care physician in the Ohio State Uni medical system, who would order bloodwork, and the print-out would come to him, but there was no way for the transplant specialist in the same Ohio State system to access the data. Whenever I met the transplant specialist, either I had to have a copy of the results or we were starting with zero information and he had to order bloodwork. The transplant center tried for the entire five years I was at OSU to get my data from the other clinic, but no luck.

Regarding the point by Keith to just trust the doctor, that does not fit with my experience. Too often, doctors often have a vested interest. Try asking a sample of renal transplant surgeons if a transplant is better than dialysis and tabulate the results. When I was being briefed in Germany on the dangers and benefits of transplantation, the doctor painted a very rosy picture of transplantation. He even gave me a booklet which describes the advantages of transplantation in a most unrealistic way; it was a hard sell. At this point I had 25 years’ experience in being a kidney transplantee, and I knew that the lovely story they were telling was pure BS (and I was one of the successes) and was intended to get more people to be transplanted. I suspect that at least part of their extreme enthusiasm for transplantation is to allow the research hospital to have more survival data to publish their papers with and tout their success rates. I have no medical training but have 25 years’ experience of being transplanted; if I were a medical doctor giving advice on whether to get transplanted or not, I would be much more conservative about the benefits of transplantation, balancing them out with the very scary reality of actually being transplanted.

Although I have to admit that you can’t really blame the doctors for being so biased, and you can’t blame them for being uneducated about statistics. They too have to eat, and life is too short to hum and haw about the choices a patient has.

• Keith O'Rourke says:

Shravan: My point was ” find a physician you trust” not trust the physician you find!

• Shravan says:

Good point! I have to agree with that, but in practice this is an impossible task; one has to get involved, despite being an outsider and despite the danger of incomplete knowledge.

• Anonymous says:

CMS has been working hard (theoretically) on mandating better information sharing. There are actually quite severe penalties for hospitals and physicians who don’t use an electronic medical record (EMR) system for tracking their patients. But, these systems are quite poor in quality — by having to be used by so many different specialties they are not good for each one (and down-right terrible for ophthalmology). So its not uncommon for different clinics at the same institution have different systems. Even if they use the same system, one clinic might still use paper records that are then scanned into the EMR while another clinic uses customized / esoteric forms that another doctor wouldn’t necessarily understand.

Physicians and hospitals can’t afford to not use an EMR but they also cannot afford to build one or have one built. Instead, there are dozens of EMR providers — a few of the large ones are EPIC and Cerner. Although CMS mandates that they provide some interoperability, it is in software company’s best interest to provide only what is needed and no more. So I don’t expect that this will improve significantly. Instead there will be silos of data depending on the healthcare system’s medical record system provider.

• Martha says:

I have heard that since many doctors don’t like having to write up patient records on the computer, a para-profession has arisen of people who write up the record while the doctor interacts with the patient. I’m not sure of the name of such people, but “medical transcriber” came to mind. However, when I looked this up in Wikipedia, the term “medical transcriber” was used to refer to someone who transcribes from voice recording of a physician’s notes to a typed version, stored either electronically or printed. The Wikipedia article (https://en.wikipedia.org/wiki/Medical_transcription) points out various problems arising in this process. So it does appear in any event that there are problems in implementing the idea of keeping complete medical records electronically.

16. Thomas says:

I think HIPPA is the main reason no such database/resource exists. In my experience, it is not even feasible to get permission to share a small medical dataset to accompany a publication. Journals do not seem to push for accompanying data, likely because they know how much red-tape stands in the way.

It seems that in this country we all want information, but we are unwilling to share our own. Certainly, the current policies make it infeasible to share medical data publicly. Moreover, the physicians obviously benefit from this lack of publicly available information.

• So at this point, we need legal changes. I agree with that. I also think HIPPA wasn’t an out-of-the-blue thing, it was a response to a long history of problems with the way data was used, and other societal ills (such as health insurance laws and historical developments allowing for pre-existing-condition exclusions and the lack of a good insurance market) and fear of the unknown.

Some time between the national cancer institute being instituted in 1937 and HIPPA being passed in ~ 1997 could we have come up with useful guidelines and methods to share data while preserving an appropriate amount of individual privacy? I’d argue we could have, but there were political and power structures that gave negative incentives.

HIPAA isn’t the problem.
We get IRB approval, anonymize, or get exemptions.
It’s a PIA dealing with IRB (Institutional Review Blockheads), but it’s just untrue thats the a reason, let alone a main reason.

I have an “Ariely opt-out” consent and about 5% of the people sign it, which means that 95% allow permission to use their case for teaching and research.
So the changing the consent process will help.
Legislation that “good-samaritans” us along these lines would help as well.

I think the first big problem is “like me.”
There are lots of people “like me” that don’t get cancer, get indolent cancer, or get aggressive cancer and die.
So maybe they’re not “like me.”

It what Phil wants…well…not quite…Phil really wants to know his risk and what he should do…and we need you smart people to quit dinking around with Red State/Blue State, Attractive Parents have Daughters and whatnot and help us out. hehe…

17. Michael Lew says:

Great post! The topic is examined in the context of statistical foundations in the very impressive paper by Liu and Meng: There is Individualized Treatment. Why Not Individualized Inference? (http://arxiv.org/abs/1510.08539)

18. Moreno Klaus says:

No one here mentioned the role of simulation models. Given the limitations of clinical trials, it is definitely a way of synthesizing all available information and giving individualized estimates, though there are also drawbacks. There is an NCI funded programme where different simulation models per cancer site are compared together against observed data and used to make (individualized) predictions: http://cisnet.cancer.gov/about/

• Yes, I think this kind of thing is the first step. The next step is making these simulations heavily “biology aware”, that is, identifying mechanisms by which various cancers operate at the molecular level and adjusting the models based on molecular factors.

• Anoneuoid says:

You’ll find the information is just not there when coming up with such models. It will be things like getting an idea of how many cells in the tissue, what is the division rate, how many of molecule x per cell. Quickly the assumptions will build on one another until you are no longer confident in any conclusion due to nonlinearity and leave it for later.

This type of data is just not available, it is directly due to what you called above the ‘“model free” thinking … heavily influenced by the Null Hypothesis Testing paradigm.’ Getting parameter estimates is just not the primary concern of those collecting the data, which prevents them from ever actually testing the research hypothesis.

• Right, at least to an extent. But there is research along these lines being done by biologists (I know some of them). I’m not saying that we should already have molecular-based risk assessment tools, only that it should be one of the major focuses going forward, and that the appropriate data collection and analysis will also feed-back into informing candidate treatments and research.

• Anoneuoid says:

I am with you 100%, that is what needs to be done. I doubt it will go smoothly though. In many cases this will be the first real test of long and widely accepted theories. Even with legal cases, government recommendations, clinical trials, etc based on them. If/when it doesn’t turn out the “right” way there will be heavy pushback, or possibly worse, it will just be totally ignored and no one will give you useful feedback.

19. Tova Perlmutter says:

This sounds like precisely the kind of project Bill and Melinda Gates would be interested in. I wonder how you could get to them. Know any Nobel Laureates?

• Andrew says:

Tova:

Sure, I could ask Jeff Sachs (no Nobel, but I expect he knows Bill Gates). Or for that matter Angus Deaton. But it’s not clear to me why a charity would want to fund this project. From a statistical point of view it’s interesting, but ultimately the uncertainties will be large, and it’s not clear how much practical value is attained by telling someone their life expectancy is 3.4 years or whatever.

• Think of the project as much broader:

1) Collecting, organizing and creating a repository for useful medical data.

2) Building models of various kinds, not just risk models for life expectancy but also things like differential diagnosis of diseases.

3) Using the data for discovery of new candidate treatment methods, to inform and structure projects involving understanding mechanisms of disease…

I think just (1) would be a big deal honestly.

• Keith O'Rourke says:

> not clear how much practical value is attained by telling someone their life expectancy is 3.4 years or whatever
I avoided bringing that up (as it might seem a bit uncaring) but I don’t believe it is of much real value to an individual (benefits, harms and the uncertainties of treatments being more important.)

But for managing health care and funding health care researcher, it is very important – if those managing health care and funding health care research will utilize it. So I agree that it is not something for a charity to fund (at least other than a specific disease based charity).

• Anoneuoid says:

They are already on it:
https://www.tycho.pitt.edu/

20. Martha says:

Just came across this that illustrates the scope of what needs to be done:

In Section 9.2 of https://dl.dropboxusercontent.com/u/23421017/50YearsDataScience.pdf, David Donoho describes a “cross-study validation exercise” comparing 14 different methods for predicting ovarian cancer survival from gene expression data. He comments:

“The effort involved in conducting this study is breathtaking. The authors delved deeply into the details of over 100 scientic papers and understood fully how the data cleaning and data fitting was done in each case. All the underlying data was accessed and reprocessed into a new common curated format, and all the steps of the data fitting were reconstructed algorithmically, so they could be applied to other datasets.”

• Martha (Smith) says:

Thanks for the links. I found the second one particularly interesting (or perhaps just more readable for me?). Draper’s comments and Roberts’ response do seem together to cover important points. When I read Donoho’s paper, I initially was skeptical about his focus on prediction, especially since so many example where prediction is not very useful/meaningful came to my mind. But the example in section 9.2 did really seem like an important case where prediction is appropriate. Still, as Roberts points out, it doesn’t cover all types of problems; his 50-50 emphasis seems sound to me, since “statistics” applies to so many different types of situations.

21. Shelly says:

Hi folks- Have you read this piece from the NEJM on Data Sharing- http://www.nejm.org/doi/full/10.1056/NEJMe1516564? The author presents many reasons not to data share.

I plan on writing to the NEJM to express my disagreement with the editorial.

• Keith O'Rourke says:

Any selection based on observed results can result in problems which can’t be re-dressed without knowledge of the selection rules.

By the way, I am not aware of anyone writing about this problem in animal studies used to inform drug development before 1998 (seems almost everyone overlooked that.)

• Shelly says:

I wanted to mention Paul Kalanithi’s book-When Breath Becomes Air. I mentioned the author’s NYT article earlier in the comments, and I finally read his book. It is an incredible book. The author started writing his book after he was diagnosed with lung cancer. He never was able to finish it because he passed away March 2015. His wife helped to publish his book.