Retrospective clinical trials?

Kelvin Leshabari writes:

I am a young medical doctor in Africa who wondered if it is possible to have a retrospective designed randomised clinical trial and yet be sound valid in statistical sense.

This is because to the best of my knowledge, the assumptions underlying RCT methodology include that data is obtained in a prospective manner!

We are now about to design a new study in a clinical set up and during literature search we encountered few data published with such a design in mind. It has risen some confusion among the trialists whether we should include the findings and account for them in our study or whether the said findings are mere products of confusion in a mathematical/statistical sense!

My reply: You can have retrospective studies with good statistical properties—if you know the variables that predict the choice of treatment, then you can model the outcome conditional on these variables, and you should be ok. I don’t think it makes sense to speak of a retrospective RCT but you can analyze retrospective data as if they were collected prospectively, if you can condition on enough relevant variables. This is the sort of thing that Paul Rosenbaum and others have written about under the rubric of observational studies.

20 thoughts on “Retrospective clinical trials?

  1. I’m having a very hard time imagining how a retroactice RCT would look like. Can the OP or anyone else offer an example use-case?

    Is it like I conduct an RCT to see if drug-A cures acne. And later there’s speculation as to if it might have boosted mood. And luckily my questionaire captured happiness levels. So I now analyse that?

    • That sounds reasonable, though it will only give you information about interactions of drug-A:(group that was chosen to get drug A). so I think that’s what Andrew was talking about in terms of conditioning on the variables that predict the choice of treatment. In your example of acne:

      Age,
      Skin Condition,
      Appearance,
      Race,
      Geography,
      Cultural issues surrounding Acne,
      etc….

      So if you had a sufficiently broad acne study, across say 4 or 5 different countries, with different races, and different cultures, you might have a chance of finding out how drug-A affects mood in a broad population… but if it were all upper-class, white, middle-school students from California…. your ability to generalize mood effects will be very limited.

        • Well, not if it showed that in that population, acne was strongly eliminated. There are enough upper-middle-class white Californians who suffer from acne that solving their problem is a step forward.

          Put another way, instead of acne, suppose it were a heart-attack intervention and only performed on people with certain personality traits and certain high stress jobs, and 90% of those people turned out to be white males in finance, law enforcement, and the military. If you prevent 10% of that population from dying of heart attacks, is that a lousy study, or a great step forward for somepeople in high stress jobs?

          There are lots of reasons to limit your study to a target group, but the reasons when your RCT is designed around one particular issue, are different from what you’d want when you want to go back and look at an alternate auxiliary issue.

        • But what if they go back and see if the acne drug did elevate mood, ease acid reflux, cure headaches, reduce stress, ease constipation, etc.

          A retrospective RCT at some level of effort is bound to hit the jackpot?

  2. My guess is its a confusion of terminology. For instance in the well known example of latitude and tuberculosis or vaccine effectiveness 6 or 7 of the labelled RCT were actually observational studies.

    But if not, I would revisit the problem discuss here http://statmodeling.stat.columbia.edu/2014/02/09/keli-liu-xiao-li-meng-simpsons-paradox/ and especially so if all covariates determined after the exposures.

    ” Hence, the “fruitful” question to ask is not “Do we condition on the third variable,”
    but rather “Are two fruits which appear similar, actually similar at their core?”

  3. Below is the link to the hilarious large sample, retrospective RCT on intercessory prayer done in Israel and published in 2001 in the BMJ.

    http://www.ncbi.nlm.nih.gov/pmc/articles/PMC61047/

    “This was a retroactive intercessory study, done 4-10 years after the 3393 patients who suffered bloodstream infection concluded their stay in the hospital. A list of the first names of the patients in the intervention group was given to a person who said a short prayer for the well being and full recovery of the group as a whole. There was no sham intervention.”
    Leibovici states, “This intervention is cost effective, probably has no adverse effects.” Further, “The very design of the study assured perfect blinding to patients and medical staff of allocation of patients and even the existence of the trial.” However, “Regrettably, the very same design meant that was not possible to obtain the informed consent of the patients.” He justifies the retroactive aspect with “As we cannot assume a priori that time is linear, as we perceive it, or that God is limited by a linear time, as we are.”

  4. I am not sure what he means by “retrospective randomized trial” – it is obviously not possible to randomly assign treatment retrospectively. One possible interpretation is that he has data from an old trial, and wants to analyze the effect of the same exposure on a different outcome. From a causal inference point of view, this is completely valid – the trial still gives an unbiased estimate of the effect of exposure on the outcome. There is no need to condition on covariates: the problem is not confounding. However, from a statistical point of view, this approach is problematic because of multiple comparisons. He will have to think carefully about how the new hypothesis arose, and discuss multiple comparisons thoroughly in the paper.

  5. Hello, anyone still here? just wondering what do you think about this now? there’s more paper on this topic, and I am heading to similar direction.
    a bit concerned since the methods seems to considered as not feasible 9 years ago.
    There’s also some books about this.

Leave a Reply

Your email address will not be published. Required fields are marked *