Kelvin Leshabari writes:
I am a young medical doctor in Africa who wondered if it is possible to have a retrospective designed randomised clinical trial and yet be sound valid in statistical sense.
This is because to the best of my knowledge, the assumptions underlying RCT methodology include that data is obtained in a prospective manner!
We are now about to design a new study in a clinical set up and during literature search we encountered few data published with such a design in mind. It has risen some confusion among the trialists whether we should include the findings and account for them in our study or whether the said findings are mere products of confusion in a mathematical/statistical sense!
My reply: You can have retrospective studies with good statistical properties—if you know the variables that predict the choice of treatment, then you can model the outcome conditional on these variables, and you should be ok. I don’t think it makes sense to speak of a retrospective RCT but you can analyze retrospective data as if they were collected prospectively, if you can condition on enough relevant variables. This is the sort of thing that Paul Rosenbaum and others have written about under the rubric of observational studies.
I’ll just leave this here …
I have a hard time telling… is that paper serious?
I’m pretty sure the prayer paper is intended to point out the goofiness of typical practices in medical studies by doing a study which couldn’t *possibly* have had an effect but using all the techniques that are used by typical studies where an effect is claimed.
It’s notoriously hard to communicate irony reliably.
I believe that it is in the same category as the dead fish MRI work: http://boingboing.net/2012/10/02/what-a-dead-fish-can-teach-you.html
I consider these kinds of papers to be of the same ilk of methods papers like this one, which “evaluates econometric evaluations”: http://www.personal.ceu.hu/staff/Gabor_Kezdi/Program-Evaluation/LaLonde-1986.pdf
Only obviously a dead fish and retrospective prayer bring an element of theatrical hilarity to the whole thing. LaLonde’s paper is not particularly funny (but very good, and there are follow-ups which are also good).
Also, because it was posted again, I will again re-post my favorite line: “As we cannot assume a priori that time is linear, as we perceive it, or that God is limited by a linear time, as we are, the intervention was carried out 4-10 years after the patients’ infection and hospitalization.”
That you are having a hard time reflects the beauty of that paper…..
It was tongue-in-cheek:
http://en.wikipedia.org/wiki/Studies_on_intercessory_prayer#Retroactive_intercessory_prayer
I once actually had an undergraduate cite that paper in a research proposal class project. Not in a lighthearted way, as part of a literature review. It’s not really well enough written to make its point and it doesn’t think about who the potential audiences are.
I think actually it’s beautifully written, but its audience is basically the same audience as this blog, yet the message needs to be sent to a much wider audience.
That’s fair.
I’m having a very hard time imagining how a retroactice RCT would look like. Can the OP or anyone else offer an example use-case?
Is it like I conduct an RCT to see if drug-A cures acne. And later there’s speculation as to if it might have boosted mood. And luckily my questionaire captured happiness levels. So I now analyse that?
The randomised bit is troubling, other than that I think it’s just a standard observational study.
That sounds reasonable, though it will only give you information about interactions of drug-A:(group that was chosen to get drug A). so I think that’s what Andrew was talking about in terms of conditioning on the variables that predict the choice of treatment. In your example of acne:
Age,
Skin Condition,
Appearance,
Race,
Geography,
Cultural issues surrounding Acne,
etc….
So if you had a sufficiently broad acne study, across say 4 or 5 different countries, with different races, and different cultures, you might have a chance of finding out how drug-A affects mood in a broad population… but if it were all upper-class, white, middle-school students from California…. your ability to generalize mood effects will be very limited.
If they were all upper-class, white, middle-school students from California the original acne study would have been pretty bad as it is, right?
Well, not if it showed that in that population, acne was strongly eliminated. There are enough upper-middle-class white Californians who suffer from acne that solving their problem is a step forward.
Put another way, instead of acne, suppose it were a heart-attack intervention and only performed on people with certain personality traits and certain high stress jobs, and 90% of those people turned out to be white males in finance, law enforcement, and the military. If you prevent 10% of that population from dying of heart attacks, is that a lousy study, or a great step forward for somepeople in high stress jobs?
There are lots of reasons to limit your study to a target group, but the reasons when your RCT is designed around one particular issue, are different from what you’d want when you want to go back and look at an alternate auxiliary issue.
But what if they go back and see if the acne drug did elevate mood, ease acid reflux, cure headaches, reduce stress, ease constipation, etc.
A retrospective RCT at some level of effort is bound to hit the jackpot?
My guess is its a confusion of terminology. For instance in the well known example of latitude and tuberculosis or vaccine effectiveness 6 or 7 of the labelled RCT were actually observational studies.
But if not, I would revisit the problem discuss here http://statmodeling.stat.columbia.edu/2014/02/09/keli-liu-xiao-li-meng-simpsons-paradox/ and especially so if all covariates determined after the exposures.
” Hence, the “fruitful” question to ask is not “Do we condition on the third variable,”
but rather “Are two fruits which appear similar, actually similar at their core?”
Below is the link to the hilarious large sample, retrospective RCT on intercessory prayer done in Israel and published in 2001 in the BMJ.
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC61047/
“This was a retroactive intercessory study, done 4-10 years after the 3393 patients who suffered bloodstream infection concluded their stay in the hospital. A list of the first names of the patients in the intervention group was given to a person who said a short prayer for the well being and full recovery of the group as a whole. There was no sham intervention.”
Leibovici states, “This intervention is cost effective, probably has no adverse effects.” Further, “The very design of the study assured perfect blinding to patients and medical staff of allocation of patients and even the existence of the trial.” However, “Regrettably, the very same design meant that was not possible to obtain the informed consent of the patients.” He justifies the retroactive aspect with “As we cannot assume a priori that time is linear, as we perceive it, or that God is limited by a linear time, as we are.”
I love this study, because by all appearances the experimental methodology is completely legitimate.
I am not sure what he means by “retrospective randomized trial” – it is obviously not possible to randomly assign treatment retrospectively. One possible interpretation is that he has data from an old trial, and wants to analyze the effect of the same exposure on a different outcome. From a causal inference point of view, this is completely valid – the trial still gives an unbiased estimate of the effect of exposure on the outcome. There is no need to condition on covariates: the problem is not confounding. However, from a statistical point of view, this approach is problematic because of multiple comparisons. He will have to think carefully about how the new hypothesis arose, and discuss multiple comparisons thoroughly in the paper.
Hello, anyone still here? just wondering what do you think about this now? there’s more paper on this topic, and I am heading to similar direction.
a bit concerned since the methods seems to considered as not feasible 9 years ago.
There’s also some books about this.