Anna Ostropolets et al. write:
Observational studies can impact patient care but must be robust and reproducible. Nonreproducibility is primarily caused by unclear reporting of design choices and analytic procedures. . . .
Nine teams of highly qualified researchers reproduced a cohort from a study by Albogami et al. The teams were provided the clinical codes and access to the tools to create cohort definitions such that the only variable part was their logic choices.
What happened?
On average, the teams’ interpretations fully aligned with the master implementation in 4 out of 10 inclusion criteria with at least 4 deviations per team. Cohorts’ size varied from one-third of the master cohort size to 10 times the cohort size (2159–63 619 subjects compared to 6196 subjects). Median agreement was 9.4% (interquartile range 15.3–16.2%). The teams’ cohorts significantly differed from the master implementation by at least 2 baseline characteristics, and most of the teams differed by at least 5.
I’ll just add that you’ll often see forking paths in different analyses of the same sorts of data within a subfield, or even different analyses by the same researcher on the same topic. We’ve discussed many such examples over the years.
This is part of the reason I think having reproducible code (even in scenarios where the underlying data cannot be shared) is very critical. Verbally describing procedures is often very difficult to translate to code without having loss of information, let alone considering errors in the code and/or verbal description.
This is so jargony I can’t even quite parse it. What does “reproduced a cohort” even mean?
My guess, based on looking at the paper’s abstract and first section is that they’ve given multiple research groups an english language description of the groups that were used in a study, and then given them access to the same database, and ask them to choose the patients that should go into each group based on the english description of the criteria for inclusion… then compared the chosen patients across the different research groups.
Is that a fair characterization here?
Also, yeah, code is better than english language. But also if your results of your research depend strongly on relatively minor differences in interpretation like “recent user of X medication” meaning 1 month vs 6 weeks etc then probably you’re not doing it right. I think one of the issues is that in a Frequentist setting the preference seems to be to group people (or other units) together on some criteria and then look at observed statistics like mean, standard deviation, or percentage that have a certain quality. Even in non-bayesian regression analysis there’s a tendency to use binary dummy variables instead of much better specifications. For example suppose you’re looking at number of people who experience some complication through time, it’d be a common technique to break time into short, medium, and long term followup, maybe short is less than 1 month, medium is between 1 and 3 months and long term is 3 to 6 months… Then you’re looking at count of people vs a dummy variable for each time and some continuous predictors… when in fact you’d be much better off just using the data on individual events rather than regressing sums across groups vs dummy and continuous predictors.
I blame a lot of this on complete lack of knowledge about function approximation, aliasing, and other “signal processing” type knowledge. If you are categorizing things into bins then where you put the edge of the bin definition can have a big effect on the measurement in the bin, the discontinuous nature of the data masks a more continuous underlying rate. The amplification of this noise enables “statistical significance” but it’s just amplified noise.
Sure, there’s some things that really are discrete, like if you gave drug A vs drug B but in most cases if you’re working with time, dose, weight, size, etc and you’re breaking things into groups, you’re doing it wrong.
Some of these comments (not just Daniel’s here) seem to lack appreciation for the complexity of the situation when working with incomplete medical history data.
So here is an example, say I am interested in whether an initial Opioid RX following some procedure results in long term dependence. From a practical perspective, a DR. may be interested in whether dosage volume (eg stronger drugs) and length of dosage (eg 3 days vs 7 days of pills) results in higher likelihood of future dependence. You can define “dependence” different ways:
– from the first Rx, are they still getting Opioid Rx filled 3/6/9/12 months later?
– have they been *continuously* refilled an RX prescription over the same time period?
– does the average morphine milligram equivalent equal a particular value over those time periods? (Some people do on/off in cycles to prevent addiction.)
– the person has been officially diagnosed with a dependence (and has been say prescribed Buprenorphine)
This is maybe amenable to modelling a continuous process (I have not seen one, I would be interested if you have an example!) Or you may say all of these definitions are superficial (fair enough), but we typically don’t have a golden rule. I take the question from the Drs perspective to be quite reasonable — even if “dependence” may be better to describe as a continuous variable, how you define dependence matters for the subsequent analysis.
This is focused on the outcome, we have decisions we need to make about who to even include to study. What is the index Rx? People can cycle on/off different insurance policies or hospitals, censoring here is gnarly. If the first example Rx we have is Buprenorphine we can be pretty sure they are already addicted — not much point in seeing if they still have Buprenorphine Rx 6 months later is there? You may also say “if they are censored less than 6 months prior to index Rx, we do not include them in the cohort”, because we cannot be sure if they are actually already dependent before the first observed Rx.
This does not even get to questions about what you want to count as an opioid Rx. Do we want to count in-hospital fentanyl? Or only worry about after the person is out of the hospital and has more discretion how they take their drugs? Do we want to analyze each opioid Rx drug independently? Or say pool them into a continuous model based on MME — choices, choices.
A not unreasonable response may be “this data is not sufficient to answer that question”. (Lulz to the post not too long in the past here about massive claims data solving cancer!) I don’t see any obvious way to turn this problem into a purely continuous process though that entirely avoids decisions on who belongs in the cohort given realistic medical records data. You will have to make some decisions on what to model and who to look at outcomes for.
>This is maybe amenable to modelling a continuous process (I have not seen one, I would be interested if you have an example!)
I don’t have an example of anyone doing this, but I can certainly think of an example of “modeling this as a continuous process”.
Take some measure like “morphine milligram equivalent”, and fit a curve based on information about the Rx history, and measure “area under the curve” from the initial Rx to now, and predict that area under the curve as a function of the area under the curve windowed by a continuous window function for the first ~ 3 weeks (and other covariates).
I just don’t think most people doing these analyses think like that, and they don’t think like that because their background isn’t guiding them to think like that.
Daniel, you observe MME (assuming no censoring) — it is a step function based on what you are prescribed. How is the area under the not stochastic curve here different than just measuring average MME over a particular horizon? (Picking 3 weeks is making an ad-hoc “binning” decision as well :)
If your answer was “you need a better underlying latent variable model for dependence” I grok that (even if not feasible with the transactional medical record data we have). I don’t understand fitting a curve though here in your example.
This is even getting a bit off-topic, the post is about building the cohorts on who to study. For a relevant example of opioid, many studies that start from an index event will have medical exclusions (people with cancer) and exclusions based on censoring (prior history measured in the dataset). The index event itself is assuming a discrete event in time to subsequently measure some outcome.
I am open to continuous variable models (I have debated on doing something like https://brucehardie.com/notes/021/palive_for_BGNBD.pdf, that is not a time series, but based on the transaction history). It isn’t easy or obvious how to go from discrete medical transactions to continuous based models though.
You just said things like people going on and off to avoid addiction etc. If a person is getting the same Rx every month for 3 years, then yeah it’s a step function, but if they’re changing drugs, having intermittent Rx, or not taking the drug every day then it’s different.
Also I said a continuous windowing function with a 3 wk timescale. So no it wouldn’t be an arbitrary bin (ie. If they got a refill at 22 days that would be incorporated by the continuous tail declining smoothly to 0 on the window function). 3 weeks is chosen because most docs Rx on a few week horizon post surgery.
A window function would be something like a raised cosine curve or a bump function… https://en.m.wikipedia.org/wiki/Bump_function
https://en.m.wikipedia.org/wiki/Hann_function
The dosage curve is not observed but rather inferred by the Rx refill dates
I’d look at the individual cumulative (prescribed) dose curves, then try to model the process that generated those.
I’d imagine some clustering would be involved. Eg, the people who cycle on/off will have temporary plateaus. Some people are getting what appears to be way too much, suggesting they are selling part or all of their prescription. And so on. Some will drop from the records for whatever reason after a few weeks, fine that is just another outcome.
If the goal is to guess what will happen for a given patient then that is a machine learning problem. Train using standard methods on what you have then plug in the patients data in to get a prediction for their curve.*
Either way, I don’t really understand the purpose of defining these cohorts. It seems like the plan is to describe some average patient that doesn’t exist.
* Its quite possible the data is crappy so your model will have poor performance. The solution is to look at your generative model to see what data needs to be collected in the future. If getting it is impractical, then that is just how it is. Get out your magic eight balls because its better to be up front about your arbitrary decision than to pretend it is rational.
btw: continuous interpolation of cumulative prescribed dose and area under the curve of the daily consumption (which i advocated above) are close to the same idea.
I noticed that too when I read your post.
My principle is to process the data as little as possible, then model that. The more you process/transform the data the more assumptions get involved. Both yours and whoever devised the measuring device, analytical method, etc. Especially the implicit ones that can take forever to track down.
This is why when I come across something new, I like going all the way back to the first person to use a method. Usually they do point out at least some of the issues and other interpretations. Then you can see how each possibility was examined historically.
They go on to list other variations implemented based on the same prose. Yea, others being able to repeat the same thing is a bare minimum requirement. But why is this so seemingly difficult? Just put the search query and software used in the paper.
I think they’ve lost sight of the purpose* of these studies though. It is like a cell culture experiment that only works if you use the medium produced in Vat 4 at Facility F of Company C. (Maybe that vat has a rust spot that affects iron levels just right, who knows but it works.)
If that kind of thing matters so much, whatever you are studying probably isn’t going to generalize to your problem of interest. Actually, in that case you’ve probably observed the (cell medium) effect multiple times, while it is exceedingly unlikely anyone is going to repeat your exact observational analysis in 10 years to see if it is stable.
* The purpose is to distill some pattern out of the data that can be used to predict the future under various scenarios.
How about just using “date of diagnosis with T2D” and “date of diagnosis with CLRD” and “time series of dispensing of drug” and soforth.
Why do we take continuous observations and group them into groups? It’s just so people can calculate averages over groups instead of doing the harder work of fitting continuous functions through time. Because people don’t understand function approximation basically. How many medical researchers understand what a spline or a orthogonal polynomial basis or a radial basis function are? It’s like zero to first order.
This “grouping behaviour” is because the researchers are trying to look for (statistically significant) differences rather than find general laws we can use to predict the future.
In cancer research the standard is aggregating into 10 year age groups along with 85+. This has created a huge obstacle to understanding cancer because it obscures the interesting peak in incidence that happens ~50-80 years old for various cancers.
Such a peak is exactly whats predicted by the armitage-doll model (some kind of “errors” accumulate into a cell over time). But also, it is totally plausible that diagnosis rates play a role. Ie, a doctor wont bother with a tumor in the elderly because they figure heart disease or whatever will be an issue first.
Then regarding diagnosis rates, the easiest to diagnose, and by far most common, cancers are excluded from all the cancer registries (basal and squamous cell carcinomas). So the least confounded data is now scattered around in hundred/thousands of different databases.
This issue has been known since literally 1954 and not yet resolved due to these choices. It is once again another way that NHST is an active obstacle to progress.
I see grouping/binning as not really related to NHST. People often innocently group things because it is easier to envision and communicate – and they don’t realize that they are destroying (or at least undermining) the value of the granular data. It is also a long-standing (and very slow to change) habit resulting from the era where software was generally not available to do any statistical analysis. Sometimes it is not innocent and the groups permit finding a “significant” difference (there is the link to NHST) between groups rather than perhaps not finding it for a marginal change in a continuous variable. In my view, the worst offender is when age is grouped. Most studies do collect age on a continuous basis but then quickly group it into bins – and the bins are arbitrarily chosen to be in 5 or 10 year groups rather than looking at the data to inform the bins. Even worse, sometimes age is 50-65 and 65+ as I’ve seen in some cancer studies (where the patient’s age is actually known).
Dale, I think grouping is largely due to most people operationalizing “statistics” as “finding statistically significant differences between groups”. Like, they think that’s basically what it is, running tests to see if groups have different means, or different risks, or different standard deviations or different histograms (chi-squared tests etc).
If you think of statistics as “the process of finding ‘true’ differences between groups” then you just won’t be able to conceive of doing statistics without grouping… and that’s the level of understanding that dominates out there, particularly in medical studies done by non statisticians.
I fully agree with you about the evils of grouping ages into bins, but also failure to incorporate dimensionless ratios, failure to estimate continuous curves related to dosing or related to duration of illness or exposure to risk (for example an STD study should consider frequency and diversity of sexual partners at an individual level, a study on malaria should consider duration of exposure to mosquitos etc.)
Easier to communicate what though? It is whether rates are higher/lower in some age groups than others, which is NHST.
I also have to say, at least this binned data lets us see a time series. I have no idea what the point is of those age-standardized rates. I have never once had a use for such a number.
The various comments above show that there are many ways that an experimental design can throw a monkey wrench into the experiment.
Serendipitously, I recently came across the book Measuring and Reasoning: Numerical Inference in the Sciences, by Fred L. Bookstein. It attracted me for two reasons. First, it seemed to address an important but often neglected point in statistical analysis; and second, I remembered Fred as an extremely bright person in my high school and college math classes.
The various comments above show that there are many ways that an experimental design can throw a monkey wrench into the experiment.
Serendipitously, I recently came across the book Measuring and Reasoning: Numerical Inference in the Sciences, by Fred L. Bookstein. It attracted me for two reasons. First, it seemed to address an important but often neglected point in statistical analysis; and second, I remembered Fred as an extremely bright person in my high school and college math classes.
Maybe this is a little known factoid. OHDSI grew out of the Observational Medical Outcomes Partnership (OMOP) that was started around 2005 when I was working at PhRMA. When one of my colleagues left, I became the project manager and got the $20M initial seed funding from PhRMA companies. PhRMA companies then funded it for a second year.
Patrick Ryan, the senior author of the paper mentioned and an adjunct professor at Columbia, was on the design team back then. David Madigan, then in the Statistics Department at Columbia was involved when the project began to take shape. We did a lot of good work and transferred it over to the Reagan-Udall Institute for the FDA but for what ever reason FDA dropped their interest and the those involved spun it off into OHDSI in 2014. I still look at this project as one of the highlights of my career in the pharma industry.
Oh interesting history there. OHDSI has produced a good amount of seemingly (to me, not in medicine) quite useful work.