The Journal of the American Medical Association recently contacted me to ask for a comment on a recent Draft Guidance for Industry from the U.S. Food and Drug Administration on the use of Bayesian methodology in clinical trials of drug and biological products.
I’ve recently been doing a lot of work with Erik van Zwet and Witold Więcek–we meet every week and we’re writing a bunch of papers related to statistical inference and decision making, especially for clinical trials (our two completed papers are Meta-analysis with a single study and A statistical case for qualified scientific optimism) so I invited them to join in writing this. Erik is a biostatistician who works with clinical researchers all the time, and Witold’s involved in policy analysis, also he posted on the FDA draft guidance back when it appeared in January.
What with all the bad things going down at the U.S. Department of Health and Human Services, I was ready to get riled up–also there was this problematic document from HHS back in 2015–but actually this new Draft Guidance for Industry was just fine, very serious and professional. I guess the clowns at HHS were too busy doing pushups or whatever to interfere with this one.
So, Witold, Erik, and I wrote this short article for JAMA. The review process was smooth. The journal was on a tight schedule so we submitted our first version right away, while simultaneously sending it to several colleagues who made various useful suggestions, which we implemented for our revision. Our first version was too technical for JAMA–maybe we’ll put it into another article–so we were able to cut to the required length.
I’m happy with our final version, and here it is:
Some of our key points:
The prior is sometimes described as a subjective belief, but the FDA guidance frames it as pre-study information, which we think is a good framing. The formal and transparent incorporation of prior knowledge into the design and analysis of trials is the main advantage of the Bayesian approach. The use of prior information, even if not in a formal Bayesian framework, is not new in the context of clinical trials. For example, it is used during the planning stage of a trial using traditional frequentist methods to determine the sample size. . . .
The guidance still insists on type I error control when Bayesian methods are used, but only when the prior is noninformative. We see the logic to this: Bayesian analysis with noninformative priors is functionally similar to a traditional frequentist analysis, and in these cases there is no compelling reason to break with the current paradigm. In contrast, when substantial prior information is available, the required assumption for defining type I error of no treatment effect may already be inconsistent with that prior information to some extent, invalidating the type I error rate as a meaningful design metric. . . .
We suggest two simple rules for Bayesian analysis of clinical trials: (1) the prior should be clearly stated and (2) readers and reviewers should be able to assess the prior’s influence on the result. . . . The guidance is right to stress the importance of justifying the prior. However, we also remind readers that it is just as critical to have a prespecified and valid data model. Both the prior and the data model should be scrutinized when the data arrive, and it may be necessary to revise either. We also applaud the recommendation that computations be evaluated using simulation studies: in the Bayesian framework, these would be posterior predictive checks, in silico replicated datasets simulated from the fitted model that are then compared with the observed data, with systematic discrepancies indicating problems with the model. . . .
Although often criticized as subjective, Bayesian approaches, when implemented transparently, can improve on informal principles of clinical judgment that often inform the current FDA model.
Two other discussions
Along with our article, JAMA ran two other comments on the FDA guidance:
– Embracing Bayesian Methods in Clinical Trials: FDA’s Long-Awaited Draft Guidance, by
Jack Lee, Frank Harrell, Lisa LaVange, and David Spiegelhalter.
– Reflections on FDA Draft Guidance on Bayesian Methods in Trials—Protecting Scientific Integrity and Evidentiary Standards, by Scott Evans, Thomas Fleming, Holly Janes, and Lori Dodd.
Both groups of authors have lots of experience on the statistical analysis of clinical trials, and both articles are thoughtful. I recommend reading them both.
Comments on the pro-Bayesian discussion by Lee et al.
Lee at al. share our general perspective, and their comment is usefully complementary to ours. We focus on priors, hierarchical modeling, and meta-analysis, while they go into more detail on the way that Bayesian methods connect to classical methods and existing regulatory approaches.
I only have three nits to pick on their article. First, they refer to “skeptical, optimistic, or noninformative prior distributions.” I’d like to clarify that there are too sorts of “skeptical prior.” One version of a skeptical prior is centered at a negative value, i.e., you assume the prior model in which the effect is more likely to be negative than positive, so that to get a positive estimated effect you need strong positive information from the new trial. The other version is centered at zero, which corresponds to a world in which most effects are small (as in the priors on the signal-to-noise ratio discussed here). By default with clinical trials we recommend that second sort of skeptical prior.
Second, Lee et al. write, “By contrast [to classical p-values and confidence intervals], Bayes tackles the real question of interest head-on and provides a direct answer to the question ‘Does the new drug or biologic work?’ by computing the probability of treatment benefit.” I get what they’re saying, but I don’t like the binary framing of whether something works or not. Effects will vary by person, across situations, and over time; there’s not really a stable effect or average effect, and even if we’re summarizing by an average, the question is not whether this average is different from zero or even whether it’s positive, but how large it is. Sure, you can take your Bayesian inference distribution and summarize it by the posterior probability that the average treatment effect is positive, but that’s just one thing to look at, and I think it’s a mistake to identify this as “the real question of interest.”
Third, for reasons I’ve discussed many times, I’m not happy with their use of the term, “prior belief”–unless they’re also willing to refer to other aspects of statistical models as “belief.”
But these are all just minor issues of emphasis. In practice I expect that Lee et al. are recommending the same sorts of Bayesian methods for design and analysis of clinical trials as we are.
Comments on the Bayes-wary discussion by Evans et al.
The article by Evans et al. expresses much more skeptical about the use of Bayesian methods for clinical trials. As noted above, I find their article to be thoughtful and it is worth reading. But they do say some things I disagree with.
The good news is that I think that some of their concerns can be directly addressed, and I hope that after reading my comments they will revise their view somewhat.
You might say that Evans et al. have a strong prior against Bayesian methods, and in this comment I’m sharing information that should shift their prior a bit. As a Bayesian, I can hardly complain that they have strong priors, and it’s my duty to provide arguments that they will view as convincing evidence.
Evans et al. conveniently provide a list of key points:
I’ll go through their five points in turn.
Item 1: I agree 100%. “Replicability, integrity, and reliability of evidence for disease treatment and prevention.”
Item 2: Again, I agree completely in this list of settings where Bayesian methods have been useful.
Item 4: Again, I agree! Transparency about all aspects of design, data collection, and modeling is absolutely necessary. This is the case for non-Bayesian method and for Bayesian approaches as well. I also agree that Bayesian inferences based on informative priors should be accompanied by results with noninformative priors or non-Bayesian estimates. It’s important to see the effect of the prior.
Item 5: Again, complete agreement. FDA guidance is crucial.
So it all comes down to item 3.
Item 3: They write that Bayesian methods “can compromise evidentiary and integrity standards.” I agree: any statistical method can be used inappropriately, and we should be aware of failure modes, especially in a high-stakes situation such as clinical trials. They list three concerns: “concession of the benefits of randomization,” “loss of objectivity by incorporating sponsor- or investigator-specific priors,” and “reduced robustness via reliance on strong and sometimes unverifiable assumptions.” I think they’re just wrong on the first of these items. We discuss the issue more fully in Bayesian Data Analysis (see chapter 8 of the third edition), but, just quickly, Bayesian analysis of clinical trials makes strong use of the benefits of randomization, as this is what allows you to set up a likelihood function corresponding to an unbiased estimate. I don’t see any concession of the benefits of randomization, in design or analysis. For their second item, yeah, if you put in a bad prior you can get a bad posterior, and that’s one reason we emphasize transparency in any statistical analysis. See the last paragraph of our article above. Finally, yes, I agree that Bayesian inference has additional assumptions beyond a classical analysis. I don’t think this “compromises evidentiary standards,” though! The prior is based on evidence too.
Let me put it another way. Sometimes you’ll have a nice clean trial, no problems with recruitment, dropout, or missing data, precise and well-targeted measurements, and a large enough sample size to get precise inference for endpoints of interest. You can do a Bayesian analysis if you want, but it won’t make much difference, and the classical confidence interval should be just fine. In other settings, your data are noisy, there are various sources of bias, you’re interested in small subgroups, and classical inferences aren’t enough. You’re in the position of making decisions based on incomplete information. And here the Bayesian approach can be helpful. Yes, additional assumptions are required–I’d like to frame that as, “additional information can be added to the analysis”–; that’s the price you pay for adjusting for bias and noise.
All this explains why I agree with Evans et al. that it’s good to compare any Bayesian inference to the (potentially biased and noisy) classical estimate, and it’s important to be transparent about all assumptions–including the past data and theory used to construct the prior. Data models can be constructed badly (for example, adjusting for age or smoking using binary indicator variables, or using curves that don’t fit the data, or not accounting for hierarchical structure), and priors can be constructed badly too (for example, by centering your prior from a noisy past estimate). Transparency doesn’t eliminate these problems but it should help us notice and correct them when they happen.
Summary
General caution by skeptics of Bayesian methods is valid and should be applied more generally, to all of our statistical models.
A useful analogy is to regression adjustment for experiments and observational studies. If your experiment is clean and your noise level is low, you might be able to get away with a simple treatment vs. control comparison with no adjustment. But in real-world studies there are a lot of reasons to adjust for pre-treatment predictors–it can help even in the simplest experimental settings. That said, it’s good to compare adjusted to unadjusted results and to understand your adjustment procedures as they get more complicated.
The reason for Bayesian methods in clinical trials is that classical methods are often not enough. It’s good to see these new FDA guidelines, and I think JAMA got some good discussions that should help move the ball forward.
P.S. There’s this funny thing . . . the JAMA style required us all to write “bayesian” as lower case. I told them this was nonstandard but they insisted on it. Weird, huh?
P.P.S. More here from Frank Harrell.
I discovered Bayes rule via MCMC because I wanted to fit my own model rather than some generic regression or default null model. The priors are more like fine tuning knobs.
Relatively easily fitting the actual theory seems like the main advantage to me. I’d guess the FDA is going to do NHST with Bayes factors instead of p-values, and the fancier math will be used to excuse the lack of replications and control groups.
Anon:
Click through the links and read the document. The FDA is not proposing to do null hypothesis significance testing. As we write, “The guidance still insists on type I error control when Bayesian methods are used, but only when the prior is noninformative.”
I skimmed this and it still sounded like looking for “treatment effects”, ie same old inverted science story:
https://www.fda.gov/regulatory-information/search-fda-guidance-documents/use-bayesian-methodology-clinical-trials-drug-and-biological-products
Instead we want some model of how the treatment works. When we can successfully predict the outcome of the trials, THEN we become confident.
Maybe I missed where that was included. I mean the document was generic enough to cover both uses (but, if allowed, people will do NHST because its way faster/easier and that’s what they are used to).
Andrew says,
The typographical standard in the US is to lowercase common adjectives and nouns. Here’s a list of examples: https://atkinsbookshelf.wordpress.com/tag/what-is-term-when-a-brand-name-becomes-a-common-noun/. Almost all of these are now conventionally written lowercase (e.g., heroin, escalator, dumpster, jacuzzi, laundromat, etc.)
I find it strange that Andrew objects to quirky capitalization as he uses it all the time. He writes acronyms British style, e.g., “Nuts” when they can be pronounced rather than “NUTS”. For example, the Guardian newspaper’s style guide says
But he writes most things American style, like “color” instead of “colour”.
He also downcasts all camel-case names, styling the package named “ArviZ” as “Arviz” and “NeurIPS” as “Neurips”, for example (I don’t know if Andrew’s used either of those two terms, they wee just examples that popped into my head).
So it’s weird he wrote “JAMA” rather than “Jama” :-).
Bob:
I’m not talking about any general principles here. I’m just talking about common use, which you as a linguist should respect! As you say, some originally capitalized words such as dumpster, jacuzzi, etc., are typically written in lower case. Other words such as Orwellian and Kafkaesque are typically written in upper case. In statistics, the terms Laplacian, Hamiltonian, Gaussian, and Bayesian all are used in upper case. Except not for JAMA!
I’m not saying JAMA is “wrong” here–it’s their journal and they can do what they want. It’s just nonstandard to write “bayesian,” in the same way that it would be nonstandard to write “gaussian” or “orwellian.” The fact that people also write “dumpster” doesn’t change the fact that “bayesian,” “gaussian,” and “orwellian” are nonstandard.
basicaly this CASE shit doesn’t matter.
emphasize whatever you want to emphasize.
I see the humor in the back and forth script …. but it’s very narrow
> heroin, escalator, dumpster, jacuzzi, laundromat, etc.
When a brand name becomes a common noun seems unrelated to the word Bayesian. More relevant nouns would be ‘diesel’ and ‘zepelin’ and there are also adjectives like ‘braille’ or ‘abelian’ but I agree with Andrew that Bayesian is usually capitalized like Freudian, Victorian and some many other adjectives.
Carlos:
That’s funny about “abelian.” It’s unusual; in math I think the capitalization is usually kept: Euclidean geometry, the Gaussian distribution, the Laplace approximation, Hamiltonian dynamics, Newtonian physics, non-Markovian, Gibbs distribution, etc.
In may be related to the influence of French mathematicians in modern algebra. In French you also have “l’operateur laplacien”, “la distribution gaussienne”, etc.
Hi all. JAMA uses the AMA Manual of Style. An older version of the styleguide can be found here:
https://www.utica.edu/academic/library/JAMA%20Network%20-%20AMA%20Manual%20of%20Style_%20A%20Guide%20for%20Authors%20and%20Editors-OUP%20USA%20(2020).pdf
A quick search for “Bayes” shows that the styleguide does indeed use “bayesian” for something like “bayesian analysis”, but curiously, also uses “Bayes information criterion.”
Why is this?
On page 499 of the styleguide, it’s noted that for eponyms, the proper name should be capitalized but the common noun that follows should not be. For example: Down syndrome and Breslow thickness.
However, words derived from proper nouns are not capitalized. For example: mendelian, darwinian, petri dish, arabic numerals. In general, the style manual recommends following the current edition of Merriam-Webster’s Collegiate Dictionary.
Well, wait a minute, what does Merriam-Webster use? https://www.merriam-webster.com/dictionary/Bayesian A capital letter for Bayesian! Ok, so there is some inconsistency here.
I’m wondering if the specific copy editor in Andrew’s case just referenced the style guide itself, and since there was an explicit recommendation to use “bayesian”, they went with it.
This is all pretty encouraging. I’m still always baffled at the sheer amount of attention the prior model gets in these types of discussion. If one was really out to cheat, the prior model wouldn’t be the most effective place to tinker. The Evans et al. key point three is amusing – “compromised integrity” and “unverifiable assumptions” are hardly the product of Bayesian methods but rather the investigators themselves who could just as easily be Frequentists. I’m reminded of this lovely IJ Good quip, “Cookbook statisticians, taught by non-Bayesians, sometimes give the impression to their students that cookbooks are enough for all practical purposes. Anyone who has been concerned with complex data analysis knows that they are wrong: that subjective judgments of probabilities cannot usually be avoided, even if this judgment can later be used for constructing apparently non-Bayesian procedures in the approved sweeping-under-the-carpet manner.”
Its because they are putting a prior on the “treatment effect” itself, rather than on parameters of their mechanistic model (eg, cell division rate, number of cells, etc).
That’s why I say this looks like its still just NHST. The treatment effect should be something you predict (eg, 5 year survival is increased because the drug slows rate of cancer cell division) not a parameter itself.
The problem with the NHST approach is that messing up the study looks the same as a treatment effect. So all the same bad practices are still incentivized.
Another way of putting this:
In science, you test the prediction of *your theory* against observation. This is fail-safe because messing something up is unlikely to closely match your predictions.
NHST is the opposite, messing up the experiment is extremely likely to introduce some kind of difference between groups. This is fail-deadly.
The FDA should use the fail-safe one.
I partially agree with you on this.
If you put a prior on the treatment effect, and the prior makes a big difference — e.g. if you are looking at five-year survival rate for a cancer treatment and you get a very different answer for a flat prior than for a truncated normal with mean 0.5 and s.d. 0.3 — then you know your study doesn’t give you enough data to quantify the effect very well…and that you probably shouldn’t trust the central estimate that you get from your study.
This is not a good situation to be in, but I would still prefer this approach, because if you want to estimate the effect and its uncertainties you have to get the numbers from somewhere, and where else?
I agree if you have a mechanistic model you should put priors on the physical/chemical/biological quantities in the model.
Andrew can tell us about this himself, but I remember that he worked with Frederic Bois on putting prior distributions on parameters in pharmacokinetic models. (Here, for example https://www.tandfonline.com/doi/abs/10.1080/01621459.1996.10476708 ). When Andrew first described this work to me, way back in the mid-90s, he said that the models that were in use at the time often had parameters describing the body and its processes, and that the model parameters would be chosen to best fit experimental observations (e.g. the concentration of a drug in the blood decreased at such-and-such rate, the concentration of a by-product in the breath had such-and-such temporal pattern, and so on)…and that the best-fit parameters often, or at least sometimes, were nonsense if interpreted literally. I still recall the hypothetical example he used: a model might include the fact that the liver can process a chemical at such-and-such rate per gram of liver per hour, and then when you fit the data you find that the best fit is that the person has a 10 Kg liver. So Bois (with Andrew’s help) was developing models that included prior distributions on all or most of the physical and chemical parameters, just as you are suggesting. It’s the right thing to do, for sure.
But I think that very often there is no mechanistic model, or at least not one that is remotely sufficient to predict how a drug or treatment will behave.
I am all for using Bayesian analyses in clinical trials, but I do not understand why a regulator would allow the drug company to choose the prior and other essential aspects of the design. People are phrasing this as a problem of subjectivity, when I think it is better understood as a conflict of interest. If the regulator is making the decision on whether to approve the drug based on a Bayesian analysis, then the staff of the regulatory body should be more than capable of telling the drug company what prior distribution to use if they want their application to be taken seriously.
Ben:
I think it’s ok to allow the drug company to choose the prior, if that choice is justified. Drug companies get to choose the data model, right? And drug companies get to choose the hypothesized effect size for the power analysis, right?
If they get unlimited choice and no vetting over the justification, then, sure, that’s a conflict of interest. I guess I agree with the point in your last sentence, as long as this is applied to other aspects of the analysis, not just the prior.
Ben is pointing to a broader issue in his “other essential aspects of the design”: a conflict arises whenever you allow the institution that is supposed to be regulated to create and operate the entire infrastructure, which will then be used in making regulatory calls. In the U.S., people seem to be fine with it, as evidenced by the Bank — SEC and Drug Sponsor — FDA setups. Ideally, we would have third parties (not necessarily the government) designing and running trials, but I doubt it will happen in the US.
The drug companies already choose the primary and secondary endpoints (measurable outcomes) and the thresholds for success and all the other aspects of the design of a trial. They already discuss with the regulators if it seems acceptable.
An excellent and thoughtful response Andrew. Could I suggest an additional point? In Evans at al’s second part of their 3rd bullet I see “loss of objectivity by incorporating sponsor- or investigator-specific priors”, this is a naive model of the use of Bayes. When I do a Bayesian analysis I’d only use my prior if I was the only person that was going to act on the results. If I’m going to do a Bayesian analysis to convince someone else (e.g. the FDA) I *have to use their prior*, or at least a prior they agree to. It’s pretty bloody obvious point, but almost universally overlooked in criticisms of the use of a Bayesian analysis.
What does “unbiased” mean here? I’m not sure how that word connects with my understanding of randomization’s benefit for Bayes which is that the assignment mechanism becomes ignorable.
As an aside I think the informal use of “unbiased” has a harmful effect in this context (and in a few other statistical contexts), namely that it makes zero-centered skeptical priors, which are “fair” in the technical sense of equivariant, seem “unfair” because they are not in the technical sense unbiased (i.e., they will tend to introduce bias in the frequentist sense for many parameter values).
In a regulatory contexts there is often a binary decision to be made that is very proximal to the trial results, e.g. approve or not approve.
So a binary framing is important, even if cashed out more carefully as something like “is the new drug or biologic [sufficiently] effective [with sufficiently high probability for sufficiently many people or a sufficiently reasonably skeptic]?”.
Leon:
I was going to make the same comment as your second one. All this analysis necessarily comes down to a binary decision in the end – whether or not the drug should be approved – so it’s strange to criticise that framing. It would be like critisiing a professional footballer for never practicing their basketball skills or something.
It’s perfectly reasonable to critique *how* that binary decision is made, of course, which is maybe more what Andrew meant to say.
Leon:
1. By “unbiased” I’m referring to E(y|theta) in a model such as
y1 ~ binomial(n1, p1)
y0 ~ binomial(n0, p0)
theta = p1 – p0
theta_hat = y1/n1 – y0/n0
This goes into the likelihood part of the model. My point is that design tools such as randomization are very important in Bayesian inference, not for the prior, but for the likelihood part of the model. So I don’t think it’s correct to say that Bayesian methods involve “concession of the benefits of randomization.”
2. I’m fine with binary regulatory decisions. Approve vs. non-approve is such a decision. That does not mean that we should consider the underlying state of nature to be binary. Erik, Witold, and I write of the problems with the 2×2 model of science.
I am a bit confused by the comment that the prior breaks randomization. As pointed out above the assignment mechanism becomes ignorable which is what randomization is supposed to do. There are so many other things that break randomization, eg. inter-current events, so I fail to see that as a major issue.