How large is that treatment effect, really? (my talk at NYU economics department Thurs 18 Apr 2024, 12:30pm)

Posted on by

19 W 4th Street, Room 517:

How large is that treatment effect, really?

Andrew Gelman, Department of Statistics and Department of Political Science, Columbia University

“Unbiased estimates” aren’t really unbiased, for a bunch of reasons, including aggregation, selection, extrapolation, and variation over time. Econometrics typically focus on causal identification, with this goal of estimating “the” effect. But we typically care about individual effects (not “Does the treatment work?” but “Where and when does it work?” and “Where and when does it hurt?”). Estimating individual effects is relevant not only for individuals but also for generalizing to the population. For example, how do you generalize from an A/B test performed on a sample right now to possible effects on a different population in the future? Thinking about variation and generalization can change how we design and analyze experiments and observational studies. We demonstrate with examples in social science and public health.

8 thoughts on “How large is that treatment effect, really? (my talk at NYU economics department Thurs 18 Apr 2024, 12:30pm)”

  1. I’ve always found the foundational assumption in econometrics that it is possible to estimate an ‘unbiased effect’ at any level to be a completely bizarre and somewhat unhinged perspective. If you know anything about measurement, then the best you could ever hope for would be an estimate of direction (sign) and a qualitative sense of scale (i.e., small, medium, large).

    Reply ↓

  4. “But we typically care about individual effects (not “Does the treatment work?” but “Where and when does it work?” and “Where and when does it hurt?”).”

    Many physicians (me included) have a *big* problem with the hype around personalized medicine, mostly because 1) those who promote it rarely understand the massive (and arguably insurmountable) barriers to achieving it, and 2) it diverts money and attention away from research into public health problems that are a lot more pressing (e.g., social determinants of health).

    In principle, doctors would love to be able to predict how a treatment WILL affect a given patient, but this isn’t usually a realistic expectation. We have enough trouble reliably showing, through A/B testing, that a treatment CAN work.

    A well-known article on this topic:

    https://pubmed.ncbi.nlm.nih.gov/26415869/

    Some key excerpts:

    “In the context of clinical trials, a consequence of this is that the identification of differential response to treatment requires replication at the level at which differential response is claimed… Such replication is only achievable with great dif!culty at the level of the patient, involving as it would repeated period cross-over designs or equivalently designs in which a number of patients were given so-called n-of-1 protocols…

    …A practical problem is that for many indications cross-over trials, let alone repeated cross-over trials, cannot be undertaken. Because replication is the key to identifying interaction, identification of interaction must thus be accepted as only being possible at the group level…

    …Thus, I am not claiming that elements of individual response can hardly ever be identified. I am claiming that the effort necessary, whether in design or analysis, is rarely made and that labelling patients as ‘responders’ and ‘non-responders’ according to some largely arbitrary dichotomy…is not a sensible way to investigate personal response…

    …Managers (or other with decision-making capabilities) who intervene in a system without understanding it adequately was what Deming identified as being an important adverse influence on quality…

    …There is a considerable danger, however, that an obsession with personalising medicine before a reasonable average policy has been established may actually introduce harmful variations into the system. Again understanding components of variation is key.”

    It’s incredibly frustrating to see so much money thrown at “personalized medicine” research ventures when we already have our hands more than full with problems that require *population*-level solutions. The ever-increasing spending, in wealthy countries, on the holy grail of optimal *individual* health, in the face of ever-worsening aspects of *population* health, is, ethically speaking, pretty gross.

    Reply ↓

    • I think your criticism of ‘personalised medicine research’ is not inconsistent with the message that the abstract says this talk will deliver. It says that the talk will be about ‘variation and generalization’, which can help ‘design and analyze experiments and observational stud[ies]’. In fact, I think it might fit very well into your view of ‘personalised medicine research’! I understand from the abstract that it denies that there is an invariable effect that affects all individuals in the sample, and even the out of sample population, equally. You can see in some experimental studies that the authors claim that the confidence (or credible) interval of the average treatment effect (‘the effect’) contains zero, so either there is no effect or it is negligible. Or worse, the model is tortured until the interval around the average treatment effect no longer contains zero. I think this is the message rather than arguing against population-level estimates.
      For what it is worth, I agree that there are narrow limits to what medical research can do at the individual level. I think it really depends on the research question. If we consider an individual exercise and diet plan to be ‘personalised medicine’, then sure, that will work. But calculating the exact amount of a drug to give based on twenty variables and their interactions (just an example, no reference to real-life studies) sounds like a fool’s errand – unless you shrink eighteen of those twenty variables to zero, or something like that.

      Reply ↓

Leave a Reply

Your email address will not be published. Required fields are marked *