(again) why I don’t like to talk so much about “p-hacking.” But sometimes the term is appropriate!

Posted on by

Part 1

Jonathan Falk points us to this parody article that has suggestions on how to p-hack.

I replied that I continue to be bothered by the term “p-hacking.” Sometimes it applies very clearly (as in the work of Brian Wansink, although it’s a mystery why he felt the need to p-hack given that it seems that his data could never have existed as reported), but other times there is no “hacking” going on. So I prefer the term forking paths.

Two things going on here:

1. Saying “p-hacking” when it’s forking paths is uncharitable, as it implies active “hacking” when it can well be that researchers are just following the data in what seems like a reasonable way.

2. Bad researchers looove to conflate the professional and the personal. Say they’re p-hacking and they’ll get in a huff: “Who are you to accuse me of misconduct??”, etc. Say they have forking paths and you remove, or at least, reduce, that argument. OK, in real life, yeah, people will say, “Who are you to accuse me of forking paths?”, but forking paths is just a thing that happens, an inevitable result of data processing and analysis plans that were not decided ahead of time.

So, yeah, humor aside, I don’t like the p-hacking talk, for similar reasons to my not liking the “file drawer” thing: in both cases, the focus on a specific mechanism can serve to minimize the real problem, to conflate scientific mistakes with intentional misconduct, and to provide an easy out for many practitioners of bad science who don’t seem to realize that honesty and transparency are not enuf.

Falk responds:

I agree completely with that.

But honestly, I feel like both the garden of forking paths and p-hacking are just versions of Bitcoin’s Proof of Work method. You get rewards for showing how much effort you had to go to get SIGNIFICANCE. If you have a study with a p-value on your first try of 1e-8, people will say “But that result was obvious! Why do they even bother with a test?” If you garden-of-forking-paths or p-hack your way to 0.047, you will be credited for your perspicacity.

Part 2

Ethan Steinberg writes:

I just came across an article that will probably be interesting to you and your readers. Back in 2022, the Florida Surgeon General released a report that the COVID vaccine appeared to be statistically significantly correlated with cardiac arrest “In the 28 days following vaccination, a statistically significant increase in cardiac-related deaths was detected for the entire study population (RI = 1.07, 95% CI = 1.03 – 1.12).” is the full report.

This was then used to recommend against COVID vaccines for young men in particular.

A local Florida paper just obtained and released the original versions of the reports:

Here are the drafts, from first to last.

The TLDR is that the original analysis did not find significant increases in cardiac related deaths. They had to go through a lot of analysis variants / drafts to get the result they were looking for.

I guess the real question here is how this could be avoided in the future. Maybe we should expect public health officials to register their analysis in advance?

I don’t think we should ask public health officials to register their analysis in advance, as that just seems like more of a mess. But in any case the above seems like an example where there really was p-hacking.

P.S. Just to clarify: As always, the problem is not with the “hacking”—looking at data in many different ways—but rather in only reporting some small subset of the analyses. It’s fine to go through a lot of analyses of the data; then, you should publish all of it, or publish a single analysis that incorporates all of what you’ve done using multilevel modeling.

9 thoughts on “(again) why I don’t like to talk so much about “p-hacking.” But sometimes the term is appropriate!”

  2. Why does relative risk of non-covid death go like this by week after vaccination (from the pdf link):

    1: 0.37
    2: 0.63
    3: 0.79
    4: 0.84
    5: 0.91
    6: 0.94

    Also, the premise you can compare covid infection vs vaccination as if they are mutually exclusive is very wrong. Those are hardly the only issues, but anyone with critical thinking should have been stopped in their tracks there.

    Regarding p-hacking, that behavior is exactly what scientists should be doing. The problem is that NHST is incompatible with science.

    Imagine researchers trying to slice and dice the data in every kind of way to try to disprove *their theory* (not a default null hypothesis). Then other people try to replicate the study to check if they also see that deviation. In this case we would gradually learn all the various factors that need to be monitored/controlled to get reliable results.

    That would be cumulative progress, the opposite of NHST which generates an endless series of conflicting results until you just move on to something else and don’t talk about it any more.

    Reply ↓

    • I can think of one example mechanism which would lead to the pattern of numbers you mention. If people are avoiding leaving the house until after they are vaccinated they would be at significantly reduced risk of car accident, falling off a ladder, crime, etc. After getting vaccinated if they resume more normal activities then they’d be exposed to all those risks.

      Reply ↓

      • It shouldn’t be that unless there is some major classification error. The use:
        “Natural all-cause deaths (i.e., excluding homicides, suicides, and accidents)”

        Really, you have to read the methods in the pdf because the numbers do not mean what a normal person would think.

        Eg, “baseline” comes *after* the “risk period”.

        Essentially we have unknown pre-vaccine risk, then “all cause” (non-covid, natural) mortality of people (who did not report a positive covid test within 18 weeks of vaccination) gradually increases to 3x over 6 weeks. Then weeks 7-18 are aggregated together, so maybe that represents a plateau.

        What we want to know is whether pre-vaccine mortality was closer to week one or week six rates. And also if weeks 7-18 are really about constant, or is the average more like a cross section of a spike.

        Whatever it is, that obviously needs to be explained before moving on.

        Reply ↓

        • Also, the entire population consists of people who died within 6 months of last vaccination. There are no survivors.

          So this is a study of people with terminal illness and must be interpreted in that context.

          I’d guess they were allowed to see their families one last time so stayed alive for that. Eg:

          Human aging is characterized by physical and physiological changes that reduce the immune system and modulate vaccine responses (5, 6). Whether the coming COVID vaccine will generate an effective response in the oldest and frail older people and how these responses will be modulated by age or other clinical factors such as frailty is currently an open question. In older individuals who show increased COVID-19 disease severity, an impairment in the protective effect of COVID-19 vaccine responses in older adult may be observed. Preliminary data suggests a reduced antibody response to COVID19 vaccine in older persons.

          Lockdown during the COVID19 pandemic has led to older persons living at home and especially those in nursing homes to experience increased isolation and loneliness (7, 8). Loneliness leads to depression, poor sleep, decreased cognition, cardiovascular disease, disability and death (9). Staff wearing Personal Protective Equipment further causes socialization distance. This has created a major reason for older persons and their carers to receive the COVID19 vaccination.

          https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7816745/

          Also, the vaccination may have been triggered by some contact with the healthcare system. Perhaps there was some short term benefit unrelated to the vaccine.

          But why would anyone assume you can extrapolate from the terminally ill to the general population? This study is bizarre in so many ways.

        • Why would someone assume that you can attribute mortality causality to NPIs rather than the pandemic itself, unless they’ve at least attempted to address the underlying counterfactual assumptions that are necessarily implied to do so?

          I can thnk of some reasons why someone would do that.

          What would have happened in congregate housing for elderly people, absent attempts to restrict the rate of infections – with, say, unrestricted access to visitors?

        • A thought occurs to me.

          For all the many comments I’ve seen where people focus on the age gradient in COVID mortality, and quality years of life lost, and the “dry timber” effect (where a jump in deaths is considered just a basic outcome of the most vulnerable dying off a little sooner than they would have otherwise) to be critical of the outcomes of NPIs (why were they implemented to only reduce the impact of relatively unimportant factors?), I’ve never seen those issues raised from the same people in the context of the harms caused to seniors “caused” by NPIs.

          Prolly just a coincidence.

  4. Here is another of these self-controlled case studies on covid vaccines where we can see more detail:
    https://www.nature.com/articles/s41467-023-36494-0#MOESM1

    In figure S7 we see that the relative mortality rate starts at ~50% of the peak, then increases. It peaks at 12 weeks then at 24 weeks it has gradually dropped back down to the low week 1 rate.

    They speculate the initial low rate is:

    a pattern that is consistent with a time-varying healthy vaccinee effect due to postponement of vaccination during periods of illness or recovery25.

    Then the later drop in mortality:

    Risk periods exceeding the interval between doses are subject to negative bias (the median interval was 10 weeks between dose 1 and 2 and 27.5 weeks between dose 2 and 3)

    This “explanation” doesn’t fit the data since we see the same pattern of increasing for 12 weeks then dropping back down after all three doses.

    It was also discussed in the peer review file:

    “Lines 201-2: the interpretation of the decline seems to imply a positive association with deaths before 12 weeks after dose 1, implying an RI > 1 for dose 1, yet none is observed. This might suggest that the decline over time is actually attributable to temporal effects, rather than dose 2.”

    We agree that a decline in deaths after dose 1 could be temporal as well as due to dose 2. However, it is unlikely in this case to be temporal as first vaccinations were given out over a range of calendar times so a drop would be unlikely to be sudden and the drop after dose 3, where there would not be a dose effect, is much more gradual indicating that a temporal drop after dose 1 would also be more gradual than is observed. Temporal effects such as registration delay that means deaths are less likely to be observed at later calendar times are taken into account in our model by the inclusion of calendar time. Dose effects are taken into account by the structure of the stacked dataset, which is designed to deal with eventdependent exposures and multiple doses of vaccination [1]. Therefore, a change in RI after vaccination should not be related to either temporal effects or dose effects.

    Isn’t this what we would expect if there is some side effect killing these terminally ill people that peaks at about 12 weeks after vaccination?

    I also note neither the authors of the UK paper, the peer reviewers, the florida authors, or anyone commenting in the news comment on the generalizability of these results. I ask again, why would we expect such results to generalize to the general population?

    Reply ↓

Leave a Reply

Your email address will not be published. Required fields are marked *