Probability problem involving multiple coronavirus tests in the same household

Mark Tuttle writes:

Here is a potential homework problem for your students.

The following is a true story.

Mid-December, we have a household with five people. My wife and myself, and three who arrived from elsewhere.

Subsequently, various diverse symptoms ensue – nothing too serious, but everyone is concerned, obviously. Video conference for all five of us with an Urgent Care doc who interviews each of regarding identity, onset, severity, etc. of symptoms. This leads to him scheduling virus tests for all of us. Doc says he’s seen too many cases where patients with symptoms postpone tests because they think it’s not serious, then it both gets serious and others are infected. No surprise there.

We all agree to self-quarantine if anyone gets a positive test. (Maybe we should have anyway, but that’s another story.)

The symptoms resolve themselves, and all five tests are negative.

In email follow-up from the doc (good for him) he observes that since all five tests are negative they probably do not include a false negative.

I recount this story – and the doc’s observation re reduced likelihood of false negative – to various smart folks, but no statistics/probability nerds, with fascinating results. It shows how hard it is for normal folks to deal with these concepts. One friend who heard the story consulted his quantitative expert who confirmed the doc’s observation.

I think it’s a wonderful example because we probably all had the same “bug”, but, apparently, not THE virus. Thus, if we all had THE virus, it does make it more difficult to get five (false) negatives. Again, the implication from the doc was that he had had to deal with false negatives, generally – apparently, they were frequent enough to be a concern (ignoring the technical challenge of confirming a false negative in real practice).

The test was some kind of PCR thing; I could get the exact name if that ever mattered.

So, the homework problem is to assign probabilities to the various inputs in a way that justifies the doc’s observation that false negatives were unlikely (or not), and unlikely vs. the false negative rate for a single test on a single random person with symptoms.

35 thoughts on “Probability problem involving multiple coronavirus tests in the same household

  1. On the face of it, the bare statement that with five negatives, probably none of them were false seems silly. But in context, it makes sense. If one of the family really had had covid, then probably some or all of the others would have gotten it from them. In that case, the chance that all of the covid ones would all have had a false negative test would not be high.

    • I suppose it depends on the transmission dynamics. For example if you were exposed yesterday, and tested everyone today… it’s too early to have enough virus to detect. If one person was exposed 2 days ago, it’s too early for that person to have transmitted to others and have them be definitely detectable, so if that one initial person gets a false negative the others would be expected to be negative too.

      Now, if everyone was exposed 4 days ago… then it’s unlikely they’d all be false negative, yes.

      If one person was exposed 6 days ago and the others were in close contact, then yes it’s also unlikely for them all to be false negative.

      The viral dynamics are such that from initial exposure to peak viral concentration is something like 3 days. so it’s something like 6 days from initial exposure of one person to detectability for others in the family.

  2. There’s information I don’t have about how COVID tests work and what causes them to be falsely negative. Basically, are they randomly falsely negative, or are they non-random in ways that could be correlated within a household? Like, are there particular strains that are likely to cause false negatives? If that were the case, then the likelihood of a bunch of false negatives among people who’ve infected each other is the same as the chance of one false negative. (Or, and this doesn’t apply here, but are there people who are because of some genetic thing more likely to test falsely negative, in which case that could also be correlated within a household.)

    If those aren’t the case than this seems correct to me. Conditional on n people having COVID, the chances that none of them will test positive for it goes down as n goes up, if these are independent events – it’s a (1-p)^n problem. I am assuming they all have the same thing, but that seems likely since they all have something and they live together.

    • For the Covid PCR test
      – The false negative assay rate is very low: ie, if you get a negative test, it’s safe to assume there was little or no viral SARS-Cov-2 RNA on the swab
      – The false negative rate due to not picking up virus on the swab is appreciable. It’s probably reasonable to model it as roughly independent between tests, but it varies strongly over the course of the infection. Early in the symptomatic period and just before it, the false negative rate gets as low as 20%, but it’s much higher immediately after infection.

      • I am not so sure whether it is reasonable to model it as random. A swap as you said depends on the person who receives the test and the one taking the sample. Someone can make the same mistake over and over again and therefore cause false negatives. On the other hand it is possible that due to the kind of infection it differs where it sits. Also there was a publication a while ago investigating testing procedures in different centres in the same city (Muenchhoff et al.) seeing that the outcome was correlated to the location. They argue that simply due to the kits which were used this could cause missing of infections.
        Considering that they were probably tested together one could argue that they were exposed to the same bias at least to some degree.
        The main issue as you pointed out though is the time point of infection in relation to the time of when the sample was taken.

      • There are at least two causes for little or no virus on a swab: high cycle threshold (Ct) value or bad swab technique.

        Check out figures 1 and 2 of this paper, which back up what you (Thomas Lumley) are saying

        https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7427302/

        Figure 1 plots Ct value vs. days since onset of symptoms (not since infection). Figure 2 plots sensitivity of PCR tests as a function of Ct. They’re plotting days since onset of symptoms because there’s not enough viral load right after infection to test positive. These were from April/May 2020 and aren’t based on huge amounts of data. I’m looking for something more recent that had enough info for me to estimate sensitivity as a function of days since infection (not just symptom onset).

    • Keep it simple. For a prevalence of 1:100 and a sensitivity of 70%, what is the likelihood that if you test negative twice in a row, you are still true positive?

  3. For this sort of matter, you should probably also consider the timing of the events (possible exposure, symptoms onset, testing).
    If you test too early for covid-19, i.e. before the viral load is high enough, the test may come back negative, but I’m not sure if it officially qualifies as a false negative, i.e. if this aspect is already included in the advertised sensibility of the test.

  4. “the homework problem is to assign probabilities to the various inputs in a way that justifies the doc’s observation”

    I just don’t get this. Why make up numbers that no two people will interpret the same?

    This is a logic problem.

    1. The circumstantial evidence – everyone got similar symptoms at approximately the same time – makes it unlikely that a single member of the household got coronavirus while the other members of the household became sick from a different virus.

    2. The false positive rate has to be less <10% for a test to even be used.

    Combine 1. and 2. and the likelihood of there being a single family member with coronavirus getting a false negative is vanishingly small. How would putting numbers on this make it any better?

    • A few points:
      1. “Exercise to the reader” is just to engage with the problem I think, not to converge on a canonical model. That said, sharing approaches to problem modeling is how consensus is built.

      2. False positive rate false negative rate. False positive rate is tied to specificity, false negative rate is tied to sensitivity IIRC. You can tune a test to be more likely to report false positives or false negatives – ideally we minimize both, but usually there is a priority and these two don’t move completely independently.

      This article is filed under Bayesian thinking, so I think it relates more closely to examining your priors.

      Finally, you pivot from logic to probability in your last sentence, and hand-wave away with “vanishingly small.” I would rather have a number on that ;-)

      • “Finally, you pivot from logic to probability in your last sentence, and hand-wave away with “vanishingly small.” I would rather have a number on that ;-)”

        Well, let’s parse this.

        One of three things happened:

        1. Everyone had coronavirus.
        2. One or more, but not all had coronavirus.
        3. No one had coronavirus.

        The provided fact that everyone had similar symptoms supports 1. and 3. and refutes 2.
        The provided fact that everyone tested negative supports 3. and refutes 1. and 2.

        So only 3. is supported by the known facts, unless there was a false negative.

        Now we have to deal with bad test results.

        For 1. to be true, all five tests had to be botched. Not false negatives, per se, but botched tests. Five out of five properly performed tests being false negatives is not worth considering.

        For 2. to be true, two things have to be true. One is that the test of the coronavirus-infected person(s) had to be a false negative, and the second is that the other family members had to be sick at the same time with a different disease.

        So all the doctor claimed was that it was more likely that no one had coronavirus than that 1. all the tests were botched, or 2. there was both a false negative AND two different diseases in the household at the same time.

        Works for me. Now if you want to embellish the logic with false positive rates or whatever, it does no harm. But it doesn’t add much either.

        This is a great example to use when explaining to folks how to diagram causality!

        • >Not false negatives, per se, but botched tests. Five out of five properly performed tests being false negatives is not worth considering.

          Not that I’m arguing the basic logic, but as people have mentioned there could have been a shared site-collection problem for all the tests, or a contamination problem at the lab where all the tests were evaluated in close sequence. How do you determine those possibilities aren’t worth considering?

        • “How do you determine those possibilities aren’t worth considering?”

          It is not that they are not worth considering. Everything is worth considering. Not only would the tests have been botched, but they would have been botched in a way that produced valid-looking results, without the testing entity having any clue. And botched tests do not mean that testees had coronavirus. The scenario is very unlikely, that’s all.

          Or a comet could have hit the lab. That was always the running joke when I was building these diagrams (retired now).

    • Most answers above are not noting that this group of 5 is sharing viruses. (They all got sorta sick at the same time; viruses are shared frequently within households; COVID is thought to be eminently sharable.)

      I think we’d want to estimate p(n) = P( n housemates get COVID | 1 gets COVID ), for n = 1 to 5. If p(5) = 1, then what happens in the story is as good as 5 negative tests. p(3) = 0 means the pandemic is a probably a hoax, right? So we’re somewhere in between.

      I guess I am modeling false negatives as noise.

    • Specificity (accuracy on negative cases) for PCR tests is very good—well over 99% by most estimates. Sensitivity (accuracy on positive cases) is very bad—well below 80% by most estimates, even at peak viral load, given the variation in Ct values for infected individuals. It’s the sensitivity (false negatives) that matters here, not the specificity (false positives).

      Check out the plots of the raw data in the first two figures of this paper:

      https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7427302/

      Before figure 1, the paper says in the first week after symptom onset, (geometric) mean Ct is 28.2. That gives a roughly 60% test sensitivity according to figure 2. But you can also see that the individual variation in Ct value is huge from figure 1, even during the highest viral load around time of symptom onset.

  5. Now this is an intriguing exercise b/c it does require an explanation of assumptions being made by the physician and family members. In my observation only a handful of experts have the knowledge to undertake this exercise. They have knowledge of the biology of the virus as well as the kinetics of PCR and Antigen testing. So in some sense, I wonder of what use it is for statisticians to assign probabilities without the input of this handful.

    What do we mean by ‘bug’? And if the PCR was used, did the physician get the CT values alongside?

    The case presents an overarching question that I have had. If you have a coronavirus other than SARS2, will the PCR test be able to specify accurately that it is SARS2? To date no one has answered this question. Period. Now if it is a question that is silly or stupid. Just tell me.

    Frankly, from early on, I came to the conclusion that the PCR was the wrong SCREENING tool: it is typically referred to as a Diagnostic. Given that the PCR is being used equally as a screening tool, the PCR test has been identified as potentially resulting in false positives b/c the PCR test is so sensitive. It can yield a false positive for up to 2 or 3 months.

  6. Great question!
    The main issue – are PCR tests just finicky and fail randomly, or do they fail for good reasons? The main answer is – they fail for good reasons.

    Were the swabs taken by the same technician? That could lead to a coordinated failure of the tests. Were they run through the same lab, using the same PCR primers, etc? Some tests were much worse than others…

    If all the infections were ‘in phase’ from a single source – since many false negatives come from people with low titers – then having tested all of them is unhelpful. However, if the first case was from outside and the rest travelled around the home, then the five tests might be informative if any of them are positive. Further, a question of variants – particularly if negative results were being called because of the ‘drop out’ antigen – then having 5 tests is also potentially not helpful.

    So these are several ways the simple multiplication of false negative rates could be undermined by the real world.

    • Great answers! It’s all about correlation, including viral load, swab, and lab tech! We only get (1 – p)^n decay in false negatives with n tests of sensitivity p if the tests are independent. If they were 100% correlated, n tests would still have a (1-p) chance of a false negative.

  7. > So, the homework problem is to assign probabilities to the various inputs in a way that justifies the doc’s observation that false negatives were unlikely (or not), and unlikely vs. the false negative rate for a single test on a single random person with symptoms.

    I guess this is a statistics course?

    From my point of view, what’s interesting is how much difficulty many people have understanding the statistical implications of multiple tests.

    I’ve come across many people who think that if there’s an X probability of inaccuracy, there’s no change to that probability with multiple tests.

    Something like:

    “Why take the tests multiple times if it’s only accurate half the time? It’s just not as good as the PCR test which is more accurate.”

    I guess it’s about conditional probability. Like when people say “What’s the point of wearing a mask if it doesn’t guarantee to keep me from getting infected/infect other people?” It’s very hard to get across the message that a marginal benefit in an individual risk event compounds across a population.

    • “It’s very hard to get across the message that a marginal benefit in an individual risk event compounds across a population.”

      Does anyone know of an explainer (video, article, simulation tool?) for this concept? Does it mean that if each individual has a 5% lower chance of getting infected, there is a greater than 5% reduction in overall cases?

  8. Ah, but here’s a different story. In January, my wife and I (there are no others in the household) came down with the same flu-like and respiratory symptoms on the same day. We got tested three days later. My test came back negative. But we got no result on my wife for another three days, at which point her test came back positive. Very mild symptoms continued for another week, during which time we both quarantined. I have assumed that there is a much higher chance that I had a false negative than that my wife had a false positive. Given that we had virtually identical symptoms, there is really a negligible chance that one result wasn’t incorrect; given the fact that the symptoms were pretty classic mild Covid symptoms, I assume the false report was mine. Does anyone disagree?

    • It would be relevant to know the rate of COVID in your area that that time, and then to estimate how many people you encountered.

      When someone tells you “Yeah, it was probably a false negative,” reply “Oh, that’s great, I can assume I’m immune now!” and see if their expression or story changes. Or if they slap you.

      • January was of course a spike period here in NY, and I left out the info that we ate at a restaurant three nights before symptoms started that, a night from which (apparently) several others had confirmed positives. While that greatly raise the chances of a false negative I left it out of my description because I wanted to see people’s take on the bare facts. And yeah, I absolutely assumed I was immune, which greatly relieved psychological stress while waiting for the vaccine line to diminish.

  9. The problem here is less probability interpretation and more one of language use.

    Assume
    C_i is the i-th person in the household has Covid.
    T_i is the i-th person tests positive.

    For algebraic simplicity I’ll assume only 2 family members from here on but the point clearly generalizes.

    What the doctor really means is that P(C_0|~T_0) > P(C_0|~T_0&~T_1). What a pedantic translation would interpret him as saying is that P(~T_0|C_0 & ~T_1) <P(~T_0|C_0).

    This later claim is going to be false under simplified toy model style assumptions since it would require how my Covid test turns out to be affected by something besides whether I'm infected. OTOH the indented claim is clearly true on reaeonable toy model assumptions since for you not to have Covid you must have both had a false negative and either not infected your partner or they just also have had a false negative and the effect grows with number of family members.

  10. Huh, interesting homework problem. On one hand, you could note that the familywise error rate increases with the number of tests. So the chance that at least one is a false negative naturally increases with the number of tests. So if you take that perspective, the doctors statement seems wrong.

    But… There are important background assumptions. 3 people traveled, and all five people in the household develop symptoms. So if you assume that they all have the SAME disease (a reasonable assumption, given the circumstances) it is indeed much more unlikely to have five false negatives if everyone really did have COVID.

    Good example of how context and underlying assumptions are important for a student.

Leave a Reply to Joshua Cancel reply

Your email address will not be published. Required fields are marked *