How to incorporate new data into our understanding? Sturgis rally example.

A colleague writes:

This is a very provocative claim about the Sturgis rally—can you do a stats “fact check”? I’m curious if this has been subjected to statistical scrutiny.

I replied that I’m curious why he said this study is provocative: It makes sense that when people get together and connect nodes in the social network, that they’ll spread disease, right?? I don’t know about the exact numbers, but the general idea seems reasonable. The only part of the abstract that really rubs me the wrong way is the last sentence, “We conclude that the Sturgis Motorcycle Rally generated public health costs of approximately $12.2 billion.” I suspect double counting (not that I’ve read the paper in detail; I’m just generally suspicious of this sort of claim).

But, the $12 billion dollar thing aside, what is it about the paper that you would consider controversial? Indeed, why would it get a lot of attention? I feel like I’m missing something.

My colleague responded:

I’m not into converting everything into money-equivalents so I’m not concerned with that part.

But just to emphasize that your view isn’t widely held, see here.

I think my point is that I find it plausible so sure in some sense it isn’t an outrageous claim, but not sure the evidence really shifts my prior that much. Someone sent me some data on mask rules claiming that there was observational evidence that masking reduces transmission. I strongly disagreed based on the data he sent – I just felt we couldn’t learn anything. So my priors are still strongly that masking helps. But looking at the timing of community masking rules in that other study didn’t really change that view.

I replied: So you’re saying the result is believable but that this is just one (noisily measured) data point so it doesn’t carry much information? That makes sense to me. But that’s different from the link in that twitter thread, which calls the claim “ridiculous.”

More generally, this reflects a problem with scientific communication: individual studies are supposed to be definitive, so what do you do with an analysis that is consistent with a generally plausible claim but would not represent strong evidence on its own? My colleague referred to the claim in that paper as being “provocative,” but I think the provocative part is not the substantive claim but rather the meta-claim that the paper represents strong evidence in favor of the substantive claim, rather thank weak evidence that is consistent with the substantive claim.

P.S. Thanks to Zad for the most adorable version of quarantining with your family in a small apartment.

54 thoughts on “How to incorporate new data into our understanding? Sturgis rally example.

  1. There is nothing at all that should be controversial about a paper suggesting that Sturgis caused enormous costs to society. I mean the precise number is hard to estimate but if anything I’d expect it to be higher. I imagine more than 10% of participants had the disease by the end of it, and that each one started a chain of at least depth 5 with branching factor 3 on avg. So that’s 50000*3^5 = 12M infections… Just basic order of magnitude. So each infection ~$1000 of costs seems reasonable. The order of magnitude is basically very reasonable. Remember that an ICU stay is likely $1M so if only 1/1000 had ICU and the rest had nothing it’s already done. I’d guess Sturgis cost between $5B and $50B no problem.

    • I think you’re overestimating by at least a couple orders of magnitude. Just try applying that to Thanksgiving or Christmas. Assuming half of Americans held Christmas gatherings and 2% of those caught it, that’s over 3 million. Keeping parameters the same, that’s 3,000,000*3^5 = 729 million.

      • You obviously can’t extrapolate in the same way from later in the pandemic because there are only 320M people to infect not an infinite pool. But at the time of Sturgis, prevalence country wide was still in the few percent range.

        The question we have to ask is why just as Sturgis happened, did the cases go from plateaued to clearly exponential growth continuously from mid Aug to about Nov 15 and then start to linearize and eventually decline? Thanksgiving and Xmas actually are still in the concave down region. Things only look like they’re spiking right after because no one files paperwork during a holiday.

      • That’s kind of been my question about a lot of these claims. My expectation all along has been that an essentially fixed final % of the US will be infected, measures mostly will change the time scale*. If vaccination scales up fast enough, that could change, but I am not sure it will.

        *for the US specifically, as opposed to somplace like Iceland or New Zealand(this

    • Both Rahul and Anonymous have points worth considering.

      But here’s the thing. Sturgis was mid Aug… a 5 layer branching “tree” would take til mid Nov to play out and that’s just an arbitrary number I mentioned.

      https://ourworldindata.org/coronavirus-data-explorer?zoomToSelection=true&time=2020-03-01..latest&country=~USA&region=World&casesMetric=true&interval=smoothed&perCapita=true&smoothing=7&pickerMetric=total_cases&pickerSort=desc

      Shows that basically from Sturgis onward, we had a massive explosion in cases nationwide. Thanksgiving and Christmas don’t even really show up there except as a dip caused by probably reporting issues followed by surges after reporting.

      We’ll never know the counterfactual: what would the nationwide cases have looked like if Sturgis hadn’t occurred. But I find it entirely plausible it might have been a slowly oscillating function between 100 and 200 cases/million throughout the fall with appropriate messaging including “do not come to sturgis” and “you will be arrested if you enter the sturgis quarantine area” etc.

      Since about 50% of the area under the curve since the start of the pandemic is under the post-sturgis region. I think it’s entirely plausible that say 1/3 of all infections in the US are due to holding a massive motorcycle rally for participants from all 50 states in mid Aug.

      1/3 of all infections is probably 20 Million since we have ~30M confirmed and probably 50-100M including uncaught ones.

      We simply won’t ever know the real answer. All we can say is that you can’t rule out the idea that on the order of 1/3 of all infections are “due to sturgis” but you CAN rule out the idea that “Sturgis caused fewer than 10000 infections”

        • I think a better framing of questions of this sort could be in terms of time than number of cases.

          Rather than ask how many cases a motorcycle rally caused wouldn’t it be more useful to see how much forward in time it displaces an infection curve?

          Basically think of the scenario as a superposition of two curves: covid mortality and vaccination cohort. Ideally you want to push one forward and the other backward.

          So what matters is something like the first moment of the integral.

      • There’s another factor – somewhat hard to quantify.

        The attendees at Sturgis, I would guess, had a rather high level on average of contempt for social distancing measures. My guess is quite a bit beyond, on average, than the average attendee at a Thanksgiving dinner. Also, I would guess that at Sturgis there would have been a relatively high level of mixing with people beyond one’s own pod – which would not have been the case nearly as much with Thanksgiving dinners.

        • There are lots of documented cases where people went to Sturgis and came back and explicitly said how they didn’t think COVID was real, but then they got sick and it changed their mind. So yeah, they were basically actively superspreading things + alcohol, and generally zero regard for prevention. Immediately afterwards the country went exponential and didn’t recover control (sub-exponential spread) until mid Nov.

        • And there were stories (in the Post) about poor souls in the ICU in ND who *still* refused to believe it was real. Of course those stories, in normal times, would have fallen into the back-page category of items like “Chicken With Three Legs Found in Emporia”.

        • When I estimated what would happen over Thanksgiving (and Christmas) I assumed that after the holiday, most people would go back to whatever they had been doing so far as protective behavior is concerned. And the data seem to me to bear out that idea. So why not after Sturgis, too. Of course, we don’t know what the behavior of the participants in the aggregate really was beforehand…

          After Christmas and Thanksgiving there was no sign (to my eyes) that there was any carry over or delayed transmission. Most of the bulge of cases in the US came about because the growth rate had become fairly positive (a few percent per day, using a heavily smoothed data set) for long enough that by the time that growth rate relaxed back below zero, the integrated number of cases was large.

          I imagine something similar went on after Sturgis.

        • That whole long history of cases growing exponentially so that “the integrated number of cases was large” started like Sept 1 and continued through Nov 15 or so. That’s basically the “post sturgis period” as far as I’m concerned. Sturgis ended Aug 10, so people were back home and had transmitted by Sept 1… It’s also the “move into University” period and it’s the “a whole bunch of BLM protests” period.

          I’m not saying it’s ALL Sturgis, I just think it’s plausible that Sturgis was an important part of raising that to exponential spread for the next several months post event. Sturgis probably brought the virus into a lot of communities where it otherwise wasn’t really spreading, people in those regions hadn’t really needed to be very compliant, and weren’t yet taking things seriously. That’s certainly true for the Dakotas themselves. It seems plausible it’d be true for many of the communities people returned to from sturgis.

          Also, I don’t actually see any effect of Thanksgiving and Xmas in the nationwide data. The cases already went concave down by Nov 15 or so, and continued that way if you do a 15 day smooth, which is a more appropriate smoothing time for a period of time when multi-week holidays were in play (fewer tests, more people on vacation instead of writing up reports, etc)

          It looks to me like you can attribute all of the “holiday” period to “people actively trying to get things back under control after summer and early fall screwed everything up”.

        • I’m going to disagree with you about the effect of Thanksgiving and Christmas. Using the Johns Hopkins data, the upward swells are pretty clear; the Thanksgiving one, for example started 3 – 4 days after the actual date. Before then, the US daily case count had just started to go down again.

          It’s much harder to see without heavy smoothing, and heavy smoothing widens any narrow features, so overlaying both smoothed and unsmoothed versions of the data is very helpful.

        • There are large drops caused by no reporting during the holiday followed by big “post holiday surges” which last only a few days. If they were real surges you’d see continued growth post holiday… But you don’t. What you see is a strong concave downward trend starting about Nov15 and continuing to now, with very short term disruptions around the holidays. You wouldn’t see the overall concave downward trend continue if we’d had true surges caused by holidays.

        • >>The attendees at Sturgis, I would guess, had a rather high level on average of contempt for social distancing measures.

          I have to wonder if this factor could work both ways, though – that is, given that (I would say very sensible) assumption, wouldn’t we expect most of those people to have caught it by now (~6 months later) due to other incautious activities even if Sturgis hadn’t happened?

          I am sure it accelerated the spread, especially in rural areas like the Dakotas that hadn’t yet been hard-hit. But I’m not sure how much of the cost/harm was *accelerated* cost/harm that would have happened by now anyway…

      • What would be your estimate of the costs to society of the protests and riots in late spring/early summer? Can you rule out that 1/3 of all the infections are due to them? I think a paper about that would be quite controversial.

        A massive motorcycle rally for participants from all 50 states in mid Aug was surely a bad idea. But it’s not as if it was the only thing happening that may have contributed to the spread of the epidemics. There were 400’000 domestic flights in the US in August alone (more than 24 million passengers).

        • Data I saw suggested that the Sturgis rally included people from something like 50-60% of all counties in the US. It specifically was a location where people who **didn’t believe** that COVID was a risk got together to drink beer and party in close contact… Then they all went back to their original counties, and undoubtedly didn’t take any precautions against spreading the infection. It’s not just the size, but the concentration of the particular demographic that was the factor that makes me think it was particularly bad.

          The protests across the US had a totally **different** demographic, most of them were masked, and specifically tried hard to maintain distancing, and were much more likely to quarantine afterwards, etc.

          I’d say that University students are the other big demographic that spread the disease during that time period, rather than protesters.

        • There were around 100 people going to Florida this summer for every person participating on that event. From how many US counties would you say the came from? Do you think the amount of unmasked beer drinking and close-contact partying was really higher in ten days in Sturgis than in four months in Florida?

        • There were 500,000 people at Sturgis, all in a town that normally holds 6627 people, and an area of 4 square miles. That’s a crazy high density.

          If it’s true that 100x as many people went to Florida that’s 50M people, which is one in 6 americans. That’s hard to believe, what the hell is wrong with people? But let’s even say it’s true. The density of people was way way lower, and there were hopefully some level of controls on behavior, and it took place over months.

          After Sturgis, the Dakotas became the worst place on the face of the earth for covid deaths per capita for several weeks. Florida didn’t. That’s not a coincidence.

          I think density and intensity of activity makes a big difference. I’ve run a lot of agent based simulations as part of some projects I was working on, and it was striking to me how much one big spreading event could cause long term consequences that take months to resolve. Cut off those events and you don’t get 3 or 4 month long explosions in cases. Qualitatively that’s what I’ve seen.

          So, I stand by my claim that it’s quite plausible. I’m not at all claiming that I have proof that it’s definitely true.

        • My understanding is that the 500’000 figure counts people multiple times if the enter the area in multiple days. The actual number of people attending would be lower. And only a minority of them would be coming from outside South Dakota and its neighbouring states.

          It seems plausible that it was the main driver for subsequent infections in the region (which had a relatively low incidence at the time). It doesn’t seem plausible that it was the main driver for infections for the whole of the US.

          https://www.wired.com/story/how-much-do-crowds-contribute-to-covid-its-complicated/

          “But the bigger issue, Jha says, is that the real-world data just doesn’t seem to line up with the study’s estimates—except in South Dakota. The state’s health department has so far identified 124 people who attended the rally and later tested positive for Covid-19. A spokesperson for the department declined to say how many close contacts of those individuals are currently being monitored to see if they become ill. But in the month following the rally, South Dakota’s daily average for new diagnoses more than tripled from 82 to 307. That’s a visible spike that lines up pretty well, timing-wise, with Sturgis, says Jha.”

          Based on cell phone data, though, 90 percent of the rally-goers came from places outside the state. Yet when you look at the counties identified by the Sturgis study authors as contributing the highest number of attendees, you don’t see similar surges. According to their model, in the three weeks following the rally, Covid-19 infections rose the most in places like Maricopa County, Arizona; Hennepin County, Minnesota; and three counties surrounding the Denver metropolitan area in Colorado. In those counties, the authors found the Sturgis rally to be associated with a 13.5 percent increase in Covid-19 cases. But according to state public health department data, Maricopa County actually saw a decrease in daily reported cases—from a seven-day average of 775 at the start of the rally to 266 a month later. In two out of the three Colorado counties, the number of new cases declined or went flat following the rally. That trend was also observed in some other high-flow counties, including in San Diego, Los Angeles, and Clark County, Nevada, home to Las Vegas.”

        • Thanks Carlos. Interesting, obviously a county-by-county analysis is a better one than just looking at the whole US.

          I do stand by the idea that you can make agent based models go nuts by inducing a sudden surge of infected people and then spreading them around the map, which is what occurred here. So the observed pattern is consistent with one explanation. It’s also probably consistent with other explanations, that’s why I say it seems plausible but by no means proven.

      • You’re talking cases, and I’m talking infections. There are 26M cases, but probably 50-100M infections, so is it plausible that cancelling the Sturgis rally would have cut infections by 12M / say 60M, which is 20% or so. Yeah, I think it is quite plausible, certainly within a 95% high probability set.

        If we canceled sturgis and canceled in person college attendance together I suspect we could have cut infections between Aug 1 and today by 50% or more.

    • > I imagine more than 10% of participants had the disease by the end of it, and that each one started a chain of at least depth 5 with branching factor 3 on avg.

      No! You cannot compare to the counterfactual of zero covid, that’s intellectually dishonest! You’re assuming that each of the 10% of people ended up (via chains) of infecting 3^5 = 243 people that otherwise would not have been infected.

      Also, there were (according to Wikipedia) 250,000 attendees, so you should be doing 25,000*243.

      You’re also distinguishing a single event in a long causal chain and saying it “caused” the whole chain. If someone who went to Sturgis infected someone else ‘because’ they were both not wearing masks, does that mean the infection was ’caused’ by Sturgis or ’caused’ by not wearing masks?

      • Obviously it’s caused by both sturgis and not wearing masks.

        Here’s the basic underlying issue that makes me thing Sturgis wasn’t just a small effect. If you run an agent based simulation in which you have some anti-infection steps in place, there’s a bunch of people who are trying to avoid infection and a bunch who aren’t. And people are mingling… you get some pandemic graph. It takes “two to tango”, so if the non-compliant people are in a mix of compliant people, they don’t all tend to get infected and cause everyone else to get sick. But if you perturb it by forcing a super-spreading event by suddenly infecting a whole bunch of non-compliant people all at once and spread them around the map, the infection graph diverges wildly, increasing much more rapidly than it would have and taking a long time to come under control. It’s not just 250,000 or 500,000 people go to sturgis, and then 25000 or 50000 of them get sick, and then maybe they each infect 2 or 3 people and then everything goes back to normal… no it’s a whole separate trajectory for months on end. That’s precisely what we saw, a massive acceleration right after sturgis for 3 months, and it finally came under control in late Nov.

        Now at the same time, we were also doing start-up of Colleges and Universities. But they tend to be localized to the campus area, not suddenly going to a campus, having a big party, and then driving off across 7 states. Also universities did have SOME kind of testing and quarantine programs. So that was a bad idea, but not nearly as bad as basically infecting a whole bunch of noncompliant people and sending them all around the country.

        The action of forcing that superspreading into the agent based simulation produces a radically different trajectory which makes me think that it’s plausible (not in any way proven) that an event like Sturgis (where I’d read closer to 500,000 participated) could be driving a huge fraction of the cases since Aug.

        This is all just private speculation, I haven’t published anything on it or anything like that, I just asked myself what happens if you suddenly perturb agent based simulations in different ways.

        • >Obviously it’s caused by both sturgis and not wearing masks.

          Right, but morally speaking I feel like “not wearing masks” takes some of the blame away from Sturgis.

          >That’s precisely what we saw, a massive acceleration right after sturgis for 3 months, and it finally came under control in late Nov.

          Correlation is not causation! That said, who is “we”?

          >then maybe they each infect 2 or 3 people

          You’re suggesting a reproduction number of 2 or 3 on average in summer? Wow.

          I’m also not sure how a large proportion of Sturgis attendees were infected if a large proportion were not already infected going into Sturgis. There’s only so many people a super-spreader interacts with, and “noncompliant” people are not as stupid as most people think.

        • reproduction number of 2 or 3 among this crew? Sure. Seems easy. If you were looking to make a list of people who are likely to be much higher above average, you couldn’t do a better job than to advertise a big event for them all to come to right?

          The thing about COVID is that it’s a long-tailed distribution of reproduction. The median is 0, the average is around 1 when prevalence is in the 0.1 to 1% range and everyone is working hard to keep infection down, it’s probably 2 or 3 when people aren’t working hard, and the 90 percentile is probably 80.

          In mid august among the Sturgis crowd, let’s say coming into the event they had 1% prevalence. Now let’s have it grow about as fast as it did in uncontrolled conditions at the beginning of the pandemic… doubling every 3 days for 10 days, so that’s 2^(10/3)*.01 ~ 0.10

          I don’t think that’s at all out of the question when you put several hundred thousand people into a 4sqmile town with parties and drinking and free flowing bars and dancing and etc. So coming in there were ~5000 people with COVID and going out there were ~50,000 most of whom were noncompliant, and some of whom defiantly so.

          Now these people go back to their families, and within-family spreading occurs relatively actively because at home people aren’t masking inside their house and keeping 6ft etc. So you take 50,000 people and turn that into 150,000 over the next 2 weeks all around the country, including places where before there was extremely low prevalence. We’re not talking LA County, but more like say Bakersfield, or medium size town in Ohio or places like Tucson or whatever. Suddenly in regions where things would have been relatively stable because there wasn’t a lot of influx of COVID, and it wasn’t spreading much within the town… you’ve got spreading. Furthermore, maybe you’re talking semi-rural places where testing isn’t great, so case counts are not necessarily going to reflect it.. until much later when spreading has occurred sufficiently to create a bunch of sick people. Now, most people are going to spread to 0 others… but those who don’t because they’re non-compliant, will generate 20 or 40 or 100 other infections. The tail is long… and we’re still talking about a group enriched for noncompliance, even defiantly. So maybe that’s 10% of the 150,000 … they generate 30 a piece on average… so you’re at early October and there’s 450,000 new infections. Now things are out of control and spreading relatively wildly, in rural cold areas where people head indoors… so that 450,000 goes to 4.5M over the next month… we’re in the Thanksgiving region.

          But if the original event hadn’t occurred… lots of moderately rural regions wouldn’t have hit the threshold for uncontrolled spread, because many infections die out on their own, as people head indoors etc the starting point would have been dramatically lower prevalence and spreading. I don’t think anyone is going to argue that this didn’t happen in the Dakotas. The question is how much did it affect say Arizona, Utah, Colorado, Idaho, Ohio, Pennsylvania, Michigan, etc other places where people came from.

          I just think it’s plausible. Not proven, not even highly likely, it just isn’t within the realm of things you can rule out. There’s a mechanism by which you can imagine it occurring, and agent based models seem to be able to reproduce that kind of effect.

          It’s a really really bad idea to attract a bunch of highly noncompliant people into a massive highly dense area and then send them off home to spread disease. I don’t think anyone should disagree with that statement, the rest is all questions of just how big could the effect be across other regions? I don’t claim to have an answer, just that you can’t say the magnitudes are definitely obviously below say 12M extra infections between Aug and Feb

        • >I don’t think that’s at all out of the question when you put several hundred thousand people into a 4sqmile town with parties and drinking and free flowing bars and dancing and etc.

          4sq miles isn’t so little over 10 days; I’m sure people were more careful than usual. From videos on youtube it seems most of the action was outside, i.e. in a well-ventilated space.

          >reproduction number of 2 or 3 among this crew? Sure. Seems easy. If you were looking to make a list of people who are likely to be much higher above average, you couldn’t do a better job than to advertise a big event for them all to come to right?

          There’s this weird contradiction in your thinking in which all of the “non-compliant” people being met are free of covid (maybe sturgis prevalence of only 1% pre-event is plausible, but isn’t 50% immunity amongst the worst spreaders also more than plausible at this point?) but at the same time all of the “non-compliant” people are spreading covid. Is it possible? yes. It is plausible? no. If some biker from Sturgis gets infected and infects 3 family members, those 3 family members don’t again infect the already infected family members. Observe also that data from the UK suggests that only about 10-15 of contacts there lead to an infection (https://twitter.com/kallmemeg/status/1347555411940028417).

          >In mid august among the Sturgis crowd, let’s say coming into the event they had 1% prevalence. Now let’s have it grow about as fast as it did in uncontrolled conditions at the beginning of the pandemic… doubling every 3 days for 10 days, so that’s 2^(10/3)*.01 ~ 0.10

          I thought most people agreed that the fast growth at the beginning was also caused by the ramp up in testing?

          >But if the original event hadn’t occurred… lots of moderately rural regions wouldn’t have hit the threshold for uncontrolled spread

          What on earth is the “threshold for uncontrolled spread” and why should it exist? I don’t believe there is any “controlled spread” worth speaking about.

          People with similar arguments to yours projected that Thanksgiving/Christmas/New Years would lead to millions of new cases. These spikes largely did not materialize. If something as small as a few hundred thousand noncompliant people outdoors (many of whom surely were already infected in the first wave) caused as many cases as you suggest, then Thanksgiving travel surely caused an order of magnitude more, and business travel yet another order of magnitude more. I don’t buy the idea that all the Thanksgiving travelers were “compliant”, given that they were traveling despite everyone telling them not to.

        • “There’s this weird contradiction in your thinking in which all of the “non-compliant” people being met are free of covid”

          non-compliant with what?

          In South Dakota, one can argue that “non-compliant” means wearing a mask, even today, as the governor of the state has not only refused to issue a statewide mask mandate but has said they’re worthless. She’s trying to build a national conservative political presence based in large part on her covid denialism, and has an eye on a future Presidential candidacy.

          Beyond that, she very publicly did not mandate mask wearing at the Sturgis rally, and publicly encouraged people to come to Sturgis.

        • There is a “threshold” for uncontrolled spread precisely because the reproduction number distribution is so heavy tailed. If you give the virus to 100 people it’s entirely possible for say 80 of them to not spread to anyone, and a few only spread to 1 other or 2 others and you get lucky and none of them spread to 40 others.

          But if you get the numbers high enough the super spreading occurs and the average reproduction number goes from 0.6 to 3 because of that one to 3 events.

        • My model is explicitly this, and it could be just wrong, and I’d be happy to have data to prove otherwise… but basically based on the reporting such as: https://www.washingtonpost.com/health/2020/10/17/sturgis-rally-spread/ I think the model is plausible

          1) Lots of people who went to Sturgis were coming from areas that hadn’t yet been strongly impacted by COVID. Rural and semi-rural areas. And of course the Dakotas themselves, which had virtually nothing going on before the event.

          2) The even wasn’t a “family friendly” event, it was largely individuals coming to drink beer and party without their kids, wives, etc.

          3) On the order of 1% of the participants had the disease coming in, maybe somewhat less. Maybe largely those who did come from more populated areas.

          4) During the event the spread was completely uncontrolled: one person in a bar singing and dancing and shouting could easily give it to 40 others, and that could happen multiple times a day. This is well understood. The reports were clearly that this was going on: unmasked indoor attendance at bars, restaurants, tattoo parlors, and similar.

          5) Leaving the event, ~10x as many people had it as coming in. An amplification factor from say 2000 to 5000 coming to to 20,000 to 50,000 going out was not at all out of the question. It could easily have been 100k leaving even.

          6) People return to regions where covid hadn’t yet taken hold, they immediately give it to their families who were left behind… obviously there’s heterogeneity but this would be a common occurrence. Not necessarily just children and wives and things, but brothers, cousins, uncles, parents, etc. The group is specifically not just maybe non-cautious but politically anti-cautious thanks to Trump etc. As shown by the fact that in all the videos I’ve seen of Sturgis there was hardly a single mask visible.

          7) The particular group that was involved clearly didn’t care about covid, didn’t believe it even existed (see ron bloom’s comments elsewhere) and believed masking was for wusses or whatever, so coming back to their towns they’d be much more likely to spread the disease outside their families: grocery stores, hardware stores, their buddy who fixes their motorcycle, whatever. Igniting a stable enough flame that it didn’t just die out at 2 or 5 people in this newish town.

          8) The communities, not having had much real-world experience with covid, were primed to have a lot of spread. The general level of caution would be low in semi-rural and rural environments with populations who would be interested in going to Sturgis.

          Clearly **ALL** of what I’m saying above **did happen** in North and South Dakota. Shortly after Sturgis the Dakotas became the worst place on the planet for COVID. I believe though that it probably happened in at least 1000 other semi-rural counties around the country precisely because of Sturgis, thereby causing a lot of spread which didn’t get under control until Nov.

          What I consider “under control” is sub-exponential spread. Clearly without any effort to control it, spread is exponential. If you bring spread to below exponential levels, you’re obviously doing some control.

          Did Sturgis also cause major run-ups in more far-spread and urban areas, like Southern California, or Florida, or Texas? I don’t know there. My sense is those more urban areas had their own problems. If sturgis did affect those areas, it probably didn’t have as big an effect.

        • In reply to https://statmodeling.stat.columbia.edu/2021/02/06/how-to-incorporate-new-data-into-our-understanding-sturgis-rally-example/#comment-1701612 , where you claim that long-tails introduce a threshhold:

          My intuition says that due to linearity-of-expectation and law-of-large-numbers reasons, your claim is mathematically unsound.

          You say that

          >But if you get the numbers high enough the super spreading occurs and the average reproduction number goes from 0.6 to 3 because of that one to 3 events.

          But you confuse “low probability of super spreading” with “no super spreading” and “high probability of super spreading somewhere” with “high probability of super spreading everywhere”. A single person can be unlucky, become a super spreader and by your reasoning the growth now explodes. In the same way, a superspreader can get lucky, and have only non-super-spreader descendants.

          Let’s model this: Assume that the number of people infected by one person is given by a random variable X. It can have long tails or really any distribution you like. If you sum lots of Independent variables of Identical Distribution (henceforth iid) as X, the expected value of the sum is linear in X (we don’t actually need the independence assumption here). So for one step, the reproduction number (or at least its expectation) does not depend on how many people you have. With the independence assumption, we can look at further steps: if we assume that the process is memoryless, then the total number of currently infected people forms a Markov chain on the natural numbers. I haven’t worked with Markov chains in a while, but I’ll conjecture that under reasonable assumptions on the distribution of X, that the expected number of cases from any starting state is either infinite or not. This would disprove your “threshold” theory.

        • I put threshold in quotes because of course the probability of explosive growth in a community is continuous. But whether explosive growth does or doesn’t occur is a discrete outcome, and the probability though continuous is a nonlinear function of the number of various variables.

          I’m not talking about what happens in the country I’m talking about what happens in one particular town when 1 or 3 or 11 Sturgis riders show up and comparing that to 1 or 3 or 11 people who fly in wearing masks, and quarantine for 14 days… The latter people mostly spread to 0 others. Those they do spread to are similarly cautious often… And after 1 or 2 generations the spread has extincted. the Sturgis riders presumably have a much higher probability to spread initially and a much less cautious cohort of contacts. This is an assumption about their social and behavioral “phenotype”. Under such conditions the probability of extinction in short generation times plummets. Under this assumption, when say 11 Sturgis riders show up, in 10 days there are now 30 or 60 cases and the explosion is on.

          None of this is super controversial. Agent based models more accurately model extinctions than continuous models and they are well known to show this kind of effect.whats potentially controversial is the idea that Sturgis riders are more careless etc. But it’s not that controversial if you were reading the news and interviews and watching the videos out of sturgis

        • From the perspective of mathematical argument, remember that the number of people that an individual infects is necessarily an integer. Furthermore that integer is most frequently 0. The product of 0 and anything is 0.

          If you have a distribution over the nonnegative integers whose average is say 1.5 but 90th percentile is 0, multiplying 1.5^n suggests exponential growth. Let’s give an example distribution… the distribution where 90% chance of 0 and 10% chance of 15.

          But if you simulate say 10 people initially infected, and infecting others according to this distribution… you’ll find a lot of extinctions. To get “reliable” growth, you’ll have to start with a large number of initially infected people so you can get “unlucky” almost always, or you’ll need to skew the distribution towards the 15 by “spreading behavior” so that perhaps it’s only 50% chance of 0 and 50% chance of 15 for example.

          For example, in R, to simulate 10 people infecting according to the 90% = 0 and 10% = 15 distribution:

          > replicate(10, rbinom(1,1,.1)*15)
          [1] 0 0 0 0 0 0 0 0 0 0

          Hey we got “lucky” and none of those 10 spread.
          > replicate(10, rbinom(1,1,.1)*15)
          [1] 0 0 0 0 0 0 0 0 0 0
          > replicate(10, rbinom(1,1,.1)*15)
          [1] 0 0 0 0 0 0 0 0 0 0
          > replicate(10, rbinom(1,1,.1)*15)
          [1] 0 0 0 0 0 0 0 0 0 0
          > replicate(10, rbinom(1,1,.1)*15)
          [1] 0 0 0 0 0 0 0 15 15 0

          Turns out 4 different towns had no spread… the final town got 30 cases. Let’s chase the 30 cases:

          > replicate(30, rbinom(1,1,.1)*15)
          [1] 15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 15 0 15 0 0 0 0 0 0
          [26] 0 0 0 15 0

          now the town has 60 cases.

          > replicate(60, rbinom(1,1,.1)*15)
          [1] 0 0 15 0 0 0 0 0 0 15 0 15 15 0 15 0 15 0 0 15 0 0 0 15 0
          [26] 15 0 0 0 0 15 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 15
          [51] 0 0 0 0 0 0 0 0 0 0

          Now it has 165

          So extinction is common. If you can make the superspreading events very uncommon, you can cause avg reproduction number to drop below 1 rather easily. If there’s a cohort of people whose behavior makes them more likely superspreaders, then simply keeping them from getting sick or from doing their superspreading behaviors will rapidly bring the avg reproduction number below 1.

          The distribution of case spreading is not just a “Constant” one it depends on who gets infected. For example if a person is staying home alone all the time except grocery store visits. They get sick at a grocery store visit, and then don’t go shopping until they’re better… They have 0 spreading reliably.

          if you build up a population of a mix of “mobile, non-careful” people, a group of “mobile careful” people, and a group of “non-mobile, careful” people with reproduction number distributions for each… you’ll see that if you drop one or two infected mobile non-careful people into a town, you’ll often get extinction… But if you drop a few more in the probability of rapid extinction drops nonlinearly. You’ll get an explosion among the mobile non-careful people, leaking into the mobile careful people, and into the nonmobile people as well, until you burn out your ability to propagate reliably and start exponential decay. That burnout could easily take 5 to 7 generations, which is say 50 to 100 days, which is mid Aug to mid Nov, then you see concave downward cases per day, and have a couple more rounds and you start to see decaying… That’s Mid Jan… just like we saw

        • I don’t think you understand my argument. My point is that I don’t see why expected number of cases should grow super-linearly in the number of initially infected people. I understand that the probability of getting a super-spreader is lower with fewer people, what I don’t understand is why the effect on the expected value should be nonlinear. Let’s take a simple example – a random variable X that is zero with high probability and 1000 with probability p. We should agree that E[X] = 1000*p. We should also agree that E[sum of n iid X] = n*1000*p. Of course, we also agree that with low n the “sum of n iid X” has higher probability of being zero. However, I care more about the expected value, and that scales linearly with “n”.

          I don’t see why the expected value of covid-cases should scale more than linearly with the number of people infected. I don’t see why your reasoning so far shouldn’t apply to the random-variable X. You are confusing “high probability of being zero” with “expected value being small”.

  2. I’ve written critique threads on three of that group’s papers now, and was motivated to publish a small research note (ungated https://rexdouglass.github.io/DataScienceAndCovid19/DataScienceAndCOVID19.nb.html) . They’ve been using what I consider a broken approach over and over again to pump out results on specific events all year (between 3 and 6 last time I checked).

    The main problem is measurement- their outcome of interest is the change in the rate of infections on a given day in a given place R_ct, but they don’t have that information so they directly plug in change in confirmed cases as a proxy. There are ways to do that appropriately, but shoving them into a regression and star hunting isn’t it.

    To contrast it with the most careful work I know asking that kind of question, look at Kubinec et al.”Fear, Partisanship and the Spread of COVID-19 in the United States,” (https://osf.io/preprints/socarxiv/jp4wk). I discuss an older draft of their paper here (https://twitter.com/RexDouglass/status/1301208895961149442) where the punchline is you have to at a minimum incorporate both rates of testing and number of positives to even begin to make a guess at true number of infected. It’s incredibly hard, and until recently only possible at the state level.

    If you don’t do all that hard work you end up making two kinds of inferential mistakes.

    With Sturgis, they had a death rate that remained flat that they arbitrarily chose to completely ignore, and a rising confirmed cases rate that was some combination of rising testing and confirmed cases that they completely attributed only to infections. Their ridiculously large estimate should have been called an upper bound, if that, but instead it was presented as if it were the right answer in expectation. My discussion of that paper here (https://twitter.com/RexDouglass/status/1303379252742479872).

    Two other times they use the same model to make a different mistake for social protests and a Trump rally (my discussions here https://twitter.com/RexDouglass/status/1275186049455583232 and https://twitter.com/RexDouglass/status/1282708491656814594). In those cases they find a non-effect and declare those activities must have been safe and try to come up with a story why.

    Again, because they don’t have an actual epidemiological model, they incorrectly assert that absence of evidence is evidence of absence. What they needed to do was estimate the actual number of infected from testing and case data, show us the giant compatibility intervals that estimate has, and then give us the bound on the biggest effect they could have reasonably measured with coarse location level counts. Like trying to measure whether my child has a fever using the home’s thermostat. I suspect that bound would have been far too large to be meaningful, and giving people false certainty that their mobility was safe was unbelievably reckless.

    Our shop is working on a Bayesian estimate of infection rates over time that we hope will be able to definitively establish these critiques above and provide a public good that lowers the overhead cost of using rates of infection in county level analysis instead of bad proxies.

    • I don’t think it’s possible to back out very reliable estimates of most of these effects. There are several main reasons:

      1. The people getting tested are not a random sample of the population. Some will have been required to get tested by their employer, some because they want to travel, and probably (but who knows, really?) most because they feel some symptoms and want to know if they are actually sick with covid;

      2. The PCR tests mostly haven’t reported the replication cycle numbers, so who knows if a given positive result represents a large viral load or a tiny one, perhaps small enough not enough to matter;

      3. Testing methods differ from state to state, and sometimes have changed over time.

      In my view, some ways to check to see if a rise in reported cases is due to more testing or to more actual cases are:

      a. If the death rate changes more or less track the reported cases – with a suitable time delay which may vary from state to state – then we can have more confidence that the case count changes are “real”. The same would be true of hospitalization rates, but reliable data on that seem to have been hard to come by;

      b. Exponentially rising or falling cases will generally indicate some “real” effect, while a lower, more linear change is more likely to be a result of testing rates. By “exponentially rising”, I mean an actual straight line on semi-log plot, not some woolly metaphorical trend.

      Obviously none of these points can be definitively established, and efforts to quantify them can only ever be coarse estimates. That’s why I don’t have a lot of faith in very detailed analyses. If someone wants to work out the consequences of a very careful model, I say fine, more power to them. But when we look at actual data, we need to be very cautious about trying to wring too much out of them.

  3. Philosophically speaking, how much should one attribute to a single source, infections after a chain of transmissions? Surely most of the sturgis infections can be traced back to people in New York, and those back to Italy, and those to Wuhan. What fraction of the extra 266,796 cases—love the precision—would better be attributed to people’s actions post-sturgis?

    I could buy counterfactual value of the analysis, but even that feels like it dilutes with time. Try to estimate the difference in infections if there was a Europe travel ban in February. At some point it crosses from counterfactual analysis into alternate history fan faction.

  4. >I replied: So you’re saying the result is believable but that this is just one (noisily measured) data point so it doesn’t carry much information? That makes sense to me. But that’s different from the link in that twitter thread, which calls the claim “ridiculous.”

    I feel a bit weird right now because isn’t that exactly the issue with the replication crisis? Authors saying “this is entirely plausible, I guess the data only shows it if you massage it a bit but since it’s plausible we shouldn’t criticize the study” is suddenly not a big deal?

      • The article being discussed, http://ftp.iza.org/dp13670.pdf, has plenty of p-values in it.

        Can we now only criticize bad statistics if their result seems inplausible? Why bother with doing statistics at all if we can just take the plausible results, do bad statistics and call it ‘science’, but reject the bad statistics of the implausible results… Himmicanes are plausible to lots of people too.

    • I think the relationship to the replication crisis is interesting. Perhaps the veracity of the priors and the manner of falsifiability is important. This particular study is falsifiable in a way that provides evidence for existing and testable alternative hypotheses and so it would not put fundamental or foundational principles of infectious disease spread into question. I think it is also important that testing those alternative hypotheses could rely on independent experimental data (t-cell immunity, serological tests, genomic sequencing, etc).

      Epidemiology has traditionally been thought of as a mix of hard and soft sciences and I think there is probably some very interesting philosophy of science work to be done in a post-replication crisis and post-pan-academy-covid-analysis world.

  5. The controversy should be obvious. Why was this event the subject of covid-19 spreading investigation when all summer long we had BLM protests which have been spared similar investigation?

    • Chrisare:

      Why do you say that BLM protests were spared similar investigation? I don’t know this literature at all but a quick google search turned up this article from June 2020, “Black Lives Matter Protests and Risk Avoidance: The Case of Civil Unrest During a Pandemic.”

      • Their conclusion is kind of funny: they find that the net effect was probably positive because protests caused business closings, traffic disruptions, and non-participants preferred to stay at home to avoid catching a nasty virus or a stray bullet.

        • Carlos:

          Yes, I noticed that. I didn’t read the paper carefully and am neither endorsing or criticizing it; I was just sharing the link to counter the claim that the coronavirus-spreading effects of Black Lives Matter protests hadn’t been studied. Lots of people were talking about this issue at the time, so of course it got studied. I was not at all surprised to find a research paper on the topic in the first page of my google search.

      • Story time:

        “Event-study analyses provide **strong evidence** that net stay-at-home behavior increased following protest onset, **consistent with** the hypothesis that non-protesters shifted their activity in response to the perceived heightened risk of contagion and protest-related violence. ”

        my **

        The evidence is **strong** that net stay-at-home behavior increased; but that the size of that increase was so big it wasn’t worth mentioning :) And it’s only **consistent with** the idea that people shifted their behavior in response to protests. Not to mention 500 other hypotheses that it could be “consistent with”.

        Not that I care about these conclusions. There were no major outbreaks in cities with large BLM protests so we already know they weren’t major infectious events. I just think it’s amusing to deconstruct research papers. Writing an abstract that sounds like it says something is meaningful is an art form. It’s interesting because the language describing the conclusions is usually very precise – far more precise than the data – saying all that can be said and not one thing more. That means what isn’t said is as important as what is said, which is an excellent clue to the quality of the work that’s coming up.

    • To the extent you’re right, I can think of a few reasons for differences in criticism: 1) The BLM protests came on the heels of a wave of anti-lockdown protests that consisted by and large of unmasked protesters. The media/scientists were (consciously or not) contrasting BLM protests devoted to ending an old and persistent injustice, with protests entirely devoted to ending COVID precautions. 2) I recall seeing a lot of pics and vids of BLM protesters in masks, and many social distancing, so at least there was some effort to be responsible in terms of super-spreading. 3) I also saw a lot of vids of white guys with rifles menacing peaceful BLM marchers, which tends to make the latter seem awfully sympathetic. 4) BLM protesters were forced to weigh a grave threat to their lives and their communities from, on the one hand, the virus, and on the other hand, institutional racism. For many, it was not an easy choice. But the level of attention from white America, and their overall momentum, were literally a once-in-many-lifetimes opportunity for change. 5) In contrast, the Sturgis rally was a music festival, a frivolity held and attended in defiance of public safety. When you compare the cost-benefit for each, BLM is at least debatable, while Sturgis was all cost to society and virtually devoid of benefits.

      • “I recall seeing a lot of pics and vids of BLM protesters in masks, and many social distancing, so at least there was some effort to be responsible in terms of super-spreading.”

        Where I live, at least, there was consistent messaging from those organizing BLM protests that people should wear masks and attempt to maintain social distancing. I think this was pretty much true nationally. In large-scale events the latter was impossible in some cases, but in my anecdotal, local experience attendees at rallies and marches were almost all wearing masks.

        The messaging before Sturgis regarding covid was … different.

      • Many communities depend on summer tourism and events. The communities that lost tourist income in 2020 are suffering financial losses that are having many negative externalities, as I learned to say in micro-economics 101. I am not defending how the festival operated. Rather, calling it a frivolity while deeming all BLM protestors as virtuous risk-taking advocates has no value.

        I like the work the authors did. First of all they explained what happens at Sturgis–it is a flow of people over more than 10 days. The media never explained this and claimed an impact that seemed low for the reported 500,000. Cell phone data is a marker but it seems to lag with purchasing in the town. I think we should appreciate the results on a descriptive level as it is difficult data.

        The hard part about COVID is that the spread varies so much between clusters. Ten Sturgis visitors might have an impact in one county but not in another. I think we should take this work for its descriptive value and as Andrew said, treat this as one point of evidence.

  6. “provocative” here simply means incompatible with (1) the priors of many people and (2) the politically-charged policies based on those priors. To those for whom the study is compatible with their priors, it’s merely another piece of annoyingly (or appropriately, since it’s just one study) ineffective ammunition.

Leave a Reply to confused Cancel reply

Your email address will not be published. Required fields are marked *