Philippe Lemoine writes:
I [Lemoine] just published a blog post in which I explore what impact population structure might have on the transmission of an infectious disease such as COVID-19, which I thought might be of interest to you and your readers. It’s admittedly speculative, but I like to think it’s the kind of speculation that might be fruitful. Perhaps of particular interest to you is my discussion of how, if the population has the sort of structure my simulations assume, it would bias the estimates of causal effects of interventions. This illustrates a point I made before, such as in my discussion of Chernozhukov et al. (2021), namely that any study that purports to estimate the causal effect of interventions must — implicitly or explicitly — assume a model of the transmission process, which makes this tricky because I don’t think we understand it very well. My hope is that it will encourage more discussion of the effect population structure might have on transmission, a topic which I think has been under-explored, although other people have mentioned the sort of possibility I explore in my post before. I’m copying the summary of the post below.
– Standard epidemiological models predict that, in the absence of behavioral changes, the epidemic should continue to grow until herd immunity has been reached and the dynamic of the epidemic is determined by people’s behavior.
– However, during the COVID-19 pandemic, there have been plenty of cases where the effective reproduction number of the pandemic underwent large fluctuations that, as far as we can tell, can’t be explained by behavioral changes.
– While everybody admits that other factors, such as meteorological variables, can also affect transmission, it doesn’t look as though they can explain the large fluctuations of the effective reproduction number that often took place in the absence of any behavioral changes.
– I argue that, while standard epidemiological models, which assume a homogeneous or quasi-homogeneous mixing population, can’t make sense of those fluctuations, they can be explained by population structure.
– I show with simulations that, if the population can be divided into networks of quasi-homogeneous mixing populations that are internally well-connected but only loosely connected to each other, the effective reproduction number can undergo large fluctuations even in the absence of behavioral changes.
– I argue that, while there is no evidence that can bear directly on this hypothesis, it could explain several phenomena beyond the cyclical nature of the pandemic and the disconnect between transmission and behavior (why the transmission advantage of variants is so variable, why waves are correlated across regions, why even places with a high prevalence of immunity can experience large waves) that are difficult to explain within the traditional modeling framework.
– If the population has that kind of structure, then some of the quantities we have been obsessing over during the pandemic, such as the effective reproduction number and the herd immunity threshold, are essentially meaningless at the aggregate level.
– Moreover, in the presence of complex population structure, the methods that have been used to estimate the impact of non-pharmaceutical interventions are totally unreliable. Thus, even if this hypothesis turned out to be false, we should regard many widespread claims about the pandemic with the utmost suspicion since we have good reasons to think it might be true.
– I conclude that we should try to find data about the characteristics of the networks on which the virus is spreading and make sure that we have such data when the next pandemic hits so that modeling can properly take population structure into account.
I agree with Lemoine that we don’t understand well what is going on with covid, or with epidemics more generally. I agree, and, as many people have recognized, there are several difficulties here, including data problems (most notably, not knowing who has covid or even the rates of exposure etc. among different groups); gaps in our scientific understanding regarding modes of transmission, mutations, etc.; and, as Trisha Greenhalgh has discussed, a lack of integration of data analysis with substantive theory.
All these are concerns, even without getting to the problems of overconfident public health authorities, turf-protecting academic or quasi-academic organizations, ignorant-but-well-connected pundits, idiotic government officials, covid deniers, and trolls. It’s easy to focus on all the bad guys out there, but even in world where people are acting with intelligence, common sense, and good faith, we’d have big gaps in our understanding.
Lemoine makes the point that the spread of coronavirus along the social network represents another important area of uncertainty in our understanding. That makes sense, and I like that he approaches this problem using simulation. The one thing I don’t really buy—but maybe it doesn’t matter for his simulation—is Lemoine’s statement that fluctuations in the epidemic’s spread “as far as we can tell, can’t be explained by behavioral changes.” I mean, sure, we can’t tell, but behaviors change a lot, and it seems clear that even small changes in behavior can have big effects in transmission. The reason this might not matter so much in the modeling is that it can be hard to distinguish between a person changing his or her behavior over time, or a correlation of different people’s behaviors with their positions in the transmission network. Either way, you have variation in behavior and susceptibility that is interacting with the spread of the disease.
In his post, Lemoine gives several of examples of countries and states where the recorded number of infections went up for no apparent reason, or where you might expect it to have increased exponentially but it didn’t. One way to think about this is to suppose the epidemic is moving through different parts of the network and reaching pockets where it will travel faster or slower. As noted above, this could be explained my some mixture of variation across people and variation over time (that is, changing behaviors). It makes sense that we shouldn’t try to explain this behavior using the crude categories of exponential growth and herd immunity. I’m not sure where this leads us going forward, but in any case I like this approach of looking carefully at data, not just to fit models but to uncover anomalies that aren’t explained by existing models.
I’m pretty sure metereology influences behaviour.
People are more likely to take the bus than the bicycle when it rains, less likely to sit outdoors, etc.
And we know the seasonality of influenza isn’t due to consciozs behaviour changes.
The hitting of pockets was obvious in a low-prevalence situafion, when you could see a care home outbreak or a meat factory make a bump in the numbers. We’ve had “superspreader” events for quite a while, which typically aren’t “1 guy infects a group” but rather “virus enters an interconnected group and spreads rapidly”.
With high prevalence and endemic virus, I doubt these boundaries are observable. The “Diamond Princess” seems to be proof of that.
And I’d want to not even be speculative about statements like “this demographic group are the plague bearers”, for fear of what kind of populists would gladly run with that.
Lemoine’s conceptual model sounds a lot like Richard Levins’s metapopulation model.
On the importance of behavior, I think the low incidence of flu last winter speaks pretty louldly.
I don’t know Richard Levins’s model specifically, but as I note in the post, my approach is similar to the metapopulation approach, although there are differences of implementation and interpretation that I only briefly discussed in the post. As I note in another comment, I don’t deny that behavior matters (indeed I have been complaining that people were ignoring the role of voluntary behavioral changes for months), my claim is only that the effective reproduction number sometimes undergoes large fluctuations that don’t seem to be responses to behavioral changes.
The Levins paper is a classic: Levins, R. (1969), “Some demographic and genetic consequences of environmental heterogeneity for biological control”, Bulletin of the Entomological Society of America, 15 (3): 237–240, doi:10.1093/besa/15.3.237
I agree about variation in the effective reproduction number. Why on Earth did cases go down so much in Florida, for example.
What is this based on?
As I explain in the post, it’s based on the literature on the effects of meteorological variables on transmission, which did find some connection but in my opinion not strong enough to explain the kind of sudden fluctuations I find puzzling. Perhaps their real effects is stronger and we just can’t pick it up due to bad data or methods, but overall I think it’s unlikely to be a major factor, though I expect it will become dominant as the virus becomes endemic. See the post for more details on this.
Phillipe –
Depending on how you define “major factor,” perhaps you’d like to explain what it is about this analysis that would cause you to dismiss their conclusions?
https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2021GH000413
Just plot degree days vs cases in the US, the correlation is obvious. The cases rise when it gets uncomfortably hot or cold.
https://www.eia.gov/outlooks/steo/data/browser/#/?v=28&f=M&s=&start=199701&end=202012&linechart=ZWCDPUS~ZWHDPUS&maptype=0&ctype=linechart&map=&id=
@Phil, still havent done it.
Anon, you’re overthinking it for sure! This is a blog! Use the data you already have in hand, stick in a few plots and a few paragraphs, and that’s it. If you feel unsure of how to write it, you can do a draft and I’ll spend ten minutes editing, but you’ll probably prefer to do it yourself.
“The Best is the enemy of The Good.”
@Phil
I am not kidding. I keep trying to sit at a computer to do it but then go outside instead.
It is kind of a tedious task because I gotta combine a few different datasets. But I haven’t been getting any work done really due to this weather.
Thanks for sharing the post!
To be clear, what I’m saying is that some fluctuations in the epidemic’s spread can’t be explained by behavioral changes, not that behavioral changes don’t affect the epidemic’s spread. (Indeed, in my previous work on the pandemic, I have complained a lot that many epidemiological models used to estimate the effects of non-pharmaceutical interventions didn’t take into account the impact of voluntary behavioral changes.) As I explain in the post, this claim is based on the fact that, after the first wave, mobility data predict the effective reproduction number very poorly, but for what it’s worth I also think it’s supported by anecdotal evidence. However, I also acknowledge that mobility data may be a poor proxy of the relevant behavioral changes, but still this disconnect between mobility and R should encourage us to look at other potential explanations and that’s what I’m trying to do in this post.
One paper I briefly discuss in the post that provides some evidence that, with better data on behavior, we could find a stronger connection with the epidemic’s spread is Rüdiger et al. (2021). It uses a very rich dataset based on cell phone GPS tracking data to reconstruct a contact network and, instead of just using the average number of contacts, it also computes another index that is motivated by theoretical work on the spread of infectious diseases on complex network and shows that it predicts the effective reproduction number much better. As I note in my post, this paper has been completely ignored, which is a pity because I think it’s very important work. I hope my post will also help to bring more attention to it.
Perhaps you could put the point more effectively as: it’s not plausible that all the apparent changes in the reproduction number are merely the result of behavioral changes.
Sure, some of them might be but you can’t explain why so many different countries with different traditions, responses etc.. all seemed to be able to bring the reproductive rate low enough that they avoided burning through most of the population but yet virtually none of them (excepting NZ and a few islands) could bring it below 1 for long enough to actually get close to containing the virus.
Yes, I think it’s a good way of putting the point, and this sort of considerations is definitely what led me to take seriously the possibility that population structure could play an important role. Again, I certainly don’t deny that behavior also matters, I just think it leaves a lot of unexplained variation and I don’t think it’s plausible that the other factors people have brought up to explain the dynamic of the epidemic can explain more than a relatively small part of that residual variation. I think a lot of people have had the same thought, but as you, I was surprised that epidemiologists have not been taking it seriously in their work on the COVID-19 pandemic.
Just let me add that I’m glad to see someone taking an interest in this. I’ve been wondering why the virus reproduction rate seemed to behave so oddly and figured it was an issue with population structure but been frustrated by the lack of researchers taking this seriously.
I mean we do have datasources that could help us better model people’s social connections and I’d think that better modeling the population structure could really improve models.
Phillipe –
> To be clear, what I’m saying is that some fluctuations in the epidemic’s spread can’t be explained by behavioral changes, not that behavioral changes don’t affect the epidemic’s spread.
Of course that’s true. I don’t recall seeing anyone dispute it. At the very basic level the properties of the different variants in themselves likely have some sort of effect, as obviously do other features like number of people per household or ability to work from home or testing and isolating or baseline health across different populations etc.
Lemoine’s blog post is entitled:
“Have we been thinking about the pandemic wrong? The effect of population structure on transmission”
Who’s thinking wrong? If you put “epidemiology population structure” into google scholar you get 3+ million hits. There are already lots of papers on Covid 19 and the exact point Lemoine is making, like this one:
https://www.medrxiv.org/content/10.1101/2020.04.27.20081893v1
Or this one:
https://nicholaslewis.org/why-herd-immunity-to-covid-19-is-reached-much-earlier-than-thought/
In the linked post Lemoine writes “In this post, I propose that such fluctuations could be the result of population structure…” and then proceeds to describe an approach that looks an awful lot like Nic Lewis’ work.
As I note in the post, there is a vast theoretical literature on the spread of infectious diseases on complex networks, but as far as I can tell, it has played virtually no role in applied work during the pandemics. In practice, people typically divide the population into age groups with a contact matrix for the contact rates between the groups, but otherwise make the homogeneous mixing assumption within age groups.
In my approach, this amounts to a very simple network of a handful of subpopulations — corresponding to the age groups — in which each node is connected to every other with limited variation in connectivity, so there is no community structure and the epidemic has a behavior that is qualitatively very similar to that of a simple homogeneous population mixing model.
As I explain in the addendum on the relationship between the hypothesis I explore in this post and the previous debate on the impact of heterogeneities, the kind of heterogeneity that I discuss in this post is not at all the same as that people were arguing about last year. In particular, this applies to the work of both Gabriela Gomes and Nic Lewis, which you mention in your comment and is very interesting but just isn’t the same thing as what I’m doing in this post. See the addendum for more details on why it’s not the same thing.
Matt –
Yes.
A lot of people have been modeling the effect of heterogeneity in transmission across populations, in comparison to homogeneity, since the beginning of the pandemic (see Gabriella Gomes for example – that she was largely wrong in predicting an early “herd immunity” doesn’t make her wrong to consider the effect of heterogeneity).
And before the pandemic started there was considerable analysis as well.
Pehepas I’m missing something (I’d be happy to be corrected) but I have to wonder if this is an example of Phillipe focusing the discussion on Phillipe.
Oh, and BTW, if I recall correctly Nic’s work was largely in reponse to someone directing him to Gabriel’s work.
That he was horribly wrong in his modeling also doesn’t in any way detract from the sound logic of trying to consider heterogeneity in transmission across populations.
But again, there is a large body of work of other epidemiological modelers addressing this issue.
Did Phillipe even bother to consult the literature before characterizing what’s out there? Or maybe he’s differentiating his analysis from that of others in some way I don’t understand?
As I noted in response to Matt’s comment, and explained in some detail in an addendum to my post specifically about that issue, the kind of heterogeneity I explore in this post is totally different from the kinds of heterogeneity people discussed last year when there was a debate about this, including the heterogeneity in susceptibility that Gabriela Gomes and her team modeled in their work. In general, I try to read a paper before I comment on it, but I know that other people sometimes adopt a different approach.
Phillipe –
I didn’t see your reponse to Matt before writing my comment.
I’m going to take a pass on reading your actual post (along with the addendum) as in the past, imo, you’ve responded quite poorly to critiques when I’ve done so.
So instead I’m responding to a excerpt I’ve seen here, which seems false at face value. It’s a kind of statement where the veracity, imo, shouldn’t hinge on an addendum.
What I’d rather read, if indeed you’ve developed a unique and uniquely useful way of predicting the behavior of the pandemic as you have claimed, is a peer-reviewed analysis you’ve written and submitted to the critiques of people who are intimately familiar with the previous literature on modeling the effects of heterogeneity (age-stratified or otherwise) on infectious disease pandemic trajectories.
I am only very superficially familiar with that literature, having seen Nic’s blog posts and Gabriela’s articles and at that time doing a bit of Googling on the issue. At the time I was looking at their work, it was quite clear that there were, in fact, MANY other modelers who were considering the effects of heterogeneity and questioning the validity of models based on an assumption of homogeneity.
That Nic’s and Gabriela’s modeling seemed to have been quite wrong in the sense of their conclusions about when “herd immunity” would be reached (and I wrote many comments on Nic’s blog posts critical of his lack of inclusion of behavioral differences and other vectors of heterogeneity) is certainly interesting but doesn’t seem to me to put into doubt the necessity of considering factors of heterogeneity. (I should not that Gabriella argues that she wasn’t wrong – because her reference to “herd immunity” is based on a definition where populations can cross into and out of “herd immunity” depending on various factors).
Hopefully, you will submit your analysis to extensive critique from those who are well positioned to offer one. If indeed, you’ve revolutionized the methodology of pandemic modeling, you iwwe it to society to do so.
Joshua, your comment seems a bit odd.
1. I don’t see that Lemoine claims to revolutionize epi modeling in the post. He seems to make a point about assumptions in epi models and then do a bunch of simulations.
2. Reading his blog post will allow you to make informed comments. As many people (like myself) read these comments, they certainly don’t need to be aimed with only a response from Lemoine in mind.
3. Peer review?? Isn’t that what he has done by writing a public blog post and then sending the link to Andrew Gelman to further publicize it at a rather well-read blog with rather critical commenters? Why should this blog post go to a peer review journal any more than one of Gelman’s posts?…this seems like a decent format for starters at least. (and peer review at journals seems rather lacking in quality often times; writing a blog first and seeing responses seems much more efficient to me)
jd –
> Joshua, your comment seems a bit odd.
Could be.
> 1. I don’t see that Lemoine claims to revolutionize epi modeling in the post. He seems to make a point about assumptions in epi models and then do a bunch of simulations.
Perhaps a fair point, but in the title he asks whether “we” have been thinking about the pandemic all wrong. That could go into the category of “just asking questions,” but I’ve become very cynical about that form of engagement. It just seems totally unnecessary to me. Just write an analysis where you propose a particular methodology. He also says that his modeling is distinct from what has come before. Maybe it is, but I think such a statement should come with an elaborated literature review. I’m not in a position to evaluate that claim – so what is the value to me of a blog post based on such a claim if I can’t see that claim detailed thoroughly? If you don’t provide a means for evaluating the claim, imo, you should just leave it out.
> 2. Reading his blog post will allow you to make informed comments. As many people (like myself) read these comments, they certainly don’t need to be aimed with only a response from Lemoine in mind.
I’m more interested in the discussion here than in reading his blog post (not the least because I likely wouldn’t be able to evaluate the technical details). If I get something wrong because I didn’t read his post I’m happy to be corrected. I get that might be “unfair” in some sense. But I’m sure there’s no harm that will come out of me being wrong, and reading his posts and commenting on them in the past didn’t net anything useful for me. Commenting here on his blog posts has.
> 3. Peer review?? Isn’t that what he has done by writing a public blog post and then sending the link to Andrew Gelman to further publicize it at a rather well-read blog with rather critical commenters? Why should this blog post go to a peer review journal any more than one of Gelman’s posts?…this seems like a decent format for starters at least. (and peer review at journals seems rather lacking in quality often times; writing a blog first and seeing responses seems much more efficient to me)
Peer review (of the journal type) is no foolproof mechanism – for sure. But I think that if someone’s going to claim that they’re doing epi modeling in a unique fashion with uniquely useful results, that work should be subjected to review from the people who are best in the position to evaluate such a claim. That would be true with one of Gelman’s posts as well if they contained a similar claim. On top of that, my assumption is that one of Andrew’s blog posts on a topic where he’d make a claim of unique value would likely be read by at least some % of the people who are best equipped to evaluate his claim. I would have no such confidence regarding Philippe’s blog posts either on his blog or as referenced here.
That isn’t to say, in any way, that I think there’s zero value in Philppe doing his simulation or putting up his blog post. Not at all. It’s just that I think the value is limited, and in a large part the value is limited, at least in my eyes, because it hasn’t been subjected to a critique, specifically, from those best situated to evaluate some of his assertions. I think the value of his analysis would only be enhanced by such a process.
Replying to @Joshua:
>But I think that if someone’s going to claim that they’re doing epi modeling in a unique fashion with uniquely useful results, that work should be subjected to review from the people who are best in the position to evaluate such a claim
The things Philippe are claiming havehopefully been obvious to anyone who knows what the SIR model is, and what a graph is I’ve been waiting to read his post for a while not because it offers anything particularly creative – “SIR models assume the complete graph when modelling transmission, but this is obviously nonsense” has been a pet peeve of mine for a while, (proof: https://statmodeling.stat.columbia.edu/2021/03/05/alan-sokal-on-exponential-growth-and-coronavirus-rebound/#comment-1744401) and I’m sure I’m not alone . But Philippe has gone and written an article which says this, and this is valuable because it makes this point plateable to those who read articles. Yes, it isn’t “peer reviewed”, but “peer review” is gatekeeping nonsense; I say this as someone who has reviewed manuscripts in the past.
To go on the offense a bit: Even economists think epidemiologists have shoddy models (see e.g. https://marginalrevolution.com/marginalrevolution/2020/04/what-does-this-economist-think-of-epidemiology.html), so I don’t see any reason to think that they are uniquely qualified to review Philippe’s work. If you’re an academic who cares about being “first”, maybe it is important to cite all the right people, but this vain nonsense shouldn’t be encouraged. Even I understand it, calling for “peer review” is silly. Philippe is definitely not the first to have made this point, but it seems like he is the first with this point to be featured on this blog; which is the bigger achievement?
>Pehepas I’m missing something (I’d be happy to be corrected) but I have to wonder if this is an example of Phillipe focusing the discussion on Phillipe.
>I’m going to take a pass on reading your actual post (along with the addendum) as in the past, imo, you’ve responded quite poorly to critiques when I’ve done so.
I wonder why…
Matty –
> I wonder why…
Yes, well chicken and egg, my friend, chicken and egg.
My view in philippe’s reaction to critique wasn’t just spontaneously generated.
Now you may interpret matters differently but I’ll add that you and I have differed in our views on many issues and I’ve never said something similar about your response to critiques.
If you feel “I wonder why” sums it up, pleass pint me to a comment I made where I responded to you in bad faith so I can be accountable.
Or perhaps you have ahriey why I’d respond to you differently?
Every time I publish something, you make uninformed comments that show you haven’t actually read it, so I don’t want to waste my time replying to you when everything you say either misrepresent my argument or is explicitly addressed in the post. This isn’t new, you have done this several times before, so please don’t pretend that you don’t want to read it because I have responded poorly in the past. It’s the other way around: I responded poorly because, then as now, it was obvious that you hadn’t read what you were criticizing and I have no interest in debating with someone who doesn’t even bother to read what he is criticizing.
You have just demonstrated that once again. I just explained to you that the kind of heterogeneity I discuss in this post is completely different from the kind of heterogeneity that people were debating last year, and pointed out to you that I even wrote an addendum specifically to explain the relationship between the hypothesis I explore in this post and previous work on the impact of heterogeneity in contact/susceptibility, but your reaction is to repeat that many other modelers have considered the effect of heterogeneity. It’s like talking to a wall.
Anyway, I never claimed to have “revolutionized the methodology of pandemic modeling”, you just made that up. I’m sure that plenty of other people have had similar ideas, indeed one commenter above said that he’d had been thinking about something like that for the same kind of reason I did, but as far as I know nobody had tried to model this and discuss how it could be relevant to the COVID-19 pandemic. I thought it was interesting and wanted to make people think about it, so I wrote this post and shared it with people who I think might be interested.
If you don’t think it’s interesting, this is perfectly fine, but I don’t see what purpose making uninformed comments suggesting that I didn’t even bother checking the literature on the pandemic and that my claim that the role of population structure in the dynamic of the pandemic had been under-explored was just “an example of Phillipe focusing the discussion on Phillipe” (not even spelling my name correctly) serves. And now I will just proceed to ignore you again.
Philippe –
A longer response, mis-nested below, is stuck in moderation. Not that you particularly care. But anyway, as for this specific aspect:
> Anyway, I never claimed to have “revolutionized the methodology of pandemic modeling”, you just made that up….but as far as I know nobody had tried to model this and discuss how it could be relevant to the COVID-19 pandemic.
No, you didn’t say “I have revolutionized the methodology of pandemic modeling.” But you have indicated that you have found a key to explain errors in how pandemic modeling has been approached previously, and corrected for those errors in application through your simulation with COVID 19.
That certainly seems to me like a claim that you’ve revolutionized pandemic modeling – at least in theory, if not that you’ve done so in a way that the convinces the entire field to change and follow your lead to greater accuracy in the future.
Not that anyone’s asking for my opinion, but FWIW (which is exactly what you paid for it), I think there’s just an inherent problem with the modeling of the pandemic trajectory.
The rate at which the virus transmits is necessarily downstream of many different factors: the biomechanics of the different strains, baseline health status of populations, factors like number of people per household, the level of mask-wearing, the effects of weather/climate on the behavior of aerosolized particles, behaviors that are a function of weather/climate (like in Sweden, traveling to vacation homes in remote areas), ventilation, vaccination levels, average age, etc. So in a sense you should just be able to look at a reproduction rate and consider it to be a consolidation of all those factors, be they as they may be. And so you should be able to predict the trajectory based on the reproduction rate (and related trends). But ALL of those underlying factors either change depending on the conditions on the ground (which could be political), or feedback from the reproduction level itself, or are stratified significantly (and can’t be accurately averaged) across any meaningfully sized population.
I suppose statisticians have ways of dealing with all of this, but my own personal take is that maybe we shouldn’t be basing policies on modeling. Maybe the best we can do is get a ballpark sense of worst-case scenarios, and a rough common sense idea of what we can do to prevent those, and a rough common sense idea of what the associated costs/and benefits of the potential solutions might be. Yuck. That sounds so unscientific!
Look at the relatively binary and simple issue related to whether or not it’s a good idea to shut down travel in response to Omicron. I’ve seen smart, knowledgeable people absolutely convinced on both sides as to what’s the best thing to do. My own reaction was initially that it’s best to err on the side of caution, that it’s better to act too soon than too late, and just shut it down. But now I really have no clear opinion – given that it’s probably ALWAYS too late to shut travel down by the time you’ve discovered a variant of concern and given that effectively punishing countries where variants are discovered could well be counterproductive in the long run.
What a mess.
Given the expected benefit of a shutdown, I think it’s pretty clearly a bad idea. I mean, let people not travel if they don’t want to, maybe do more testing requirements to board planes… but a total travel ban. That should be at most a few days to a week long to prepare the new testing rules or such.
The fact is we should assume Omicron is spreading in most populated countries already.
> maybe we shouldn’t be basing policies on modeling …
An astute point made by Youyang Gu is that correctly explaining past variation is totally equivalent to predicting future variation. So given that basically all forward-looking predictions evaluated in a true out of sample setting appear to have zero predictive power after two weeks — i.e. when the linear trend fades — it’s fair to conclude existing “explanatory” models are worthless.
Basically: if you want to convince others your model is anything besides epicycles, make a meaningful novel forward prediction and observe it come true.
(this isn’t intended as a criticism of Lemoine specifically. institutional modelers (IHME, etc.) and policymakers are far more guilty of over-indexing to meaningless models).
To be clear, I share this skepticism of modeling as a policy tool (in this case), so I don’t take it as a criticism at all since I take that to be one of the upshots of my post. Indeed, a point I make in the post is that, in the presence of community structure in the network, except in the short-run (because it has some inertia), it’s probably impossible to forecast the evolution of the epidemic.
> my own personal take is that maybe we shouldn’t be basing policies on modeling.
I dont agree here
> Maybe the best we can do is get a ballpark sense of worst-case scenarios, and a rough common sense idea of what we can do to prevent those, and a rough common sense idea of what the associated costs/and benefits of the potential solutions might be.
I do agree here.
Acting on ballpark estimates are better than acting on no-idea-at-all, and modelling is a good way to get ballpark estimates (i.e. modelling supports common sense)
Mikhail:
Yes. Modeling is not just about getting the answer, or even about getting the answer given assumptions. It’s more about the path between assumptions and conclusions. Modeling (or, mathematics more generally) won’t in general give us the right answer to an applied problem, but it can reveal incoherence in our assumptions and reasoning. I’m sure Charles Peirce would agree. (Just throwing that last bit in to see if Keith is reading this far into the comment thread.)
Mikhail –
Yes. Of course you are right. I got carried away..
There is some (anecdotal) evidence of this kind of low mixing in populations, see e.g. https://slatestarcodex.com/2014/10/16/five-case-studies-on-politicization/ .
But it seems like everyone here is ignoring Philippe’s main post, which is that he demonstrates that lack of a good model can systematically bias scientific results towards results that do not hold up. The fact that a physically plausible model results in completely wrong estimation of epidemiological parameters does not seem to be widely appreciated.
So sorry, wrong Slate Star Codex post. I meant to post https://slatestarcodex.com/2014/09/30/i-can-tolerate-anything-except-the-outgroup/, more specifically the section reading:
>There are certain theories of dark matter where it barely interacts with the regular world at all, such that we could have a dark matter planet exactly co-incident with Earth and never know. Maybe dark matter people are walking all around us and through us, maybe my house is in the Times Square of a great dark matter city, maybe a few meters away from me a dark matter blogger is writing on his dark matter computer about how weird it would be if there was a light matter person he couldn’t see right next to him.
> This is sort of how I feel about conservatives.
Matty:
I can appreciate that the post you link to was written with heartfelt sincerity. I also think he got some things wrong.
For example, he writes, “So what makes an outgroup? Proximity plus small differences.” But lots of people express hostility to outgroups that are neither proximate nor similar to them. For example people living far from the border who are angry about immigrants from faraway places. I’m not saying it’s wrong for people to feel this way—ultimately, these are policy questions—I just can’t see how Alexander can say that an outgroup needs proximity plus small differences. Lots of outgroups don’t fit that description.
And he writes, “According to Gallup polls, about 46% of Americans are creationists. Not just in the sense of believing God helped guide evolution. I mean they think evolution is a vile atheist lie and God created humans exactly as they exist right now.” I guess we’d have to do some careful survey research to check this, but I have no reason to think that all or even most people who believe that God created humans “in their present form within roughly the past 10,000 years” (that’s the wording used in the Gallup poll) “think evolution is a vile atheist lie.”
Overall, I’d say that I see where he’s coming from, but I think he’s overstating things.
>I guess we’d have to do some careful survey research to check this, but I have no reason to think that all or even most people who believe that God created humans “in their present form within roughly the past 10,000 years” (that’s the wording used in the Gallup poll) “think evolution is a vile atheist lie.”
Perhaps the phrasing is a bit exaggerated, but I think you and I live in a bubble that very strongly selects for certain ideas. I mean how many “unreasonable” creationists do you have 15-minute discussions with regularly (pick some definition of “unreasonable”)? If covid-spread concentrates amongst people who hold 15-minute discussions – and I believe UK track&trace data suggests that even for symptomatic cases, only about 1/10 close contacts end up getting covid (see https://www.imperial.ac.uk/news/209673/covid-19-spread-different-social-settings-imperial/) – it’s not a stretch to assume that self-selected bubbles play a huge role in covid-dynamics.
Funnily enough, the post I *actually* meant to link to was this one, also from Slate Star Codex, part (III) of https://slatestarcodex.com/2017/10/02/different-worlds/ . There are several further examples of such “bubbles” in that post.
>For example people living far from the border who are angry about immigrants from faraway places.
This is really a discussion orthogonal to the point, but I’d like to point out that people are not angry about immigrants in faraway places, but about immigrants who come to their country from faraway places. These are typically selected for liking the place they want to immigrate to, so I think “proximity + small differences” fits here too… But I do agree that this model is not perfect, though I think it does make sense that the people one feels like one is competing with most are typically those that are similar.
Matty:
Yes, I was thinking about people who live far from the border. I remember when I lived in California in the 1990s, talking with a cousin from Chicago who was super annoyed at all the Mexicans coming into California and taking all the spaces in the public schools. My cousin had neither proximity nor similarity toward these immigrants—but he was thinking a lot about them. That makes sense: my cousin had policy concerns. There’s no reason you can’t get angry at people who live far away and are different from you. I just think that, in writing his post, Alexander got carried away and made some universal statements that probably sounded good when he wrote them but don’t really hold up in the light of day.
>Alexander got carried away and made some universal statements that probably sounded good when he wrote them but don’t really hold up in the light of day.
Maybe, but I do think the point he makes is interesting and in the category of ideas where a single counterexample doesn’t doom the theory.
Matt –
I really liked the post you linked to in error (in particular I think that Alexander has a good angle on the climate wars).
Reading his post, I started to think about the interesting juxtaposition of the left/right divide on Ebola quarantining with the left/right divide on quarantining for COVID. The orientation of the two sides on the tension between violation of basic rights on the one hand versus protecting the safety of our community on the other have pretty much reversed from then to now. IMO, that’s hard to reconcile with the view that these types of polarization are about rights or values, IMO. Seems to me it makes it rather clear that views on rights and values as they map onto particular contexts are more a function of ideological/identity orientation (and division) than an explanation for it.
Philippe –
Needless to say, I think you have mischaracterized our previous encounters, where in my view you have made claims that weren’t substantiated and then doubled down to say they were substantiated when they weren’t.
We can go back and forth on this forever. It would be pointless.
So I’m content for you to ignore me
However, I actually did just spend a couple of minutes looking at your post and here’s what I came across right off the bat:
The COVID-19 pandemic has been ongoing for more than one year and a half now, but we still don’t understand its dynamic well. What is perhaps more surprising, however, is that the fact that we don’t understand it well is not more widely acknowledged. …
A totally unqualified statement made as fact. From what I’ve seen, it’s been widely acknowledged that there’s a lot of uncertainty. Now perhaps if you actually verified your statement in some way, as someone might be expected to do in a more serious treatment, I could be convinced that your unqualified statement of opinion as fact was indeed correct. But you didn’t even bother to try.
… people have been continuously surprised…
Well, I do agree that people have been surprised, but given that this is a largely unprecedented even with a novel virus, I find that amount of surprise to be unsurprising.
…For the most part, governments continue to rely on projections based on models that have systematically proved to be massively unreliable,…
Again, totally unqualified. What does “rely on” mean there? Governments have had to make decisions on the fly based on the best evidence they’ve had available. They have “relied” on the best evidence they’ve had available. What else would you expect for them to do?
And what does “massively unreliable” mean, exactly? All models are wrong. Any expectation otherwise would be totally unrealistic. Massively unreliable is an inherently subjective assessment here. You could try to quantify that kind of statement so one could get a foothold on the subjectivity involved, but you don’t bother to even try. Instead you just state and opinion as fact and move on.
…while this fact receives almost no attention in the public debate….
Again, an unqualified statement of opinion as fact. I think I’ve seen a ton o’ attention to the lack of certainty and the difficulty of predicting the trajectory, in the public debate. What does “almost no attention” even mean? You don’t bother to quantify.
…Occasionally, one can hear people briefly acknowledge that we don’t really understand why waves of infections come and go (typically after the epidemic took a turn that wasn’t predicted by the models used to make projections), …
So now after saying almost no attention, now it’s occasionally one can hear… That seems a bit inconsistent but even if it isn’t, it’s a totally unqualified opinion stated as fact. I’ve seen a lot of talk about the lack of clear understanding about what explains the waves over time in different locations.
…but they almost never follow up with a real effort to try and figure out why the pandemic exhibits this cyclical pattern….
More of the same approach. I know that your mind is made up about these things but am I to just take your word for it because you say so?
…People often claim that it’s because respiratory infections are “seasonal”, …
Ok, that seems like a pretty much straight up contradiction. After saying there’s almost no attention to explaining the waves, now you say that it’s “often” claimed that seasonality is an explanation. Worse still, you lump all related analyses, with attached probabilities and error bars and confidence intervals to imply that they are “claims” that seasonality is the whole explanation. Well, first of all, I’ve seen many analyses that explore seasonality that introduce it as just one element in association with other factors. You don’t even vaguely reference who it is that you are saying is making such claims, how prevalent they are, etc.
…but meteorological variables are not associated with transmission strongly enough to explain this pattern, …
Opinion stated as fact, again. No actually analysis attached. And further, again, it’s just sloppy. meteorological variables covers a very wide range, and necessarily interact with behavioral variables to help explain “seasonality” as in influence. You equate seasonality with “meteorological variables,” in a facile manner.
… so in practice this boils down to the claim that infections rates fluctuate over time, which is not a genuine explanation but just a restatement of what is to be explained….
Wow. A total dismissal of multiple, extensive analyses, with a single, simple hand wave. No matter what ANYONE has written about the effects of meteorological variables or the interaction between those variables and season-associated behaviors, we should just take your word for it that it all just boils down to “infection rates fluctuate over time.”
…In theory, transmission should ultimately be determined by people’s behavior, but the effective reproduction number often fluctuates wildly even when, as far as we can tell with the data we have, there were no behavioral changes….
Now I suppose it’s possible that you’ll clear all that up in the body of the text. But I can’t get past the unreferenced and unqualified hand-waving, with a complete lack of references to existing literature. Now Andrew does say in his introduction that: In his post, Lemoine gives several of examples of countries and states where the recorded number of infections went up for no apparent reason, or where you might expect it to have increased exponentially but it didn’t.
And indeed, perhaps you do and perhaps you do back up all those unqualified statements well-enough to justify everything you stated as fact in that introduction. If others have the interest to get past the introduction to assess your proof, more power to them.
Joshua:
Without getting into any more general questions about tone of debate, etc., I do agree with you that Philippe’s post, interesting as it is (otherwise I wouldn’t have linked to it) does make some strong claims. To get into a couple of these:
“the fact that we don’t understand it well is not more widely acknowledged. …”: I agree with you that it does seem to be widely acknowledged that we don’t understand things well. It seems to me that uncertainties and failures have been discussed widely in news reports from the start. Back in 2020 there were tenured buffoons acting all certain about things—but these sorts of commenters were outliers, and they were recognized as buffoons even at the time.
“governments continue to rely on projections”: I agree with you that “rely on” doesn’t make so much sense here. Governments use projections because you have to use something. Some of these government projections were ludicrous and recognized as such even at the time, but, again, I don’t think it makes sense to say they were “relying on” these projections. Governments were making decisions, as they needed to do, and projections of varying quality made their way into those decisions. The point is that using projections of some sort is unavoidable when making decisions about the future.
These issues arise more generally. I’m always yammering on about how people should be acknowledging uncertainty. The corollary to that is that we need to make decisions under uncertainty, and the corollary to that is that we need to learn from our mistakes. I think Philippe’s post is useful because he’s pointing to anomalies in our theoretical and practical understanding. It’s indeed difficult for policymakers to act, given all the contradictory messages they’re getting. I got a similar vibe from Trisha Greenhalgh’s article that we discussed recently.
Belongs here:
https://statmodeling.stat.columbia.edu/2021/11/29/have-we-been-thinking-about-the-pandemic-wrong-the-effect-of-population-structure-on-transmission/#comment-2032607
As noted in quite a few comments, there are a lot of sweeping statements about “what everyone is doing” (and how it’s all wrong). The literature on population structure in epidemic modelling is vast!
Any recent paper on current epidemic modelling includes a section about the difficulties in accounting for population structure, eg https://royalsocietypublishing.org/doi/10.1098/rspb.2020.1405#d1e1840
“A common challenge faced by epidemic modellers is the tension between making models more complex (and possibly, therefore, seeming more realistic to stakeholders) and maintaining simplicity (for scientific parsimony when data are sparse and for expediency when predictions are required at short notice) [76]. How to strike the correct balance is not a settled question, especially given the increasing amount of available data on human demography and behaviour. Indeed, outputs of multiple models with different levels of complexity can provide useful and complementary information. Many sources of heterogeneity between individuals (and between populations) exist, including the strong skew of severe COVID-19 outcomes towards the elderly and individuals from specific groups. We focus on two sources of heterogeneity in human populations that must be considered when modelling exit strategies: ***spatial contact structure*** and health vulnerabilities.”
To then argue “If real populations have the kind of structure my theory posits, then a lot of what people who study the pandemic have been doing is completely wrong. Thus, if I’m right that we should take that hypothesis seriously, it should at least make everyone nervous. First, since the beginning of the pandemic, we have been obsessed by quantities — the effective reproduction number and the herd immunity threshold — that are essentially meaningless at the aggregate level in a model with complex population structure” is just a bit silly. Clearly SEIR models are mis-specified, that goes without saying. The real question is how useful they are for precise questions. Forecasting long terms dynamics is one question, but there are many others.
I find this stuff quite hard to take seriously.
Here is what I specifically wrote on that issue in the post:
Thus, not only do I not deny that a vast theoretical literature on the spread of infectious diseases on complex networks exist, but I explicitly acknowledged it and even referred the reader to a review of that literature in a footnote at the end of the passage I just quoted. What I’m saying is that it has played virtually no role in the applied work on the COVID-19 pandemic.
I think this claim is correct and, in any case, the example you gave doesn’t show that it’s not. Indeed, what they do in the section from which the passage you quoted comes from is cite some of the theoretical literature I refer to, note that we still have a lot to understand about the role of population structure and call for collecting more data that would allow us to understand this better, which I guess is also what I’m saying, but again this has been largely ignored in the applied literature on the pandemic.
For instance, I think I have read every single modeling paper that was cited by the Scientific Council (roughly the French equivalent of SAGE in the UK) in support of its recommendations to the French government, but I don’t recall a single one of them even mentioning the role that community structure in the network might play. It certainly played no role whatsoever in the models used to make the projections that were presented to the government, which assumed quasi-homogeneous mixing, i. e. the population was divided into age groups between which the contact rates were taken from a contact matrix and the homogeneous mixing assumption was made for within age group contacts. Similarly, in all their work about the effects of government interventions, the same assumptions were made, even though as I point out in my post the methods they used to estimate the causal effects of government interventions can be extremely reliable in the presence of community structure.
I don’t necessarily blame them for not taking that into account. I’m well aware that it can be extremely difficult and that, as you point out, we have no choice but to use models that we know to be misspecified. But I do blame them for never conveying that their methods crucially hinged on those simplifying assumptions which they knew to be false, although we don’t know how much it matters. Thus, in the reviews of the scientific evidence they wrote for the public and decision-makers and even in their scientific work, true model uncertainty was systematically underestimated because — among many other things — the role that population structure might play was completely ignored.
When their projections turned out to be widely off the mark, which happened pretty much systematically, they did not once point out that it may have something to do with the presence of community structure in the network. Neither did they explain that, if there was in fact community structure in the network, their estimates of the effects of government interventions were completely unreliable. Instead they happily plugged those estimates, which even setting aside the issue of population structure were based on no serious causal inference, into the models they used to make projections.
So in my opinion, not only is the fact that a voluminous theoretical literature on the spread of infectious diseases exist not inconsistent with the claims I make in my post, but I actually think it makes things worse, because it means that a factor that researchers knew could play a significant role was ignored in their applied work. Again, I think there can be legitimate reasons for this (as I note in my post it’s very difficult to take population structure into account and we generally lack the kind of data we’d need to do it correctly), but it’s still a fact that they did not and, perhaps more importantly, that they never seriously discussed in their applied work how it might weaken their conclusions.
A related topic I’ve yet to see discussed here:
SIR-type models are seeded with an R0. Supposedly the R0 for a typical flu, original and delta variants are around 1.5, 2 and 6. However, actual annualized case counts show (ballpark!) the same number of infections for all three.
So what is the more parsimonious explanation of the raw data:
(A) that all three have roughly the same transmission, or
(B) that delta’s transmission is really 3-4x faster but is **near perfectly** cancelled out by vaccines/etc.
As a skeptic of “just so” stories I’m inclined to the far simpler (A). Without knowing where the covid R0 headline numbers even originate it’s hard to critique the underlying methods; but certainly sampling biases could easily result in dramatically overestimating R0 from small populations. (for ex: initial estimates come from the first cluster noticed, which by standard arguments is larger/faster spreading than typical)
Something to keep in mind for the (supposedly) R0=10 omnicron…
R0 isn’t constant over either space or time. Or at least if you want to average over enough different scenarios to calculate a constant then it isn’t very useful.
Basically it is a nice concept but too underdetermined in reality to get something quantitatively useful out.
In the SIR setting R0 is definitionally constant. Maybe your point is that SIR models are totally inapplicable to the actual data we have and thus estimating R0 from this data is pointless… no objection there!
I think the basic point stands anyhow even if we replace R0 with any metric M of transmissibility: we’re told M(delta) >> M(alpha) > M(flu) but annual case counts are similar. Clearly something is off.
Most people never get tested for the flu, the number of cases/deaths is an extrapolation. Despite the mass testing, for covid that is probably the same. The flu seems to transmit predominantly via droplets though, so I would expect covid is more transmissible.
There is nothing stopping you from using a varying R0 in your SIR model though. It can be seasonal and/or differ among compartments.
Also, SIR models (originally developed for measles) are not appropriate for respiratory viruses except in the short term (~6 months):
https://journals.plos.org/plospathogens/article?id=10.1371/journal.ppat.1009509
It would be very exceptional if herd immunity is possible for covid, even due to infection. Basically unheard of.
I was hoping to answer my own question and the most cited link I found is: https://academic.oup.com/jtm/article/28/7/taab124/6346388
The entire paper is one page and estimates the widely cited R0=5.08. Their methodology? The mean of 5 basically random arxiv+baidu links. Only a single paper has a stated 95% CI (2.0-4.8) and 3 of the other 4 are well above the CI. Three of the papers are based off Chinese data; one is a policy brief from JP Morgan.
This is about as garbage in / garbage out as I expected.
Andrew –
I can see where Philippe’s simulation would be useful. I thought Nic Lewis’ modeling was useful even though he predicted Sweden would reach a HIT (at the time he meant HIT in a sense that few people would get infected from that time forward) in May 2020, and London and NY and Barcelona and India shortly thereafter. Exploring what the models turn up as you tweak parameters and consider heterogeneity of transmission more comprehensively seems useful to me. It’s information to put into the databank as you go forward.
I take issue, however, with the attached overly-broad, emotive, and unqualified statements, particularly when they are laden with political overtones, all sheltered under an umbrella of plausible deniability and stealth advocacy, and clamoring for apple pie and “freedom.” All of that I find less than useful. Nic dressed his modeling in the same kind of overcoat, while arguing that he was just asking the hard questions o one else would ask.
And it just seems unnecessary
Joshua:
I agree with some of what you write, but (a) I wouldn’t call it “stealth” advocacy. Philippe’s advocacy seems pretty open, and (b) I don’t think advocacy is always a bad thing. These are important issues so I can understand that people have policies they want to advocate. I don’t think advocacy is necessary in science—I think it’s possible to do science without advocating a position—but I don’t think it’s necessary not to advocate, either.
Andrew –
Fair points. Perhaps the stealth label was misapplied. And I am also a fan of advocacy. Advocacy gets a lot done.
I actually went over and read the blog post and I have to say the meat of the post is much more equivocal and circumspect than I anticipated. There were a lot of important, and explicitly stated caveats.
All in all, I think my reaction was overstated.
That said, I still think there’s much in the meat of the post that carries forward the attributes I was criticizing – in particular the characterizations of exiting modeling (say the binary depiction of how seasonality has been discussed) and their place in the public discourse and government policy – but I think they were more tempered than what I expected and then saw in the introduction.
I’m left with a question regarding how the disparity and fluctuation over time between identified cases and actual cases (iow, testing rates) would be incorporated into his conclusions about the putative level of disconnect between “waves” and behavior?
I’m not personally convinced that he’s actually accounted, to the extent that he’s claimed, for the complex interaction between behaviors, biological mechanics (variants, meteorological effects), etc., to say that they don’t mix together to sufficiently explain waves – particularly as they occur in similar periods across very different population structures – but assuming his simulation doesn’t have some glaring imbedded bias (I couldn’t judge) it does seem to me to be something worthy of putting into the mix.
..
embedded…
Joshua:
Yes this reminds me of a discussion we had several years ago regarding a brash, overconfident style of internet writing.
Andrew –
> Perhaps I’ll write a blog entry in this style at some point, just to see how it comes out.
That would be fun.
I was thinking a bit more about this.
Usually, when I think of an academic style of writing, I think of stuffed shirts and overly formal and passive voice in ways that are intended to make people sound smarter than perhaps they really are.
But on the other hand…
There is kind of a function to a lot of that. Say writing that “next we did an analysis..” or perhaps even… “Next, an analysis was done,” instead of “Next I analyzed.”
A lot of that syntax is focused on de-centering the individual, with the connotation that the point isn’t the individual, but the academic topic at hand, removed to an “academic” arena of discourse.
When I think of the alternative, the blog type of analysis, with the aggressive, angry blog-style, the de-prioritizing of formal academic style and protocol, perhaps in a reactionary way, against the very notion of peer-review that is “gate-kept” by the phony self-aggrandizing “elites,” and instead opened up to the non-hierarchical proletariate, the real democratic approach to analysis, the true meritocracy, ready to throw out all that formal and stuffy nonsense, I think for all the advantages there may well be something lost as well.
In a sense I think it might be captured by a blog analysis that contains no elaborated literature review, where overly broad statements are just thrown out there without the constraint of citations or reference to existing literature, without any guarantees of having been surveyed by anyone who’s really familiar with the topic.
Hopefully, maybe eventually we’ll reach some kind of middle ground. But I kind of doubt it.
Interesting perspective. This study may also be relevant: “Effective containment explains subexponential growth in recent confirmed COVID-19 cases in China”
https://www.science.org/doi/pdf/10.1126/science.abb4557
And here maybe even more relevant work from the same group using German mobility data:
“COVID-19 lockdown induces disease-mitigating structural changes in mobility networks”
https://www.pnas.org/content/117/52/32883
Anecdotal counterevindece. Here in Helsinki (Finland, about 6_000_000 people) I was able to get access to covid cases numbers on a finer resolution. I was looking on numbers by postal code and mother language, and… found no partucular hererogenity. And if epidemic so easily crosses the border between language groups and high-low class areas, then I dont think any other social border could stop it for a year.
So far I saw only one case of hererogenity. After the euro football cup, fans were bringing delta variant from Russia. We had a relatively higher numbers in the city center, there the bars are. But after a few weeks epidemic mixed very well again, and all the clusters disappeared.