Matthieu Domenech de Cellès writes:
I am a research group leader in infectious disease epidemiology at the Max Planck Institute for Infection Biology in Berlin. I read your recent post on how to respond to fatally flawed papers.
Here is a personal anecdote from my field, which you might find interesting.
The flawed paper in question was published in the New England Journal of Medicine (NEJM), supposedly one of the most prestigious medical in the world (at least if one cares about its impact factor, which was 158.5 in 2022). The study (https://pubmed.ncbi.nlm.nih.gov/22970945/) was a case-control study aiming to estimate how the risk of pertussis varied over time after receipt of 5 doses of pertussis vaccines in children aged 4–12 in California. The main result was that, in this fully vaccinated population, the risk of pertussis increased by 42% each year after the fifth dose. The analysis itself was OK, but the fatal flaw came from the interpretation of this result, as reported in the discussion: “[…] we estimated that the fifth dose of DTaP became 42% less effective each year […]”. From this flawed interpretation, they claimed that the protection conferred by pertussis vaccines waned very fast (maybe on a time scale of a few years), hence the title. As I write this, this paper has been cited 606 times (according to Google Scholar) and has largely led the field to believe these vaccines are no good.
As the authors acknowledged, however, their study did not include an unvaccinated group, which is needed to estimate vaccine effectiveness (VE). In other words, it was impossible to conclude about the rate of waning VE from the 42% increase they estimated. Incidentally, this basic fact should have been obvious to the editor and the reviewers from the NEJM (but as you wrote in another blog post, sometimes the problem with peer review is the peers). In a follow-up study (https://pubmed.ncbi.nlm.nih.gov/31009031/), we showed that, given the high transmissibility of pertussis, this 42% was entirely consistent with vaccines with sustained effectiveness whose protection wanes, on average, slowly. More generally, we demonstrated that their claim (i.e., a 42% annual increase in pertussis risk after vaccination is equivalent to a 42% yearly decrease in VE) was baseless.
Overall, we adopted a strategy close to option 5 in your post. We initially submitted the paper to NEJM, which rejected it immediately (probably for the reasons you mentioned). Our paper was ultimately published in 2019 (in JAMA Pediatrics), with 28 citations so far. Although this was a good outcome, I don’t expect our paper ever to reach the impact of the NEJM paper, whose damage is probably irreversible.
I took a look at the JAMA Pediatrics paper, and . . . wow, this is complicated! These cohort issues really confound our intuitions. This fits into our recently-discussed “It’s About Time” theme. I feel like there should be some clever graph that could make this all clear to me. The paper does have some graphs, but not quite what it takes for me, in that I still struggle to hold this model in my head. I kinda believe that Domenech and his colleagues are correct—they sent me the email, after all!—but I don’t have the energy right now to try to untangle all this. Given that, I can see how the NEJM editors could’ve thrown up their hands here. I’d be interested in seeing their stated reasons for rejecting the Domenech et al. paper.
P.S. When I was a kid, I never heard of anyone who had whooping cough. We all wanted to get it, though, because the only thing anyone knew about whooping cough was from the school rules that they would give out every year which had a list of the minimum number of days of absence from school if you had various illnesses: measles, etc. For some of these, the minimum period of absence was 1 week or 2 weeks. The thing I remember was that whooping cough was the only illness where you’d have to stay out of school for a full 4 weeks—or maybe it was 8 weeks, I can’t recall the exact number, just that whooping cough was the longest. From a logical standpoint, this should’ve made us super-scared of whooping cough—it’s so dangerous you have to miss a month or two of school—but, to us, the lure of getting to miss 4 or 8 weeks of school was so appealing. Recess and gym class aside, elementary school was such a horrible waste of time.
Also, we had no idea whether it was supposed to be pronounced “wooping cough” or “hooping cough.”
Some of the younger people who read this blog may not be all that familiar with pertussis so for them, go to
https://en.wikipedia.org/wiki/Whooping_cough
and scroll down so as to watch the 11 second video of “a boy with pertussis.” Anyone who watches that video clip and still is opposed to vaccinations is beyond the reach of medical science.
Andrew’s obiter dictum, “Recess and gym class aside, elementary school was such a horrible waste of time,” begs for more specificity and possible rebuttal.
With regard to pronunciation, this is what I found: /ˈhuːpɪŋkɔːf/
I certainly learned a lot during my elementary school years, although I suppose that doesn’t answer the question of how much I learned in school, nor how much I would have learned during those hours if I had been doing something else. Perhaps watching Sesame Street and Zoom and The Electric Company and whatever else was on PBS for kids in those days would have been just as good, or better. By the time I met Andrew, in 7th grade, we could do basic algebra and some geometry, we knew something about cells and other basic biology — mammals, reptiles, insects, yada yada), and we could (and did) read pretty well. I guess a big question is: if we hadn’t spent all that time in elementary school, what would we have been doing instead?
One thing about elementary school as I remember it is that we spent way too much time learning about the American Revolution. Maybe it’s a trick of memory and we only spent a couple of weeks on it, and I just happen to remember them — I remember very, very few specific moments from elementary school — but in my memory we spent hours and hours learning about the Boston Tea Party, and Thomas Paine, and Bunker Hill, and Ethan Allen and the Soggy Bottom Boys (j/k), and Francis Scott Key, and on and on and on. It’s not that that stuff isn’t somewhat important — although I do question the present-day importance of knowing about specific battles and other details — but from what I remember it feels really out of proportion with other important historical knowledge…or, for that matter, contemporary knowledge: hugely important events of the Civil Rights Movement were playing out while I was in elementary school, and I don’t think we learned about them at all!
Hmm, riffing on that last point: Ruby Bridges’ first day in elementary school was in 1960; mine was 1970. When I was in second grade she was still in high school! But I don’t think we studied anything about the contemporary civil rights movement until I was in junior high school. Even the Civil War and Reconstruction didn’t get much attention in elementary school as far as I recall, although, as I said, that may be due to my faulty recollection.
Anyway my assessment is that I did learn quite a bit in elementary school. But I admit to not remembering enough about what I learned, and when, to really be sure.
Phil — We are the same age. My impression of elementary school is both (1) I learned important things (2) at the cost of enormous amounts of “wasted” time, because, of course, it is day care. (And my father was the principal!)
Phil:
Every year on January 15 they would show us a filmstrip called I Have a Dream, about the life of Martin Luther King. It was the same filmstrip every year; by fifth grade I’d pretty much memorized all the slides.
Every year in elementary school my kids had to write some stuff about a Black American for Black History Month (Feb). Every year kids were encouraged to write about MLK or Ruby Bridges or Harriet Tubman or some black astronaut or whatever. There was always a suggested list, though you could choose someone else.
Every year I encouraged my kids to read and write about black musicians because one of the major cultural contributions to the US by black people since the end of the civil war has been the development of music, through Blues, Jazz, Soul, R&B, Rap, Hip-hop whatever. Certainly not the only contribution but this is an important and outstanding area of contribution!
There’s such a wealth to explore too. Just a small selection: Duke Ellington, Louis Armstrong, Ella Fitzgerald, Sarah Vaughn, Lena Horne, Billy Holiday, Charles Mingus, Thelonius Monk, Miles Davis, John Coltrane, Alice Coltrane, McCoy Tyner, Pharaoh Sanders, Yussef Lateef, Marvin Gaye, James Jamerson, Isaac Hayes, The Supremes and Diana Ross, George Clinton, Bootsy Collins, Eddie Hazel, Jimi Hendrix… the list goes on and on.
Anyway, invariably not a single person on that list makes it to the list of suggestions from the teachers.
Phil and I went to different elementary schools, so who knows? I can pretty much confidently say that essentially 100% of elementary school was a waste of time, again with the important exceptions of gym and recess. A teacher once spent a couple minutes teaching me long division, which was cool, and another time I asked a teacher how to compute the area of a triangle and she said it took two steps, and that was perfect because I figured it out from there. And I guess the first time we say that Martin Luther King filmstrip, that was something I learned in school that I would’ve have otherwise read in a book. I think that was everything, though. The American Revolution stuff I was reading in books anyway; school added nothing there. School didn’t even teach me how to behave well with other kids: with the exception of gym and recess, we weren’t really interacting, we were just being bossed around and told to sit still.
I’m not saying that elementary school is, or was, useless for everyone; this was just my experience.
Andrew,
if your claim was that you didn’t learn anything in elementary school that you wouldn’t have learned anyway if you had done something else with that time, OK, that’s possible. But if your claim is that you learned nearly nothing in elementary school…there’s no way. That would require that everything you knew in seventh grade had been learned outside of school — somehow you weren’t learning anything from people who were purportedly trying to teach you for hours every day, but you were learning a lot of stuff at home, such that you ended up just about exactly matching your peers. That seems extremely unlikely.
That said, now that I think of it I do recall that 6th grade was almost a complete waste of time. At the end of fifth grade we had moved to a new school district, and I was going to the very small Colesville Elementary School. My previous school (Magnolia Elementary, in Prince George’s County) had been a lot more challenging, or at least I had learned a lot more, so by the time I got to Colesville I had already learned all the math they were teaching in sixth grade at Colesville, and I was way beyond the stuff they were teaching in English, and so on. That really was a wasted year. My mom, who had some education background, ended up making up a bunch of reading and language coursework for me, and for an hour each day I would sit in the library and work on that stuff rather than sit in class re-learning stuff I already knew. One of my life regrets, maybe top 30 or so, is that I never got back in touch with my 5th grade math teacher (Mr Neelon) or English teacher (Ms Melee? Meely?) to thank them for giving me a great head start…although maybe it didn’t do me much good since I didn’t really gain additional ground in 6th grade. I guess it gave me some confidence, though, so I guess that’s good.
Anyway I think you’re not giving elementary school enough credit, I’m sure you learned something there. I just think that one learns pretty slowly. How many times did my parents ask “what did you learn today in school”, leaving me flummoxed because I couldn’t think of anything specific? Many many. And yet, somehow I learned a lot of stuff by the time I was in junior high.
Phil. My grandparents pushed my mom to consider putting me in a private school in the Bay Area. We couldn’t really afford it, but the school allowed me to visit and they gave me some assessments of my reading level, this must have been about 4th grade. The school went K-8 I believe. They started me with a textbook for 4th graders and had me just read some random stuff in the middle, then bumped me up to the next year and the next year until they had me in the 8th grade book and basically didn’t have anything more to offer me. I was probably reading at an 11th grade level in 4th grade. My favorite books in 3rd grade were The Three Musketeers, Connecticut Yankee in King Arthur’s Court, and excerpts from Moby Dick. Looking back it’s probably not surprising that I thought the next 10 years of school were really super boring. 😂
In 7th grade I asked my school librarian for a copy of Don Quixote and she refused to get one for me. I read Robin Lee Graham’s book Dove, about his mid 60’s voyage around the world by sailboat in 7th grade. I spent my time imagining how much better it’d be if I could just sail around the world with a bunch of books :-) Anyway at some point in 8th grade I got sick and stayed home from school for like 2 months. I’m guessing looking back that I was only sick for like 2 weeks, but then had severe allergies during the spring of that year… anyway when I went back basically nothing had happened and I didn’t miss out on anything.
We didn’t really start reading stuff at my reading level until about 11th grade when we read things like Huckleberry Finn and The Sun Also Rises and As I Lay Dying or something like that. Math from 4th to 8th grade was completely a waste of time, except the one section for a few weeks in 6th grade where the math teacher did logic puzzles… other than that it was long division in 4th grade, and we were still basically doing long division in 8th grade. By 8th grade I was acting up a bunch in math class and was once sent outside to sit by myself where I got in trouble for playing blues Harmonica as if I were in a jail scene in a movie set in Mississippi.
My kids seem to be repeating the same issues. At least my youngest who is really into math got into the accelerated math program in Pasadena which teaches all of AP Calculus by 8th grade. But I’m 100% convinced that my youngest has learned more from YouTube this year than any of his other classes. He comes home and watches YouTube videos about Gobekli Tepe https://www.youtube.com/watch?v=xJU973IbG7I and the physics of flames that chase themselves around a track https://www.youtube.com/watch?v=SqhXQUzVMlQ and stuff like that. Every couple days he tells me how amazing some video was and I should watch it… We also have long conversations like yesterday I discussed an overview of the history of US university protests in the context of the Vietnam war and today’s protests over Gaza, we discussed the creation of Israel in the 1940’s and the lead up to and history of US involvement in Vietnam and its affect on US politics and the music of the 1960s and 1970’s all on the way to his Volleyball practice. I’m 100% completely confident that his school has never even once mentioned any of that.
People really underestimate how much parents are responsible for their kids education.
I’ve analyzed test scores in the past for Pasadena Unified Schools. What I remember finding was that kids math skills rose rapidly between 1st and 4th grade, plateaued from 4th to 6th, declined slightly 6th-8th grade, and then rose between 9 and 12 (they only test and 9 and 12). Similarly for reading scores. The decline in middle school seemed to be pretty consistent… At least that was on *average*, if you subset by educational attainment of the parents there’s a more steady growth for the ones whose parents have at least undergraduate degrees.
So, there’s something to be said for the idea that between about 4th and 8th grade it really is about babysitting and having the slower kids catch up and maybe some indoctrination in social sciences, there isn’t all that much skill growth.
I agree with Phil. Elementary school deserves more credit. But also more blame. In my case it did plenty of damage in terms of peer pressure and (light) bullying. I’d add that junior high school was probably even a bigger waste of time for me.
Dale:
I’m not saying elementary school had no effects on me. I’m sure it did. I’m just saying that I didn’t learn anything in elementary school, except at recess, gym, and the three examples given above (long division, the area of the triangle, and life of Martin Luther King). We did a lot in school, I just didn’t learn anything. Everything I knew when I started seventh grade, I knew because I read it in a book or saw it on TV or because I figured it out on my own. OK, you could say I learned some general concepts that would be useful in later life, for example that people in authority can be mean, they can play favorites, and they can use their power in arbitrary ways—so, sure, some valuable life lessons—but I don’t that’s what Phil was referring to when he talked about learning things in school. We had a 10-year-old World Book Encyclopedia at home, and that pretty much covered anything they could possibly have taught us in school. We did sometimes have art and music classes, but I never learned any art and music from them. I didn’t learn art and music at home either. For a couple years they also had some early morning French lessons; I guess that from that I learned about 10 words. I learned more in the first two days of French class in seventh grade than I did from years of the teacher talking in French to us and trying to teach us the names of the colors etc.
Andrew, I feel like your’re talking about facts, as opposed to skills. Like, did you learn to read at home before you went to Kindergarten? Or did you learn phonics and reading in K,1,2 grades? Did you learn place value of numbers at home? or in school? I could believe either, but it does seem likely that in the K,1,2 years you probably learned some important foundational stuff which later allowed you to learn things like facts on your own.
It’s also possible you were sufficiently precocious or started school sufficiently late that you learned all that stuff at home too… I just think that’s rare.
My impression is K-12 school is really good for K,1,2,3 … doesn’t have a ton of value in 4,5,6, kids actually decline in 7,8,9 when they’re ultra distracted by puberty etc… then learn something in 10,11,12 unless they get sidetracked to the criminal justice system.
That seems to track both my own experience and so far my kids experience (7th and 8th grades). I expect high school to be harder and more interesting for my kids next year.
Daniel:
Yes, I learned how to read on my own. Street signs, cereal boxes, the newspaper, etc. I agree that this is rare, and I’m not saying that elementary school in the 1970s was a waste of time for everyone! I’m just sharing my experience here.
Back when I was a boy, all the kids had these things called “parents” – we all had two of them – and they taught me how to read long before I went to school. They also taught my two sisters and I sophisticated math like fractions – we each got 1/3 of a can of coke when we watched The Wonderful World of Disney on Friday nights!
But I guess we were privileged. My dad worked for the US Air Force as a tail gunner! And my mom had so much free time she could just hang out and sew clothes for us *and* she sewed all the curtains in every house we lived in until I was out of high school. I still have a picture of myself in this really stupid embarrassing hat she made for me that I had to wear to make her happy.
So I guess it was our extraordinary economic privilege that allowed me to have amazing toys like wood blocks – even with different colors – where you can learn the names of the shapes and colors before you go to school. And really I mean I had an *entire set* of Legos so my mom taught me how to classify and categorize by sorting by size and color. Back then Legos were an expensive toy for fabulously rich air force enlisted men’s kids, but today you can buy used ones by the pound on eBay.
So I guess it’s only privileged people like me and Andrew who could learn things outside of school. I wonder if Andrew had “parents” too. Andrew’s kind of right: school doesn’t matter that much if you’re as economically privileged as I was, having two parents that lived together and your dad has a high paying job like B-52 tail gunner.
I don’t agree with Andrew though that school was useless. I liked things in school like learning all the rivers of the world, which was largest, what the names of the mountains were and all the countries and continents. I don’t think I would have learned my times tables if not for school. I also learned things like why we have a constitution and how it works. I guess Andrew missed out on the whole constitution thing. The purpose of the constitution – this is the part he missed – is to set out some basic rules that can’t be violated or changed on the whim of a temporary and/or emotional majority. Everyone agrees to the constitution before the emotional issues come up, so that way it’s in place and the rules still have to be followed when people start losing their cool. You can’t learn that from a ceral box I guess.
chipmunk
Since you felt compelled to suggest (more indirectly than usual) your usual themes, let me comment on the Constitution. I probably learned something about that in school (I’m not sure it was elementary school, however) as well. But I didn’t learn some of the most important things about the Constitution – things that I am only learning now in my advanced age. I had little idea of just how incomplete it is, just how much depends on trust, and just how fragile that is. About the only thing the current Supreme Court immunity case shows universal agreement on is that if the president seeks legal counsel from the Attorney General, then they are immune (assume that the Attorney General says it is on sound legal ground). And the protection against collusion between the President and Attorney General apparently relies on the checks and balances established in the nomination of the Attorney General and the subsequent task of the Senate to decide whether to approve the appointment. All of these things are useful and welcome protections – but they now strike me as more fragile than I ever learned in school.
We do learn a lot from our parents and also from school. Often what we learn is not what we need to learn – from both sources. In some cases, useless learning may even be better than the alternative.
Chipmunk:
I’m not recommending that any basic rules “be violated or changed on the whim of a temporary and/or emotional majority.” I’m just sharing my experiences here! Also, my parents were around, but I was good at reading and I figured it out on my own. I’m not so good at music and I never learned much of how to do it, on my own or in school. The rivers, mountains, constitution, etc., I learned about from reading. They didn’t tell me anything in elementary school on those topics that I hadn’t already read. But, again, kids differ; that was just me. In college I took an excellent course on the constitution and learned a lot. I learned a lot from many courses in college. Somewhere I’m sure there’s someone who learned a lot in elementary school and nothing in college.
It explains a lot if you weren’t ever taught to read by a professional
That was directed at Chipmunk by the way.
Somebody:
That’s ok. I was taught statistics by professionals!
It makes sense, they assume contact rates increase with age. Increasing contacts *or* increasing proportion susceptible (waning immunity) will increase cases according to their SEIR model.
There are a few questionable assumptions being made though. From the supplement:
Assuming the mother is immune, newborns are protected by maternal antibodies for the first few months.
And related:
The model assumes long lived immunity is independent of contacting children with whooping cough. Ie, ignoring the role of “natural boosters”.
If there are fewer infections around, we’d expecting waning rate to increase. That makes both the adult and newborn populations *more* vulnerable. (Newborns since the mothers are less likely to pass on antibodies.)
And
fig 2b
looks interesting.
The flawed interpretation they are correcting looks due to the same confusion as for the “turnover” in cancer at old age.
The cumulative incidence increases but the age-specific incidence peaks. So for high probability of infection/mutation the rates will actually start decreasing.
Flip 3 coins in parallel, for each coin stop when it lands heads. Write down how many flips for the last heads.
Repeat 10 times or whatever to see the distribution. Id guess their model reduces to something like that.
PS. The model has not really been validated. You can’t validate a model (especially one with so many parameters) by checking fit to the same data used to develop it. You gotta make predictions and check the fit to new observations.
Children won’t be protected by their mothers unless circulating antibodies are relatively high, but they won’t be high unless they’re recently exposed… But if they were recently exposed then so was the child. Empirically, small children used to die of whooping cough routinely.
My guess is that protection from getting whooping cough probably wanes much faster than protection from severity. The cough is caused by a long lived toxic that basically destroys your lungs and takes weeks and weeks to clear if I remember correctly.
I tried looking in to newborn (under 2 months old) pertussis pre-vaccine, apparently it was very rare. But that can also be chalked up to changes in reporting.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5106622/
But more interestingly, it seems the entire premise of the model in the OP has been questioned:
[Same source]
My impression is there’s good reason to believe that Pertussis vaccines don’t really prevent the disease as well as many other vaccines and that the cyclic nature is due to fairly rapid decrease in immunity from both vaccine and natural immunity. I do think there’s reason to believe the severity is less though.
I read up on all of this when my kids were young so it’s been 10 years maybe, but there were a number of models which were concluding that the vaccine effectiveness might wane exponentially fast over a 3-4 year timescale. I’m not going to claim any special knowledge here just that when I looked into it it wasn’t at all fringe to be suggesting that pertussis vaccine and natural immunity both wane quickly. Perhaps that’s due to poor models and bad data, or perhaps it’s true.
I got pertussis in my 20’s when my wife an I went to the Cook Islands. Started coughing about 2-3 days after we returned to the US, and then coughed for like 6-8 weeks straight, my wife would cough so hard she threw up, but fortunately didn’t break any ribs. It’s a horrible disease. I do think it becomes not-communicable rather rapidly if you get on antibiotics, but the toxin has done its thing already by then, and you still cough for 6-8 weeks after that.
Doesn’t seem like anyone knows.
Whats really going on here, then, is multiple competing explanations for the same observstions. But no one involved knows to frame it that way. The journal may not even have a mechanism for this situation, or be rejecting papers that disagree with what they’ve previously published. Crazy!
“I feel like there should be some clever graph that could make this all clear to me.”
Sometimes the clever graph is there, but no one bothered to look at it… clever like Figure 1 here, where a cohort of women stop dying.
https://jamanetwork.com/journals/jama/article-abstract/2818061
The article you link to doesn’t have any figures. Was this a mis-paste?
Peter:
Here it is:
Apparently immortality is ensured by answering ~25 years of nurses health study questions, but only if you’re not gay.
Is this cumulative probability of death?
Yes.
Hmmm…Well, the heterosexual nurses aren’t immortal, because the line still goes up a little bit. The bisexual nurses appear to be immortal, but the numbers at the bottom of the figure suggest that 1 (out of 287) of them died. Saying “1 out of 287 of this cohort died in a ~3 year period” isn’t unreasonable. But the fact that the (very high n) heterosexual line levels off after appearing to increase exponentially suggests there is an issue with the data, at least for y>25. I’m tempted to chastise the authors for not addressing this, but then again they could have just truncated the graph at y=25 (a la Matthew Walker) and had a much clearer message, so maybe we should appreciate their transparency. The graphs provides strong support for the article’s conclusion, assuming the issue with the y>25 data doesn’t affect the y<25 data.
Adede,
I agree – A truly unique thing about this graph is that it is both a) incredibly convincing evidence for the authors’ main claim up to 25 years post followup, and b) so egregiously unconvincing from 25+ that it makes you question the validity of the entire thing because you can’t believe the data are actually meaningful anymore.
Seeing the heterosexual women go “almost immortal” after 25 years makes me think they likely have major issues with non-random nonresponse / drop out in the control group of 90k people and that Kaplan Meier curves are therefore inappropriate and probably exaggerate the longevity of the heterosexual group leading to very poor estimates of the difference. Also the fact that their accelerated aging model produces *slower aging* estimates in the second figure but then shows faster aging in the above figure gives me pause… probably just means they’ve done a weird parameterization, but the fact that they don’t feel they need to explain their second figure seems weird. I no longer can click through to the full text… Anyway the idea you’d publish a figure that shows 90k people going essentially immortal with no commentary on that seems to give no confidence at all.
LOL, man you can’t expect that kind of dry humor to transmit through text.
Coincidentally, my university sent out a notice just a few minutes ago that “[The County Public Health Office] is investigating multiple cases of community spread of pertussis/whooping cough …and cases have been confirmed on campus.”
Your blog post scheduling is prescient, Andrew!