Joel Schwartz writes:
After reading your recent post on Langer et al. (2024) (the Demography article that concluded Trump’s election caused an increase in rates of preterm birth and low birth weight among infants of Black, Hispanic, and Asian/Pacific Islander mothers, relative to White mothers), I downloaded the birth data to see if I could reproduce their study. I did everything in R and put together the results in an html document produced with Quarto (I provide a link to that at the end of this email). Below is a summary of key results. The full document has additional analyses, plots, tables, and discussion.I tried to follow Langer et al.’s description of how they selected and cleaned the data, but ended up with a substantially larger set of observations than they reported. After limiting the data to their four maternal race and two maternal nativity groups, and removing missing values for all of the variables they included in their modeling, I ended up with 19.68 million observations, as compared with 15.57 million observations reported by Langer et. al. This is out of 23.89 million total observations to start with for the period November 2012 through November 2018, which is the time range that Langer et al. focused on. I was a bit surprised that Langer et al. ended up removing 35% of the data. I can’t see how their data selection process could remove that much data, but maybe I’m missing something.I ran into some other issues as well. For example, the public use data includes mother’s nativity starting in 2014, so I don’t have mother’s nativity for November 2012 through December 2013 (it can be obtained by requesting restricted-use data, which I haven’t attempted yet). I excluded that time period in the rest of my analysis.Below is a table comparing some key means and percentages for U.S.-born White and Black mothers for the period after Trump’s election for the data Langer et al. used (from Table 1 of their paper) with my version of the data. I had access to the same variables as Langer et al. for the after-election period. As you can see, for the two outcomes, Low Birth Weight and Preterm Birth, my version of the cleaned data had lower rates than reported by Langer et al. (I confirmed that I used the same cutoff values as they did for defining these binary outcomes). For the covariates, most are similar to the values reported by Langer et al. while a few are different. And, as I mentioned above, I ended up with a lot more observations than Langer et al.
Below are Langer et al.’s figures 1 and 2, along with my attempted replications. My versions of the graphs also include data through December 2023 (the most recent year available in the public use files) so you can see what was happening after November 2018. As you can see, my attempted replications of the Langer et al. regression lines often have a similar pattern, but not always (and we wouldn’t expect the pre-Trump-election period to necessarily have the same pattern, since I’m missing the first 14 months of data). Also, as already shown in the table above, the rates of low birth weight and preterm birth are lower in my versions than in Langer et al. Since the data go through December 2023, you can also see that rates of low birth weight continued to increase after the election of Biden, as shown by the generalized additive model (GAM) fit to the full data range.
I generated graphs like the ones above, but with Biden’s election as the cutoff date for the linear regression lines (using the exact same data, but limiting the date ranges to the four years before Biden’s election and all of the months available after Biden’s election). After Biden’s election, the probability of low birth weight continued to increase for all sub-groups.
I also tried to replicate Langer et al.’s regression model for low birth weight (bearing in mind, once again, I ended up with more observations than Langer et al. even though I tried to replicate their data cleaning process). Based on their description, these appear to be linear probability models. Below are the results of my attempt to replicate their Model 2, using the following formula and R’s lm function (Trump is a dummy for before/after Trump’s election and low.birth.weight is a dummy for being below Langer et al.’s 2,500-gram cutoff for low birth weight). I also fitted a similar model for Biden, using the same time ranges as the Biden Election regression lines in Figure 6 above (replacing the before/after Trump dummy with a before/after Biden dummy in the model formula).low.birth.weight ~ Trump + mother.race + mother.nativity + birth.year + Trump:mother.race:mother.nativity +marital.status*mother.race + mother.education*mother.race + mother.age*mother.race +prenatal.care.adequacy.index*mother.race + medicaid*mother.race + total.birth.order.recode*mother.race +sex.of.infant*mother.race(By the way, in addition to the big picture validity issues with their analysis, I think using the prenatal.care.adequacy.index causes information leakage. This index is calculated by comparing the actual number of prenatal care visits to an “expected” number of prenatal care visits that is based on weeks of gestation. But weeks of gestation is an outcome variable, and low birth weight, the other outcome, is highly correlated with weeks of gestation as well.)Figure 8 shows the model-predicted disparity in the probability of low birth weight for each race, relative to White mothers. For example, Langer et al. reported that U.S-born Black mothers had a 4.7 percentage-point (pp) greater rate of low birth weight than white mothers before Trump’s election and 5.2pp greater after (see Table 2 of their paper). I found greater disparities. The 95% confidence intervals are narrower than the widths of the value labels and are therefore not visible in the graph.Figure 9 shows the model-predicted change in disparities from before to after each election (the Langer et al. results are also in Table 2 of their paper). For Blacks, the increase in disparity was actually greater after Biden’s election when compared with Trump’s. All of the marginal means and confidence intervals in figures 8 and 9 were calculated using the avg_predictions function from the marginaleffects package.
Although my analysis seems to call into question the conclusions of Langer et al., I also want to point out that I could, of course, have made errors myself. In some cases I wasn’t certain which variables Langer et al. used (there are several race/Hispanic origin variables, for instance) or all of the details of which specific observations they included or excluded, as they describe what they did in words, but don’t appear to have provided code. I’m also not sure whether I used the same method as Langer et al. to calculate the disparities shown in figures 8 and 9 above. There could also, of course, be errors in my R code.If you’ve read this far, you can see that I obviously spent way too much time on this, but it has been a fun little project. There are other variables in the birth data that could have been included (maternal smoking, fathers’ characteristics, etc.) and no doubt better ways to model the data. Also, Langer et al. collapsed the data into a smaller number of categories for many of the variables they included. In addition, regressing on a cutoff value, rather than using the actual continuous data for birthweight and gestation weeks as the outcome, throws away information that could be used to derive better predictions.I put a very low prior probability on the validity of their hypothesis, so I kind of feel like digging into this further might not really be time well spent. On the other hand . . . post-publication review.
I’ve not looked into any of this since writing that post a few months ago, so you’ll just have to judge all this new stuff for yourself, if you’re interested.
I think there will not be many responses to this post (given its length and complexity) so I’ll offer one despite the fact that I haven’t looked into this in any detail. But I am struck looking at the figures presented. The impression I get from the graphs on the right are that regression discontinuities are a stupid way to model this data. What I see looks like fairly continuous curves – the discontinuities appear to only arise if you zero in and ignore most of the data. The quoted conclusions don’t seem to mention this however – my takeaway would be that it should not be modeled in that way, unless the focus is to determine where the most meaningful discontinuity arises (and then examine whether it is meaningful). Perhaps I am misreading this though. But my prior is that regression discontinuity models are usually bad ways to model data, and these examples seem to confirm that.
The other takeaway for me is the difficulty of trying to replicate someone’s creation of data to be analyzed. So many forking paths, so little time.
On the last point, I have more than once been told by a reviewer that I’ve included too many details in a methods section, so I don’t even necessarily fault the authors (although everyone would be happier if there were shared code, probably. But of course, when I was still publishing, it was rare to do that in my field and I didn’t do it either)
In general, regression discontinuity is a recipe for noise mining. Take any time series that includes noise of any sort and fit a model to the first half and the second half, there’s probability 1 the coefficients will be different. It’s easy to simulate this. When my blog was working right I had a post like this with a PDF where I fit two models to Normal(0,1) noise, so we know the mean is a constant 0. You could get a publication with a “statistically significant” effect almost every time.
There are right ways and wrong ways to do these kinds of analysis. Basically the right way is to set up a representation for functions which includes the ability to have a sudden change in behavior at any time, run the regression and see if the posterior probability of a sudden change at the time location of interest is high relative to other times.
It seems like virtually noone does this.
I started out just trying to replicate the study, but it turned out that the description of the data cleaning was insufficient (at least for me) to understand exactly what they did. Starting with the raw data of 23.9 million births, I ended up with 19.7 million births, compared with 15.6 million the original study. As you noted (and as Andrew has discussed many times here), without code it can be difficult to figure out was actually done.
In terms of looking for a discontinuity, in the paper, the authors fit a model where they collapsed birth weight into a “low” (below 2500 grams) or “normal” outcome. Since this throws away information, it seems like it would make more sense to fit models directly on birth weight, and maybe use quantile regression to look for discontinuities in the lower percentiles, rather than at the mean.
I’ve been experimenting with GAMs, but there are seasonal patterns in the data and I’m not sure I’m accounting for them correctly (birth weight seems to generally be lower in colder months than warmer months, and the presidential elections occur right around the transition to cold months). I’d be interested in any suggestions you might have for specific approaches. I don’t have the computing power for a Bayesian model (millions of observations), but can fit GAMs in R using mgcv::gam and quantile GAMs using qgam::qgam, but of course let me know if I should be looking at other approaches.
Taking this a step further, to look for discontinuities, maybe it would make sense to include more information in the model that might be directly related to the hypothesis. For example, I believe the restricted use data includes county of residence. That would allow inclusion of things like party vote share by county and demographic and economic data at the county level.
Joel – You should totally write this up and submit it to a journal — not as a comment on Langer et al, but as a de novo publication, along with plausible-sounding explanations, such as that the excess of robocalls during election years produces lingering health impacts.
I know I’m missing the main point of the post, but: why has the low-birth-weight percentage been going up for the past ten years (or more) for everyone, at least according to Schwartz’s work? (And why hasn’t it been going up for whites in Langer’s analysis?)
Most births in the US are covered by Medicaid and an even higher amount of at-risk births occur in that population. I would guess that changes in Medicaid rules for enrollment for each state and other policy choices over time are confounding these results.
Well, it appears to be “many” but not quite “most” – https://www.medicaid.gov/medicaid/benefits/downloads/2024-maternal-health-at-a-glance.pdf. Still, your point is relevant. Given the large variation across states and races, there is probably some good analysis that could be done with that data.
In the raw birth data for 2011-2023 (excluding births where the payer was unknown; about 1.2% for 2011-2013 and 0.8% after) the Medicaid fraction peaked in 2017 at 42.7% (it was 37.1% in 2011) and ranged from 40.6%-41.1% during the last three years. Changes in Medicaid rules and incentives for states has probably changed the demographic composition of Medicaid enrollees somewhat. Also, not every state adopted the Medicaid expansion. So that probably means a regression adjustment for “Medicaid births” probably isn’t doing what one might naively think it’s doing.
Doh! The values above don’t exclude rows where the payer was unknown. One more time, with Unknown payer excluded:
Medicaid percent (excluding unknown payer)
year medicaid
2011 37.6%
2012 38.3%
2013 39.3%
2014 42.0%
2015 42.2%
2016 42.5%
2017 42.9%
2018 42.2%
2019 42.0%
2020 41.9%
2021 40.9%
2022 41.2%
2023 41.4%
Phil,
I actually found the same low birth weight trend for whites as Langer et al in the post-election period. If you look at my Figure 2, the slope of the regression lines is negative in my graphs for the period Nov 2016 – Nov 2018 (the post-election period in Langer et al.) for both U.S. born and Foreign-born whites, just like Langer et al. reported. For Langer et al.’s pre-election period, I’m missing mother’s nativity in the first 14 months (November 2012-December 2013; not provided in the public use data). The longer-term trend through December 2023 is, however, upward (the incidence of low birth weight has been rising for a long time, e.g., https://time.com/6965173/americas-birthweight-crisis/).
I didn’t mean to imply that literally every month has a higher low-birth-rate percentage than the previous month, I’m just saying that the general trend is upwards for all of the demographics. I didn’t know about this phenomenon. Doubtless I shoulda, but I didn’t.
Perhaps more preterm pregnancies survive long enough to get counted?
Better pre/peri-natal care or fewer abortions could account for that. You’d be looking for about 50k extra births per year.
Also, could be the definition of “live birth” is being applied more (or less) uniformly as well. When I looked into this in the past each hospital had its own guidelines which may (or may not) line up with WHO/CDC definitions.