Ethan Steinberg writes:
A while back, you briefly blogged a bit about a very nicely done pre-K RCT on results up to the third grade:
I thought you might be interested that the authors have just published their sixth grade results.
It looks like the negative effects found in the third grade analysis seem to have gotten stronger:
Data through sixth grade from state education records showed that the children randomly assigned to attend pre-K had lower state achievement test scores in third through sixth grades than control children, with the strongest negative effects in sixth grade
I think the really interesting question about this study is, if the study is correct, how were we so wrong previously? Someone posted the above plot from a 2013 article that really shows the difference in how much earlier pre-K studies differed from later pre-K studies.
Is this a change in types of pre-K studies being done, a change in the environment (maybe something about the internet really changed the effectiveness of pre-K?), or a publication bias issue?
I don’t know! It’s my impression that those old studies were so noisy as to be essentially useless for any quantitative purposes. It’s funny how that could happen. I’m guessing that all these designs included power analyses but with massively overoptimistic hypothesized effect sizes, which can happen if you don’t fully think through the implications of treatment effect heterogenity. Kinda scary to think of all this money, effort, and statistical analysis that was missing this basic point. To really understand this, we have to go back to the gung-ho Cold War mindset of the 1950s and 60s, the attitude that, with sufficient fortitude, all problems could be solved.
“To really understand this, we have to go back to the gung-ho Cold War mindset of the 1950s and 60s, the attitude that, with sufficient fortitude, all problems could be solved.”
Hey! What happened?? The boat was just here and now it’s gone!! Wha???
Back when it was vogue to use teaching methods that actually worked, like phonics in reading! Darn optimists!!! What were they thinking by using teaching methods that were known to work???
Thankfully, in these days of modern science, we’re strongly focused on **scientifically tested** methods! Like the 30 minute video that improves lifetime earnings!! Like we know that the color of the teachers skin accounts for almost 0.00000073652288% of learning acquisition!! This is extremely important!! Like getting rid of AP classes to prevent competence across the board and create equality of ignorance among all students!!! Like blaming social media for the disaster created by closing schools for two years!!! Like getting rid of testing because it makes students feel bad, or because numbers are racist!
Yes, with modern science we’re *right on top of it*. We just missed a few things about the stats. That’s all.
Note that the forerunner of the Department of Education was created shortly before this graph in 1953. The Department of Education itself was created in 1979. Great job for US government!!! Our tax dollars screwing things up!! Would love to have another graph on top showing federal education spending.
I dunno: is truth stranger than fiction, or is this just a great example of the classic outcome government policy?
“is truth stranger than fiction, or is this just a great example of the classic outcome government policy?”
Oops.
Your pearl-clutching about the school closures suggests that you think that they were terrible for those poor neglected children who were not allowed to be in a classroom. And now we have a study that suggests that being in a classroom is detrimental for young children, with pre-K being a “classic outcome government policy” that somehow goes in the opposite direction. So yes, if you can believe both before breakfast, I’d call your truth stranger than fiction.
I don’t think there’s really any point in engaging with comments like this. It’s transparently just a disorganized spontaneous rant, barely related to the original topic; there’s no content here you can’t get by watching Grandpa Abe on the Simpsons.
I think somewhere within Chipmunk’s bizarre rant there’s a kernel of a good point; it may not necessarily be the case that the older studies are wrong, it could be the case that it *was* beneficial (on average) at some point, but has since become detrimental for whatever reason. I have no idea if that’s actually the case or not (I’m not American, so I’m not sure what “pre-K” even is), but it’s a possibility.
I didn’t read the blog post properly and spoke too soon – I see now that this was already raised as a possibility in Steinberg’s message. Nevermind.
Yeah, I think Chipmunk is correct that it’s unlikely the treatment itself is time invariant, let alone the response to the treatment. I’ve always wondered how much of a response there is to studies like this by parents whose kids don’t partake in the treatment. A lot of communication regarding a child’s abilities during school is based on their relative position in the classroom. If a treatment is successful (let’s also assume it’s effect is homogeneous), wouldn’t parents whose children aren’t treated respond to the fact that their child is no longer in the top x% of the class? I know there’s a small literature on this (IIRC it’s referred to as the Rat Race Effect), but it seems relatively unexplored as an explanation for why treatments don’t scale.
Pre-K is a grade before Kindergarten. Kindergarten is usually for 5 year olds, so Pre-K is for 4 year olds. It’s basically a govt subsidized govt run single year of preschool.
Thanks. Of course it’s easy to guess it’s something like that from context but it’s nice to have confirmation.
I do wonder if this is a case of technology helping. In the 1950s we didn’t have education TV/video games, toys were less sophisticated, and low-income kids would not have many things to do that were at least partially educational. These days even a poor household likely has PBS and other kids TV; picture books, and other enrichment toys. The true effect of pre-k is then a lot smaller as a kid left on their own with modern toys and kiddie Netflex will do a lot better than with a few wooden blocks.
I very much doubt there’s any simple answer. My wife and I are good friends with a long-term Head Start teacher. Among other things, the population mix that shows up has changed dramatically over the years she’s taught.
Just one data point from a rural area, but there have been lots of other changes.
I’m guessing that by RCT, there are both HeadStart and non-HeadStart students in the same classrooms?
I wonder if pre-K is one of those things where it really doesn’t help any student that much unless it is implemented universally. A possible mechanism for this… non-HeadStart students not ready for the classroom cause disruptions and slow the pace of learning for both HeadStart and non-HeadStart students. In later years, perhaps educators have to teach to the level of the non-HeadStart students, instead of at the higher pace the HeadStart students are ready for?
Are there similar longitudinal studies comparing universal pre-k with no pre-k, that are able to well control for other confounders, while removing this ‘network effect’?
And I have no intuition on how the effect of a pre-k program could end up being negative. Is a 4 year old’s time better spent sitting at home in front of the tv than in a classroom?
do the authors believe their negative effect estimate? Do they have a hypothesis for why this could happen?
It wouldn’t be hard to imagine a 4 year old who finds it suddenly extremely stressful to be dropped off at a school rather than taken care of by a parent, who rebels against the whole procedure and actively subverts the schools benefits through acting out, refusing to listen, even violence. We had a 6yo shoot a teacher in the US within the last year. Many things are possible.
The full preprint is here, though I think there was at least one correction to the results in the final version.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9716729/
There are a few things that i’d like to know more about. First, why not run a within subject analysis? Well, one issue is that there is quite a lot of attrition by 6th grade. And just like all these studies, it could be that the people who, for example, left the state may be different. They spend a lot of time talking about the 141 students who did attrited before the kindergarten year, but much less on what happened to the others.
As the authors say, the effect sizes are small, though they get their *. Having been through the elementary school thing with two kids, I’d say we don’t necessarily know what an IEP means and the same goes for disciplinary incidents (and the two might be associated in some cases).
“Early childcare program” is not a thing; it’s a category of things.
Here’s a description of the “Abecedarian” program that is pointed out on the figure, according to the 2013 article the figure comes from: “The Abecedarian program, which served 57 low-income, mostly AfricanAmerican families from Chapel Hill, North Carolina, provided even more-American families from Chapel Hill, North Carolina, provided even more-intensive services than Perry Preschool. Beginning in 1972, children assigned to the Abecedarian “treatment” received year-round, full-time center-based care for five years, starting in the child’s first year of life. The Abecedarian preschool program included transportation, individualized educational activities that changed as the children grew older, and low child–teacher ratios of 3:1 the youngest children and up to 6:1 for older children. Abecedarian teachers followed a curriculum that focused on language development and explained to teachers the importance of each task as well as how to teach it. High-quality health care, additional social services, and nutritional supplements were also provided to participating families.
Do current ‘early childcare’ programs last for five years and involve a teacher for every three to five children? I don’t know — I know nearly nothing about such programs — but I know which way I would bet.
It seems possible (at least to me) that programs back in the sixties, seventies, and early eighties used to be fairly effective on average, but that they were so costly and labor-intensive that the current programs aren’t nearly as good.
The economics of education people talk a lot about the kinds of children who were enrolled in these programs changing over time. Partly it relates to where the programs were rolled-out, partly to the outside options for children and partly to changing groups of parents taking advantage of these programs (which is related to the first two points). There are also some serious differences in what “Head Start” or “Pre-K” meant at different places at different times – it’s not like it is a single suite of activities/inputs, exactly what is being provided is very context dependent.
Which is all just to say it isn’t crazy to think that the actual effectiveness of Head Start/Pre-K has declined over time relative to the “what they would otherwise do” counterfactual. It might really have worked well in the 1960’s and 1970’s in the places it was implemented.
Well the question becomes: What happens if you take it away? As opposed to what happens if you add.
The 6th grade results from the evaluation of the Tennessee program are not consistent with the broader literature, or with earlier time points for the same study, which did show positive effects of public preschool on student outcomes.
Here is my analysis based on a recent review of this literature (Sulik, Townley-Flores, Steyer, & Obradovic, 2023):
We expect public preschool to have positive effects on school readiness to the extent that it provides a higher quality learning environment relative to what children would have experienced in the absence of public preschool (i.e., the counterfactual; Feller et al., 2016). The effectiveness of public preschool has likely declined over time not because the quality of instruction has decreased, but instead because the average quality of the counterfactual has increased substantially in recent decades (Duncan & Magnuson, 2013). For example, the average quality of home learning environments increased between 1998 and 2010, with gains concentrated among low-income families (Bassok et al., 2016).
There is broad consensus that public preschool has positive effects on students’ academic skills at school entry (Phillips et al., 2017). For example, Duncan and Magnuson’s (2013) meta-analysis of 84 “methodologically sound” experimental and quasiexperimental impact evaluation studies concluded that public preschool positively impacts students’ academic skills. Drawing on data from some of the most recent evaluations of preschool programs, Barnett and colleagues (2018) used regression discontinuity (a quasi-experimental causal inference technique that compares preschool students to kindergarten students who are around the same age) to estimate the impact of eight different targeted state and local programs in the United States between 2004 and 2015. Across these studies, the average short-term (e.g., 3 month) effects were large for literacy (ES = 0.85) and moderate for math (ES = 0.56).