Daniel Arovas asked for my impressions of this article, “Ethnic studies increases longer-run academic engagement and attainment,” by Sade Bonilla, Thomas Dee, and Emily Penner.
I responded that the article looked reasonable to me and asked if there were particular concerns that he had.
Arovas replied, “None in particular but seeing as this is both topical and in a fairly politicized field of research, I was a bit wary.”
I’ve been known to be suspicious of regression discontinuity analyses, but in this case the running variable (high school grade point average at the time of treatment assignment) is strongly predictive of the outcome, as we would expect it should be, and the curves are clean and monotonic; they don’t show the wacky edge effects that we’ve seen in problematic discontinuity regressions such as here, here, and here. As I wrote in a recent textbook review, “when you look at the successful examples of RDD, the running variable has a strong connection to the outcome, enough so that it’s plausible that adjusting for this variable will adjust for systematic differences between treatment and control groups.”
What about the conclusion, that, for these students, taking an ethnic studies class will have such huge effects, reducing high school dropout rate by 40% and increasing college enrollment by 20% (that’s what it looks like from eyeballing Figures 1 and 2 of the paper)? I don’t fully understand these numbers (in particular, it seems that more kids are graduating from high school than were attending in year 4; they get into some of these details in the paper regarding students who drop out or move to another district or state, so I guess someone can check how they analyzed these cases), but the patterns seem clear. The paper also considers subgroup effects but I’m pretty sure those will be too noisy for us to learn much.
The thing I don’t fully understand is what are the treatment and control conditions. If a student was required or strongly urged to take the ethnic studies class, was this added to the schedule or did it take the place of another class? What they say in the article seems plausible, that you’re less likely to drop out of school if you’re taking a class that’s more relevant to your experiences. I’m still hung up on how large the estimated effect sizes are. I guess one way to think about this is that, if this class can reduce dropout rate by 40%, then lots of kids must be dropping out who really don’t need to, who are in some sense just on the edge of dropping out, and there should be lots of ways of keeping these kids in school. Which is not to say the ethnic studies class is a bad idea; just trying to put these results in context.
I had to check the date to see if it was April 1.
Two points that I assume Andrew would normally make:
1. The nice clean graphs are a result of post-processing the data. The underlying graduation “rate” is extremely noisy – each actual student is either a zero or a one. The usual request to show the raw data would reveal a much less obvious pattern here, even if the authors were clever about displaying the heaped data.
2. The chief concern is that there might be *another* policy that kicks in at the *same* GPA. That seems reasonably likely and I am not sure how to rule it out. I see this as much less of a concern if the intervention is something like suspension or throwing kids out of school – that’s the end of the line, so to speak. But if you are in the middle of the distribution and your GPA falls below some salient cutoff you might also get a bunch of other extra help.
Anon:
Both good points; thanks!
Really? I disagree with (1). Precisely because the underlying data is binary, the authors are correct to plot means rather than the raw data. The mean p and number of samples n is a complete description of the data. The only possible issue is if the bin widths are too big and they obscure meaningful differences between p at the left and right sides of the bin, but it doesn’t look like that’s an issue here. Am I missing something?
Side note: the reCAPTCHA required for commenting wasn’t appearing for me in Firefox. Seems to work in Safari though.
Daniel:
Yes, good point: you wouldn’t want to plot raw binary data. Let’s just say that the coarser the averaging, the smoother you’ll expect things to be.
With (2), the authors do mention this. There is an Early Warning Indicator (EWI) that kicks in at the same cutoff. These interventions are at the district level, they say. They say they fail to find an effect at the cutoff in the schools within the district that aren’t participating in the ethnic studies pilot program.
I still think the non-participating schools should be a baseline. It’s way too common for people to drop parts of their model based on significance.
It also leaves us to wonder if the schools that chose to participate in the ethnic studies program implement their EWI interventions better. The participating schools are self-selected after all!
Yes, seems like it should be a diff-in-diff RDD
RE: The thing I don’t fully understand is what are the treatment and control conditions. If a student was required or strongly urged to take the ethnic studies class, was this added to the schedule or did it take the place of another class? What they say in the article seems plausible, that you’re less likely to drop out of school if you’re taking a class that’s more relevant to your experiences. I’m still hung up on how large the estimated effect sizes are. I guess one way to think about this is that, if this class can reduce dropout rate by 40%, then lots of kids must be dropping out who really don’t need to, who are in some sense just on the edge of dropping out, and there should be lots of ways of keeping these kids in school.
I think this is the same thing you’re talking about but to me those large effects (40% lower dropout rate!) from what seems like it could be an almost trivial “intervention” suggests something along the lines of a Hawthorne effect. It could well be that certain students at risk of dropping out would respond to almost any intervention or program or special attention by remaining in school. Maybe it’s my public health background but I’d want to suggest some “usual care” comparison condition. Perhaps adding a special class in a STEM area or music as the alternative to “Ethnic Studies”.
I took special ethnic studies every day and all day long. My exposure to Anglocentric, Protestant, and male dominant culture was uninterrupted. It didn’t harm me and actually made me appropriately cautious about accepting the predominant paradigm. It might have harmed my classmates who incorporated it in the amusingly mistaken belief that they were beneficiaries rather than being groomed for exploitation.
Oncodoc:
Good point! I took took the ethnic studies program at school when I was a kid. It featured special units such as “U.S. History,” “gym class,” and “Christmas carols.” I guess it didn’t seem like ethnic studies because everyone was in the program.
“Anglocentric, Protestant, and male dominant culture”
i.e., the culture that doubled human life expectancy, freeing humans from the Malthusian trap after ~200K yrs of hunger-ridden subsistence. Can’t imagine why anyone would teach that stuff. Criminal.
Anon:
Read Oncodos’s comment carefully. He said that the ethnic studies course he took didn’t harm him. His concern was that it may have instilled a dangerous attitude of complacency in the other students in his school, those of the majority culture.
Also, I guess the ethnic studies curriculum you received may have been different from mine. In my school, the ethnic studies curriculum had just about nothing on doubling human life expectancy and freeing humans from the Malthusian trap. It was mostly about things like wars, colonial exploration, sports, and religion. All important aspects of life, no doubt, and I don’t think there’s any question about covering these topics; the question is what will the perspective be. It would kinda be cool to have an ethnic studies curriculum that focused on changes in life expectancy; this isn’t what we received back in school, that’s for sure!
Andrew, oncdoc said:
“It might have harmed my classmates who [mistakenly believed] that they were beneficiaries rather than being groomed for exploitation.”
They might not have noticed the exploitation, given that it gave them 35 years of additional life (since 1900), with half of those years coming in the retirement they wouldn’t have lived to see in 1900. Now it’s possible that !Kung Bushmen were on the threshold of a doubling of life expectancy when interrupted by wars and colonialism. We’ll probably just never know. I wonder if we could do a regression discontinuity analysis…
Right! There was nothing wrong with slavery or Jim Crow laws, no need to learn about them or their effects. And sure, the genicide of the Indians was ethically questionable at the time, but now we can look back on them with the full knowledge that the ancestors of the Indians who -did- survive have life expectancy higher than they would have. The higher life expectancy not only justifies all of history but also eliminates any interest people might learn about any culture other than that of Europeans and their descendants. Totally agree. Seems obvious to me.
Phil:
Be careful with sarcasm on the internet!
Phil said: “There was nothing wrong with slavery or Jim Crow laws…” (blah blah)
I learned about slavery and conflicts between whites and Native Americans in my vanilla history class. I didn’t have to take a special cultural studies class for that. Did you?
But teaching slavery only in the context of the US south is misrepresentative. Slavery was common throughout the history of civilization and most slaves that came to the US were captured by other blacks. It’s not an atrocity to capture and sell slaves if they’re the same race? Native American groups routinely captured slaves in their battles with one another and of course the Aztecs practiced ritual human sacrifice. At Cahokia (1300s, shortly before European contact) entire coteries of servants were sacrificed and buried with their masters. That’s not Native American history? It’s not a barbaric atrocity? After all, Native Americans claim the bones of the 10,000 yr old Kennewick Man as their ancestor.
This doesn’t justify any atrocity. But perhaps we should study the atrocities of non-white groups among one another alongside interracial atrocities, lest we give the impression that human nature differs across races. I think that would be a more important lesson: skin color is just one of many excuses for atrocities, as we’re seeing in Ukraine right now.
Not very effective sarcasm.
Anon:
Setting aside any political disputes, I like your implicit idea of teaching history as a series of bad guys. As it is, I feel like history is taught as a series of heroes. OK, there are one or two bad guys in the story, but it’s mostly heroes. So you’ll hear all the good things about some historical figures and then it’s balanced out by one or two flaws. What about flipping it around and treating all the historical figures as assholes with one or two redeeming characteristics? That would be going too far but there’s an appeal to it too.
Anon:
Southern United States slavery was part of the largest economic system ever supported largely by slave labor. That qualitative distinction seems worthy of special attention.
Anonymous,
I learned about the -existence- of slavery in my vanilla history class. But I learned nearly nothing about the culture of the slaves, or the culture they left behind. Ditto for what I now know to be the extremely distorted (and very short) descriptions of the Indian cultures that used to populate this continent.
I’m going to go waaaaay out on a limb here and guess you’re white. If so, you’re serving as a great illustration of what has come to be called “white fragility.” It’s a phenomenon that interests me. This is only the second time ive experienced it in direct conversation, although I’m told it’s widespread.
Ignoring infant mortality, modern hunter gatherers have a life expectancy of around 70 years old. This is corroborated by anthropological evidence, particularly the length and age of bones from pre-agricultural societies. The evidence suggests that hunter gatherers lived longer, healthier, and grew larger than their counterparts in agrarian cities. “We’ll never know”—you’ve never tried, all you’ve done is guess!
Again, the drop in infant mortality is wonderful—I’m not advocating for a return to hunter gatherer society. But regarding the actual lengthening of adult lives, the rise you’re describing is actually a reversal of what had been the effects of urban labor.
Hopefully this all works!
Curious said:
“Southern United States slavery was part of the largest economic system ever supported largely by slave labor”
I’ve heard that too, but I’m not sure it’s true, but I agree that American slavery was a huge commercial operation, and in fact that was part the reason many northern whites to wanted to end slavery: they couldn’t make a living farming against slave plantations. I don’t know what to make of this data. I’m not sure exactly what it represents but it seems to say that, at least in terms of the Atlantic trade, the US was a far smaller player than the European powers. I’m not wanting to make too much of it, it seems almost too much to believe.
Phil Said
“But I learned nearly nothing about the culture of the slaves, or the culture they left behind. “
I didn’t learn much about that in regular classes either I guess. I watched “roots” when I was younger. I’ve been learning about slave culture in bits and pieces since, especially through music: “Wake up momma, turn your lamp down low; You got no nerve, baby to turn uncle John from your door” (Blind Willie Mctell).
Somebody Said:
“Ignoring infant mortality”
It must be pretty large:
For human hunter-gatherers, mean life span at birth is about 31 [ranging from 21 to 37 in several populations (3)]. For Swedes, it was about 32 in 1800, 52 in 1900, and is 82 today (17). So life expectancy increased by about 165% from hunter-gatherers to modern Swedes and at a rate of about 12% per generation since 1800.
That’s , That’s a little different than what you quote. What do you make of it? I’m certainly not an expert. But it has a lot to like from the modern culture perspective:
We show that human mortality has decreased so substantially that the difference between hunter-gatherers and today’s lowest mortality populations is greater than the difference between hunter-gatherers and wild chimpanzees. The bulk of this mortality reduction has occurred since 1900 and has been experienced by only about 4 of the roughly 8,000 human generations that have ever lived. Moreover, mortality improvement in humans is on par with or greater than the reductions in mortality in other species achieved by laboratory selection experiments and endocrine pathway mutations. This observed plasticity in age-specific risk of death is at odds with conventional theories of aging.
I’ve had the unfortunate experience of working in remote areas around severely disadvantaged communities, often ridden with substance abuse – at far higher levels than in average American society. It’s really tragic. I sometimes feel their groups are holding them captives to protect their culture, but inflicting them with hopelessness and isolation.
I was very specific about infant mortality vs adult life expectancy for a reason. You should learn what infant mortality is, and also how an expectation value works.
anonymous –
> I sometimes feel their groups are holding them captives to protect their culture, but inflicting them with hopelessness and isolation.
It’s interesting that you see these serverly disadvantaged groups living in isoloated areas has having enough agency within the larger social context such as that they are holding themselves captives to protect their “culture” (whatever that means) and causing themselves hopelessness and isolation. I don’t think I share your view of the magntitude of their agency relative to other factors.
But you say that you only feel that way sometimes. What do you think at other times? And what is it that explains the difference in your views at different times?
“I was very specific about infant mortality vs adult life expectancy for a reason”
But you don’t care to share it with other readers, apparently? In a debate your response is “go read a reference”?
It’s the same reference, you just have to learn to interpret curves and quantitative data. The increase in life expectancy is driven primarily by a thirty-fold decrease in infant mortality and a hundred-fold decrease in early childhood mortality. You can see the mortality curves progressively converging as age increases. It’s still true that the hazard rate (in the Cox model sense) is higher for hunter-gatherers throughout their lifespan, but to interpret this as an “equivalent age” is misleading. Medical science has not slowed down or reversed aging–it has reduced the mortality of acute disease and injury, especially in early childhood. A 35 year old bushman does not have the body of a 75 year old Japanese salaryman. If you interpret it that way, hunter gatherers would almost never live to be 65, which is simply not true. Looking at the same dataset from your reference
https://content.csbs.utah.edu/~hawkes/Blurton%20Jones%20et%20al.%202002%20AJHB.pdf
some of the tribes studied have about 8.5% of their population as over 60s. Again, I’m not saying that hunter gatherer life is great. I am specifically correcting the statement
Because it is a common misconception, where people wash away the details in the averaging. Average lifespan was short because lots of babies died. That’s terrible, but adult’s weren’t on death’s door when they were 40. Medical science has not slowed the natural aging process.
On the broader point–I generally agree with you. I love living in the United States today, I would rather live here than anywhere else at any other time. And I’m glad I learned a lot about American history. Nobody is arguing that people should know *less* about it. But I do think learning about other cultures is valuable too–we understand ourselves better by placing ourselves in context.
Another minor correction
if anyone was responsible for freeing humans from the Malthusian hunger trap (though again, for much of those 200K years humans were just at a lower population rather than going hungry), it would be Haber and Bosch for the Haber-Bosch process for artificial nitrogen fixation. Both of whom were German, and one of whom was Jewish. Not a part of the “Anglocentric, Protestant” culture at all. Again, not taking away from the broader point that the history of White Anglo-Saxon Protestants is valuable to learn, but pointing out that it might be worth learning about other people as well.
Why do historians write of the coastal Northwest Indians that “Food more or less came to them, and it was always available” (long ugly link source provided upon request), contradicting Malthus’s evidence-free story? How does the reported doubling of life expectancy square with my brother’s suicide at age 49, despite enjoying academic and career success (BA from Berkeley in Economics, well-paying job as Accounting Manager at a multinational company)? Why are overdoses and suicides rising? Would I be happier living a simple Gandhian lifestyle than I am now, trapped in Western materialism with ubiquitous ads for meat bombarding me?
By Gandhian, I have to assume you mean “racist towards Blacks and into sleeping with young teens”. You know, things that happened.
It’s almost like cherry picking is easy!
“By Gandhian…”
“Anglocentric, Protestant, and male dominant culture” causing so many problems! It even corrupted Gandhi!
That seems irrelevant. The point is that actual empirical evidence suggests that small scale societies tend to have very high quality of life. That’s true whether the people are racist or not.
“empirical evidence suggests that small scale societies tend to have very high quality of life.”
We just need about 50-100 more earths to all live in “small scale” societies.
‘historians write of the coastal Northwest Indians that “Food…always available” ‘
The plagues brought by Europeans decimated American native populations. Some estimates claim that the population of S America may have fallen by 90% due to European disease. That would leave a relative abundance of food for generations.
Again, irrelevant. Yes, the Earth would not support our current population living as tribal hunter gatherers. But hunter gatherer societies do not and did not experience the same explosive population growth.
The point here is not a prescriptivist call for people to move back into the wild and start gathering fruits and berries. I am not denigrating civilization. The point is specifically rebutting your ahistorical claim in
That’s just you uncritically regurgitating enlightenment-era propaganda, when it was trendy to try and divine history and observable facts by meditation and pure reason. It doesn’t stand up to confrontation with the actual anthropological evidence.
“hunter gatherer societies do not and did not experience the same explosive population growth”
Isn’t that why they are almost extinct? :) Apparently purportedly amazing benefits of the hunter gatherer society weren’t enough to keep people from wanting something else. Are you going to be the Cass Sunstein of anthropology and claim that their abandonment of H/G society was irrational? What nudge would you design to instigate people to recreate the hunter gatherer society? Nuclear war? :)
+1 for gratuitous Sunstein reference.
The selection pressure againdt from hunter gatherer society is that agricultural cities can raise standing armies and conquer hunter gatherer societies. The moralization of selection pressures and narrativizing history as things getting monotonically better is, again, uncritical regurgitation of speculation from interested parties.
And again, no I do not want to live in a hunter gatherer society.
Anonymous, is brute force the “nudge” that Anglos used to “assimilate” Native Americans? “Kill the Indian, save the man”? How will you persuade the North Sentinel Island hunter-gatherers to adopt your clearly superior lifestyle, without a violent invasion?
I’m skeptical.
1) those seem like enormous effect sizes.
2) the effect size grows with each year after they took the class! That’s a unicorn!
3) the effect is serendipitous. The authors say the district-wide interventions for at-risk students (gpa <2.0) had no discernible effect on graduation rates, but the ethnic study course did, even thought it wasn't designed to help them graduate.
4) their sample size is smaller than I first thought. From their preregistration:
Sample size
The total sample includes 1,405 students. These students are part of five different high school year cohorts. Within each cohort students were assigned to take Ethnic Studies as 9th graders if their 8th grade GPA fell below the mandated threshold (i.e., 2.0). Of the 116 students with an 8th grade GPA below 2.0, 64 (55 percent) ultimately completed the Ethnic Studies course.
I'm not sure what "ultimately completed" means. Took or passed? Also, they say in the paper that students below a 2.0gpa were 27 percentage points more likely to take the class. So how many students were actually affected by this treatment? 30? I'm actually unsure.
The effects could be true. The data certainly look nicer than most RDDs on here, but I'd need to look into it more.
Side note: I dislike that they dropped everyone below a 1.25 GPA in 9th grade (post intervention!). Probably not many students, but it feels arbitrary and unjustified.
The data presented looks like it tells the story they claim, but the effect sizes are implausibly high. Leads me to believe they’ve preprocessed the data waaaaaay too much. Frankly, I don’t believe it, and with a claim this big I’m leaving towards it being total incompetence or outright fraud.