Part 1: Reading what different sources say
The other day, as part of a long discussion about the estimated effects of Mississippi’s education plan, I quoted some education researchers, Wainer et al., who wrote:
The 2024 NAEP fourth-grade mathematics scores rank the state at a tie at 50th! The eighth-grade scores also qualify for 50th place.
I also quoted a different critic of the Mississippi claims, Ravitch, who wrote:
In math, [Mississippi’s test scores] zoomed from fiftieth to twenty-third. Adjusted for demographics, Mississippi now ranks near the top in fourth grade reading and math according to the Urban Institute’s America’s Gradebook report.
And I found this from the wikipedia page on the Mississippi Miracle:
After adjusting for demographics, in 2024, Mississippi was the nation’s #1 state in Reading as well as in Mathematics.
I wrote, “But Wainer et al. say that Mississippi is tied for 50th in math. Can they really be worst in the nation, but best after demographic adjustment? I guess it’s possible.”
Part 2: Anomalies!
Wainer et al. said Mississippi’s 4th and 8th grade math scores were the nation’s worst in 2024.
Ravitch said their 4th-grade math scores have increased to 23rd in the nation and that they’re near the top when adjusted for demographics.
Wikipedia said that Mississippi’s math scores were best after adjusting for demographics.
So, Wainer et al. and Ravitch flat-out disagree on Mississippi’s absolute ranking in 4th-grade math; Ravitch and Wikipedia disagree slightly on the result after demographic adjustment (“near the top” or “the nation’s #1 state”); and I can’t be sure, but it also seems doubtful that a state could be #50 unadjusted and #1 after adjustment. As I wrote, it’s theoretically possible but it seems like a stretch.
Part 3: I do nothing.
One of my sayings is that an important characteristic of a good scientist is the capacity to be upset, to recognize anomalies for what they are, and to track them down and figure out what in our understanding is lacking.
In this case, though, I just let the anomaly sit there like a rotting fish. I went around it and I kept writing.
Why did I not explore this 4th-grade math test thing more closely? Partly because I didn’t have the data and hand. It turned out that a quick google was all that was needed, but I didn’t take that step. Another thing is that, in any investigation, many anomalies will come up (one of these was the average age of the students being tested; more on that below), and we can’t look into everything at once. In that way, it’s a like an Agatha Christie-style mystery, where various inconsistencies and anomalies arise and are noted in turn, but then the story moves on, with the explanation happening later. The other day we saw the new Knives Out movie–it was really great! If the original Knives Out was a 10 and the sequel was a 3, this third installment was a solid 9–and it did that thing were anomalies would pop up and get discussed but then set aside. If you stopped the train at every anomaly, you’d never get to the destination.
And the math scores were not a key part of the story, so I just let my bafflement sit there and I did not follow up.
Part 4: Let’s look at the numbers.
In the discussion of our post, two commenters said that Wainer et al. were wrong on the math scores. Steve wrote:
You can look the data up on the 2024 NAEP report:
https://nces.ed.gov/nationsreportcard/
I have no idea how these researchers came up with these claims: “The 2024 NAEP fourth-grade mathematics scores rank the state at a tie at 50th! The eighth-grade scores also qualify for 50th place.”
My reading of the report is that Mississippi’s 8th grade math scores had trailed the national average by 18 points in 2000 but by only 3 points by 2024.
And SD wrote:
“The 2024 NAEP fourth-grade mathematics scores rank the state at a tie at 50th! The eighth-grade scores also qualify for 50th place.”
This is just literally made up
So I looked it up, and . . . yeah, Wainer et al. had it wrong! Here’s what it says on the NAEP page:
4th grade math: National avg 237, MS avg 239, above average!
8th grade math: National avg 272, MS avg 269, but rank is approx 35th, not 50th.
Also I went to the Urban Institute page to see their demographically adjusted numbers (“The demographics we use for the adjustment include gender, age, race or ethnicity, receipt of free and reduced-price lunch, special education status, and English language learner status”) for 2024:
4th grade math: MS 248.6, they are indeed #1!
8th grade math: MS 281.3, also #1!
You can make of this adjustment what you will. But, in any case, no way were they ranked #50. I contacted Wainer et al., and Dan Robinson, one of the authors on the paper, confirmed that this was a mistake and that they would remove those two sentences from their paper.
Part 5: Where are we now?
As I discussed a couple days ago, I’m coming at this from two directions.
On one side, Wainer, Grabovsky, and Robinson are experienced education researchers, and they are not impressed by the claimed large effects of Mississippi’s policies.
On the other side, Wainer et al. are making their arguments in general terms, and the specific numbers from Mississippi seem impressive. This “on the other side” point is even stronger when we consider that Wainer et al. based part of their argument on math scores on garbled numbers.
There’s also a political angle, which I did not discuss in my original post but which came up in the comments, and it’s interesting because both side’s arguments have a politically conservative flavor. It’s a conservative vs. conservative battle. The proponents of the Mississippi plan offer the conservative argument that back-to-basics education work, also the conservative (in the U.S. context) argument that Mississippians are as good as anyone else. The skeptics of the Mississippi plan offer the conservative argument that there are no miracle cures, that schooling can’t do much to alter the natural order of things, and that government statistics can’t be trusted. I’m exaggerating the political slant in both directions here, but I do think that the arguments are taking place on a conservative turf, which is interesting, and I guess reflects the discrediting in recent years of education practices associate with the left.
Before ending this discussion, though, I wanted to go back to the statistics. Not the details but more of a view from 30,000 feet.
– An intervention was done in Mississippi in the mid-2010s, and people studied state-level aggregate test scores before and after. Mississippi’s test scores improved a lot relative to the nation during this period. This was part of a longer-term improving trend.
– The estimates of the program’s effects are observational. There was no control group. The implicit control is to imagine that previous trends in the state would have continued, or that the trends in Mississippi would be like trends in other states afterward.
– We don’t have easily accessible data on individual students. Robinson asks, “For example, what students benefited most from the intervention? What happened to the scores of the retained students once they took the NAEP reading test again?”
– The critics were coming into this from a generally skeptical position based on their view of previous hype in the education field, also the clear statistical issue that if you delay the kids who are performing poorly on the test, that averages will go up, also the lack of a control group. They did not do the work to quantify these concerns in this particular case, in part because relevant data were not easily accessible, but their distance from the details was a problem, as we could see with the gross error regarding the math tests.
– Mississippi’s average test scores have been going up. How much is this due to selection of who takes the test and when they take it, how much is due to changes in accommodations for disabilities (as discussed by Kelsey Piper in comments), and how much is due to targeted test preparation, I don’t know. It is a luxury of blogging that I can openly admit my uncertainty here.
– Stepping back, it’s clear to me why Wainer et al. remain skeptical, while Piper and other reporters have a more positive take on the Mississippi program.
– Finally, it’s not all about average test scores and it’s not all about the students being held back. I’m still thinking that a key outcome is reading and math ability at the time of school leaving. The idea of the program seems to be that if you hold some kids back a year, that will help them learn by keeping them in classes that are closer to the right level for them, and that this will also allow a higher level of education for the kids who are not held back. Some commenters also argued that the threat of being held back would motivate kids to learn more in third grade. I don’t know about that, but the point is that the problem is complicated enough that I can see the virtue of a “reduced-form” approach that just looks at effects on average test scores–but then you have to be concerned about the lack of control group and about compositional effects, which is where we started!
Part 6: Summary
– I should’ve looked into those math-score claims more carefully! Once I noticed the discrepancy between different reports, that was the time to track down what was happening. I’ve criticized statisticians for just accepting unreasonable numbers without checking, so bad on me for sloppiness here.
– As before, I don’t have a strong take on what’s happening in Mississippi. I see good arguments on both sides and no easy way to resolve them. My Bayesian inclination is to split the difference and say there’s some evidence that these policies are working but not to the extent that is advertised, but I don’t really know. Indeed, I can think of this Bayesian splitting of the difference as a kind of frequentist procedure in the sense that, on average, I think we will do well by splitting the difference in this sort of dispute. In any given problem, I’ll often come down stronger on one side or another (as here, for example), but in this case, nah, I don’t really have more for you.
P.S. I get that many readers of this post and my earlier post on the topic are frustrated because I don’t come to a strong conclusion for or against the Mississippi program. But that’s because I can’t: it’s an observational study with a lot of uncertainty about key aspects of the data. We can criticize particular aspects of various reports on the topic, but that’s not the same as coming to a strong conclusion about the effects of the program. Meanwhile, though, policymakers need to make decisions. And this sort of decision can’t wait on definitive evidence; they’ll need to rely on some mix of theory, judgment, and an assessment of political possibilities.
P.P.S. In part 5 of the above post, I remark that the Mississippi discussion has turned into a conservative vs. conservative debate with not much from the liberal direction. Jonathan Chait discusses this too: at a liberal journalist who supports Mississippi’s school policies, he’s surprised that liberal pundits are taking the conservative line that the policies don’t work.
P.P.P.S. I received the following email from Jean Gordon Cook of the Office of Communication and Government Relations of the Mississippi Department of Education:
The Mississippi Department of Education was made aware of an upcoming article that appears to be set for publication in January in Significance magazine. The article casts doubt on the accuracy of Mississippi’s gains on the National Assessment of Educational Progress
We have noted several errors/issues with the article that we sent to the editor of Significance yesterday. We are sharing these items with you because you reference this article in a blog post.
• Incorrect information is in the second-to-last paragraph on p. 33 when it states that the “2024 NAEP fourth grade mathematics scores rank the state at a tie at 50th! The eighth-grade scores also qualify for 50th place.” 2024 NAEP state profiles show Mississippi’s fourth-grade mathematics scores rank the state No. 16 in the nation, and eighth-grade scores rank No. 35.
• Regarding the discussion of retention, the article does not address the fact that students can be retained for reasons other than the Literacy-Based Promotion Act. If you look at the 2018-19 LBPA Annual Report, you will see that 5,049 (14.4%) of third graders did not pass the third-grade reading test on the initial or two retests. Of those students, 4,131 were promoted to fourth grade with a good cause exemption. That means only 918 of the 3,379 third graders who were retained that year were held back because they failed the third-grade reading test.
• The article suggests that students who are held back in third grade may never advance to fourth grade and possibly be in the sample of students who take NAEP. It also doesn’t discuss the fact that students who are retained and students who are promoted to fourth grade with a good cause exemption are required to receive intensive remediation. This is a key part of the Literacy Based Promotion Act (LBPA) and Mississippi’s work to ensure students become strong readers.
The first point is covered in my post above, but I thought it was simplest to share the whole message.
In my reply to Cook, I apologized for not checking the numbers myself the first time. The funny thing is that, as I explain in the above post, those numbers did look odd to me, but then I didn’t follow through and try to look them up.
I would like to point out that Mississippi, as most states, does have a state longitudinal data system that would contain individual level student data: https://slds.ms.gov/. Most of the questions here, like tracking retained students, are one researcher request away from being definitively answered.
Seems inane to me to throw heavy statistical machinery at the publicly-available aggregate numbers when disaggregated data exists and is accessible.
As far as I can tell Gelman is not very competent at checking publicly available data.
Anon:
I’m pretty competent. But competent people make mistakes too! Indeed, it’s a mistake to think that a researcher (in this case, me) is “not very competent” just from one example where I reviewed some literature and did not check up on an anomaly. I made a mistake, someone pointed it out, I looked into it and fixed it a few days later.
I’ll clarify I meant nothing against you with this original comment.
Moreso I wanted to point out that for all the discussion of the Mississippi Miracle, as far as I can tell, no one has done the analysis that I would instinctively reach for.
Also, I’ll add here because I can’t go five levels deep to reply to another comment.
State longitudinal data systems are set up by many states, owing to Obama-era grants, to assist public policy and decision making. They keep track of student-level data – keyed by SSN – through postsecondary, workforce, and welfare. They exist explicitly to make this data available to researchers and policy-makers. The data is not “publicly” available in that it can’t be downloaded directly from the website, but if you email them they will likely send back a researcher request form that can be filled out to be returned deidentified individual level data, with use restrictions to protect sensitivity.
There are access problems, owing to understaffing and state agencies being generally over-cautious to release data to people outside their state, but they work well.
Sorry that some people in this thread are being so venomous.
> also the clear statistical issue that if you delay the kids who are performing poorly on the test, that averages will go up
I don’t think this is a “clear statistical issue”. Those students don’t disappear. They will appear in the next year’s data.
You might see boundary effects when the policy is initially applied, but there should be no steady state statistical effect.
Ethan:
I think the concern is that it’s not clear what’s happening with individual students. But I agree that the effects of selection will be complicated.
most states have something like this, how are you so obtuse?
https://slds.ms.gov
Robert:
You linked to a website that has some reports on Mississippi education. If I click through, there’s an interactive database on “Mississippi Community College and Public University Student Educational and Workforce Outcomes,” several reports on various outcomes of high school and college graduates in the state, and some miscellaneous reports such as “In-Demand Occupations in Columbus, MS.”
This could all be great stuff. I don’t see how it answers questions about what’s happening with individual students in third and fourth grade. I’m willing to believe that such information could be there somewhere; I just didn’t see it.
I looked up the definition of “obtuse.” According to Merriam-Webster, the first definition is ” slow to understand what is obvious or simple.” It’s not obvious or simple how the link you gave can answer the questions being asked above. Feel free to explain.
P.S. I guess this goes without saying, but I don’t find it helpful for people to sling around insults without some basis for it. I mean, sure, laugh at me for not checking the anomaly (that’s the point of the above post), but if you really think this website you’re linking to has the data we’re talking about, you should explain. This is a blog, not twitter, and if you have something to say, you can say it and not just give an insult and a vaguely related url.
One reason I think blogs are better than twitter is that in blogs we can take equivocal stances (as I wrote in the above post, I don’t have a strong take on what is happening in Mississippi, and that’s ok; I don’t need to act as if I have more certainty than I have), and we can point to evidence in as much detail as necessary.
“… They will appear in the next year’s data. …”
But when they do they will be a year older and will generally have learned something in the extra year. Suppose Mississippi relabeled first grade zero grade, second grade first grade and so on. Then their fourth graders would be everyone else’s fifth graders and would be expected to score higher than the real fourth graders.
Over the last few decades, a lot of states have changed the starting date for kindergarten, in part to get older students on average who score better on tests.
Not necessarily, the question is why were they not at the cut off level when they got to the end of third grade.
Yes there is some secular improvement at those ages related to whatever is happening in brain development. But those students will already be behind the upper half of students in the new cohort.
I don’t understand the Bayesian prior you are applying. It sounds like it is “the very worst readers in the state would have gotten so much better without any intervention that merely being a year older without holding them back to relearn what they failed on would have caused the scores to rise”.
But that seems like a rather surprising prior.
The prior is that children are learning stuff. Smart children are learning stuff faster but even dumb (but otherwise normal) children are learning stuff, just at a slower rate. So suppose the dumb kid is only learning half as fast as normal. So at the end of third grade they are at grade level 1.5, 1.5 grade levels below expected. A year later you expect them to be at grade level two, two grade levels below where they are supposed to be at the end of fourth grade. But if you hold them back at the end of third grade it will be two years before they finish fourth grade. So their expected score at the end of fourth grade will be grade level 2.5. Still lousy but half a grade level better. Do this with a bunch of kids and you will bring up average scores a little.
You can cause a steady-state effect if you’re willing to assume that student’s hypothetical scores on a test will rise over time regardless of whatever happens in their education. If that’s true, holding low scorers back will mean you get early scores from the smart kids and later scores from the stupid kids, and the fact that later scores are higher than early scores makes it look like you’ve improved outcomes for the stupid kids. (Even though, in this model, 100% of that improvement is artifactual.)
The Urban Institute adjustment strategy recodes student ages two years above the mean weighted national age. Description of the age variable from the UI NAEP variable appendix (https://apps.urban.org/features/naep/naep-technical-appendix.pdf):
“Age on February 1 of testing year, using date of birth estimated as the 15th day of the birth month in the birth year, with ages more than two years from the mean weighted national age recoded to the mean.”
MS has high rates of retention in K-2nd, plus a more recent increased retention in 3rd graders based on the outcome variable, so this adjustment may artificially keep the average age constant over time (as seen in some of the graphs) as well as affect the demographic adjustments.
There’s some evidence that these policies are “working,” but not to the extent advertised.
I don’t think questioning “to the extent advertised” captures the problem in full
The key question is: what does “working” actually mean here? Taken at face value, raising average reading scores could simply reflect changes in who is being assessed. Lower-scoring students may be retained or excluded, and students with learning disabilities may receive accommodations that alter the measurement without really nearing the the advertised causal mechanism. This is a reminder not to attach too much meaning to correlations without a solid understanding of the causal mechanisms.
As I talked about in the earlier thread, much of the justification for retention is supposed to be about improving outcomes among the retained students. But is measuring average scores really a measure of that? Imo, it seems highly inadequate. Outcomes among promoted students are being assessed directly, but those results may be confounded by hidden variables—selection effects, exclusions, or artifacts of policy. If the goal is to improve retained students’ outcomes, then their progress should be independently measured. If the goal is to raise average scores among promoted students, that should be stated explicitly—and then it becomes essential to ensure that the apparent improvement isn’t just an artifact of other outcomes, outcomes that *could* in fact reflect harm to the retained students.
What if holding back the bottom 3 students in a class of 35 improves performance among the other 32 students more than it worsens performance among the next set of students?
I’m saying that should be measured before anyone determines that the policy “works.”
I will say this. Among many hats as an educator, I was for a few years as a special education teacher in a middle and a high school. In my experience, an association between SES and designation as a special ed student was unmistakable.
It would be great if Mississippi found a way to address the needs of students who would be likely to be retained without directly addressing socio-economic factors. But I think a prior of skepticism is warranted.
Improving educational outcomes among other cohorts without improving (or perhaps worsening) outcomes in those students is a choice people can make – but doing so should be explicitly identified as a goal before you assess the policies.
My goal as a teacher (at all levels) was to help all the students I worked with to access a love of learning and to become empowered as learners). I don’t believe that there are inherently tradeoffs that need to be made between the progress of different cohorts of students – but that is a tradeoff that’s largely embedded in our educational paradigm. In my view, those accepted tradeoffs perpetuate class status quo, to the detriment of us all.
I’ve read different accounts of how the “Mississippi miracle” as impacted achievement gaps. For now, I’ll remain agnostic with acknowledging my priors. And I’ll remain opposed to advancing the achievement of some students at the expense of others. What about you, Steve?
“I’ll remain opposed to advancing the achievement of some students at the expense of others. ”
I won’t.
The fact is that there is a distribution of abilities. If we provide eveyone with equal resources – the term “resources” specifically meaning cost of providing education – we achieve greater benefit to everyone. If we focus our resources disproportionately on people with lower ability, we achieve less benefit to everyone – including the people with lower ability. In fact it makes sense to provide the people with greater ability a disproportionately large share of the resources – which is traditionally how American society has operated, to great success.
Families with limited resources have deployed this strategy throughout history and still do, because it generates the most future resources for the family, which in turn benefits the people of lesser ability in the family. IOW, this is an evolutionarily tested and successful method.
Win –
The fact is that there is a distribution of abilities.
As a long time educator, I have a lot of questions about the causal mechanisms behind this “distribution of abilities,” as well as the about what we choose to measure, and how well our measurement tools actually match the abilities we think we’re measuring
If we focus our resources disproportionately on people with lower ability…
You’re reading something into what I wrote. I didn’t say anything about focusing resources disproportionately. Also, I think your scope of “resource” is much to narrow. Simply measuring costs of education, it seems to me, isn’t very useful. I think you need to look at a much more expansive definition of “cost,” and you need to compare cost against return (say, in terms of incarceration rates of lifetime earnings) to get a very useful assessment.
In fact it makes sense to provide the people with greater ability a disproportionately large share of the resources – which is traditionally how American society has operated, to great success.
I’d dispute both parts of that claim. First, I’m not convinced America has consistently allocated resources proportionally to “ability.” Second, I’m skeptical that our national successes can be explained primarily by such allocation. One could just as easily argue that exploitation of the less advantaged has fueled much of that “success.”
Families with limited resources have deployed this strategy throughout history and still do…
I’m not sure what that means in practice.
IOW, this is an evolutionarily tested and successful method.
Evolution doesn’t operate on the time scale of public schooling policy history, so I don’t find that analogy persuasive.
Let me turn the question back to you: Do you see social mobility as a key principle of American success? If so, do you favor an educational paradigm that largely perpetuates the class status quo, or one that actively works to disrupt it?
Also embedded in CW’s reply is a certain vision for the purpose of education which is not universally held. If I remember correctly (although I wasn’t around at the time), the purpose of general education was to create a population that could meaningfully be entrusted with a democracy. Such a purpose would be optimally satisfied by raising everyone above the same common threshold, which would mean resource deployment inversely proportional to ability.
This is a bad framing. No students is being excluded from the test. They are being held back one year (or very very rarely two). Yes this could cause a selection effect rise in scores *during the first year after the change*. But we have eight years after the change. In those years the low performing students were tested. So the selection effect we can worry about year one is gone.
The only remaining selection effect is that they are one year older. If your prior is that the very worst readers would, without intervention, be better enough to significantly increase average test scores purely due to the one year delay, that would be a big concern. But is that a defensible prior? I don’t think so.
In this document (https://mdek12.org/sites/default/files/reports/LBPA/2025/LBPA_2024-25.pdf) 23% of grade 3 students are classified as being marked for retention based on initial MAAP-ELA taken in early/mid April (https://www.mdek12.org/sites/default/files/Offices/MDE/OA/OSA/Documents/2024-2025_mississippi_statewide_testing_calendar_8.28.2024.pdf). After the retest, taken in early May only 15.5% are classified as being marked for retention. (And in the end only 6% got retained).
That must mean 1 months worth of teaching does wonders for those at the bottom or the reading abilities distribution, so an extra 12 months worth of such teaching must create a big shift in the mean.
Or there is incredible regression to the mean i.e. the reading abilities of kids at the bottom of the distribution aren’t measured very well at all.
Or perhaps they make the second test easier but that would be incredibly unethical e.g. scaring people into believing their abilities are worse than they are and that they are on course for being held back.
Sebastian –
Do you know the average age of the more recent 3rd grade test-takers compared to the test-takers before the increased retention rate, or compared to 3rd grade test-takers in other states? Do you know whether any students effectively dropped out from the test-taking after being retained? If you think improved test scores are not merely a function of age, do you think the better aversge test scores are because the students who are retained test better the next year, and if so, to what intervention do you attribute the better test scores? More resources devoted to their education? More attention from teachers? More accommodations? More social service support? How do you find the better average test scores useful?
You appear to be almost completely non responsive. But as it turns out I do know how the average age changed—it largely didn’t. So now that you know that, does that change your analysis?
https://jabberwocking.com/mississippi-revisited-the-mississippi-reading-miracle-looks-to-be-real-after-all/
Again I see quite a few people who seem to use a Bayesian prior that seems questionable: that the lowest performing readers tend to get significantly better at reading merely by getting a year older.
Sebastian –
Thanks for that link. It doesn’t make logical sense to me that holding significantly more kids back wouldn’t increase the average age, all things being equal. So I’m perplexed by those data and I would guess there might be some issues with the “all things being equal” part. The obvious example would be that the students who were retained became home-schooled or moved to another school or for one reason or another not tested. And while I’m no statistician, I’m not sure that there are enough years post the retention policy being in place (3 years) to make a valid comparison to that 15-year average prior to the intervention. Especially since there was a pretty big jump from 2013 to 2017, and especially since there could have been any number of confounding variables that could affect enrollment numbers.
As for this:
that the lowest performing readers tend to get significantly better at reading merely by getting a year older.
First, yes, it’s not merely an issue of age, but that age can also be a proxy for an additional year of instruction and an additional year for any other kind of intervention to take place, or for accommodations to be provided for that student in the testing.
I still think, the data from your link notwithstanding, these raw averages are inadequate to draw much of any solid conclusions, let alone that a “miracle” has taken place. It seems to me we don’t k or much without more information at a more granular level.
I still think the useful question is how the interventions in MS affected different types of students, not the overall average. I mean, if all the relevant groups improved to approximately the same extent the average would be the story, but we wouldn’t know this until we looked at disaggregated outcomes. But as a former teacher, I know that just about everything that’s done in education has differential effects.
Note: this is not simply about a formulaic approach to equity. It’s about understanding better who is being helped (or hurt), how much and why. Really, without that I don’t know if you can even call it “understanding”. So many things in social policy are like this. (Sorry for the tone, but I’ve had to deal with this issue in one form or another too many times.)
All theyd have to do is provide average age and average score for each of the quartiles of age each year and youd have enormously improved visibility and understanding.
Individual schools are probably too small to break down a lot further but if you broke down age, race, income, and parents educational attainment into 3 groups each youd have 81 groups. Providing the average values for those variables and average scores, and N in each of the 81 groups at the state level (for any geoup that had N larger than say 20) would tell the distributional story we really want to hear.
I think these things arent done because we discourage distributional thinking in stats education. Its always about the average.
Daniel:
It’s tricky though because you don’t just want average age and other variables for each grade in the state; you’d also want it for the kids in the test. But, yeah, if you had the distribution the ages in which kids are entering first grade in Mississippi and other states, and how these distributions are changing over time, and also the proportion of kids held back each year, and their ages and test scores, and when they get tested . . .
One of the challenges here, as discussed above, is that this is an observational study, with implicit and explicit comparisons to baselines defined by other states and by a hypothetical Mississippi with no changes in policy. Another challenge is that sometimes people are talking about a particular policy adopted by Mississippi in the mid-2010s and other times they’re talking about everything that Mississippi has done in this area in the past few decades.
In the previous post you quote Wainer et al:
“Third, Patrinos, and others who have praised the Mississippi miracle, should know that extreme educational reform success stories are non-existent. History has shown us that a little bit of digging has, in the past, always revealed such claims of miracles to be false. This does not mean that we should give up hope. Small successes are common in education. But dramatic huge successes should always alert us to scepticism.”
This strong skepticism may be justified (at least up to a point) but it is also dangerous as it can lead to confirmation bias and embarrassing mistakes.
Agreed!
Id guess that extreme success would be things like switching to the metric system, or using pi = C/r ~ 6.28 (like it was originally) that make lots of things simpler. But this will require a period of disruption, perhaps even lasting a generation, where things get worse. Thus, the current method of studying these interventions will *reject* the most promising ones.
Of course, the worst thing is to constantly AB test various minor interventions (disruptive) without learning anything from it because you averaged away all the information.
Well, we know from America’s top multidisciplinary scientist Malcom Gladwell that being the oldest in one’s age group is highly beneficial in sports. It’s hard to come up with a reason why it would be less beneficial in academics. So I think it’s worth remembering that while there is an important statistical question about how much “retention” contributes to the success of the 2013 intervention, there’s probably not a question about whether or not its beneficial to the kids who are struggling – it almost certainly is.
There are still quite large questions about whether retention is better or worse for kids who are struggling. Doing more challenging work can lead to greater improvements than repeating the same work again. Being with classmates with higher order thinking can lead to better outcomes than being with classmates one or more years younger. The social stigma of being left behind and the loss of friendship groups can lead to emotional problems that interfere with learning.
“Doing more challenging work can….”
You speak in purported potentials rather than certainties.
One certainty is that the brain develops as kids age. Based on this actual physical real irrefutable fact, it’s reasonable to suppose – no, it’s almost certain – that most kids really can’t learn certain things until their brains have developed to a certain point. In fact we see this argument has been used in the legal system to protect younger people from “adult” punishment. So if you believe it’s true that a teenager doesn’t get what “murder” is or has impulse control problems due to inadequate brain development, then it’s reasonable to believe that a pre-teen or 8-10yo kid or a kid in grade 3 or 4 might not get long division or more complex sentences or whatever is on the educational menu; or have age-related impulse control problems that interrupt their learning.
This seems all the more likely at younger ages where, say, in grade two, the youngest kid, roughly speaking, would be 10-15% behind the oldest kid in development even if they have the same ability and are developing at the exact same rate. OK, this is highly simplified, sure, but now throw in the spread in ability from the most capable to the least capable and it starts to get ugly.
A kid in my HS became locally famous for graduating but not being able to read. Imagine the social stigma of that.
BTW, “Conservative for the Win” was kind of a joke on Andrew’s “Conservative vs Conservative” thing. I’ll stick with it though so we know who we’re talking to.
I speak in counter-examples to your assertion that being held back is beneficial to struggling students.
Kids have mental growth spurts as well as physical growth spurts. A kid struggling in grade 1, may come back to grade 2 able to master the work easily. And that is besides all the social and health reasons why a kid may struggle in grade 1 but master grade 2.
Here’s an article I wrote 23 years ago: “Should You Redshirt Your Child?”
https://www.upi.com/amp/Archives/2002/07/25/Analysis-Should-you-redshirt-your-child/9231027569600/
A lot of affluent private school parents hold their sons, including future NFL quarterbacks, back so that they can be a year older than their classmates. Maybe that’s not such a terrible strategy for poor public school parents too?
Let’s all do it!
on the topic of numbers not looking right, someone sent this nice story to me today about a 75yo albatross:
https://indiandefencereview.com/world-oldest-wild-bird-returns-again/
‘With a life span for her species typically around 30 years, her longevity has far surpassed expectations. As Jonathan Plissner, a biologist at the refuge, told BBC Radio 4, “We don’t know of any others that are even close to her age.” ‘
So we have a 75yo bird from a species with a typical life span of 30yr. What are the chances that…er…errors were made?
I don’t think it’s impossible for this bird to be that old. From what little I know about the age of animals in the wild, my understanding is that, for small birds like nuthatches or chickadees, it wouldn’t be that surprising for an individual bird to live several times longer than the average age, since many birds get picked off young by predators. But, OTOH, this is not a high-reproduction prey species, it’s a low reproduciton species higher on the food chain, so we might expect the mean and the max to be closer together. It’s not impossible but it seems unlikely.
When I read this article the first thing I thought of was the controversy over blue zones.
Maybe someone has some thoughts about this.
I think (not sure) that bird, Wisdom the albatross, has warn a tag (band) continuously since 1956. When one band is getting illegible or becoming weak from damage, another band is replaced. Here’s a relevant video: https://www.youtube.com/watch?v=FDmr2pnNu6I&t=269s that says the first band lasted 46 years and was then replaced by the same scientist (86 years old at the time) who had banded her in the first place. She has since been re-banded again, if we believe the video.
If we believe the video (we don’t have to!) then unless we look at crazy possibilities — like someone capturing Wisdom, removing her band without damaging it, and putting it on a new bird; or creating a counterfeit band with Wisdom’s numbers — I think the most likely place to look for an error would be in that original 46-year gap. Perhaps the researcher recorded the wrong band number when he tagged Wisdom, and much later a band with that number was given to a different bird. I say “much later” because if it didn’t happen much later then all that changes is the identity of the very old bird that is still laying eggs, not the fact that a very old bird is still laying eggs. I think a mistaken band number is very unlikely to be an issue here. For one thing, the type of band used by researchers has changed over the years and I think an experienced albatross researcher would recognize immediately if , say, a 1970s-style band was supposed to be from 1956.
So, although I think you raise a valid point — which I will generalize even farther to say “never blithely assume that the data are correct” — I think in the case of Wisdom, she really is a very elderly albatross that is still having chicks.
As long as we are digressing about things not looking right, this topic reminded me of this https://www.youtube.com/watch?v=P8QVBt2hh9M
Interesting thoughts Phil, thanks.
Well GPT tells me that there is another albatross aged 50. So its possible the distribution may eventually fill in. Another possibility is that the average lifespan is not so well known and is too low.
Regarding the canon balls story: the best part of the video was the fake smoke coming off the cigarette! Hilarious, maybe not a great way to convince people that your documentary is real? :) But as to the canon balls: I didn’t spend much time on it but the moved rocks / gravity theory isn’t that convincing. The rocks moved, fine but gravity moved them in the time between when he set up his camera and took it down? If you’re familiar with Racetrack Playa, you’ll know that moving rocks leave tracks. I didn’t see that evidence. My question was where did the canon balls come from? And if “off” was before “on”: why were there no canonballs on the road in the first place? I wanted to watcht he vid again but ScrewTube kept forcing me to watch multiple commercials. Google fits my Bank of American model: own the stock, avoid the product.
One more thing here. In NAEP, Mississippi in 2024 is slightly ahead of New York state on 4th grade math and 4th grade reading, a couple points behind on 8th grade math and reading. This is with *no* demographic adjustment.
When people talk about the Mississippi Miracle, what more do you need? What percentage of Americans would believe that Mississippi, one of the poorest states, with huge historic black-white gaps, and per-student education spending *sixty-three percent lower* than New York state, has essentially identical student performance, *unadjusted for demographic differences*? To me, that really is a miracle. And if I were an education researcher in New York, I would be banging down the state house door asking where the hell the money went (answer, at least in part, is not to phonics and Asia-style math revision!)
Kevin:
The main concern is that what we’re talking about here is average scores among students who took the test, and if there’s enough selection in who takes the test, that can distort the comparisons.
That is not the case! There’s no selection bias–the kids who are retained are not murdered, and I assure you that they still take the test, just one year later.
While selection is clearly an issue, teaching students to be better test takers can also put a nearly invisible thumb on the scale.
I had the unusual experience of attending school outside Chicago in what was then the third richest county in the country (no thanks to my family), then being moved to rural Georgia in the seventh grade and one of the poorest counties. The idea that a focus on phonics would propel those Georgia students upwards to the point of exceeding the Illinois students in proficiency seems impossible to me. Many of my seventh-grade classmates in Georgia had never left the county, and I got the distinct impression that some of them had hardly left the farm. Their parents called what happened at school “book learnin'”. They went to school because Georgia was very aggressive with truancy, since the schools got their funding only if the students showed up.
My brother’s SAT score was tied for highest in the state. He tied with a student from the toniest private school in the toniest Atlanta suburb, Lakeshore Academy or something like that. An educator at my school later told me that phone calls were subsequently made to the school about this suspicious Bill Skaggs guy, and could he really have gotten a score that high? When it was explained that he was educated elsewhere, the situation seemed less suspicious. Lakeshore Academy basically spent the year prior to the SAT teaching to the test, and the result was a long string of wins in terms of the highest score in the state.
New York state may be stuck in a mode of trying to produce well-rounded, truly educated students who can think for themselves, while Mississippi may be pouring all its limited resources into developing expert test takers. If so, they have been resoundingly successful!
It would be helpful to have 12th grade NAEP scores for Mississippi. Unfortunately, sample sizes aren’t big enough for 12th graders to rank them by state to see if any of that big 4th grade advantage in state rank still exists in 12th grade or whether Mississippi has fallen back to its traditional 50th place.
How so? Just put the numbers in order. I guess youre using some arbitrary threshold to decide on “large enough”?
There are surprises about the Mississippi NAEP perf. NWEA is a better proxy for NAEP as the same test is given to all students so that the perf of all student of all grades and percentiles can be compared. The NAEP data assumed that 5% of disability students are not included.
sorry, wrong url
https://x.com/dux_ie/status/1998298773982990663
For Mississippi there are the complications of student repeating grade. Thus 4 set of expected perf data are needed at the 7.2% cutoff for retention and the 5% that might not be included in test for grade 3 and 1 year later. There is no NAEP rule for not including students in the test.
The Mississippi data are not avail so using the national as proxy. This allow forecast of the perf of additional year old repeat students and they are significantly perf better than the bottom of the non-repeat students. Including repeat students will raise the avg NAEP score.
So, what are the differences between the MAAP-ELA, NAEP, and NWEA? 3 tests for the same thing? Who takes them, who is excluded, what do they measure, etc.? Please try to refrain from acronyms and abbreviations (e.g. perf) for those of us who are unfamiliar with these tests.
The extra question is whether the kids who repeated would have got a higher perf score had they been promoted than if they had been repeated.
And hidden amongst all of this, when looking at the tails of distributions, is regression to the mean.
And it would be interesting to know what a difference of 10 in a score actually equates to. A difference of 1 question wrong out of 30 questions?
anonymous –
The extra question is whether the kids who repeated would have got a higher perf score had they been promoted than if they had been repeated
Indeed. A great question.
I’ll be completely honest. I have no idea how you can even remotely begin to take these people seriously when they make a mistake as big as this.
If someone trying to lecture me arrogantly preached that Earth had two moons, I would not take them seriously when they talked about comets.
So when a trio of “respected researchers” say “Need more proof that Mississippi public education is without miracles? The 2024 NAEP fourth-grade mathematics scores rank the state at a tie at 50th! The eighth-grade scores also qualify for 50th place. This is certainly
consistent with the Mississippi that most of us know”…
How in the world can anyone take ANYTHING else they say seriously, at all?
Andrew, I fear that you’re being far, far too soft on them because they are your friends.
Matthew:
Howard Wainer has done good work in other settings, so I would not take the mistake in this paper to invalidate his entire career. But, fair enough regarding this particular paper. I guess the question is how this error got into that paper and how the authors didn’t catch it. As discussed in my above post, I did catch the anomaly but I didn’t fully see the problem until it was pointed out to me by blog commenters.
On the plus side, blogging served as a useful pre-publication review, as that particular error got caught before publication.
That’s certainly a plus! And I would not suggest his entire career to be invalidated.
However, the paper’s principal argument is that Mississippi cannot have really had the improvement that test scores say they do. And the math performance is a load-bearing part of their argument against the reading improvement being valid!
They talk a lot of talk about other places, where there actually was fraud… but in those places, individual teachers had specific and heavy incentives to produce high scores (at least in Atlanta, the one I’m most familiar with), which I believe is not the case in Mississippi. Then, they talk a lot about the statistical ramifications of removing students from the pool of test-takers, as if these retained students were being taken out and shot, rather than… uhh… taking the exact same test, just a year later.
Why are they making these bizarre arguments to support their case? Because they have no better ones? Perhaps, but then why are they making the case at all?
The mask comes off in that paragraph I quoted. “This is certainly consistent with the Mississippi that most of us know.” THAT is why they wrote the piece at all–contempt. With respect, Andrew, does this sound like an objective assessment?
When forced to reckon with Mississippi’s equally impressive math resurgence, there will be no basis left to support their extraordinary claims: that an entire state is cooking the books on a reading test that no one there has any financial stake in.
I am glad that this error was caught, but the entire piece needs to be thrown out. And, it may be out of line for me to say this, but I hope they sincerely take a hard look at themselves and how they managed to write out of such spite that they willfully ignored facts that would’ve taken literal seconds to verify.
Matt:
I remain interested in hearing what you have to say, but I can’t argue with you that the paper’s flaws are serious enough not much stands after removing that error about the math scores. The general point about selection is important, but it would be possible to make that point, along with the point about the difficulty of learning from an observational comparison with no direct control group, without all the general unsupported arguments. Regarding your point about kids taking the test a year later: that won’t make the selection problem go away, but it makes it more complicated, for sure. I also remain interested in learning how the error got into the paper in the first place.
My guess as to why the risk-adjusted MS performs so much better than raw scores MS – two things:
1. Mississippi has proportionally the highest black population of any US state (~40%), so any estimates that treat race as a confounder will be most consequential for MS
2. A hypothesis, or a stylized fact if you like – the achievement gap between black and white students in MS will be smaller than in the average US state
A quick way to show 2 – I used this data tool to generate reports of reading scores in 4th grade by race in PA and MS:
https://www.nationsreportcard.gov/ndecore/xplore/NDE
Here is the table for 2019 data:
US | White | 230
US | Black | 204
US | Hispanic | 209
MS | White | 203
MS | Black | 209
MS | Hispanic | 221
So the margin white to non-white students in MI was already smaller than it was nationally. I show hispanic students for completeness although be aware, the hispanic population in MS is tiny compared to nationally (3 % vs 20%) so it probably does not bear on the risk-adjusted number
With the Urban paper, I don’t know exactly what they did. But I assume they throw MS into the regression and give them a big boost for having a large black population, even though their particular black students are more high-achieving than is the case nationally. The achievement gap in MS is 20% smaller.
Now whether that is appropriate/meaningful just depends on what you’re trying to do by risk-adjusting. Do we think that the system is actually compensating and then some for the social factors? Or, could it be that behind racial categories two people who are on paper identical actually belong to different latent/unobservable groups? And then if you think there is some unobservable difference in the “social terrain” that makes teaching kids in MS easier than other states, I guess it is important to education researchers to understand the same way you wouldn’t say a school is “bad” just because it is selecting for kids with harder backgrounds.
Worth noting that the error in the Wainer paper was pointed out and corrected some months ago. I am assuming there is a paper floating around somewhere without the error in it.