Gur Huberman points to this document from Stanford math professor Brian Conrad, criticizing a recent report on the California Math Framework, which is a controversial new school curriculum.
Conrad’s document includes two public comments.
Comment #1 is a recommended set of topics for high school mathematics prior to calculus. I really agree with some but not all of these recommendations (I like the bits about problem solving modeling with functions; I’m skeptical that high school students need to learn how to add, subtract, multiply, and divide complex numbers; I don’t really buy how they recommend covering probability and statistics; and if it were up to me I’d drop trigonometry entirely). I think their plan is aspirational and the kind of thing that a couple of math professors might come up with; I wouldn’t characterize those topics as “crucial in high school math training for a student who might conceivably wish to pursue a quantitative major in college, including data science.” Sure, knowing sin and cos can’t hurt, but I don’t see them as crucial or even close to it.
Comment #2 is the fun part, eviscerating the California Math Framework report. Here’s how Conrad leads off:
The Mathematics Framework Second Field Review (often called the California Mathematics Framework, or CMF) is a 900+ page document that is the outcome of an 11-month revision by a 5-person writing team supervised by a 20-person oversight team. As a hefty document with a large number of citations, the CMF gives the impression of being a well-researched and evidence-based proposal. Unfortunately, this impression is incorrect.
I [Conrad] read the entire CMF, as well as many of the papers cited within it. The CMF contains false or misleading descriptions of many citations from the literature in neuroscience, acceleration, de-tracking, assessments, and more. (I consulted with three experts in neuroscience about the papers in that field which seemed to be used in the CMF in a concerning way.) Often the original papers arrive at conclusions opposite those claimed in the CMF. . . .
I’m not sure about this “conclusions opposite those claimed” thing, but it does seem that the CMF smoothed the rough edges of the published research, presenting narrow results as general statements. Conrad writes:
The CMF contains many misrepresentations of the literature on neuroscience, and statements betraying a lack of understanding of it. . . . A sample misleading quote is “Park and Brannon (2013) found that when students worked with numbers and also saw the numbers as visual objects, brain communication was enhanced and student achievement increased.” This single sentence contains multiple wrong statements (1) they worked with adults and not students; (2) their experiments involved no brain imaging, and so could not demonstrate brain communication; (3) the paper does not claim that participants saw numbers as visual objects: their focus was on training the approximate number system. . . .
The CMF selectively cites research to make points it wants to make. For example, Siegler and Ramani (2008) is cited to claim that “after four 15-minute sessions of playing a game with a number line, differences in knowledge between students from low-income backgrounds and those from middle-income backgrounds were eliminated”. In fact, the study was specifically for pre-schoolers playing a numerical board game similar to Chutes and Ladders and focused on their numerical knowledge, and at least five subsequent studies by the same authors with more rigorous methods showed smaller positive effects of playing the game that did not eliminate the differences. . . .
In some places, the CMF has no research-based evidence, as when it gives the advice “Do not include homework . . . as any part of grading. Homework is one of the most inequitable practices of education.” The research on homework is complex and mixed, and does not support such blanket statements. . . .
Chapter 8, lines 1044-1047: Here the CMF appeals to a paper (Sadler, Sonnert, 2018) as if that paper gives evidence in favor of delaying calculus to college. But the paper’s message is opposite what the CMF is suggesting. The paper controls for various things and finds that mastery of the fundamentals is a more important indicator of success in college calculus than is taking calculus in high school. There is nothing at all surprising about this: mastery of the fundamentals is most important. The paper is simply quantifying that effect (this is the CMF’s “double the positive impact”), and also studying some other things. What the paper does not find is that taking calculus first in college leads to greater success in that course. To the contrary, it finds that for students at all levels of ability who take calculus in high school and again in college (which the authors note near the end omits the population of strongest students who ace the AP and move on in college) do better in college calculus than those who didn’t take it in high school (controlling for other factors). The benefit accrued is higher for those who took it in high school with weaker background, which again is hardly a surprise if one thinks about it (as Sadler and Sonnert note, that high school experience reinforces fundamental skills, etc.). If one only looks at the paper’s abstract then one might get a mistaken sense as conveyed in the CMF about the meaning of the paper’s findings. But if one actually reads the paper, then the meaning of its conclusions becomes clearer, as described above. . . .
Here’s a juicy one:
Chapter 12, lines 221-228 . . . the CMF makes the dramatic unqualified claim that:
“if teachers shifted their practices and used predominantly formative assessment, it would raise the achievement of a country, as measured in international studies, from the middle of the pack to a place in the top five.”
Conrad goes on to explain how this claim was not supported by the study being cited, but, yeah, in any case it’s a ridiculous thing to be claiming in the first place, all the way to the pseudo-precision of “top five.”
One problem seems to be that the report had no critical readers on the inside who could take the trouble to go through the report with an eye to common sense. This is important stuff, dammit! The California school board, or whatever it’s called, should have higher standards than the National Academy of Sciences reporting on himmicanes or the American Economic Association promoting junk science regarding climate change.
I don’t agree with all of Conrad’s criticisms, though. For example, he writes:
The CMF claims Liang et al (2013) and Domina et al (2015) demonstrated that “widespread acceleration led to significant declines in overall mathematics achievement.” As discussed in §4, Liang et al actually shows that accelerated students did slightly better than non-accelerated ones in standardized tests. In Domina et al, the effect is 7% of a standard deviation (not “7%” in an absolute sense, merely 0.07 times a standard deviation, a very tiny effect). Such minor effects are often the result of other confounders, and are far below anything that could be considered “significant” in experimental work.
I agree that effects can often be explained by other confounders, but I wouldn’t say that a 0.07 standard deviation effect is “very tiny.” A standard deviation is huge, and 7% of a standard deviation is not nothing. I agree that the report isn’t helping by using the term “significant” here. The thing that really confuses me here is . . . did the report really claim that Liang et al. (2013) that acceleration caused significant declines, but Liang et al. actually found that accelerated students did better? Whassup with that? I’m not completely sure but I think the paper he’s referring to is this one by Jian-Hua Liang, Paul E. Heckman, and Jamal Abedi (2012), which doesn’t seem to say anything about acceleration leading to significant declines, while at the same time I don’t see it relying that accelerated students did better. That article concludes: “The algebra policy did encourage schools and districts to presumably enroll more students into algebra courses and then take the CST for Algebra I. However, among the students in our study, the algebra-for-all policy did not appear to have encouraged a more compelling set of classroom and school-wide learning conditions that enhanced student understanding and learning of critical knowledge and skills of algebra, as we have previously discussed.” I don’t see this supporting the claims of the CMF or of Conrad.
Summary
It’s a complicated story. Conrad seems to be correct that this California Math Framework is doing sloppy science reporting in the style of Gladwell/Freakonomics/NPR/Ted, using published research to tell a story without ever getting a handle on what those research papers are really saying or whether the claims even make sense. Unfortunately, the real story seems to be:
(a) The different parties to this dispute (Conrad and the authors of the California report) each have strong opinions about mathematics education, opinions which have been formed by their own experiences and which they will support using their readings of the research literature, which is, unfortunately, uneven and inconclusive.
(b) We don’t know much about what works or doesn’t work.
(c) The things that work aren’t policies or mandates but rather things going on at the level of individual schools, teachers, and students.
The most important aspect of policy would then seem to be doing what it takes to facilitate learning. At the same time, some curricular standards need to be set. The CMF has strong views not really supported by the data; Conrad and his colleagues have their own strong views, which are open to debate as well. I don’t feel comfortable stating a position on the policy recommendations being thrown around, but I do think Conrad is doing a service by pointing out these issues.
The major problem with K-12 education is the content. Until principles-based school mathematics is being taught, fiddling with the course structure won’t make much difference.
The main author of the latest version of the California Mathematics Framework is Jo Boaler. Regarding her past work, see
A Close Examination of Jo Boaler’s Railside Report
https://nonpartisaneducation.org/Review/Articles/v8n1.pdf
https://web.stanford.edu/~joboaler/
David:
Wow—those are some ugly-ass 3D-style bar graphs! They remind me of that self-contradictory drawing that looks like three bars in one place and two bars in the other. I guess we can blame it on poor statistical education.
From Jo Boaler’s book, Mathematical Mindsets. She gets bonus points for repeating widely debunked claims about Albert Einstein. Enjoy:
… Some people revel in the inaccessibility of mathematics as it is currently taught, especially if their own children are succeeding, because they want to keep a clear societal advantage. Others, thankfully, are willing to embrace the change needed, even if their children are succeeding now, especially when they learn that their children’s perceived advantage is often based on a math that is really not going to help them in the future.
The Myth of the Mathematically Gifted Child
Some people, including some teachers, have built their identity on the idea they could do well in math because they were special, genetically superior to others. People try really hard to hang on to the idea of children who are genetically gifted in math, and the whole “gifted” movement in the United States is built upon such notions. But we have a great deal of evidence that although people are born with brain differences, such differences are eclipsed by the experiences people have during their lives, as every second presents opportunities for incredible brain growth (Thompson, 2014; Woollett & Maguire, 2011). Even the people whom society thinks of as geniuses actually worked really hard and in exceptional ways to achieve their accomplishments. Einstein did not learn to read until he was nine, and he failed his college entrance examination, but he worked exceptionally hard and had a very positive mindset, celebrating mistakes and persistence. Rather than recognizing and celebrating the nature of exceptional work and persistence, the U.S. education system focuses on “gifted” students who are given different opportunities, not because they show great tenacity and persistence but often because they are fast with math facts. The labeling of students as gifted hurts not only the students who are deemed as having no gifts but also the students who are given the gifted label, as it sets them on a fixed mindset pathway, making them vulnerable and less likely to take risks. When we have gifted programs in schools we tell students that some of the students are genetically different; this message is not only very damaging but also incorrect. Not surprisingly, perhaps, studies that have followed people who had been labeled as gifted in their early years show that they go on to average lives and jobs (https://ireport.cnn.com/docs/DOC-332952).
Malcolm Gladwell unpacks the nature of expertise in his best-selling book Outliers. Drawing from extensive research…
Mark:
Wow. I was curious about the claim that “studies that have followed people who had been labeled as gifted in their early years show that they go on to average lives and jobs.” The link no longer works but the original is here on the internet archive and it’s really bad. Here it is:
Yeah, right. The best thing about this sort of argument is that it can’t lose. For example, if we google Hunter College Elementary School we’ll find that its graduates include a supreme court judge, a former member of the U.S. cabinet, TV star Cynthia Nixon, and playwright/songwriter Lin-Manuel Miranda. But if someone were to point this out, you could just argue that this has nothing to do with IQ and everything to do with their educational opportunities!
Anyway, yeah, it’s kind of horrible that this book by a Stanford education professor is making a false claim and backing it up with a really bad news article—actually, not a news article at all, just an opinion piece that once appeared on a CNN-linked website and is explicitly tagged “NOT VERIFIED BY CNN.” Not that being on CNN would make it correct; I’m just saying that to cite this as a source . . . this is poor scholarship. Boaler’s the education researcher, right? She should go to the goddam original sources if she wants to make this claim, or else refer to a credible secondary source. As it is, her sourcing makes the Why We Sleep guy look like a careful scholar by comparison!
Agreed. Boaler should have looked into SMPY, which, as far as I know, is the most extensive study of talent in young people and their long-term outcomes. Its findings tell a very different story, with students who score in the top 1% having vastly higher odds of earning advanced degrees and patents. You can read more about it here: https://my.vanderbilt.edu/smpy/files/2013/01/Article-NATURE-2016.pdf
Mark:
Indeed. As a participant in that study, I know all about it!
I came across this and thought of that post:
https://www.insider.com/why-am-i-tired-after-sleeping-more-oversleeping-side-effects-2017-1
It’s Ericsson, so you know what you’re in for.
One problem with Hunter as an example is that Subotnik et al completely ignore statistics & psychometrics concerns, their whole book ignores it (or concerns about how apparently they casually repeatedly test the kids using IQ tests, completely destroying their meaning via practice effects). When you select kindergartners for having ‘IQ >130’, they as adults will not all be >130 because of considerable regression to the mean (they will regress towards 100 far more than SMPYers or Hunter High School, who are selected much older and thus when IQs have stabilized more). You can’t use them as an example of high-IQ being useless when their IQs aren’t actually going to be high!
This has a fun counter-argument: Hunter College Elementary School has few of those like Pulitzer or MacArthur award winners, yes…* but Hunter College High School does, at a much higher rate. Yet, every HCES kid is an auto-admit into the high school and therefore gets a double-dose of these supposed educational opportunities, and so the HCES kids ought to outperform, not vice-versa. This is both qualitatively and quantitatively as predicted by regression to the mean, but totally inconsistent with the opportunity claim.
* it’s no longer true to say ‘no Pulitzer Prize’. Lin-Manuel Miranda was a HCES kid and won a Pulitzer for Hamilton.
Gwern:
Yeah, it’s horrible all around (also there could be various illustrious Hunter College Elementary students who happen not to be listed on the wikipedia page), and, yeah, that’s why I mentioned Miranda in my comment above. In some way, though, the worst thing is Boaler relying on a bunch of word salad from some rando on the internet. If she can’t even be bothered to read the literature in her own field, what she doing at Stanford in the first place??
Of particular interest to readers of this blog, some of the statistical arguments used by Dr. Boaler to support her work are laughable.
In particular, she tries to argue that a lack of statistically significant differences post program are evidence that the program worked at reducing inequality.
https://www.youcubed.org/wp-content/uploads/2017/09/Creating-Mathematical-Futures.pdf is the railside study in question.
> The Railside mathematics teachers were also extremely successful at reducing the achievement gap between groups of students belonging to different ethnic groups at the school. Table 4 shows significant differences between groups at the beginning of the ninth-grade year, with Asian, Filipino, and White students each outperforming Latino and Black students (p
> At the end of Year 1, only one year after the students started at Railside, there were no longer significant differences between the achievement of white and Latino students, nor Filipino students and Latino and Black students. The significant differences that remained at that time were between white and Black students and between Asian students and Black and Latino students (ANOVA F=5.208, df=280, p=0.000). Table 5 shows these results.
And by “significant”, she means statistically significant. If you look at the tables, the gap is still there and actually grew by some metrics. Before the program, white students had a median of 21 and black students had a median of 12 (a gap of 9). After the program, white students had a median of 28 and black students had a median of 16.5 (a gap of 11.5).
Trigonometry is really crucial to Newtonian Mechanics which is crucial to mechanical and civil engineering, and physics. It comes up in breaking down forces into components in a coordinate system and such. Obviously not everyone is going to do physics or engineering but those who are so inclined really need to be familiar with trig functions.
Daniel:
Trig is ok and if people want to learn it in college cos it might be useful to them, that’s cool. Also, yeah, if you take physics in high school it would be good to learn trig then too. I’d say it would make sense to cover it in physics class, the same way they cover differential calculus and call it “kinematics,” you can cover trig and call it “decomposition of forces” or something like that. I’d put it at a higher priority than learning how to multiply and divide complex numbers!
> cos it might be useful to them
I think you can also use sin here as a shorthand for since. Google tells me that since can be used as “for the reason that; because”, and in this hand cos is short for because. So now in the context of trigonometry we have established a mechanism by which sin == cos.
And so we can enjoy the fruits of our labor:
> Trig is ok and if people want to learn it in college sin it might be useful to them, that’s cool
It is vectors that are important for decomposing forces. Speaking as a physics professor, I would replace trig with vectors.
I actually enjoyed trigonometry. But it seems to me that much of what was done in that course was manipulating various trig identities in order to derive formulas that were forgotten by the end of the week after the semester ended.
I don’t ever remember having to do anything with the cosecant in my life. In contrast, working with exp(i theta) has taken up many hours of my life.
I don’t think that such practice in formula manipulation is a good investment of a student’s time.
I would think that a few weeks of looking at various force problems—how a sailboat works, what makes a car go into a spin, etc. and examining various applications of the sine, cosine, and tangent functions would be useful.
Bob76
PS. A cool book on the history of trigonometry is Trigonometric Delights by Eli Maor. Did you know that the word sine comes from Sanskrit via a mistranslation of an Arabic word?
I find nothing more common today than the mischaracterization of cited studies. These range from aggressive interpretation of results to downright reversals of findings, with a fair smalttering of the nearly completely irrelevant. It’s not usually from dishonesty, or I at least I don’t think it is. Mostly it’s just wishful thinking.
PS: The captcha thing is still messed up
“Trig is ok and if people want to learn it in college cos it might be useful to them, that’s cool.”
Trig is widely used by non-college graduates in the aircraft industry, the military, construction, the maritime industry and many other industries. It’s a basic mathematical tool. I wouldn’t put you in charge of any general curriculum. You have heavily overly-statistics-centered view of the world.
You can’t build a boat with statistics.
Anon:
I agree that trig is widely used. Lots of things are widely used by non-college graduates and are not taught in high school. There’s not enough room in the high school curriculum to cover everything that is widely used by non-college graduates, especially given that much of the purpose of the high-school curriculum is to prepare students for college.
In any case, I agree about not wanting to be set up any general curriculum. I have my experiences and views. If for some reason I ended up in charge of a general curriculum, I’d gather the views of many stakeholders; I wouldn’t set up the curriculum just based on my own impressions. I wouldn’t want those two math professors in charge of the curriculum either, for the same reason!
Lots of things are used by non-college-grads that aren’t taught in high school, but not lots of mathematics things. The vast majority of all people use almost no math on a daily basis. The math they do use is pretty much addition and multiplication and their inverses. A few people maybe use square roots. Mainly in association with Pythagoras theorem, stuff like laying out a foundation on a new construction site or something. If there is one thing they need beyond addition and multiplication and their inverses, it’s trig. Trig comes up in surveying, construction, machining, even auto repair (oscillations and timing) since so little math is used on an everyday basis we should really give HS grads all of the most useful parts.
“Trig is widely used by non-college graduates in the aircraft industry, the military, construction, the maritime industry and many other industries.”
It may be widely used in some industries, but that doesn’t mean it’s widely used. For construction (I grew up in a construction family), there’s not much trig used by the non-college-graduates in the industry. If there is any trig, it’s the architects, engineers, and surveyors who either went to college or completed some kind of certificate course. Nowadays, almost all of the math any non-college-graduate (or for that matter, college graduate) will use will be embedded in whatever software they use in their profession.
My freshman college course in Mechanics absolutely required trigonometry. The midterm exam had a question on how to get the second perturbation, given that the method requires cosine approximations and the first perturbation produces cosine-squared terms. Answer: Cos(2t) = sin^2(t) – cos^2(t) = 1 – 2cos^2(t), so cos^2(t)= (1 – cos(2t))/2.
The Fourier Decomposition also comes in handy in physics and engineering.
So the standard Algebra, Geometry, Trigonometry, Calculus/Probability I had in high school worked well for me. People who don’t want to work in technical fields could stop at Geometry, and did.
“…much of the purpose of the high-school curriculum is to prepare students for college.”
When did that start? As far as I can tell that’s not even close to true even in the best high schools. I live in a wealthy, liberal tech metropolis yet even here only 60% of students read at grade level. If that’s “preparing for college” somebody is doing a pretty bad job. Is that why we have so many “science for non-major” courses in colleges? Because students are so well prepared? But how is trig *not* preparing people for college? Like I said, you can’t build a boat – or anything else – with statistics.
If I were preparing the curriculum for college the last thing I would do is consult the “stake holders”. Too much “consulting stakeholders” is the problem, not the solution. Education, like statistics, has a right way to be done, regardless of the personal opinions of “stakeholders”.
But consulting the stakeholders would be perfect for a statistician. Endless dwiddling with junk data generated by untested methods for irreproducible results is their area of expertise. Job security.
Anon:
1. According to BLS, “61.8 percent of 2021 high school graduates ages 16 to 24 were
enrolled in colleges or universities,” so, yeah, preparing for college is a big part of high school.
2. Yeah, I got your point about the boats. Most college students don’t learn how to build boats. Maybe they should—if it was up to me, I’d definitely put “building stuff” on the curriculum, indeed my favorite class in college was a mechanical engineering class where we spent tons of hours in the machine shop, I learned a lot there (didn’t actually use any trig, though)—but that’s not how things are going now.
3. Your remark about statisticians and “job security” is just flat-out rude. I already have a job, and I help out lots of people for free who come to me for advice. If I were in charge of the high school math curriculum, I’d definitely solicit advice from high school and college students and teachers, employers, etc etc. To equate asking for advice with “endless dwiddling with junk data” is foolish. I would ask people for advice because I don’t know what is “the right way to be done” for education.
“Education, like statistics, has a right way to be done, regardless of the personal opinions of “stakeholders”. ” For sure what’s the “right way to do education” has to be decided by someone, according to someone’s opinion. You think non-stakeholders are better at that than stakeholders? On what basis?
The main issue is that while Conrad (and me) may have our personal views on K-12 math, we do not presume to issue guidance for the state of California.
So the question is not wether the people of California should accept Boaler’s opinion or Conrad’s. Rather it is whether they are going to accept a sloppy document as the guideline for math education for the next 8+ years or force the SBE to go back to the drawing board with a fresh set of writers
Boaz:
Yes, that makes sense.
For a sense of how such a document should look like, worth looking at Beth Kelley’s Twitter thread comparing the CMF 3rd grade guidelines to Massachusetts’
https://twitter.com/BethKellySF/status/1518991575526699008
Boaz:
Sure, but Massachusetts has terrible weather. So you have to balance these things.
Sure, rub it in :)
I think the CMF is probably not a good way of guiding math education, but Kelly’s thread didn’t seem a good faith reading of CMF.
“Proposed CMF Standard: Collect data by time of day, show time using a data visualization. Think about fractions of time and of shape and space, expressing the base unit as a unit fraction of the whole.”
“MA Standard: Ability to tell and write time in digital and analog is taught in 1st grade.”
I don’t think the proposed CMF Standard is about telling time. She seems like she has an axe to grind. Some of her criticisms felt a bit like reading between the lines rather than a plain reading of the text.
“MA Standard: Data sets and bar graphs taught in 2nd grade.”
My son was doing bar graphs in kindergarten in Virginia. Maybe MA is not all it’s cracked up to be.
What is meant by “high school”? Isn’t it in fact a continuum which serves different functions depending on what school and what particular student we’re talking about.
I have nieces and nephews who had mastered math and science content by 8th grade that I did not encounter until college. And I was a very nerdy math and science oriented kid attending a decent suburban school system half a century ago. They have access to information, materials, facilities and educational content at a very high level by the time they reach high school. But kids at this end of the continuum are disproportionately attending good schools and come from highly educated, comfortably middle class (or higher), stable families.
But there are other kids, attending other schools, who have to take out student loans and attend a two-year community college after graduating high school just to receive remedial education to address their complete innumerancy and functional illiteracy. Their level of education attainment by the end of high school is probably similar to the more privileged kids level around 5th or 6th grade.
So if California or whomever wants to promulgate standards for math or science education, do they also have the power to dictate that every kid arrives at high school prepared for that content?
Re: etymology of “sine”… A more interesting angle is found going down a deep rabbit hole (from here to here to here. Someone seems to have had an obsession similar to that of the voyageurs who were the first Europeans to see the Grand Tetons. Perhaps if the etymology of “sine” was included as part of the trig curriculum, teenagers’ learning about the topic would improve.