A question came up about the effects of school funding and student performance, and we were referred to this review article from a few years ago by Larry Hedges, Terri Pigott, Joshua Polanin, Ann Marie Ryan, Charles Tocci, and Ryan Williams:
One question posed continually over the past century of education research is to what extent school resources affect student outcomes. From the turn of the century to the present, a diverse set of actors, including politicians, physicians, and researchers from a number of disciplines, have studied whether and how money that is provided for schools translates into increased student achievement. The authors discuss the historical origins of the question of whether school resources relate to student achievement, and report the results of a meta-analysis of studies examining that relationship. They find that policymakers, researchers, and other stakeholders have addressed this question using diverse strategies. The way the question is asked, and the methods used to answer it, is shaped by history, as well by the scholarly, social, and political concerns of any given time. The diversity of methods has resulted in a body of literature too diverse and too inconsistent to yield reliable inferences through meta-analysis. The authors suggest that a collaborative approach addressing the question from a variety of disciplinary and practice perspectives may lead to more effective interventions to meet the needs of all students.
I haven’t followed this literature carefully. It was my vague impression that studies have found effects of schools on students’ test scores to be small. So, not clear that improving schools will do very much. On the other hand, everyone wants their kid to go to a good school. Just for example, all the people who go around saying that school funding doesn’t matter, they don’t ask to reduce the funding of their own kids’ schools. And I teach at an expensive school myself. So lots of pieces here, hard for me to put together.
I asked education statistics expert Beth Tipton what she thought, and she wrote:
I think the effect of money depends upon the educational context. For example, in higher education at selective universities, the selection process itself is what ensures success of students – the school matters far less. But in K-12, and particularly in under resourced areas, schools and finances can matter a lot – thus the focus on charter schools in urban locales.
I guess the problem here is that I’m acting like the typical uninformed consumer of research. The world is complicated, and any literature will be a mess, full of claims and counter-claims, but here I am expecting there to be a simple coherent story that I can summarize in a short sentence (“Schools matter” or “Schools don’t matter” or, maybe, “Schools matter but only a little”).
Given how frustrated I get when others come into a topic with this attitude, I guess it’s good for me to recognize when I do it.
Test scores are one thing, but only one thing. I might believe that funding has little effect on test scores (once a study is done carefully) but it could still have a large impact on future success measured differently. So much depends on confidence and attitude that I think those dimensions are as important as test scores. Ability certainly matters, but even then, test scores only measure some dimensions of ability – and perhaps not the most important ones. I should add that I’m not familiar with the literature, so I have no idea how much attention other measures have received in the research.
Dale –
I share your distrust that test scores are inappropriately over-valued as an outcome measure. But even if other “abilities” are measured as well, why is success measured by looking at absolute measures in students who already rank highky rather than measuring relative growth in a cross-section of students over time? Shouldn’t the goal of our educational model be to help all students reach their potential?
That would be a heavy lift; the current goal of our educational model is to get all students to reach the same standard. If you tried to help all students reach their own potential, you would have serious problems if any students ever had differing potential from other students.
> For example, in higher education at selective universities, the selection process itself is what ensures success of students – the school matters far less.
Hmmm. I think I’ve told the following story here before. One time what I was working at an “elite” university with a highly-ranked education department, they decided that they wanted increase their ranking even more. So, they decided to increase the minimum standards for (mostly test scores) incoming students. They didn’t focus on making changes in the educational experiences of the students, per se.
I found that philosophically problematic, in particular for a school of education. Going back to the quote, I question the embedded direction of causality. Maybe she was speaking of statistics that have sophisticated controls, but I have to wonder if what’s being described is a process by which the “success” of students at the school is merely a function of the “sampling,” to speak. I would think that students’ success would ideally be measured by looking at the progress that a variety of students, looking at students who were admitted with a cross-section of test scores when they were admitted, made over their time at the school.
As time goes on, I get more and more unsettled by the viewpoint that the way to measure educational outcomes is by looking at how well it serves a particular subset of society – one that already enjoys advantages simply as a product of their status at birth.
You talk like you’re disagreeing with the quote you pulled:
But the entirety of your comment does nothing but agree with the quote. What are you questioning?
So if the success of a school is a function of the test scores of the incoming students, what’s the purpose of the school? Spend nothing on their education (just build nice facilities to better attract incoming students with higher test scores).
But this is what they do!
It’s why I chose not to continue in academia. That and predatory lending tactics that fund it all.
Daniel –
> But this is what they do!
At the university level, it’s an exaggeration but too close for comfort – especially at “elite” universities. It’s broken broken model and I wonder if it’s become unsustainable.
It’s hardly even exaggeration for USC where I did my PhD. At the time I left they were in the process of building a half a billion dollar shopping mall north of campus to capture student spending and also busy settling over a billion dollars in penalties for knowingly employing rapists in the student health center and covering it up for decades.
https://comptroller.usc.edu/annual-financial-reports/
Of the $6B in revenues only $1.7B is from tuition and fees, they receive $2.5B from the hospitals they run.
They spend $3.5B on salaries but probably only 1/4 of that is people who teach, 3/4 are likely administration.
Across the country university administration is one of the fastest growing “industries”.
https://www.usnews.com/education/articles/one-culprit-in-rising-college-costs
According to that instruction is about 23% of what Universities on average spend. Administration salaries is 27%, there should probably be some facilities maintenance and such… But really isn’t it more reasonable to say that a **university** should be spending maybe 40-50% of its expenditures on **teaching**?
Daniel –
Yikes. Those numbers are scary.
Michael –
Also,
> the current goal of our educational model is to get all students to reach the same standard.
In that we mostly agree. So if we assume that model once served it’s purpose (readying a large number of students for an industrial workplace) should we assume that should still be the best model today?
I also wonder how much the (potentially!) small effect of school funding is a function of the “causation without correlation” paradox discussed here a while back. That discussion pointed out that, in a system with tight feedback control loops, you will not see a correlation between an outcome measure and the thing that actually has a causal effect on the outcome measure. One example is a car going up and down a hilly road where the driver is using the pedals to maintain a constant speed; in that setting, there will be no correlation between the degree to which the pedal is depressed and the speed of the car, even though the pedal is (literally) driving the system. Milton Friedman pointed out the same issue with monetary policy—if the fed (or whatever) is doing its job correctly and maintaining the economy at some equilibrium, then there will be no observed correlation between an economic measure and the actions of the feds.
Could the same sort of thing be going on with school funding? Schools that would have performed worse get enough funding to bring them up to some standard, whereas schools that perform at the same standard do not get additional funding. In that setting, there would be at best a weak correlation between funding and whatever outcome measure was used.
gec –
> In that setting, there would be at best a weak correlation between funding and whatever outcome measure was used.
What if you measure success as relative improvement over time (in a cross-section of students as measured by a variety of criteria)?
the best research i’m aware of on this topic is by Bo Jackson and he finds sizeable effects of funding on student outcomes, including later life outcomes https://bpb-us-e1.wpmucdn.com/sites.northwestern.edu/dist/b/3664/files/2020/10/QJE_resubmit_final_version_stamped.pdf
Sample selection problems are huge in almost all education studies. And where the selection effects are known to be the largest (higher education as Beth Tipton points out) they swamp everything else. Ivy’s turn out successful students by almost any measure because they start with the students most likely to be successful. So the amount spent on their education has very little impact. (The amount they spend to go there also has very little impact, for the same reason.)
Because charter schools are fairly rare, they have substantial selection effects too. IMO, the only reason that selection effects don’t swamp learning effects there is that the selection isn’t quite as good — it’s harder to estimate ex ante potential in a five year old.
Even ignoring charter schools, the geographic/socioeconomic sorting of public elementary schools generate a separate huge source of slelection effects that make it almost impossible to work out pure spending effects. Wealthy school districts have wealthy families living in them. This was particularly true before the drive to depend less on the property tax to fund schools, and to more closely equalize expenditures on the state level, but all that has taught us is that money spent on the schools is a very poor substitute for wealthy, achievement oriented families.
Finally, there is the damnable heterogeneity of the underlying subjects: students. Even in those fields where we have a handle on the best way to teach students on average, and the will to make teachers teach that way (they often don’t like the most effective methods) we have the unfortunate fact that individual potential may be maximized in ways that are unattainable in any organized school system, and that we may not like the social results of maximizing the potential of each individual student in any case. (Imagine everyone home schooled with competent, well-trained teachers… does anyone think the result of that would look “fair,” even if it cost a lot? And to agree with gec above, in such a world, won’t the worst students obviously require the highest resources?)
But, after selection
Jonathan –
You say:
> we have the unfortunate fact that individual potential may be maximized in ways that are unattainable in any organized school system, and that we may not like the social results of maximizing the potential of each individual student in any case.
I’m a bit confused because you say “fact” and then “may,” but…why do you think there couldn’t be an organized model that would (at least do a better job of) maximize the potential of a variety of students? And why wouldn’t we like the social results of doing so?
Nice catch. OK. I’ll modify it to “my strong supposition that organized school systems can’t possibly maximize potential at the individual level.”
It’s because I believe that the heterogeneity of individual students means that all organized models of teaching will constantly be trying to pund round pegs into square holes — that individual learning is beyond the capacity of a state-run mass institution to promulgate and monitor. Similarly, I think that an optimal tax system, carefully calculating what each household ought to pay (for whatever your definition of ought) is beyond the ability of any written tax code and tax collection public institution.
And why wouldn’t we like the results? Will we be happy when everyone’s maximized potential leaves disparities at the aggregate level little changed from (and possibly increased from) the socioeconomic disparities coming in? (If all we cared about was the individual student, we ought to be happy, but I have another strong supposition that we would not. Not to mention that the socializing and social levelling functions of schools themselves, worthy not-directly-related-to book learning goals, might have to be abandoned as collateral damage.)
Jonathan –
Having spent some time on the front lines of this issue… I certainly wouldn’t suggest that there are any easy answers. However, I do think that more progress can be made than most people envision. The first, critical step would be for there to be some level of consensus that there’s a shared goal of modifying the model. That’s the biggest obstacle to progress, IMO, and less so the difficulties of pulling it off. Bringing it down to a more practical focus, I think there’s one basic step that would have a huge impact – and that would be to move away from norm-referenced measurement system and towards criterion-referenced measurement of student progress. That one step would go a very long way, imo, towards unlocking doors for a more individualized approach. But the norm-referenced model has the function of rewarding those who already benefit from the system, so generating the energy to change that is swimming against the tide.
> everyone wants their kid to go to a good school.
More precisely, everyone wants their kid to go to a school filled with good kids, kids who are well-behaved, hard-working and smart. Most such schools are also well-funded, but some are not, like public exam schools. Many wealthy parents still send their children to places like Thomas Jefferson even if they could send them to a school which spent many more dollars per student.
No parents wants to send their child to a school, even if it is lavishly funded, filled with bad kids, kids who are poorly-behaved, lazy and dumb.
Dkane:
Sure, I get that people feel that way, but my question is not just, Why do parents want to send their kids to well-funded schools? I’m also asking, Why is it that all the people who go around saying that school funding doesn’t matter, don’t ask to reduce the funding of their own kids’ schools?
Again, I teach at a private school myself and am paid well, so I’m not arguing that money spent on schools is wasted. As noted in the above post, I find myself to be confused in this area.
> Why is it that all the people who go around saying that school funding doesn’t matter, don’t ask to reduce the funding of their own kids’ schools?
Don’t they? Honestly curious! There are lots of people (mostly on the right) who want to decrease school funding, including at schools which their children currently attend. I was one of them! Do you have a specific example in mind? Not saying such people don’t exist, but every local fight over school spending in every town in the US always features lots of parents who want to decrease spending, or who at least want to not increase it so much.
The classic example is that the elderly (who no longer have children in school) want to reduce school spending and will vote against school bond issues. Young people, particularly those with limited income, vote to increase spending on schools. I’m not providing any evidence for this claim other than it is commonly made and I can easily believe it. On the other hand, the only evidence I see that “lots of people…want to decrease school funding, including at schools which their children currently attend” is apparently you.
Dale:
One reason older people without kids might vote to reduce “school funding” is that they have time to monitor how the money is spent. However, my sister’s last child recently finished public school. She’s particularly offended by the earnings of the management who, whenever incompetence emerges, seem to be able to bail with a big fat paycheck as their reward for incompetence.
One reason why people with school age children, particularly people with limited incomes, would vote *for* higher spending is that they won’t be paying for a lot of that spending. They might be less inclined to vote for that spending if they paid for it themselves.
They do as soon as their kids graduate.
“The classic example is that the elderly (who no longer have children in school) want to reduce school spending and will vote against school bond issues. ”
Sorry for being tangential, but this phenomenon is subject to multiple interpretations.
I’m one of the elderly who votes against school bond issues. But it’s not because I want to reduce expenditures on schools. In fact, I’d love to increase them. When my own children were in school I did lots of volunteer work at the school and served on the PTA board.
But I routinely vote against school bond issues because I don’t think that we should be using debt to finance this. I think school expenses should be paid out of current tax revenues. And if this means increasing taxes to cover them, I will run to the polls to vote for that. Because I think that we parents ought to be footing the bill for our children’s education, not borrowing the money and sticking our kids with the payments.
Other mammalian species provide for their offspring until they are old enough to function alone. Why shouldn’t humans do that, too?
Clyde –
You might find this interesting. It looks at the impact of tax overrides (of prop 2.5 In MA) and concludes a beneficial effect:
https://www.bostonfed.org/-/media/Documents/nesg/wallin_zabel021711.pdf
>Sure, I get that people feel that way, but my question is not just, Why do parents want to send their kids to well-funded schools? I’m also asking, Why is it that all the people who go around saying that school funding doesn’t matter, don’t ask to reduce the funding of their own kids’ schools?
Here’s one response. As a doctoral student more than 40 years ago, I served as a consultant on a school finance case in Maryland. The poorest districts in the state, including Baltimore City, were suing the state to force it to equalize school funding. The state was joined by Montgomery County, the wealthiest district in the state, and hired noted economist Eric Hanushek to testify that money doesn’t make a difference in student outcomes. I was hired to prepare questions for cross-examination that might discredit Hanushek’s testimony.
A friend—then a sociologist and now a renowned specialty seafood purveyor—suggested a novel line: If, as Hanushek argued, spending more money wouldn’t increase achievement, wouldn’t spending less money have no effect on achievement either? “Brilliant!” I thought.
The time came for the cross-examination, and, among many other questions, the plaintiffs’ attorney asked this question.
“That—that almost follows,” Hanushek replied.
He went on to argue that increasing and lowering spending probably weren’t symmetric, and cutting spending would pose many political and practical obstacles; there was more evidence about what would happen if spending were increased, because spending had been steadily increasing in some districts.
It wasn’t the Perry Mason moment I was hoping for, but it was enlightening nevertheless. The reality is that Hanushek’s claim that money doesn’t matter was based on regression analyses of natural variations among districts in their spending patterns, and wealthy districts spent more and had better educational outcomes than poorer districts that spent less. At the time, there were no studies showing what would happen if spending were to increase or decrease precipitously over time (due to court intervention or something else). Ultimately, the judge’s decision hinged more on the language in the Maryland state constitution than the social-science evidence introduced during the litigation.
Aaron:
Interesting, especially given that, more than 40 years ago, I was a student in the Montgomery County Public Schools.
My take is that the questions about school funding and performance are ill-posed. The cognitive and emotional development of children into adults, even slightly younger children into slightly older children, is affected by a vast array of factors, not just initial personal characteristics (and “potential”), but parental input, socioeconomic and environmental conditions, community-level factors, and the interaction between all of them. For instance, individual children will respond differently to peer effects, economic constraints, etc. And there are lots of relevant metrics, reflecting different outcome dimensions. It’s all sooooo noisy.
Education is a prime example of a realm for which adaptive management models are especially well suited. The methodological sophistication is applied locally and in tandem with operation. You can’t mechanically take the results that work in one situation or for one set of kids and generalize to everywhere and everyone.
I will grant, on the other hand, that there may be narrower aspects of education that do yield to generalized conclusions. Reading methodology may be one of these, although I still wonder about individual-level or even community-level differences. But the impulse that leads policy-makers to crave universal causal frameworks is the chief problem, I think.
So 100 years of research results in “a literature that is too diverse and too inconsistent” to draw conclusions. That is like 3 generations of researchers. At some point they need to consider their methods amount to comparing arbitrary numbers to each other.
But going back to first principles, I get:
1) Funding itself doesn’t matter. If a school receives 1 billion dollars of funding then lights it on fire, how will that improve student outcomes? So the right place to look is *spending*. Stop looking at funding, which amounts to an upper bound on aggregate spending.
2) If there is an shortage of something, we expect fixing that shortage will be beneficial (this is even a tautology). There is a need to first identify actual shortages, rather than percieved ones. Also to ensure any shortage actually gets fixed via the additional funding. And it could be fixing one shortage doesn’t do anything in the presence of other shortages.
3) The school is only one aspect determining outcomes. For example networking in higher education. If you know a lot of people with (a lot of) money itll be easier to make money, etc. Just like temperature, soil fertility, sunlight, water, etc all contribute to growing a plant. How tall it is, how many seeds produced, etc. There is no independent effect of each input. Model it like that.
4) Continuing the plant growth analogy, student outcomes are not independent of each other. You will also need to model competition and cooperation at the student and school levels.
Seems the hardest part is identifying the actual shortages. I would focus on doing that (accurately and efficiently) before any more research on the effect of funding is performed. And watch out because I’m sure, eg, Apple thinks schools have an iPad shortage. Same with textbooks, cafeteria foods, etc.
Anoneuoid –
> Seems the hardest part is identifying the actual shortages.
So how would you go about doing that? Seems like one way is to pick logical issues like spending more to get better teachers, improve facilities, reduce class size.
If there isn’t a good system of measurement then either you accept that condition of uncertainty and make logical suppositions, or just give up and assume you can’t make improvements. I go for former.
It’s hard to disentangle school funding from both parent economic resources and parent education (which is really pretty commonly thought of as the best predictor). I have many times told my primarily first gen college students that one of the big payoffs will be that their children are much more likely to attend college. (Passing the Torch (Attewell and Lavin) also shows that CUNY attending parents who attended college read to their children more and are more likely to take their children to museums compared to similar peers who did not attend college. ) Even with charters the parents are different in thinking and self-efficacy than those in public schools. In fact, the first gen students in my classes who have children are often super-engaged in the process of finding good schools for them.
Sorry, also I want to say that the discussion of school funding is mainly about publicly funded schools. Your place and its funding is related to how much parents (mainly) are willing to pay and the size of the endowment. The discussion is about government funding.
Another issue that I have found is that “funding” itself is complicated. For example, low income schools get more money because they have many more students on reduced price or free lunch and sometimes breakfast. Schools with a high proportion of disabled students have larger funding because of the extra staffing they require. Schools with a larger proportion of students who do not speak English may also get more funding because they require extra services to learn English. Schools in high crime areas may get extra security funding.
Not to mention that in places like NYC that have a uniform base funding for each student (varies based on grade level) middle class schools spend much more because of parent funding. At my daughter’s public high school the parent association meetings had people with credit card readers.
So operationalization of the concept of funding requires a good deal of thought.
I actually have heard some say that school funding doesn’t matter and so the funding should be reduced at their own kids’ schools, with a reduction in their taxes. It may be that some who say (and believe) that school funding doesn’t matter also think that it is pointless to press for reduction because, they also believe, they will be ignored. It may also be that some who say that school funding doesn’t matter are signaling their alignment with a political group even though they do not believe that they are saying.
While the researchers are clearly very sensitive to the issue of class size, I would not expect that once a fairly low class size threshold has been exceeded – say, 10 students with one teacher – it makes a lot of difference to the quality of education if one increases class size significantly. Are students much more or less ‘anonymous’ in a class of 30 with one teacher than in a class of 35 or 25 and one teacher? Hasn’t it been known at least since 399 BCE (the year of Socrates’ execution) that the ideal student/teacher ratio is close to 1.0, and that the larger that ratio gets, the lower the quality of education (holding other factors more or less fixed)?
Andrew quotes Beth Tipton: “particularly in under resourced areas, schools and finances can matter a lot” and then she ruins everything for me when she adds this, “thus the focus on charter schools in urban locales.” For more on charter schools and Betsy DeVos:
https://www.npr.org/2020/11/19/936225974/the-legacy-of-education-secretary-betsy-devos
“She argued for less money from Congress, massive cuts in federal education spending and consolidating the programs that would remain.”
“government has never made anything better or cheaper, more effective or more efficient. And nowhere is that more true than in education.”
” argued for less money from Congress, massive cuts in federal education spending and consolidating the programs that would remain”
Funding amount is a limiting reagent but just because you add funding doesn’t mean it gets used to produce the good outcomes you want. Suppose for example with low funding you’ve attracted a bunch of uninspired and under-educated teachers, let’s say they teach subjects they don’t know well, maybe people with a history degree teaching middle school math, or people with an international relations degree teaching computing and robotics (both of these are real world examples I’m aware of locally).
They might be nice people, but they’re not the optimal people for the job, but the optimal people would demand a lot more money. Nevertheless perhaps we come and parachute in a bunch of money and raise these people’s salaries to double what it currently is…. Is this going to improve education a lot? Especially compared to replacing these people with people who have say Masters in Math Education and Masters in Electrical Engineering and Computer Science?
Just spending money isn’t enough. And I think in the political situation most schools find themselves in, adding money rarely results in say hiring different and more qualified teachers.
Now, suppose you’re teaching computer programming and networking technology but the school district has standardized on Chromebooks and the IT department has strict security policies excluding anything even remotely non-standard. There’s no computers on which you can do something like install Linux, Rust, Julia, C compilers, git, Apache, netcat, wireshark, etc. So someone spends a bunch of money to acquire computers, but they’re just faster chromebooks with more RAM and still don’t support the needs of the class… You *should* have spent money to hire a security / IT expert who could support the required technology for the class, but instead you bought some off the shelf technology that does the wrong thing but was easy to acquire and no-one ever got fired for buying IBM (or in this case Google Chromebooks). That kind of crap happens ALL THE TIME.
So, yeah, increasing funding by itself could have no effect or even negative effect if it’s spent in highly sub-optimal ways, which is I think very common.
Note that I am actively participating in a robotics club at a local school and I’m seeing this kind of stuff first hand.
Not Greg Cochran. He sent his kids to the underperforming local public school in Albuquerque, and they became National Merit Scholars. He has acknowledged avoiding violence as one reason to want to avoid certain public schools, but where he lives the schools aren’t particularly violent, they’re just full of kids much less smart than his own.
Wonks:
I agree, when I said that “everyone” wants their kid to go to a good school, that was an exaggeration. I think that most parents want their kids to go to a good school and that most parents want their kids’ schools to have more rather than less funding. But not everybody. There are exceptions.
Wonks –
Do you have a link for that story?
I gave a link to Greg’s blog post in which he listed NM scholars in his state that year. His kid is the only one from Del Norte High School. Same with the previous year.
It is my understanding that the research shows that if you want to predict how a child will do academically the most important factors are the characteristics of the child themselves. Things like IQ or parental income and education. Less important but measurable are peer effects, what the other students are like. A child does better when the other students are doing well. Down in the noise level are the characteristics of the school, at least for schools in range of public schools commonly found in the US. So raising funding usually doesn’t help much. If students at school A are performing much better than students at school B raising funding for school B but keeping the same students won’t affect the gap much. This by the way means that proposals for paying teachers based on student performance don’t make much sense. Teachers don’t make much difference and what difference they do make is difficult to measure because it is difficult to adjust for the much more important factors.
As for why parents generally want their kids schools to be well funded there are several possible explanations. They may falsely believe that this is important for educational achievement. They may like their children’s teachers and think they deserve more money. Or they may think that opposing higher school funding will cause the teachers to retaliate against their kids in school. They may believe more funding will improve the schools in other ways, like making them more pleasant for their children to attend, even if it has little effect on academic achievement. They may want school spending to be high to keep property taxes (often a major funding source for schools) high in order to keep what they see as undesirables out of their school district.
Wow, what a can of worms!
For starters “well funded” and “student outcomes” are not just vague but completely open to interpretation.
When you say “well funded” (school), what funding are you referring to? The pay rate for individual teachers? The number of teachers per student? The quality of the food served at lunch? The sports facilities? Money for field trips and academic extracurriculars? Football uniforms?
Often the fighting about funding in public schools is about the fact that in wealthy areas, aside from having a higher tax base for funding to begin with, parents often donate money for many extracurricular activities – sports and music activities and equipment are often costly – and school supplies, and to teachers with gifts. People in less wealthy areas then claim that this extra funding is giving the wealthy kids a leg up and entrenching entitlement.
What measure do we use for “student outcome”? Lifetime salary? Peak salary? Greater lifetime wealth accumulation? Better test scores? Educational achievement level? Some measure of “social adjustment” or “social contribution” (Joshua would like that!)? Contributions to charity? I can’t imagine how salary or wealth accumulation could be used evaluate student outcomes on a national or even state wide basis, since both are strongly tied to local economic conditions via the cost/value of housing. Certain careers also earn outsized salary compared to the investment in education – e.g., a fireman with a few years of training could earn as much as an ecology PhD with a decade of post-secondary education.
It’s amazing that sophisticated statisticians and scientists even talk about “wealth” and “income” on a national basis when there is no standardized measure to correct for the local cost of living. If someone wants to earn a name for themselves, they should create such a meausre. In the mean time, here’s a map of the cost of housing per square foot by state:
https://i0.wp.com/accidentalfire.com/wp-content/uploads/2018/11/median-price-per-sq-ft-sep18.jpg?ssl=1
West Virginia checks in at $93 per square foot vs $298 in California. But even this hides huge variation across states. If you don’t mind living in Garfield WA, chances are good you’ll pay about half the mean in WA.
So I guess unless people are going to be clear about what measurement they’re referring to, none of this matters much.
My opinion is that the only sensible measures of education acheivement are test scores, which can be nationalized without reservation, since 2+4=6 in both Bangor ME and Weed CA. I have very high confidence that current tests accurately measure educational achievement.
What measure should be used for “funding” is harder to nail down, but perhaps measuring resources by dollars is a poor choice, and the resources devoted to education should be measured more directly by student/teacher ratio or student/staff ratio.
Overall, though, my guess is that the resources devoted to education in a given jurisdiction don’t do much to determine outcomes. According to this guy funding doesn’t mean very much and most of the outcome is the result of student input.
I don’t agree with your first paragraph. I’m not at all sure what test scores really measure, and perhaps more importantly, what they do not. Don’t be fooled by substituting something that can be measured (the ability to know that 2+4=6) for what we really want to know – what are meaningful measures of educational achievement and why are they meaningful? I’m sure test scores measure something, but I am not sure what. For example, consider medical practitioners. I want my surgeon to exercise the best judgement and have the best skills. Does the score on the MCAT predict these things? Do their grades in medical school predict these things? I really don’t know. I do think there is some relationship – when I search for a doctor I do look at where they went to school, and that does correlate with their test scores and other “objective” measures. But I’m not at all confident that I am making informed choices – I am mostly using what I can readily find. It is easy to fool yourself into thinking that the information available is the right information to use, especially when better information (malpractice claims, past surgical outcomes, even number of treatments done in the past year, etc.) is not available.
Dale –
> I’m sure test scores measure something, but I am not sure what.
We’ll, one answer might be that test scores measure how well someone does on a test. I suppose if you trust the reliability scores that ChatGPT says the College Board reports for the SATs – Cronbach’s alpha over 0.9 – you might think they’re fairly good at measuring how well someone does on the test even if they aren’t valid (measuring what they’re supposed to be measuring). But I’m skeptical about that number, and I doubt it’s that high for most standardized testing, especially across much less highly controlled testing conditions.
I think it’s amusing when people who are generally highly skeptical institutional competence put so much stock in standardized testing, conducted by schools.
Have to wonder if there’s a titch of motivated reasoning creeping in with Chipmunk.
” Don’t be fooled by substituting something that can be measured (the ability to know that 2+4=6) for what we really want to know – what are meaningful measures of educational achievement ”
Test scores don’t measure the quality of a doctor. They measure the doctor’s knowledge of medicine. No, you don’t necessarily judge your doctor on only this factor, you consider it with other factors. Nonetheless, you’d probably concede that a doctor with 85th percentile test scores is more desirable than a doctor with 55th percentile scores, presuming there is no overwhleming evidence to the contrary, and you probably *should* concede that even if a specific doctor with a high test score has poor performance, on average test scores would correlate with outcomes all other things held equal.
I’m not at all fooled. Your objections to test scores strike me as contrived.
I would not conclude that a doctor scoring in the 85th percentile on some test is better than one scoring in the 55th percentile. Of course, presented with that data and nothing else, I would choose the doctor with the higher score. I suspect we agree on that. But if you ask me how much confidence I place on that decision, I think we will disagree. My confidence would be pretty close to zero. That is what I mean by being fooled – it is easy to place more weight on information that is available to you than it deserves – especially when the data you really want is not available.
Joshua:
Yes, test scores measure how well someone does on a test. That’s what they’re supposed to measure. That’s what we want them to measure.
I have very strongly motivated reasoning!!! I want a decent country to live in where doctors know medicine, engineers know engineering and everyone who has a degree has some consistent base level of knowledge. Testing ensures that.
I’m accused here often of not providing data to support my views but you have absolutely no data indicating a lack of correlation between test scores and outcomes, and in fact you openly admit the data expresses a strong correlation between test scores and outcomes. What’s the basis for your skepticism?
forblogs –
I was just somewhat indirectly pointing out that there are two fundamental issues (imo) with standardized testing: reliability – e.g., does someone get the same score each time they take it; and validity – does the test actually measure what it’s supposed to be measuring (e.g. how “intelligent” someone is or how well they’ll perform on a job) *as opposed* to just how well a person does on that kind of test.
I’m skeptical about norm-referenced standardized testing for a variety of reasons in both repects. I think they have some value but it’s pretty limited to the more extreme ends of the spectrum (IOW, they give a decent idea of outliers, but I doubt we really need standardized tests to do that).
Instead, I see norm-referenced standardized testing as mostly malign in effect because they promote the idea that the focus of education should be to evaluate students by comparing them to other students.
I do, however, see value in criterion-referenced testing (standardized and non-standardized) which actually measures what students do and don’t know in a way that’s useful for both then and for teachers in evaluating what they’ve mastered and what they have yet to master.
As I teacher, I find standardized test results as basically useless. They don’t have any meaningful pedagogical or didactic function. They don’t help me to know what a student has learned or what they need to learn. One student in the 95th percentile in a particular subject area might have a very mastery profile than another student in 95th percentile. More useful is a breakdown of what each student does and doesn’t know.
There’s some limited value in norm-referenced standardized as a tool for evaluating pedagogy more broadly, IOW how one educational practice might compare to another. But I think that value is largely lost in all the noise created by norm-referenced standardized testing, that distracts people from a more focused use of their time within educational activities.
Chipmunk –
> Test scores don’t measure the quality of a doctor. They measure the doctor’s knowledge of medicine.
It’s interesting that you’re so critical of the reliability of some institutional models for a variety of reasons but so trusting of others. Do you have evidence of the correlation between test scores and the performance of doctors as doctors, or is this just something you take as an article of faith?
Although I agree with the points around tests measuring specific components of overall competence—something quality testing organizations commonly state—and can be outweighed by more targeted evidence of the focal behavior, I think the discussion is starting to imply that the entire standardized testing industry has no validity to support the tests’ intended inferences. Because of the type of discussions occurring here, testing organizations and external parties consistently conduct/publish research exploring the abstraction from test scores to practice. This research is the holy grail of research in the industry for obvious reasons. Of course, these studies should be held to extra scrutiny given the potential conflict of interests, and they fall prey to many of the statistical faults discussed consistently on the blog. However, they shouldn’t be dismissed outright.
To the question about testing in medicine, here is a paper Andrew co-authored showing that individuals who perform higher on the United States Medical Licensing Exam have lower probabilities of disciplinary sanctions (a physician outcome that I suspect most patients would find important).
https://pubmed.ncbi.nlm.nih.gov/28562454/
A few more about medicine because that’s where the discussion has landed:
https://journals.lww.com/academicmedicine/abstract/9900/the_associations_between_united_states_medical.582.aspx
https://journals.lww.com/academicmedicine/Fulltext/2014/08000/The_Relationship_Between_Licensing_Examination.26.aspx
Anon2
Thanks for these references. They are indeed relevant and I will reconsider some of my thinking. One striking thing, however. I now want to see my doctors’ USMLE scores. They should be publicly available. So little actual data is available for choosing a physician or hospital, but we have a potentially valuable piece of evidence. Too bad that it is not accessible.
Anon2 and Dale –
> Of course, these studies should be held to extra scrutiny given the potential conflict of interests, and they fall prey to many of the statistical faults discussed consistently on the blog. However, they shouldn’t be dismissed outright.
>> So little actual data is available for choosing a physician or hospital, but we have a potentially valuable piece of evidence. Too bad that it is not accessible.
I will note that apparently, from what I can gather, the College Board doesn’t make public it’s analysis of the “reliability” of their standardized tests. Don’t know about “validity” measures.
Since there is a lot at stake – the relationship between educational spending and outcomes is important – these problems that you correctly identify should not stop research. In many ways, it makes such research more important. What is missing are processes that ensure that the research builds on past research, clarifies the issues, replicates (in the widest sense of that word) past work, and more accurately describes our uncertainty about well defined questions. It should also clarify what questions are questions of fact and which are ethical or issues of preference.
Unfortunately, most of our processes fail to do these things. Peer review, data quality, editorial practice, instructional evaluation (yes, the horrible word, “assessment”), political motivations, and analytical competence of researchers all fail to deliver these things. I wish I could say that their net effect is to move in these directions despite myriad stumbles, but I’m not convinced of that either.
We all want answers – will increased funding lead to improved outcomes? What we need is will increased funding of type X lead to improved outcomes A,B,C, etc? What we get is over-simplified answers to ill-defined questions. Of course there are exceptions – but how do we distinguish those from the reputation-enhancing clickbait?
Also – regarding your cost of living example. House prices are not a good measure of local cost of living – and reputable inflation measures are not based on house prices. A house is an asset and the price of the house represents holding one type of asset (a house) rather than another (for example, a stock). House prices vary far more than the costs of housing. To measure housing costs, you want measures of the user costs of housing (property taxes, utilities, maintenance, etc.) and some measure of the rental cost (an opportunity cost for someone that owns their home). Proper measures of housing costs will vary greatly, but not as much as the price per square foot.
Dale:
Yes, my reference to home prices was just to point out that the cost of living varies widely across the country and both weatlh and incomes vary with it, thus dollar funding levels for schools are almost meaningless on a national basis.
Housing costs are generally measured by median monthly payments. Since people use between a quarter and half of their monthly income for housing they are actually a pretty decent indicator. They are an important part of the poverty level calculation for an area.
That is a particularly poor practice unless it is carefully designed to compare apples to apples. Rents are straightforward and can be compared with income. Monthly payments on a house you own are not. If you are wealthy and have a mortgage on an expensive house, then much of that monthly payment is a form of holding assets and not part of the “cost” of owning a home. Most of the attention that gets focused on median monthly payments in relation to income involve whether or not owning a home is “affordable.” While it often is not (as when 25%-50% of income is devoted to monthly payments), this is more accurately seen as an issue for home ownership rather than an issue about housing costs. A purer approach would be to look at median rent in relation to income.
In my opinion, too much of the public debate is focused on home ownership. The US is an outlier in this respect – in many countries rental housing is much more common (and much better protected through public policies). If it is really important to focus on home ownership, then I think we need to be careful about what the true costs of home ownership are – the price tag on the house is not a good measure of that. If I spend $500,000 in a cash purchase of a home, then I have converted $500,000 from a bank account or stock portfolio into a $500,000 house. I have not “spent” the $500,000, I’ve converted it from one type of asset to another. There are consequences, to be sure, but if we change the $500,000 to $1 million, there has not been an increased $500,000 of cost.
Dale, you’re right but I think the discussion around housing is that you get a very different thing when you pay rent to occupy a unit in a 100 unit apartment complex not far from a freeway with downtown polluted air vs when you pay rent to occupy a single family home with a back yard and room for kids and dogs to play and low traffic noise and relatively better air quality etc.
When we talk about home ownership, a big part of what we’re talking about is quality of life for families with children, and affordability to have children in the first place.
The other part we’re talking about is the unreasonably good performance of investment in housing, which is undoubtedly a distorted market due to both restrictions on building, and subsidies on taxes etc.
Concerning local cost of living (or local inflation):
These measures do exist! But there is a problem: They do not necessarily exist as official statistics. Official statistics are basically correlated equilibria: We use them, because everyone agrees to use them*. There would be no point in inflation research if every researcher would measure inflation all by themselves, using their own methodology.
*Except for research that tries to address the shortcomings of official inflation measurement.
“So I guess unless people are going to be clear about what measurement they’re referring to, none of this matters much.”
I completely agree. There are a myriad of different confounding factors to consider, and never mind that education is in many ways a control/ feedback system (as gec points out). It is a really hard thing to study! I am not an educative research scholar, but I suppose I would rather try to quantify something with a strong prior (e.g. more teachers per pupil leads to better educational outcomes) than something with a weak prior (“More funding for a school may or may not make a big or small difference to educational outcomes.”).
I think the outcome variable is also difficult to choose, because school is not just about education in maths, languages, science, music, etc. It is a place where children and youth interact with each other and grown adults. The people who work within the ‘school’ universe do much more than just educate. If a school provides information to other agencies about suspected child abuse, no statistic will ever capture that. If a teacher’s good example helps a pupil to stay on the straight and narrow, or their encouragement helps another to develop their talent, this will not be captured in any test score. If a teacher helps their students to resolve a conflict, or the bus driver teaches the kids by example that there is some good left in mankind, nobody will ever know. But it makes a difference.
What I am trying to say is that the idea of measuring “the” outcome/value of school is something that policy makers may wish for, but cannot have. Let us (and them) not fall prey to the McNamara fallacy.
…that was supposed to be an answer to chipmunk’s first comment…
Raphael:
Your points are well made. I had a teacher in high school who had a masters in physics, had worked in physics research at the berkeley labs, and then decided to be a school teacher most likely out of personal altruistic/social goals. When he retired he went to teach physics in Fiji for the Peace Corps.
These days I think about that man almost every week. When I had him he was probably 60 years old. The way he taught was different from almost every other teacher I’ve had. He designed every goddamn experiment himself, he wrote out worksheets by hand to collect data and do calculations on, he built the apparatus himself, he walked through an explanation of how the damn things worked for each experiment, he programmed data analysis programs in BASIC on an Apple II himself… Essentially nothing he did was from a pre-bought curriculum. The man was a saint to put up with us and give us so much of his own effort. He also participated in teaching teachers through the Exploratorium in the summers. I don’t mind linking to the East Bay Times covering an award he won for his Peace Corps activities https://www.eastbaytimes.com/2007/08/11/two-berkeley-volunteers-honored/
If he’s still alive he’d be 94. I have occasionally stalked him online to work up the courage to call him up and tell him how important his contribution was to my own relationship with science… but I’m quite honestly afraid I’d find him in a state of such decreased capacity that the whole thing would just make me sad, or make him confused etc. I think of my own grandfather who died at 92 and how much decreased his abilities were compared to say 72…
On the other hand, in all my experience in education both for myself and my kids, there really isn’t another teacher like that. I consider myself extremely lucky.
How many inspired people like that could we have if we paid 2x current teaching salaries to attract that level of inspiration? I can tell you I’d almost certainly be teaching science at the high school or a mix of city-college and high school level using those kinds of methods if I could make $160k doing it. In reality I’d probably make $80k which after taxes and hiring a nanny to take care of my kids needs is basically $0 net. I’m not exaggerating.
On the other hand, how bad would it be to spend 2x our current salary budget and wind up with the same people currently teaching just with higher salaries thanks to politics and institutional bullshit?
I’d pay more in taxes if I could ensure every student had someone like that for at least a class or two before graduating high school. I was lucky enough to have him for Algebra, Chemistry, and Physics, so 3 out of 4 years.
My kids teachers in Middle School are burned out, they’ll tell you that themselves but it’s also just obvious from talking to them. And they’re struggling to afford housing and retirement savings and commute costs and soforth. There’s one teacher who seems a little like my Mr Ellison that teaches chemistry at one of the local high schools. He spent 30 years as a soil/environmental chemist before starting teaching. He’s part of the reason I’m planning to send my kids to that high school. But he’s doing it as basically an early retirement after his soil chemistry job did a round of layoffs 5 years ago or so.
All of this is to say, it’s not like people like this really raise the test scores as much as they do a bunch of intangible things that can be extremely important.
You need to write to him.
You’re right. That’s what I’ll do. I kept focusing on calling him… but cold-call is too much putting him and me on the spot. But I have his address, or at least at one point it was his address… so I’ll write a letter.
For what it’s worth, here is the conclusion from Chetty et al, in 2014 (https://opportunityinsights.org/paper/teachersii/):
“Are teachers’ impacts on students’ test scores (“value-added”) a good measure of their quality? This question has sparked debate partly because of a lack of evidence on whether high value-added (VA) teachers improve students’ long-term outcomes. Using school district and tax records for more than one million children, we find that students assigned to high-VA teachers are more likely to attend college, earn higher salaries, and are less likely to have children as teenagers. Replacing a teacher whose VA is in the bottom 5% with an average teacher would increase the present value of students’ lifetime income by approximately $250,000 per classroom.”
I’ve not been a fan of Chetty’s work in the past, mostly due to the fact that only the heavily processed data is made available (not the raw data, which he seems to have available to him and few others). But at least the work is carefully done. As in my earlier comment, what is missing is any clear why to tell whether this work corrects for earlier work that has differing conclusions, whether it uses better measures, whether it is done more competently, or whether it brings us closer to answering any of the questions we want answers to. I guess since he is at Harvard, that’s all we really need to know.
Keep in mind that the projected $250K/classroom lifetime earnings premium is based on a class size of 28 and a working lifetime of 40+ years. That’s about $200 per student per year. Not trivial, but the $250K/classroom may be deceptive.
https://www.barbarabiasi.com/uploads/1/0/1/2/101280322/bilaschon_2023.pdf
More from Kirabo Jackson (2018):
https://www.nber.org/system/files/working_papers/w25368/w25368.pdf
“Social scientists have long sought to examine the causal impact of school spending on child out- comes. The literature on this topic can be put into two clear categories: an older literature that relies on observation variation to correlate school spending and student outcomes, and a newer literature that relies on quasi-experimental methods to uncover relationships that are plausibly causal.
The older literature provides strong support for there being a positive economically important association between increased school spending and improved student outcomes. That is, despite claims to the contrary, the application of reasonable statistical reasoning to the patterns across studies would lead one to conclude that there is a strong statistical link between spending and outcomes. However, because this older literature is entirely observational in nature, these studies do not speak to the causal question of whether increased school spending improves student outcomes. To do this, one must examine the more recent quasi-experimental literature.
The recent quasi-experimental literature that relates school spending to student outcomes over- whelmingly support a causal relationship between increased school spending and student outcomes.
[…]
By and large, the question of whether money matters is essentially settled. Researchers should now focus on understanding what kinds of spending increases matter the most, and also in what contexts school spending increases are most likely to improve student outcomes.”
And four additional studies since then: https://www.chalkbeat.org/2019/8/13/21055545/4-new-studies-bolster-the-case-more-money-for-schools-helps-low-income-students/
Thanks