Heckman Curve Update Update

tl;dr: “The policy conclusion we draw from our analysis is that age is not a short cut for identifying where governments should, or should not, invest. There are many well‐studied interventions for children that are worthy candidates for public funding based on efficiency considerations. However, the same is also true of many interventions targeting youth and older people. . . .”

Here’s the background. David Rea writes:

Just letting you know what eventually happened with our exchange with James Heckman over our ‘Heckman Curve’ paper in the Journal of Economic Surveys.

When we last emailed, James Heckman had written a response to our paper, and this had been accepted for publication in the JES.

The journal then offered us the opportunity to write a reply, which we did, and this was published as an early view.

After this James Heckman decided to withdraw his paper. We don’t know why he changed his mind about publication, but it did seem a little unusual.

If you are interested, here is a bit more background on the debate that didn’t eventuate. This is the Heckman Curve as published in Science in 2006:

Our paper attempted to see if more recent data showed a similar pattern. We used a dataset of benefit cost ratio estimates for 314 well-studied interventions from the Washington State Institute for Public Policy database.

We couldn’t find any evidence of a Heckman Curve in the data.

To be clear, the dataset has many early childhood interventions that generate large estimated benefits relative to the program cost. There are many early intervention programs for children that are worthwhile investments from an efficiency point of view. However, the key point is that there are many well-studied youth and adult interventions that are also very cost effective.

Heckman’s unpublished response to our paper contained several criticisms. The central argument was that we were misinterpreting his work. He stated that the Heckman Curve does not describe how the return on investment of human capital interventions differs by age. It is instead a theoretical proposition or thought experiment about an optimal portfolio of best-practice investments in human capital.

We actually thought that this was a very useful clarification. Many people, particularly in the policy community, understand the Heckman Curve as describing how average rates of return of human capital interventions differ by age. Clarifying this point has important real-world implications for social policy. It’s a funny thing to say given that he is critical of our work, but we think it would be useful for James Heckman to publish his paper.

We discussed David Rea and Tony Burton’s paper, “New evidence on the Heckman Curve,” last year.

Apparently the response by Heckman will never be published.

But, in any case, here’s the reply by Rea and Burton to the response that Heckman withdrew:

Clarifying the Nature of the Heckman Curve

David Rea and Tony Burton

Abstract: In response to our paper, James Heckman states that the Heckman Curve does not describe how the average return on investment of programs differs by the age of recipients. This clarification is useful as many people in the policy community have understood the Heckman Curve in this manner.

In our paper, we interpreted the Heckman Curve as a proposition that (a) social policy interventions targeted at early childhood would generate benefit cost ratios that were higher than other age groups, and (b) interventions targeted at older age groups would often have benefits smaller than their costs (Rea and Burton, 2020).

Our characterisation of the Heckman Curve was based on the paper by James Heckman entitled ‘Skill formation and the economics of investing in disadvantaged children’. This was published in Science in 2006 and stated that ‘early interventions targeted toward disadvantaged children have much higher returns than later interventions such as reduced pupil teacher ratios, public job training, convict rehabilitation programs, tuition subsidies, or expenditure on police. At current levels of resources, society overinvests in remedial skill investments at later ages and underinvests in the early years’ (Heckman, 2006, p.1902).

The natural interpretation of the quoted statement is that Professor Heckman was discussing interventions, investments in human capital and was advising decision‐makers inside and outside government to target investments in a particular way.

The empirical component of our paper used a dataset of benefit cost ratios estimated by the Washington State Institute for Public Policy. Analysis of the dataset showed no readily apparent relationship between the estimated benefit cost ratios of interventions and the age of recipients.

The policy conclusion we draw from our analysis is that age is not a short cut for identifying where governments should, or should not, invest. There are many well‐studied interventions for children that are worthy candidates for public funding based on efficiency considerations. However, the same is also true of many interventions targeting youth and older people. A number of early intervention programs have been shown to be cost effective, as have a range of ‘remedial’ or ‘second chance’ programs targeting older individuals. Good public policy requires a case‐by‐case assessment of the evidence and benefit cost analysis for each intervention being considered.

In his comment on our paper, Heckman states that we have misinterpreted the Heckman Curve (Heckman, 2020). He states that it is not a proposition about how the average return on investment of programs differs by the age of recipients. We welcome this clarification and believe it is a useful outcome of the exchange between us. Many people including ourselves have previously understood the Heckman Curve to be advice on public investment in human capital programmes. It is helpful to understand that his work in this area is not meant to be relevant to public investment decisions in the manner we describe.

References

Heckman, J.J. (2006) Skill formation and the economics of investing in disadvantaged children. Science 312 (5782): 1900–1902.

Heckman, J.J. (2020) Comment on “new evidence on the Heckman Curve” by David Rea and Tony Burton. Journal of Economic Surveys.

Rea, D. and Burton, T. (2020) New evidence on the Heckman Curve. Journal of Economic Surveys 31 (2).

So, it all seems clear. Heckman published a pretty graph in 2006 that was misinterpreted by many people as implying that “early interventions targeted toward disadvantaged children have much higher returns than later interventions such as reduced pupil teacher ratios, public job training, convict rehabilitation programs, tuition subsidies, or expenditure on police. At current levels of resources, society overinvests in remedial skill investments at later ages and underinvests in the early years.” But that was a mistake. In fact, the data don’t seem to support the claim that human capital investments are most effective when targeted at younger ages, and Heckman appears to agree with this, to the extent of wanting to emphasize that his curve is not a proposition about how the average return on investment of programs differs by the age of recipients.

On the other hand, Heckman still has this up at his webpage:

The Heckman Curve

This graphic shows that the highest rate of economic returns comes from the earliest investments in children, providing an eye-opening understanding that society invests too much money on later development when it is often too late to provide great value. It shows the economic benefits of investing early and building skill upon skill to provide greater success to more children and greater productivity and reduce social spending for society.

This graphic is formatted for use on social media and insertion into presentations, handouts and press releases.

Summary

I’m not sure what to make of Heckman’s advice. On one hand, on his website he is clear that his eponymous curve shows that “The earlier the investment, the higher the return.”

But in his (now unpublished) response to Rea and Burton, he wrote:

None of this says that the benefit-cost ratios or internal rates of return are necessarily higher for all younger-age interventions. . . . The Curve is a technological frontier across programs (best practice) and not an average across all programs, however poorly executed . . . Policy makers need advice on best practice, not on average practice.

So we should modify the advice on his website to “In best practice, the earlier the investment, the higher the return.”

I’m concerned, though, because I think that the statistical methods used by Heckman and his collaborators cause them to overestimate effect sizes. Early-childhood intervention supposedly increasing adult earnings by 42%. Noisy studies with huge standard errors, thus the statistical significance filter yields high effect sizes. The upshot is that I have no idea what are the effects of these supposed best-practice interventions—and I think Heckman has no idea either. The main difference between Heckman and me here is that he’s expressing a lot more confidence than I am in those noisy estimates.

I have two problems with the statement, “Policy makers need advice on best practice, not on average practice.” First, we really don’t know what best practice is. These interventions that Heckman is so sure are best practice, maybe aren’t. Second, even if these interventions were best practice, to be scaled they’d have to be applied in the real world, i.e., average practice. It would seem naive to think that whatever particular interventions had been tried in some experiment many years ago could just be applied directly in new settings.

In any case, I agree with Rea that these are all good things to be discussing in the open. He and Heckman and Charles Murray and Aaron Edlin and Anna Dreber and Rachael Meager and anyone else are welcome to reply in comments.

15 thoughts on “Heckman Curve Update Update

  1. Just another publicity-seeking “scientist” publishing conclusions he arrived at before bothering to analyze any data. As it is, was and (apparently) ever shall be.

  2. I actually do think the ‘frontier’ interpretation of the Heckman curve is a more useful one than the ‘average’. However, eyeballing Rea and Burton’s scatterplot doesn’t obviously suggest that this would lead to a different conclusion (i.e. the frontier is also high for older age groups).

    Of course, as Andrew points out, there are still tricky issues with the quality of the underlying data: I expect you’d ideally want to shrink the frontier estimate based on potential bias and imprecision in the underlying BCRs, and potentially also some sense of ‘scalability’ of interventions, none of which is easy.

  3. *sigh* I haven’t looked at either paper. But my intuition is that the devil is in the “cost-benefit” calculation. I’m inclined to skepticism that social programs are so valuable. I’d guess to whatever extend the ROI is supposedly high, it would be much higher if the money were invested directly in economic assets. Rather than having a jobs program, build something that a) creates jobs and b) has economic benefit (transportation assets for example). If you can get a better job easily you don’t need more training.

  4. Oh, what a depressing worldview!

    It is more cost-efficient to train a new child that to reeducate an old man. But then what are these children are educated for? To live and work in the society where they are expendable, because it no longer cost-efficient to help them?

  5. Might interest you, they use the dummy equals to a 100 and find that “Our primary finding: when Black newborns are cared for by Black MDs, the mortality penalty they suffer as compared with white infants is HALVED.” (quoting from the researcher Twitter):

    https://edition.cnn.com/2020/08/18/health/black-babies-mortality-rate-doctors-study-wellness-scli-intl/index.html?utm_term=link&utm_medium=social&utm_content=2020-08-18T17%3A40%3A08&utm_source=twCNN

    Here is the paper:

    https://www.pnas.org/content/early/2020/08/12/1913405117/tab-figures-data

    • It is an interesting paper. Have to admit that I’m curious from the appendices that white infants with black doctors are much more likely to be on Medicaid than white infants with white doctors (but no difference among black infants depending on the doctors), plus it seems that black infants with white doctors spent significantly longer in the hospital (1.5 days longer) than with black doctors, so it’s not a lack of followup care necessarily, but possibly some other factor (including perhaps a selection bias). The mortality measured was only in-hospital mortality; it would have been slightly better to have a follow-up that measured mortality after X days especially with that.

      Also no effect on Latino infants, with Latino or Asian doctors, etc. Certainly a large effect.

  6. I am open to the idea that Heckman may be speaking out of both sides of his mouth. That said, what does the data look like for “average effect” across age cohort? Maybe I’m missing something, but the critique posted here doesn’t cite data which contradicts the assertion that early childhood intervention yields greater ROI than later age cohorts.

  7. I think Heckman is right about decreasing returns because of opportunity costs–don’t take time to do it right, so do it over.

    I think the writers’ model must be under specified, in that the environment the kid grows up in can have a huge impact on the effect of the human capital investment. In high school, several of the students in my homeroom failed courses regularly…they lived in neighborhoods where making it to school each day was the result of not being the victim of a random drive-by shooting. Years later the city was rated one of the worst places to live in the USA, because of the drugs and gangs. In fact, I knew of a guy that was a promising athlete whose career ended after being hit by bullets intended for someone else when he was walking. He didn’t die, but his dreams did.

    If controlled for environment, I believe the Heckman curve would emerge, yet I would not agree that it implied giving up on adults. Especially because those adults are often the ones creating a negative environment for the kids. If the adults study or learn new skills, rather than free-ride, it sends a signal to the kids that they can get something useful from studying.

    It must be hard to stay motivated if parents are working low skill jobs, with the exception that the kids are thinking “I don’t want to end up like my parent(s).”

  8. The graph says it all. No plot even informed by real data is perfectly smooth like that. Why did he decide it was an exponential decay? cause everything’s exponential don’t ya know? This just in scientists find that on a log log plot everything’s a strait line, now over to camera two to hear about how red wine prevents cancer and chocolate is good for gout.

    Seriously this “Heckman” is so far off into lala-land that I’m just going to bin him in the Not-Even-Wrong camp. Point and laugh at this man when you see him, make it hurt.

  9. If Heckman’s curve is supposed to apply to best-practices, isn’t it trivially true due to compound interest? One of your “interventions targeted at children” could be establishing investment accounts for them that would pay for interventions when they are adults, and it’s always more cost-effective to do that than to start with the same amount of money when they are already adults.

Leave a Reply to John Thacker Cancel reply

Your email address will not be published. Required fields are marked *