Danielle Fumia writes:
I work on research estimating the effect of college attendance on earnings. Many studies that examine the effect of attending college on earnings control for college degree receipt and work experience. These models seem to violate the practice you discuss in your data analysis book of not including intermediate outcomes in the regression of the treatment (attending college) on y (earnings). However, I’m not sure if this situation is different because only college attendees can obtain a degree, but in any case, this restriction wouldn’t be true for work experience. I fear I am missing an important idea because most studies control for one or both seemingly intermediate outcomes, and I cannot seem to find an explanation after much research.
My reply:
For causal inference, I recommend that, instead of starting by thinking of the outcome, you start by thinking of the intervention. As always, the question with the intervention is “Compared to what?” You want to compare people who attended college to equivalent people who did not attend college. At that point you can think about all the outcomes that flow from this choice. In your regressions, you can control for things that came before the choice, but not after. So you can control for work experience before the forking of paths (college or no college) but not for work experience after that choice. Also, suppose your choice is at age 21: either a student attends college by age 21 or he or she does not. Then that’s the fork. If a non-attender later goes to college at age 25, he or she would still be in the “no college attendance at 21” path.
I’m not saying you have to define college attendance in that way, I’m just saying you have to define it in some way. If you don’t define the treatment, your analysis will (implicitly) define it for you.
Here’s more you can read on causal inference and regression, from my book with Jennifer. We go into these and other points in more detail.
Reminds me of a consulting project we did a while ago determining the earnings and covariate adjusted “earnings boosts” for various universities. I had a lot of fun with that one. While I can’t go into the specifics of the modeling, I worked with a guy named Stan on it.
http://inside.collegefactual.com/methodologies/salary-data-faq?rq=earnings
Very interesting site on college rankings. However, it is most disturbing to see so much overblown hype (talk about the models not being “100% accurate” for example. Also, nowhere is there any indication of how accurate the models are or are not. I’m aware of many of the methods for inferring missing data – what I’d like to see is some measure of how well this particular modeling effort performs rather than statements about how wonderful it is and caveats that you should not expect to earn exactly what the model says.
These are good points. There were many discussions about how best to present these results given the audience. Remember that many people using this site are high school sophomores to seniors. Talk about 100% accuracy via population survey v.s. a sample based estimate is a way of explaining the idea of a statistical estimate and sample uncertainty. And while presenting an earnings density estimate and parameter uncertainty estimates for a school/major would not be difficult from a mathematical standpoint, presenting this information in a manner that is both interpretable by the audience and assists in their decision making is the challenge. My belief is that the team did a great job presenting a huge amount of information in a meaningful way.
Great care was taken to ensure that the model was appropriate and accurate. Your concerns are probably similar to those I would make looking in from the outside. The main problem is that in general, it is difficult to provide a satisfactory explanation of a methodology that is proprietary.
One thing that often doesn’t seem to be accounted for in these rankings is that schools have different distributions of majors. For example, engineers will typically have higher starting salaries than english majors. So if College A is 90% Engineers/10% English majors, and College B is 50% Engineers/50% English majors, College A graduates will probably make more on average than College B graduates, even if College B graduates are more sought after. Do your rankings adjust for this?
Indeed, that is the college’s “earnings boost.” The boost is used as part of the college’s overall ranking rather than the unadjusted estimate. Of course the estimate not removing the major compensation differences is also of interest, so that is reported to the user.
Note, it has been some time since I passed of some of this to others, so there may have been some changes.
“controlling for college degree receipt” is just another way of saying you’re really doing comparisons across 3 conditions isn’t it?
1) people who don’t attend college,
2) people who attend some college but don’t get a degree
3) people who complete college and get a degree.
or am I missing something?
Do bed nets decrease malaria rates conditional on the number of mosquito bites someone gets in the night? Does Vitamin A supplementation in children reduce blindness conditional on Vitamin A levels in the blood? Attending college does not magically increase earnings – it just increases your ability to earn (through productivity gains, minimum job requirements and social networks). So we should be interested in “the effect of a college attendance without graduating and everything that later entails,” not “the effect of a college attendance without graduating net of the things college attendance does”. The latter comparison doesn’t make sense – it would imply that we don’t care about the mechanisms through which college increases earnings, we just want to know its magical effect on earnings.
Now, as for why people control for “college completion” in this setting – I don’t think they are thinking of it as a “post treatment control” (maybe it would help them if they did, but that is another point). I think their thought experiment is comparing people who attended at least some college (whether they finished or not) with those who didn’t attend, and “controlling for completion” is like controlling for the sheepskin effect of the degree (so you get, say, 10% increased earnings per year of college, and another 10% for actually finishing). So “completing college” is one form of attending college – those people are treated, but they are also treated with a second treatment with an additive effect (graduating college).
But I think this is bad thinking. It ignores completely selection into both college attendance and college completion – as though the people who do/don’t attend college or do/don’t dropout would’ve had the same earnings had they actually all completed college (or all dropped out or all not-started). But there are probably reasons some people drop out, the same way there are reasons some people go to college and others never do, and whatever those reasons are, they are probably correlated with earnings potential.
I see no conceivable reason to “control” for post-graduation job experience, unless you just mean number of years in a job (there is an experience-wage profile, and people of the same age who did/didn’t go to college will have different number of years experience assuming you don’t work while in college). But certainly you wouldn’t want to control for, say, industry of employment.
I really don’t like the “control for” wording in this kind of research. We imagine some “life process” that produces outcomes at some mid-life point, like say 30 or 40 years old, what is that process?
1) Born into varying types of families and get varying types of early background experience
2) Early health and mental health issues (such as ADHD) develop
3) Education at the pre-college level occurs and students choose to study more or less and to “play the game” more or less
4) Young people finally get a chance to have significant control over their own life trajectory around HS graduation, but this is to varying degrees depending on the parents education, cultural, and socioeconomic conditions.
5) Some students go to college, some don’t. some of those that go have a hard time or don’t like it and drop out, others enjoy playing this game and continue through, graduating or even going on to graduate school.
6) Finally out of school people get varying jobs, and those who didn’t go to school have different experience from work.
Now… what effect did college attendance have by itself? This is ultimately the question, and in some sense I think it’s a misguided question, but let’s just be really clear about what it means. It’s related to holding *all else equal* so when we compare someone who say attended college to someone who got a job right away out of HS and didn’t attend, if we’re interested in the effect of college attendance itself, we will have to find MATCHED PAIRS of people who have essentially the same GPA, both completed HS with good enough grades to get into a similar college, both have similar cultural, health, and socioeconomic backgrounds, with similar interests… but say one of them “just decided” they were ready to head off to work and the other went on to play the college game.
My guess is, in the current US system, with the college loan scam running as hard as possible thanks to it being a lucrative source of income for the govt…. that there just aren’t enough matched pairs to make a difference here. Few people who could in fact go to college and do well will choose not to.
So, the question, if not meaningless, is at least only applicable to I’m guessing a pretty small group of people.
Now… there are plenty of people who *don’t go to college* but my guess is that they are substantially different from those who do in a variety of ways including parental income, parental education levels, cultural background, economic ability, personality, and whatever. So the difference between people who do and don’t go to college is basically down to the factors that make it so that some people aren’t a good fit for going to college, not the actual college experience itself. You could, perhaps, with enough detailed data on people’s backgrounds, fit a state-space causal model on life trajectories, but we simply don’t have enough of a good quantitative model to do this, so any attempt to simply “control for” those other factors using something like linear regression maybe with a few interactions, is a joke in my opinion. That kind of thing is fine perhaps as a sort of Taylor series around a small range of conditions, so if you’ve got approximately matched groups and they have small differences in average conditions, you could *maybe* get away with it, but across the full spectrum of life conditions it would be a joke.
To provide support for this idea that the relevant population is small, we see that college attendance for 18-24 year olds is around 70%
http://www.bls.gov/opub/ted/images/2010/ted_20100428.png
Back of the envelope, what percent of the 30% that don’t attend are people who at graduation were basically similar to the 70% that did and had all the necessary skills and personality requirements to go on to do well in college? I can’t believe it’s more than say 25% of the 30% or about 7.5% of the population of 18-24 year olds, which is itself probably around what 2% of the overall population of the US?
Well, college attendance also includes things like associate’s degrees and community college, which are probably more common than you think they are. Yes, a person going to a 4-year college is not very similar to a person hitting the job market right away, but a person in a 2-year local CC is probably a lot closer to that HS grad. Someone doing CC part-time while working is even closer to that HS grad. This division is farther from that implied question of to what degree a traditional 4-year college investment is “worth it” but closer to the lived experience of the US. I think a lot of us forget just how many CC attendees and AA carriers there are because just about all of the people talking about this in academia went to 4-year schools themselves, and likely everyone they knew in HS did the same thing.
That’s a good point. I think I was explicitly thinking of 4 year schools, that is “graduation” from a 2 year school would still be considered “some college” rather than “attended and graduated” since the 2 year degree is usually a way to signal “I’m ready for a transfer” not an end-point in itself within the industry.
In any case, as someone who got a 4 year degree and then later went to a 2 yr school to do prerequisites to transfer for a second bachelors degree, I have lots of respect for community colleges, they’re a very cost effective way to do a lot of the basic requirements for a higher level major. I’ve also experienced that CC transfer students are some of the hardest working and more motivated students. They often make better upper-class students than those who breezed straight into a 4 yr degree, so there’s some complexity in analysis.
“… since the 2 year degree is usually a way to signal “I’m ready for a transfer” not an end-point in itself within the industry.”
I’m not so sure about that. In many two-year colleges, these (often 2-year) “Associate degrees” prepare for specific careers such as dental hygienist or various types of medical or other technician or assistant. (See e.g., http://createacareer.org/associate-degree-jobs/)
A good application for Judea’s DAGs?
Potentially good, mainly to unambiguously define how your model works and to compare alternative models. I don’t really understand the formal details of Judea’s stuff so I can’t say for sure.
Also my impression is that Judea’s stuff is better for modeling things where there aren’t extended periods of time and a state-space development. The whole history of the first 20 years of a child’s life plays out in a way that affects their capacity to do well in college, so it might be cumbersome to do a DAG model of how that time-series works. I don’t know.
Daniel,
As usual, I agree with your critiques of the question, and disagree with you on how you might go about answering a similar, more useful question.
You are thinking of MATCHED PAIRS (!!!1!). How about this thought: who is the person who would be brought in to college if college was a little cheaper or closer to home or easier to get into, and how much more money would that person (the “marginal person”) make if they went to college for a bit (without graduating) instead of not going? I think that is a reasonably interesting and well-framed question, one that could have real policy implications.
Now how would you go about answering that? Well, instead of college, let’s talk about elementary/middle-school in Indonesia. Why? Because I like this paper some obscure academic person (Esther Duflo*) once wrote about it. Ignoring any questions about the execution, I just want to focus on the thought experiment (and note that my framing of the thought-experiment is a little different than hers even if hers is probably better… I’m just making a rhetorical point here, not doing science).
Rural Indonesians didn’t have a lot of access to schools in the early 1970’s. This made it very difficult for many people who would generally want to send their kids to school to do so. So the government started building schools – about 60,000 of them between 1973-1978. Some families got a school built near them, some didn’t. There was no difference between these families, it was just “as good as random” whether they ended up getting a school built near them or not. If the government randomly allocated schools, then we have a perfect experiment (a perfect “natural experiment”) that leads to some kids getting more schooling than other kids by pure (random) luck.
So then you take a look at those kids 10 or 20 or 30 years later. For every additional year of school kids got by having a school built near them, they earned about 7-11% higher wages.
There is no need for a matched pairs anything here, or for some big model of human educational attainment or the determinants of wages. She just found a thing in the world that induced some children – the children who wanted to go to school if the “price” of schooling (in terms of time to get there) was slightly smaller – to get more education than other, equivalent children who faced a smaller “price reduction”. Then she compared their earnings later on. A nice little natural experiment, with an effect identified off just comparing two groups of people (as I said, it is a bit more complicated than that, but you get the idea).
This strikes me as a policy relevant question and policy relevant answer. Translated to the above case: what is the return to college attendance for those induced to start college but who are unlikely to finish? If it is big, no problem giving everyone loans and building more capacity and encouraging more attendance. But if the “marginal college student” brought in by a decrease in price/increase in credit/increase in capacity is not likely to finish college and earns no extra money for their efforts while there, maybe we don’t want to expand access so much.
* http://economics.mit.edu/files/726
So I basically agree with your analysis, but I think it’s exactly the kind of reframing that was needed to make sense out of this issue.
First off, the Indonesia experiment is more relevant to the question “what’s the overall effect of education” because the population without education was largish and the assignment was balanced with respect to talent/drive/interest etc.
Whereas, in the US, there is already a very large attendance and any policy change would be *at the margin* which you rightfully point out. If you want to find out what the marginal effect is for a few percent of the population that are near the edge of the decision to go to college, you then need to look at the “marginal region” of the student spectrum. Since student attributes are multidimensional, you need a function that takes the multidimensional attributes into a single summary that has good correlation with the decision to go.
The experience of the majority of college students will be irrelevant, you’ll want to use this summary to define students who are going to college who are similar to those who don’t currently go, but with a small amount of incentives might (matched pairs, or at least close marginal abilities of the two populations). Then you could potentially evaluate the two groups and get a marginal effect. This marginal effect would, I expect, be quite different from the average effect over all college students, and be quite quite different from the effect on students in the upper say 10% of HS GPA.
As to Rahul’s question, yes I think that’s what Propensity score is all about. In general if you’re trying to find the effect of something on the “similar people” within two populations, then you need to define some kind of 1 dimensional summary that lets you measure a “distance” or better yet an ordering between various people, and you need to keep this distance small so that you can have a hope of a reasonable analysis. Under those conditions, the linear regression might be meaningful, because it is a kind of Taylor series around a complex multidimensional function, but over only a small region of space (the margin).
To be more specific, you might look at the attributes of students at HS graduation that you think are relevant, and then, since around 30 percent of students don’t go to college, you could select people who’s predictive prior probability for attending based on a nonlinear logistic regression on your attributes is between say 25 and 35 or maybe 20 and 40 percent and then do your analysis on just those students comparing those who decided to attend to those who didn’t.
The “matched pairs” stuff; isn’t that what Propensity Score Matching essentially does?
They are both time varying variables.
Isn’t there a bigger flaw here? That is, even if you were able to construct a model that shows that college attendance, however you define that, improves earnings that doesn’t mean we can necessarily send more people to college and increase incomes for everyone. While college presumably opens doors, the doors have to be there. And if everyone has a college degree there’s still not necessarily more job opportunities requiring a degree and commensurate pay to be filled simply because there’s an excess supply of college graduates.
It reminds me of one of my past employees who went and got his master’s degree. When he didn’t get a big raise the next year he questioned me, believing that his master’s degree made him more valuable to the organization. My reply was that the work he was doing was still the same, so why would I pay extra for it? The causality, if it exists, isn’t that direct.
+1.
It’s like the programs that encourage unmarried people to get married because married people —— (earn more, receive public assistance less, etc).
Life is not really anything like living in a bunch of orthogonal experimental designs.
You are assuming it’s a zero sum game? Maybe, maybe not.
Not necessarily. To the extent that demand exists or would be created to respond to supply then sure having a college degree ought to increase one’s earning potential. However, should there be an excess of supply then shouldn’t it follow that the price paid for that resource will drop too?
I’m suggesting that the policy I suspect they’d seek to further is already concluded and the research is being done in reverse. That is, not to figure out what policy ought to be pushed but to bolster a policy they already have in mind. And from that perspective it makes sense to question whether their conclusion would really follow from any model no matter how well constructed.
Even with an excess supply, from an individual’s viewpoint doesn’t it make sense to still get a degree?
Even if the price paid for the resource drops from its previous value you could still be with a better payoff because you weren’t the one getting paid the precious price. i.e. Bad for the incumbent but still good for the newcomer?
Of course, assuming you were smart enough to do your cost-benefit analysis of getting a degree based on the post hoc price projection.
Yes, I think that’s a reasonable conclusion as an individual actor. However, as a broad policy harming the incumbent to help the newcomer doesn’t seem like sound policy. But that gets back to the zero sum question. :)
Redistribution, a broad policy of most governments, is all about harming the incumbent to help the newcomer! :)
Touché. :)
> think about all the outcomes that flow from this choice
Isn’t this choice of post-secondary education and preferable where – largely made at birth by the parents?
For middle class and upper class kids, sure there is a lot of pressure to attend college. For the majority of college students in the US, well they are adults in community college. But it is true, most students don’t have choice in that they attend college near where they live and/or work.