Teryn Mattox writes:
I was reminded by your recent post on propensity score matching of a nagging doubt I have about this methodology. It seems as though propensity score matching actually exacerbates selection bias. I do research on childhood interventions, and am considering using a matched design to compare the outcomes of children that did and did not receive the “treatment” of high quality preschool. But…if there are two families that are very similar in every observable way, but the parents still elected to put their kids in different preschool programs, doesn’t that mean that there must be that much more difference in unobservable characteristics between the two of them? For example, we have very few highly educated families putting their kids in bad quality care – those few families must be doing something very different, no? So won’t we dramatically overestimate the treatment effect, even more than if we just did a simple OLS regression?
My response: First off, this isn’t anything specific to propensity scores, or even to matching; it also arises in regression or in any other situation where you’re controlling for pre-treatment variables. I’ll give two quick answers. First, it is generally recommended that you control for as many pretreatment variables as you can. In their classic article on matching for causal inference, Dehejia and Wahba emphasize the importance of controlling for enough variables (and they discuss what “enough” can mean in practice). Second, the hope when controlling for things (whether by regression modeling, matching, or other methods) is to reduce the selection bias that you’re referring to. I imagine there’ve been quite a few papers in statistics and econometrics discussing the conditions under which controlling for a pre-treatment variable reduces bias in estimated treatment effects.