Responding to a comment from Thomas Lumley (who asked why MRP estimates often seem to appear without any standard errors), I wrote:

In political science, MRP always seems accompanied by uncertainty estimates. However, when lots of things are being displayed at once, it’s not always easy to show uncertainty, and in many cases I simply let variation stand in for uncertainty. Thus I’ll display colorful maps of U.S. states with the understanding that the variation between states and demographic groups gives some sense of uncertainty as well. This isn’t quite right, of course, and with dynamic graphics it would make sense to have some default uncertainty visualizations as well.

But one thing I have emphasized, ever since my first MRP paper with Tom Little in 1997, is that this work unifies the design-based and model-based approaches to survey inference, in that we use modeling and poststratification to adjust for variables that are relevant to design and nonresponse. We discuss this a bit in BDA (chapter 8 of the most recent edition) as well. So there’s certainly no reason not to display uncertainty (beyond the challenges of visualization).

I’ve recently been told that things are different in epidemiology, that there’s a fairly long tradition in that field of researchers fitting Bayesian models to survey data and not being concerned about design at all! Perhaps that relates to the history of the field. Survey data, and survey adjustment, have been central to political science for close to a century, and we’ve been concerned all this time with non-representativeness. In contrast, epidemiologists are often aiming for causality and are more concerned about matching treatment to control group, than about matching sample to population. Ultimately there’s no good reason for this—even in an experimental context we should ultimately care about the population (and, statistically, this will make a difference if there are important treatment interactions) but it makes sense that the two fields will have different histories, to the extent that a Bayesian researcher in epidemiology might find it a revelation that Bayesian methods (via MRP) can adjust for survey bias, while this is commonplace to a political scientist as it’s been done in that field for nearly 20 years.

I wonder if another part of the story is that Bugs really caught on in epi (which makes sense given who was developing it), and Bugs was set up in a traditionally-Bayesian way of data + model -> inference about parameters, without the additional step required in MRP of mapping back to the population.

Also, causal inference researchers have tended to be pretty cavalier about the sampling aspect of their data. Rubin, for example, talked a lot about random or nonrandom assignment of the treatment but not much about representativeness of the sample, and I think that attitude was typical for statisticians for many years—at least, when they weren’t working in survey research. In my own work in poli sci, I was always acutely aware that survey adjustment mattered (for example, see figure 1a here), and I didn’t want to be one of those Bayesians who parachute in from the outside and ignore the collective wisdom of the field. In retrospect, this caution has served me well, because recently when some sample-survey dinosaurs went around attacking model-based data-collection and adjustment, I was able to decisively shoot them down by pointing out that we’re all ultimately playing the same game.

I don’t go with the traditional “Valencia” attitude that Bayesian approach is a competitor to classical statistics; rather, I see Bayes as an enhancement (which I’m pretty sure is your view too), and it’s an important selling point that we don’t discard the collective wisdom of a scientific field; rather, we effectively include that wisdom in our models.

You could do MRP fully within BUGS by exploiting their deterministic nodes to do the post-stratification in the same way you could use the generated quantities block in Stan to do the same thing.

I completely agree on not just “parachuting in.” Or at least keeping an open mind once you’ve jetted and rental car-ed in (I’m about to land at an applications conference).

I had to look up [Valencia bayes] and it looks like it was a series of conferences. I see the Bayesian approach as a competitor to classical stats at least in terms of focus in college curricula and in applied work. What do you teach someone in their one semester of college stats? What do you put on qualifying exams in stats departments? What techniques do you use for your next analysis of surveys?

There’s certainly no reason to throw away all the good work that came before the “Bayesian revolution.” For example, in the new Stan manual chapter on marginalizing out discrete parameters, I point out that you can go back in the literature and find most of the derivations you need in papers doing maximum likelihood via optimization, since MLE often involved marginalizing out discrete parameters. It’s sort of like putting a gasoline powered engine on your buggy.

Poststratification with deterministic nodes assumes you know the fraction of the population represented by each group exactly. You could just as easily allow that fraction to vary with some kind of distribution via a parameter, to account for uncertainty in the weights.

Just to provide one counter-example: Jeff Lax, Justin Phillips, Mike Malecki and I have a paper where we do calculate uncertainty in MRP estimates of support for Supreme Court nominees: http://www.princeton.edu/~jkastell/Noms2/klmp_paper_june_2014.pdf

I hate to embarrass myself, but what the hell is MRP?

Bill:

It’s Mister P.

Thanks, Andrew. I thought it was some sort of obscure acronym that could be looked up but only got stuff like Material Requirement Planning that made no sense whatsoever in context.

Whew thanks. Now I know.

About Rubin, I think one part of the story is that if you are into causal inference, and if you believe that the variables used in the sampling design can be considered as pre-treatment predictors, then you just need to condition on these variables to correct the bias of your causal estimates. And since the sampling weights can be considered as a crude summary of the design variables (Rubin has an 80’s paper on this), you just need to condition on the weights. Of course the relationship is probably non-linear and complicated, but you can just include the weights in your matching scheme to resolve the modeling problem. If I’m not mistaken, that’s exactly what Rubin did in his tobacco effect analysis.

In a descriptive perspective, things are not so easy: conditioning on the weights obscures the meaning of the regression coefficients. On the other hand, not correcting for nonresponse and data collection also obscures their meaning (we don’t know exactly what population our sample represents). Hence the benefit of the additional step of post-stratification to get accurate descriptive quantities.