This blog series launched with the proverb “it is the people“. So let’s talk about work by some people I’ve loved talking to over the years: Swen Kuh, Lauren Kennedy, Qixuan Chen, Andrew Gelman, and Aki Vehtari. They’ve been thinking about how to evaluate models for multilevel regression and poststratification (MRP).

The typical machine learning loss looks one individual i at a time: Loss(y_i, yhat_i).

Our post on poststratification focused on estimating a population mean E[Y] with yhat, minimizing Loss(E[Y], y_hat). This differs from Loss(y_i, yhat_i). Indeed, Kuh et al. 2023 caution that these losses may order models differently. In other words, individual-level loss may not be great for choosing models for MRP. Kennedy et al. 2024 propose a way to get at the population-level loss Loss(E[Y], y_hat).

This is one of my less-stellar tent pitches. But I have a neighbor ! This is a post about population-level errors (it is the people !), and our tents average to a good pitch.

Kennedy et al. 2024 point out that the first challenge is that we don’t know E[Y] or E[Y|X]:

However, in practice the population truth (cellwise or at the aggregate level) is not available. One option in this case is to approximate the population truth with the sample observation, .

I assume this means what we wrote as Ehat[Y | X = j, sample] in our post on poststratification, also with clumsy notation. In words, the average Y for folks in the sample with the jth possible combination of covariates X. Aggregating these sample averages across X gives the classical poststratification estimate, which we called yhat_PS. In contrast, the MRP estimate yhat_MRP uses a multilevel model for E[Y|X].

So if I understand, Kennedy et al. 2024 propose to evaluate the MRP estimate by comparing it to the classical poststratification estimate ?

The second challenge is that the same data is used for both fitting the MRP model and assessing its error. As is typical in machine learning, they propose to address this with cross-validation:

K-fold cross validation partitions the sample into K non-overlapping folds. The kth fold is removed to fit the model and score calculated using this fold.

So in their equation (9) above, we would need to compute the average Y for folks in the kth fold with X = j. For this to be nonempty, we need K folks in each cell j. With enough covariates X, this is a tall order. In the paper they propose alternatives.

But I’m still stuck on the first challenge, am I understanding correctly their proposal to get at a population-level error ?