Torleif Halkjelsvik at the Norwegian Institute of Public Health writes:
Norway has very good register data (education/income/health/drugs/welfare/etc.) but it is difficult to obtain complete tables at the population level. It is however easy to get independent tables from different registries (e.g., age by gender by education as one data source and gender by age by welfare benefits as another). What if I first run a multilevel model to regularize predictions for a vast set of variables, but in the second step, instead of a full table, use a raking approach based on several independent post-stratification tables? Would that be a valid approach? And have you seen examples of this?
My reply: I think the right way to frame this is as a poststratification problem where you don’t have the full poststratification table, you only have some margins. The raking idea you propose could work, but to me it seems awkward in that it’s mixing different parts of the problem together. Instead I’d recommend first imputing a full poststrat table and then using this to do your poststratification. But then the question is how to do this. One approach is iterative proportional fitting (Deming and Stephan, 1940). I don’t know any clean examples of this sort of thing in the recent literature, but there might be something out there.
Halkjelsvik responded:
It is an interesting idea to impute a full poststrat table, but I wonder whether it is actually better than directly calculating weights using the proportions in the data itself. Cells that should be empty in the population (e.g., women, 80-90 years old, high education, sativa spray prescription) may not be empty in the imputed table when using iterative proportional fitting (IPF), and these “extreme” cells may have quite high or low predicted values. By using the data itself, such cells will be empty, and they will not “steal” any of the marginal proportions when using IPF. This is of course a problem in itself if the data is limited (if there are empty cells in the data that are not empty in the population).
Me: If you have information that certain cells are empty or nearly so, that’s information that you should include in the poststrat table. I think the IPF approach will be similar to the weighting; it is just more model-based. So if you think the IPF will give some wrong answers, that suggests you have additional information. I recommend you try to write down all the additional information you have and use all of it in constructing the poststratification table. This should allow you to do better than with any procedure that does not use this info.
Halkjelsvik:
After playing with a few scenarios (on a piece of paper, no simulation) I see that my suggested raking/weighting approach (which also would involve iterative proportional fitting) directly on the sample data is not a good idea in contexts where MRP is most relevant. That is, if the sample cell sizes are small and regularization matters, then the subgroups of interest (e.g. geographical regions) will likely have too little data on rare demographic combinations. The approach you suggested (full population table imputation based on margins) appears more reasonable, and the addition of “extra information” is obviously a good idea. But how about a hybrid: Instead of manually accounting for “extra information” (e.g., non-existing demographic combinations) this extra information can be derived directly from the proportions of the sample itself (across subgroups of interest) and can be used as “seed” values (i.e., before accounting for margins at the local level). Using information from the sample to create the initial (seed) values for the IPF may be a good way to avoid imputing positive values in cells that are structural zeros, given that the sample is sufficiently large to avoid too many “sample zeros” that are not true “structural zeros”.
So the following could be an approach for my problem?
1. Obtain regularized predictions from sample.
2. Produce full postrat seed table directly from “global” cell values in the sample (or from other available “global” data, e.g. if available only at national level). That is, regions start with identical seed structures.
3. Adjust the poststrat table by iterative proportional fitting based on local margins (but I have read that there may be convergence problems when there are many zeros in seed cells).
Me: I’m not sure! I really want to have a fully worked-out example, a case study of MRP where the population joint distribution (the poststratification table) is not known and it needs to be estimated from data. We’re always so sloppy in those settings. I’d like to do it with a full Bayesian model in Stan and then compare various approximations.
yes – we looked into this and applied Bayesian networks to the separate registries. By conditioning on common variables we were able to integrate the networks.
See 1https://onlinelibrary.wiley.com/doi/abs/10.1002/qre.1859
and https://www.sciencedirect.com/science/article/abs/pii/S0957417417308667?via%3Dihub
I am confused by this discussion because in my mind IPF is raking. In other words they are two terms that describe the same thing. Perhaps it is the context in which I use IPF to generate weights so the sample reflects the known marginal proportions. Is there some difference between the two terms?
I also think of them as the same thing. Can anyone clarify?
I Agree
My experience is if you’re interested in the area-level estimates at the end of post-stratification, modeling the full-joint adds a lot of compute and does not do much for the accuracy of the final estimates (on US and UK election results, for example).
For me the go-to remains simple MrsP (Leemann & Wasserfallen, 2017 – “Extending the Use and Prediction Precision of Subnational Public Opinion Estimation’’ ) provided you have the area-level marginals for the variables you want to extend the stratification frame by. The extended MrsP approach presented in the paper is very similar to what Halkjelsvik is proposing above.
To my understanding, Raking or IPF has been used by Lauderdale et al. (2020 – “Model-based pre-election polling for national and sub-national outcomes in the US and UK” ) where they show it works well, but no accounting for uncertainty is included in their implementation. In my experience results are a bit mixed and is unclear how much the raking is adding to the simple extension by area-level marginals.
I tried some multiple imputation approaches in my thesis (2020, “Measuring and forecasting public opinion with non-representative samples” ) but they are very, very sensitive to the training data, so if you use `outdated’ surveys to train an imputation model, you’ll get an outdated joint in the post-stratification frame, which is fine for some applications but deadly to others (e.g. pre-election opinion polling).
`Multi-stage MrP’ has been tried by the people at Stack Data Strategy (link to the paper in this article: https://www.politico.eu/article/voter-distrust-polling-brexit-trump/) , as well as others, to impute `social trust’ onto an existing stratification frame. The basic idea is to train a model of trust, make cell-level predictions to impute trust for each cell, and then use the resulting frame to model the dependent variable you actually care about.
No systematic evaluation of the different approaches exists as far as I’m aware.
In my experiments I’ve found this sort of post-stratification model is important if you’re interested in inference at `deeper’ levels of analysis than areas, say area-by-age, or area-by-age-by-past.vote, etc.
Let’s say we have three finite covariates of sizes A, B, and C. We’d like to have the full population proportion in each of the A x B x C cells, which we can model as a simplex theta (non-negative vector that sums to one [edited]) of size A x B x C. If we only get noisy measurements of population sizes for marginals A x B, B x C, and C in three separate data sets, then we can just generate likelihoods for each of the three data sets by marginalizing theta. And then we can do poststratification using theta itself (and overall population size if absolute numbers matter). Here’s a Stan program to impute theta.
data { int A; int B; int C; array[A, B] int yAB; array[B, C] int yBC; array[C] int yC; } parameters { simplex[A * B * C] theta_flat; } transformed parameters { array[A, B, C] real theta; { int pos = 1; for (a in 1:A) for (b in 1:B) for (c in 1:C) { theta[a, b, c] = theta_flat[pos]; pos += 1; } } } model { array[A, B] real thetaAB; for (a in 1:A) for (b in 1:B) thetaAB[a, b] = sum(to_array_1d(theta[a, b, ])); array[B, C] real thetaBC; for (b in 1:B) for (c in 1:C) thetaBC[b, c] = sum(to_array_1d(theta[ , b, c])); array[C] real thetaC; for (c in 1:C) thetaC[c] = sum(to_array_1d(theta[ , , c])); // prior uniform over simplex theta_flat, but could be informative // likelihood multinomial with marginals of theta to_array_1d(yAB) ~ multinomial(to_vector(to_array_1d(thetaAB))); to_array_1d(yBC) ~ multinomial(to_vector(to_array_1d(thetaBC))); yC ~ multinomial(to_vector(thetaC)); }Now one could run this program and get a point estimate for theta and go forward with poststratification. But it would be better to fit this model jointly with the poststratification model because then the uncertainty in estimating theta will be propagated through the poststratification uncertainty. That’s similar to what Andrew and I did in our Covid PCR testing sensitivity and specficity model.
If you only get estimated proportions rather than estimated counts, then you can run a Dirichlet regresion using the same general idea using a mean/total concentration parameterization of the Dirichlet. You can even mix the two this way.
P.S. If you want to try it, here’s Python simulation code using CmdStanPy:
import cmdstanpy as csp import numpy as np import scipy as sp A = 3 B = 2 C = 3 ABC = A * B * C alpha = np.full(ABC, 0.8) theta_flat = sp.stats.dirichlet.rvs(alpha) theta = theta_flat.reshape(A, B, C) thetaAB = np.sum(theta, axis = 2) thetaBC = np.sum(theta, axis = 0) thetaC = np.sum(theta, axis = (0, 1)) NAB = 1000 NBC = 500 NC = 2000 yAB = np.random.multinomial(NAB, thetaAB.reshape(A * B)).reshape(A, B) yBC = np.random.multinomial(NBC, thetaBC.reshape(B * C)).reshape(B, C) yC = np.random.multinomial(NC, thetaC) model = csp.CmdStanModel(stan_file='population-cell-model.stan') data_dict = {'A':A, 'B':B, 'C':C, 'yAB':yAB, 'yBC':yBC, 'yC':yC} fit = model.sample(data = data_dict)I wrote everything as I did with all the conversions to make sure everything lined up with row-major ordering. But it’s inefficient to do this.
We really have to add more general array/vector/matrix reshaping to Stan along the lines of what you can do in numpy. The numpy sum operation is particularly handy for these marginalizations and the reshape syntax is much more convenient than Stan’s approach.
The model could be programmed more efficienbtly by keeping everything flat, but that would be harder to read and to modify.
*Nonnegative vector that sums to ONE
typo. A nonnegative vector that sums to zero is just a bunch of zeros.
The non-negative vector that sums to zero is knows as the incredibly simple simplex. Bluenoses among you call it the degenerate simplex. Most optimization methods have little problem finding a stable maximand. Oddly, the only ones that don’t are those that require a derivative. So people generally use a brute force search, which paradoxically requires very little brute force.
Happy New Year, everybody.
Halkjelsvik says:
“So the following could be an approach for my problem?
1. Obtain regularized predictions from sample.
2. Produce full postrat seed table directly from “global” cell values in the sample (or from other available “global” data, e.g. if available only at national level). That is, regions start with identical seed structures.
3. Adjust the poststrat table by iterative proportional fitting based on local margins (but I have read that there may be convergence problems when there are many zeros in seed cells).”
Does this mean adding a geographical region dimension to the poststratification table (instead of dealing with different geographical regions separately) and using marginals that sum over geographical region? If so that sounds like a good idea and sounds like it might be equivalent to raking on the large table.
If not, it sounds like it might be a good idea anyway. Andrew and I do something similar in this paper — https://arxiv.org/abs/2304.09126 — where our goal is to impute the entries of a surname by county by ethnicity table for registered voters in a state. We know some marginals for the full population and some marginals for registered voters. So we start with the prediction obtained by using only the marginals from the full population and then rake to marginals for registered voters.
This sounds similar to what Halkjelsvik might be proposing in that we start with some reasonable, but crude, guess for the full table and then rake to known margins. There’s a lot of theory on these estimators. For example, the resulting solution minimizes the KL divergence to the starting point of raking among all tables that satisfy the known margins. If nothing else, the theory could be useful in justifying the solution, at least to people like Andrew who prefer models to methods.