Andrew looped me into an email thread with folks working on poststratification with partial population information. (He knew I’d be interested, see “poststratification without population level information”.)
They pointed to work by Max Goplerud, Shiro Kuriwaki, Jens Wiederspohn, Adam Conner-Sax, and Philip Greengard, which was recently presented as poster at polmeth: Modeling Complex Contingency Tables.
I wish I got to attend, this is cool ! I’ve got questions about Application 1 (ACS):
- What are the covariates w?
- Suppose you also have survey data in Palo Alto. Would this be added to the training tables ?
- How is the table reconstruction error random ? (I see a density plot.)
The poster gives an answer to Andrew’s question in “Mister P when you don’t have the full poststratification table, you only have margins” and “The continuing challenge of poststratification when we don’t have full joint data on the population”:
I’d recommend first imputing a full poststrat table … But then the question is how to do this.
In Application 2, they ask if a better poststratification table reduces error in estimating the outcome using Multilevel Regression and Poststratification (MRP). Presumably this depends on the distribution of the outcome given the poststratification variables. In “toy example for energy balancing weights” I wrote that raking does well “when Y | X1, X2 is additive”.
I’m excited to read the paper and learn more !
Thanks for sharing! I am fascinated by the idea. The decomposition/approximation in step 2 is pretty smart. But I am not sure about whether it is usually possible to get a good estimate of alpha. To get a good estimate of alpha, we need good data. If we have good data to estimate alpha, it might be better to directly estimate a model of how the margins depend on each other. Please correct me if I am overlooking something here (which is possible, as I still do not work much with survey adjustments).
In some datasets, people seem to report that the true table is so close to the one implied by independence that little needs to be estimated (Cov = 0, thus alpha = 0). But I just took a quick look at the New York City ACS data that Si and Zhou’s Bayes Rake (doi.org/10.1093/jssam/smaa008) use in a case study, with GPT, and assuming independence in that data appears to have a total reconstruction error of 17% — a much larger deviation that all existing methods appear to struggle with.
Re: your intuition about directly estimating a model of how the margins depend on each other instead, I think our alpha is precisely what represents that dependency structure? (The other approach being log linear models)
Thanks, Shiro ! Do Si and Zhou (2021) look at table reconstruction error ? I could only find the downstream outcome error in their paper.
Thank you, Raphael !
Do you mean good covariates (“w” in the poster) for predicting alpha ?
@Shiro Kuriwaki and @shira:
My understanding of the poster is that we want to estimate a contingency table. However, for our target population, we only have the margins. Goplerud et al. decompose the contingency table for population X into a sum of contingency tables. This sum is weighted by unknown values of alpha. Then, Goplerud et al. use data from a different population (or different populations) X’ to estimate the alpha values. Using these values, they calculate the target contingency table using the independence baseline and the alpha values.
Please note that my term ‘populations’ refers to Goplerud et al.’s different geographies.
If I understand correctly, we assume that the data X’ has different margins to the data X, but that the relationship between the margins is the same in both.
That actually resolves my question! Initially, I thought we could just use the contingency table of X’. However, due to the differing margins, the contingency tables of X and X’ differ, too. Sorry for the confusion on my end.
Thanks again, Raphael ! Really appreciate you thinking this thru with me.
Want: y0, contingency table for target population.
Have: m0, its margins. Also {yi, mi, wi} for populations i = 1,…,N.
Compute: ytildei (independence baseline) from mi for each i (including the target 0).
Compute: alphai = B^-1(yi – ytildei) this is the deviation from independence baseline, which isn’t necessarily the same across all i, but can be modeled by covariates, see the next step. (I’m being sloppy here writing B^-1 which won’t exist in general.)
Regress: alphai ~ wi
Predict: alpha0hat from w0
Compute: y0hat = ytilde0 + B*alpha0hat
Authors of this work should definitely correct me where I’m confused !
Thanks for the pointer to Goplerud et al.’s poster. I have a pull request for the Stan User’s Guide,
https://github.com/stan-dev/docs/pull/967,
that adds a chapter on user-defined constrained parameterizations. One of the examples I work through is a constraint on a matrix to match specified margins. The following model will give you a flat prior over matrices that match specified margins. Then, like in other Stan parameterizations, you can add whatever distributional assumptions you want.
data { int M;
int N;
vector[M] row_sums;
row_vector[N] col_sums;
}
transformed data {
if (sum(row_sums) != sum(col_sums)) {
reject("sum of col sums and row sums must match");
}
matrix[M, N] E = row_sums * (col_sums / sum(col_sums)); // E has the correct margins
}
parameters {
sum_to_zero_matrix[M, N] Z; // flat over matrices whose margins sum to 0
}
transformed parameters {
matrix[M, N] Y = E + Z; // flat over matrices with correct margins
}
This makes
Yflat over the matrices matching the margin constraints. There’s nothing constraining the entries ofYto be positive. I don’t see any such constraint written into the poster, but I may have just missed it as I found the notation very confusing. The solutions given in the poster seem to be positive only. I can’t actually figure out the model from the poster, so I’ll ping Philip Greengard to see what’s up. I don’t see an easy way to constrainYto be positive since we can’t just conjoin two constraints in Stan.I think that the positivity constraint was in step 5 of part 4 of the poster where there’s a post-hoc adjustment to make $\hat{y_0} \geq 0$. But I didn’t see it stated how it was done.
The post-hoc adjustment is done by finding the closest vector (table), in the least-squares sense, on the simplex (in the case of counts rather than probabilities: the “simplex” is the surface where the sum of the entries in the table are correct and none are < 0. Is this still called a simplex?) to the vector(table) we get from the model, subject to the constraint that we still satisfy all the known margins.
At least that was the original adjustment. I think among us there have been experiments with different ways of making that adjustment, and it's possible a different one is used in the poster.
Yes Eric below is right that we do a post-hoc correction. In our application 1 with 5*5*4*3 = 300 cells on real data, we find that even in the worst target districts, the cells that initially get modeled with negative estimates are about 6% of the total cells, representing about 1% of the mass.
One advantage of dealing with ratios (not in this poster but as Little and Wu do) is that negative estimates do not arise there.
Thanks again, Shiro ! Good to know that very few cells get modeled to have negative point estimates.
Bob, thank you so much for sharing this ! Very cool. So would this prior on y0 then imply a prior on alpha0 via alpha0 = B^-1(y0 – ytilde0) ? (I am being sloppy here with the “B^-1” which doesn’t exist in general.)
I should first say that this project, with the five authors listed here, arose because Andrew connected us — people from different fields with similar interests who did not know each other — several years ago. So thank you Andrew! We hope to get a paper out soon reflecting the great comments we received at polmeth and elsewhere.
Thank you for sharing this backstory, Shiro ! Andrew has a talent for connecting people.
I just want to echo the thanks to Andrew! And to thank all of you for the helpful feedback.
This all started because I was using a (much less sophisticated) version of this method that I had cobbled together to build post-stratification tables for state-legislative districts. My contribution (the idea for the decomposition via the null-space of the constraints given by the known margins) came about mostly because I did not know about raking! My background (physics) is not in political science or statistics or data science.
I was looking for any way to incorporate more known information (I had Age x Sex x Education x Race for PUMAs but only Age x Sex x Education and Age x Sex x Race for state legislative districts).
I emailed Andrew asking for advice. He introduced me to Philip, then Max, and then Shiro (who brought in Jens).
Thanks, Adam ! Your idea is neat and your instinct to use more known information is the most important part of statistics.