I get confused when the same word is used for 2 different things. (I get confused a lot.)
In Survey Statistics, the word “calibration” is used in 2 different ways. Both are attempts to align estimates from our survey to an external source of data (e.g. census tables):
- Poststratification: Calibrate our estimates of means E[Y] to population data about another variable X.
- Intercept Correction: Calibrate our estimates of regressions E[Y|X] to aggregate data about E[Y].
Using more sources of data makes a statistical method good (a principle Andrew learned from Hal Stern). Kuriwaki et al. 2024 use both flavors of calibration to estimate Republican vote share by race and congressional district. Here is their Figure 4:
Suppose we want to estimate E[Y], the population mean. But we only have Y in the survey sample. For example, suppose Y is voting Republican. We can use the sample mean, Ehat[Y | sample], but what if survey-takers are more or less Republican than the population ?
If we have population data on X, e.g. racial group, we can estimate Republican vote share by racial group E[Y|X] and aggregate according to the known distribution of racial groups, invoking the law of total expectation: E[Y] = E[E[Y|X]]. So if our sample has the wrong distribution of racial groups, at least we fix that with some calibration. Replacing “E” with estimates “Ehat”, poststratification calibrates our estimate of population mean E[Y] to the known distribution of X, using E[Ehat[Y | X, sample]].
For example, suppose 60% of white voters vote Republican, E[Y | X = white] = 60%, and E[Y | X = non-white] = 25%. Suppose P(X = white) = 70%. But our sample has the wrong distribution of racial groups, e.g. P(X = white | sample) = 50%. Without correcting this, our estimate would be: 60% * 50% + 25% * 50% = 42.5%, too low due to our sample containing too few white voters. But with the correct population distribution of X: 60% * 70% + 25% * 30% = 49.5%, roughly the 2016 election results. So instead of assuming E[Y] = E[Y | sample], our new calculation assumes E[Y | white] = E[Y | white, sample] and E[Y | non-white] = E[Y | non-white, sample].
For more, see Lumley 2010 Chapter 7: Poststratification, Raking, and Calibration. For implementation try:
survey::calibrate()
- want: population mean E[Y]
- have: Y,X in sample, X in population
But let’s flip this around. From election results we know E[Y]. We want our estimates of vote by racial group E[Y|X] to be calibrated to those known totals.
- want: E[Y|X]
- have: Y,X in sample, X in population, AND population mean E[Y]
This is a different flavor of calibration ! With poststratification, it was the totals themselves we wanted to estimate E[Y], and the regression E[Y|X] was a step along the way. Now the regression E[Y|X] is what we want to estimate, but the known total E[Y] helps us. So when we estimate E[Y|X], we constrain it to aggregate to the known E[Y]. One way this is done is called the “Logit Shift” in Rosenman et al. 2023 and “Intercept Correction” in Ghitza and Gelman 2020 (Edit: it appears also on p.769 of Ghitza and Gelman 2013 as a “simple adjustment”). This intercept correction isn’t needed if we do a regression of Y on X in the population. As described in regression textbooks, the fitted values Yhat (i.e. Ehat[Y|X]) will aggregate to the mean Y if your model includes an intercept. What breaks that in survey statistics is that we do the regression in the sample, not the population.
For example, suppose our sample includes too many Democratic voters overall because they are more likely to take surveys. So our Yhat for all racial groups are incorrectly shifted towards Democrats. We can correct the intercept of our model by subtracting enough so that the calibrated Yhats now aggregate to the correct lower proportion of Democrats.
Simplifying Kuriwaki et al. 2024 who aim to estimate Republican vote share by racial group E[Y|X] (by congressional district, which we will ignore here):
- They first calibrate (2nd flavor) E[Z|X] to add Z (education) to the auxiliary data X (racial group), using known education aggregates from census tables.
- They then calibrate (1st flavor) estimates of Republican vote share among racial groups E[Y|X] using auxiliary data X,Z. This is done by estimating E[Y|X,Z] and then averaging over the distribution of Z | X.
- They then calibrate (2nd flavor again) E[Y|X] to known aggregate E[Y] from election results.
They use “calibration” to refer to both flavors (bolding my own):
Second, we improve upon existing survey modeling methods by developing two new calibration techniques… multilevel regression and poststratification (MRP) for small area estimation. MRP uses hierarchical modeling and calibration weights …Furthermore, we develop a two way survey calibration, which simultaneously calibrates estimates to both election results by geography and an external survey, instead of only to geography.
Calibration in machine learning
I get even more confused when other fields (e.g. machine learning) also use “calibration”.
Our 2nd flavor of calibration, the “intercept correction”, ensures that E[Y] = E[Yhat]. This is sometimes called “mean calibration“. As noted above, we get this for free if sample and population are the same, and our regression of Y on X includes an intercept. It doesn’t matter if our model is correct at all !
A stricter form of calibration requires E[Y | Yhat] = Yhat. See for example p.30 of PATTERNS, PREDICTIONS, AND ACTIONS: A story about machine learning, by Moritz Hardt and Benjamin Recht. Or Jessica’s posts, e.g. here. This holds if our model for E[Y|X] is correct, which is much harder than just throwing in an intercept term to get mean calibration.
How could you know how people voted by demographic in a US election? Its well known that you can’t trust people to tell you if they voted or how they voted, that is one reason why even the USA has a secret ballot! And collecting that much information from people leaving the polls sounds very time consuming which would drive up cost and drive down response rate and therefore demographic representativeness. Meanwhile figure 4 says that the proportion of vote per candidate is “estimated” when I would expect that election authorities even in the USA publish an exact count of votes per candidate in each district. I’m fascinated by where the numbers come from before you do statistics on them.
Great question, Sean ! I agree that we need to know where the numbers come from before doing statistics.
Kuriwaki et al. 2024 aim to estimate the Republican vote share in each racial group in each congressional district, E[Y | X, CD] (my simplification in the post ignores CD). Figure 4 shows these estimates. They use the 2016 Cooperative Congressional Election Study (CCES), which as you and the authors note, has problems with representativeness. So they mitigate this by calibrating to known totals from the census and election results (which aren’t released by racial group).
Shira:
I appreciate your link to our 2020 paper. The method of intercept correction is described on p. 769 in our 2013 paper on deep interactions with MRP. We first used it when doing the MRP analyses for Red State Blue State a few years earlier. It was a natural thing to do–I remember writing the code!–But I don’t think I wrote it up until that paper with Yair that came out in 2013.
Thanks, Andrew ! I forgot it was also in the 2013 paper, I’ll add that to the post.
This brings up a question actually:
I think the logit-shifts differ between the 2013 paper (which predicts cells) and 2020 paper (which predicts individuals) because sums can’t come in and out of logits and inverse-logits without consequence (unless we have nothing in the model that varies within the counties/states you’re calibrating to) ?
Shira:
You’d have to ask Yair about the 2020 paper because he did all the analysis for that one. For the 2013 paper (and other MRP analyses I did around that time and earlier, such as for Red State Blue State), I wrote a little program to do an iterative adjustment which was necessary because of the nonlinearity of the logit. The actual code is in the online supplement to our 2017 paper, which is a clean replication of the models in the 2013 paper.
The logit shift/intercept correction is an elegant way to integrate micro data, which allows for rich inferences about joint distributions, and macro data, which is often “ground truth.” In a paper currently working its way through the review process, we propose an extension to the “logit shift” idea to cases where we observe ground-truth data for some outcomes but not others. This is the case when we have a survey measuring vote choice (ground truth known) and policy attitudes (unknown ground truth). The idea is we fit a hierarchical model for multiple outcomes, where the geography-specific random intercepts can be correlated, then we use this correlation to predict what the intercept correction should be for outcomes w/o any ground truth data. The idea build on Andrew’s common advice to estimate everything jointly in a hierarchical model.
Another interesting idea on integrating micro and macro data was proposed in an old paper by Imbens and Lancaster. They work in a GMM framework, where there are two sets of moment conditions. First, the micro-level moment conditions derived from an individual-level model (e.g. the score function for a logit model predicting vote choice from demographics). Second, the macro-level moment conditions equate the model-implied aggregates and the known ground-truth data (e.g., the difference between an MRP-implied vote share in Pennsylvania to the actual results in Pennsylvania). This allows you to estimate the parameters all at once, rather than the two-step approach used in the intercept correction.
Thanks, Will ! Neat that your paper estimates outcomes jointly in this way. I need to think thru a simple case to try to understand this. Suppose we have 2 outcomes, where only the first has a ground-truth target. As I understand it, your approach fits a joint model p(y_1, y_2), shifts the marginal distribution p(y_1) using its target, then corrects the marginal p(y_2) accordingly using the correlation between outcomes. How would this compare to fitting models in a chain like p(y_2 | y_1) * p(y_1) where I think the shift to y_2 implied by the shift to y_1 comes “for free” … ?
Thanks also for the link to the older paper, I love seeing these connections across different disciplines !
Suppose we observe geographic ground truth marginals on y_1 but not y_2 (e.g. election results and support for a policy) and denote ground truth observations bar(y_1). Here are two interpretations of your suggestion:
1. Use geographic predictors bar(y_1) in the regression step of MRP. This is standard practice and I’d recommend it. But even if you regress sample y_1 ~ bar(y_1) — e.g. regress sample vote choice on actual election outcomes — you may still get the wrong predictions if the sample is non-representative. In other words regression itself does not ensure that E(p(y_1 | bar(y_1)) = bar(y_1). You may still need an intercept shift. The method we suggest propagates that through to y_2 as well.
2. Include individual-level y_1 as a predictor for y_2 in the regression step, so the regression model is y_2 ~ y_1 + demographics. The question then is how to do the poststratification step, which requires the joint distribution of all the predictors (demographics and y_1). We usually know the joint distribution of the demographics from Census data, and we some marginals of y_1. But generally we do not know the joint distribution of the demographics AND y_1.
We could estimate/impute it, e.g. by assuming independence p(demographics, y_1) = p(demographics) p(y_1). This is the solution proposed here. Better yet, we could infer p(y_1 | demographics) p(demographics) using a prior calibrated MRP step (upon writing this out — this approach seems closest to what you meant). This makes a lot of sense to me and is pretty similar to what we do. The main difference in our approach is that we use geographic-level correlation via correlated random effects, so your suggestion may even be an improvement. If you had multiple variables with ground-truth data (e.g. multiple elections) you’d have to chain together models p(y_1 | y_2, y_3) p(y_2 | y_3) p(y_3), so that might get a bit cumbersome to implement.
The next revision will include comparisons to similar procedures, so I’ll include this suggestion there. Thanks for the feedback.
Thank you again, Will !
Apologies for the brief reply, I’m on my phone.
I meant something more like your second interpretation. We want a model for p(y_1, y_2 | demographics X) that is shifted to the correct y_1 marginal target. So we could first fit p(y_1 | X) and perform its intercept shift. Then fit p(y_2 | y_1, X). Then multiply these two distributions to get the joint we want.
You instead model the joint, and shift each marginal, where the target for y_2 is inferred from the target for y_1. The method I’m suggesting avoids needing to calculate an inferred target for y_2 I think ?
Additional questions about your paper:
1. Model (4)-(5) assumes that correlation across issues is driven by group-g-level correlations, not individual-i-level correlations ?
2. Also in Model (4)-(5), do X or Z include intercepts ? The alphas are zero-centered so I was curious.
3. Page 15 “Consider the case of a single… ” I think maybe there’s a typo where delta^1 and delta^2 get flipped ?
I’m assuming this is the approach that Catalist uses for the What Happened report, right?
Are there any concerns that the issues we typically have in survey data, in particular non-ignorable nonresponse, is perpetuated to these subgroup estimates? Would the logit shift/intercept correction adjust for that?
Yes, my understanding is Catalist basically fits a big hierarchical model to survey responses linked to the voter file, and then calibrates to match any other administrative sources they can pull together. The other complication is the need to impute many relevant demographics in certain states (e.g., race outside of the South, age in Wisconsin, etc.)
Our logit shift paper linked above (Rosenman et al 2023) shows that logit shift is a (highly accurate) approximation to a Bayesian update of the underlying model, where you condition on the known totals. I.e., your model fit to survey data is conditional on the survey data, and then rather than compute the additional conditioning on the known Y totals, you approximate that conditioning with a logit shift. I suspect you could write down a flavor of nonignorable nonresponse which could be fully adjusted away using this approach. But in general the nonresponse patterns are going to be much more problematic, and the logit shift will probably only ameliorate the error rather than eliminate it. One area I’m interested in exploring further is whether combining this kind of survey calibration with ecological inference techniques that aren’t based on surveys lets one sidestep nonignorable nonresponse while also avoiding the identification challenges that EI faces.
Thank you for the thoughtful response, Cory!
Right, I don’t think completely eliminating nonresponse error is really realistic, but just reducing it can be very promising.
Oh, interesting, I was just thinking the other day about this possibility of using survey calibration with ecological inference. I’m not sure what it would look like, but if possible it would be a great approach to reduce the issues on both EI and survey data.
Great question, Raphael: “Would the logit shift/intercept correction adjust for [non-ignorable nonresponse]?”
And thank you, Cory, I’m interested in this: “I suspect you could write down a flavor of nonignorable nonresponse which could be fully adjusted away using this approach.”
I might call this flavor of nonignorable nonresponse “ignorability-up-to-shifts” ?
Say we want subgroup estimates E[Y|X].
We can estimate E[Y|X, sample].
We need ignorability to say that E[Y|X, sample] = E[Y|X].
In other words, conditional on X, our sample looks like the population.
But what if we also know aggregate E[Y] so we can calculate a shift such that E[Y] = E[E[Y|X, sample] + shift].
(Here I’m writing shift on the probability scale not the logit scale just for simplicity.)
Then we need “ignorability-up-to-shifts” to say that E[Y|X, sample] + shift = E[Y|X].
In other words, our sample gets the X-Y relationship correct, but we can fix a national-level lack of ignorability with the shift. (Being really informal here….)
Is this “ignorability-up-to-shifts” formalized anywhere ?