In June we discussed 2 flavors of calibration:
- Poststratification: Calibrate our estimates of means E[Y] to population data about another variable X.
- Intercept Correction: Calibrate our estimates of regressions E[Y|X] to aggregate data about E[Y].
Let’s focus on the 2nd. This is called the “Logit Shift” in Rosenman et al. 2023, “Intercept Correction” in Ghitza and Gelman 2020, “simple adjustment” on p.769 of Ghitza and Gelman 2013, and “calibration to election results” in Kuriwaki et al. 2024.
Rosenman et al. 2023 have a footnote:
Throughout this note, we focus on a binary outcome for simplicity. The same logic applies in cases of multinomial outcomes.
But something funny can happen with at least J = 3 outcomes !
Start with an N by 3 table of initial predictions for each person’s predicted probability of being White, Black, or Other. Suppose we know the correct aggregate data for this population should be P[White] = 0.4, P[Black] = 0.2, P[Other] = 0.4. So we want the column marginals to satisfy this. We also need to rows to continue to sum to 1.
As Evan Rosenman points out, we can update our predictions using Iterative Proportional Fitting (IPF), also known as “raking” in survey statistics:
Here the “u” are the known aggregates and “v” are 1.
Homework 1: take exp(log())s and combine steps to get that our updates look like:
updated m_ij = exp(log(m_ij) + shift_j) / sum_k exp(log(m_ik) + shift_k)
Homework 2: now show how if J = 2, this simplifies to:
updated m_i1 = logit^-1(logit(m_i1) + shift_1 - shift_2)
For J = 2, this is monotone, so the updated probabilities preserve rank order. In other words, if person A has a higher probability of being white than person B, this stays true after the shifts. However, this simplification cannot be done for J >= 3. And the updated probabilities depend not only on the shift for that race, but the denominators that sum across all races. So this can flip the rank order.
Here’s a little example:
import numpy as np
np.random.seed(123)
alpha = [10, 2, 5]
init_predictions = np.random.dirichlet(alpha, size=10000)
print("initial aggregates:", init_predictions.mean(0))
# One IPF iteration
targets = np.array([0.4, 0.2, 0.4])
preds = init_predictions * targets / init_predictions.mean(0, keepdims=True)
preds /= preds.sum(1, keepdims=True)
# Show two rows before/after
for r in [0, 23]:
print("initial predictions for r =", r, ":", init_predictions[r])
print("adjusted predictions for r =", r, ":", preds[r])
Homework 3: are you bothered by this ? why or why not ?
Shira,
Let me continue to say that () and [] are the same thing, and you can write E(y|x). I mean, sure, E[y|x] is fine too; I just don’t like the implication that expectations need square brackets.
:)
I was copying text from the previous post, so the brackets were still there, no implications beyond that !
As a non-mathematician who regularly reads quantitative papers in survey sampling, econometrics and political science, I have to say that I can only remember ever seeing the square bracket convention for expectations, and had assumed it was standard. I have no doubt that Andrew is right with regards to fundamental notation, but it has seemed to this reader at least that the style is extremely prevalent.
I can’t decide which one is higher:
P[ shira uses square brackets next time ]
p( Andrew replies “it’s fine but I don’t like it” | shira uses square brackets next time )
As long as the parentheses match, it’s cool.
good question, Carlos ! :)
so far the maximum likelihood estimator of both is 1, I think ?
canonically, it can reduce discrimination. i can see a few more issues. this is plausibly just fine, right (given bad choices): https://proceedings.mlr.press/v70/guo17a/guo17a.pdf
Thank you, Gaurav !
“it” = calibration ? using which method ?
“discrimination” = AURUC ? does Guo et al. 2017 discuss this ? (where ?)
One way to break this down = decide on the constraints: (a) conserve across-person order for a given class (AUROC/precision@k style) or (b) per-person top-1 class (argmax). For (a), you must keep each class’s score a monotone transform of the original; for (b), you must keep all class scores for a person transformed by the same monotone map.
What does this mean ?
In IPF (in my post),
shift_jdepends on class j.Hey, here’s a clean version: https://www.gojiberries.io/rank-preserving-calibration-for-multiclass-probabilities/
Here’s a v.1 of rank preserving calibration: https://github.com/finite-sample/rank_preserving_calibration
HUGE thank you, Gaurav !!!
for sure! i think it is a fun problem. i am surprised people haven’t worked on this.
please feel free to DM me or open an issue.
Andrew, if I might ask here, there’s a very important, just out Harvard epidemiological study on the effects of diets, and specific foods, on “healthy aging”. One part of the study runs a logistic regression of healthy aging on a large number of specific food groups, like fruits, red meat, sweets, etc., and there are many control variables, like age, sex, smoking, and physical activity.
My question is on multicolinearity with this large number of variables, which they appear to not address, although there is an enormous amount of data, 105,000 participants followed for 30 years. Fruit came out #1 for healthy aging, but how much could this be attributed to people who eat more fruit for breakfast eat less bacon, eggs, white flour pancakes and syrup? Starchy vegetables came out worse than sweets! But how much of this can be attributed to people tending to eat them with red meat, sausages, etc., and instead of broccoli and salads? Maybe people who eat desserts tend to eat them as a reward or compensation for eating their non-starchy veggies?
Could it be that in of themselves non-starchy vegetables are better than fruits for healthy aging? Likewise, are starchy vegetables really less healthy than sweets. Those vegetables at least have some fiber and phytochemicals, and there is compelling evidence for the healthiness of those.
This study may have a lot of potential to mislead, and on an extremely important subject, so any thoughts would be great. I’ll also leave a link to an interview with one of the authors.
Thank you for your outstanding work.
https://www.nature.com/articles/s41591-025-03570-5
https://www.youtube.com/watch?v=LPXWCIFDkgM&t=2547s
I guess I should add that when two explanatory variables are correlated, like, say, income and education, you hope that you have enough data points where someone has low education, but high income; and high education but low income, so the regression can tease out whether it is income, or education, or both, that are truly having a major effect on, say, Mercedes purchases. So, in this study you would hope that in this enormous data set you have enough people who, for example, eat a lot of fruit, but also eat a lot of bacon and pancakes to tease out whether fruit does so well because of its true health benefits, or largely because people who eat it, eat less harmful foods, like bacon and pancakes.
Still, I’m wondering if we can expect that in this study, with so many variables, so seemingly correlated, even though there is an enormous amount of data. In the video I linked to, the interviewer, Chris MacAskill, has a PhD in geophysics, and he is puzzled by some of the results, so any thoughts you have on this, or suggestions, would be great.