Last year we discussed 2 flavors of “calibration” in survey statistics. That post also described a 3rd “calibration” from machine learning. Andrew’s workflow article mentions “calibration” in 2 more ways. (Andrew just blogged about workflow.)
Survey statisticians (like me) often work with folks in machine learning. And we all need workflow. So let’s try to understand these 5 “calibrations” !
1. Poststratification: Calibrate our estimates of means E(Y) to population data about another variable X. Can be done via “MRP” or calibrated weights.
2. Intercept Correction: Calibrate our estimates of regressions E(Y|X) to aggregate data about E(Y). So E(Y) = E(Yhat), where Yhat is our estimate of E(Y|X). Also called the “logit shift”, or “mean calibration”, or “unconditional coverage”.
Kuriwaki et al. 2024 use both these flavors of survey calibration to estimate Republican vote share by race and congressional district:
3. Conditional Mean Calibration: E(Y | Yhat) = Yhat, where Y is out-of-sample from the data used to get Yhat. Also called “conditional coverage”, or “perfect calibration”, or “Bayesian Calibration” (p.128 of BDA3), or just “calibration” in Hardt et al. p.30 and in Jessica’s posts. How is this checked ? Jessica mentions binning Yhat and calculating expected calibration error (ECE). Have you seen folks doing a regression of Y on Yhat ?
Guo et al. 2017 contrast classification error with ECE:
4. Distribution Calibration: Y | X ~ phat(y | X) where phat is the posterior predictive distribution to estimate p(y | X). See the workflow article (and Gabry et al. 2019), which discusses checking that Fhat(Y) is uniform, where Fhat is our estimate of the cumulative distribution function F:
If we have distribution calibration, intervals (e.g. 90% uncertainty intervals) based on the posterior predictive distribution will have correct coverage. Also, we’d have the correct means Yhat = E(Y | X), so we’d have conditional mean calibration. In other words, distribution calibration is a super strong wish !
5. Simulation-based calibration (SBC): a way to validate Bayesian computation of the posterior. See the workflow article.