Alan Zaslavsky and his course drew me into survey statistics. We focused on a simple question at the heart of statistics: how can we make statements about a population, given a sample? We need to represent everyone, because we all matter and are all unique. But not everyone can be in our sample. As Andrew says, this is what makes it so hard.
Why look at survey data ? As my teammate David Shor says here, super politically engaged people are overrepresented in the media. Survey data provides a counterbalance to that, aiming to represent everyone.
So let’s jump in with a new blog series !
Who is this blog series for ? Mainly folks who already know some survey statistics and want to learn more together. Folks who have heard of, could define, or even use these concepts, but have questions about them:
- sampling designs like stratification (Sharon Lohr’s Sampling book)
- nonrandom (“convenience”) samples (Meng 2018)
- survey weights (Sharon Lohr’s Sampling book, Gelman 2007)
- design effects (Sharon Lohr’s Sampling book)
- poststratification (Holt & Smith 1979) and MRP
- design-based vs model-based inference (Little 2004)
I’ll attempt to introduce concepts along the way. But there will be gaps, which we should chat about in the comments. And I will ask you questions, so please participate. It is people that make make survey statistics (and anything) great.
p.s. The title comes from It is The People: A Pacific Crest Trail Film by Elina Osborne. She quotes the Māori proverb “What is the most important thing ? It is people, it is people, it is people”. I just did a long “solo” hike of the Virginia section of the Appalachian Trail, where I learned so much from hikers and trail town residents. As a survey statistician, I get to keep listening and learning.
Is that bear representative?
So glad you asked. The polar bear is at least representative of himself, but unclear beyond that. He is both smaller and more mighty than any other polar bears recorded.
For more on bears and survey statistics see: https://statmodeling.stat.columbia.edu/2024/10/04/basus-bears/
The link for Meng 2018 is not available :(
I think it’s meant to be this one:
https://statistics.fas.harvard.edu/sites/g/files/omnuum10116/files/statistics-2/files/statistical_paradises_and_paradoxes.pdf
Yes, and Meng (2022) is a very useful follow-up too: https://www150.statcan.gc.ca/n1/pub/12-001-x/2022002/article/00006-eng.pdf
Thank you, OliP ! Anything by Xiao-Li Meng goes to the top of the reading list.
Thank you for catching this, Phi ! And thank you for the corrected link, Stephen ! I updated.
That’s a great idea! I have a few recommendations:
For survey weighting, I’d highly recommend one of Rick Valliant and Jill Dever’s books (the firts one is in R and the second in Stata, but both cover the same principles):
* Practical Tools for Designing and Weighting Survey Samples: https://link.springer.com/book/10.1007/978-1-4614-6449-5
* Survey Weights: A Step-by-step Guide to Calculation: https://www.stata-press.com/books/survey-weights/
For design effect, I’d suggest Kish (1992) given all the confusion people have between an actual design effect and Kish’s suggestion for an evaluation of the effects of the weights: https://www.proquest.com/openview/8196b6985339ec9885193d56a5c44ca0/1?pq-origsite=gscholar&cbl=105444
Thank you, Raphael ! I am excited to learn a lot from you during this adventure.
I don’t have those books (yet) ! Which should I start with ? I use R more than Stata, but am mostly interested in the concepts.
I have been using Thomas Lumley’s Complex Surveys: A Guide to Analysis Using R
https://onlinelibrary.wiley.com/doi/book/10.1002/9780470580066
Kish (1992) makes an appearance in my draft post “Survey Statistics: Kish’s (and Meng’s) design effect ?”, where I’d like help connecting Kish’s deff from “haphazard” weights (which is confusing !) to Meng’s discussion. See Meng 2018 equation (3.13), where I wonder if A_W is almost what Kish was getting at…?
Just to note that Meng (2022), linked above, goes into this too.
Thanks, OliP !
I see in Meng (2022) in equations (2.2,2.3) there is Kish’s deff, 1 + CV_W^2.
But in Meng (2018) the deff is a mean-square error (MSE) for our design, divided by the variance under simple random sampling. These means and variances are over the distribution of R. But I don’t see where Meng (2022) takes means over R for equations (2.2,2.3) ?
To supplement the classic Lumley book, I’d recommend the recent book “Exploring Complex Survey Data Analysis Using R”; it’s available in print and for free online. It’s especially helpful for R users who might be used to working with the R Tidyverse.
https://tidy-survey-r.github.io/tidy-survey-book/
Aside from that, the Skinner and Wakefield (2017) article is a good clear, open-access overview paper with the big picture of survey statistics:
https://projecteuclid.org/journals/statistical-science/volume-32/issue-2/Introduction-to-the-Design-and-Analysis-of-Complex-Survey-Data/10.1214/17-STS614.full
Thanks, Ben !
I appreciate that these are open access. (I’m outside academia so I don’t have free library or journal access.)
I like that Skinner and Wakefield describe the design-based approach from the perspective of someone from model-based world (which is most of statistics outside of survey statistics), highlighting what is surprising.
The part starting here confused me a bit: “It may appear that the complex sampling scheme plays no role in the model-based approach. There are, in fact, two fundamental ways in which it does…” They first say it is essential to include structure like strata and clusters, but then they state assumptions about outcomes Y_k being independent of sampling indicators I_k without any conditioning on strata or clusters.
I think the Practical Tools book is much more bang for your buck, as it covers so much more than weighting and it’s incredibly useful for many other things with a lot of applications and R code.
Hmm, it might be. I just think that Kish’s intentions with that measure was not for people to use it as a design effect, not even as a weighting effect, because its underlying assumption of the weights being uncorrelated with the survey outcomes makes weighting itself completely counter-productive (it doesn’t reduce bias, and increases variance).
Thank you !
I agree, Kish’s 1965 book (section 11.7B) derives this deff for when you “give different weights to randomly selected subsets of elements”.
Have you ever seen a non-model-based derivation of Kish’s deff ? All the derivations I’ve seen have treated the outcomes Y as random, including Kish’s 1965 book and Sharon Lohr’s book (3rd edition Chapter 7 exercise 17).
In the design-based context, would “weights being uncorrelated with the survey outcomes” be in the empirical distribution like in Meng 2018 ?
Sorry, Shira, it seem like I can’t reply directly to your post, but here it is.
There are some other derivations of weighting effect, but they all rely on some sort of assumption or model. They use weaker assumptions, but are more difficult to implement. For example, Spencer (2000) [https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2000002/article/5533-eng.pdf?st=Q4NXCvqf] derived a measure that assumes the Y variable is a linear function of the selection (or response) probability. Henry and Valliant (2015) [https://www150.statcan.gc.ca/n1/en/pub/12-001-x/2015002/article/14236-eng.pdf?st=Qpa1u4Uh] expanded Spencer’s formulation for calibration estimators by adding auxiliary variables, X, in a GREG estimator. By the way, all of these are well explained in the Practical Tools book and implemented in a R accompanying package [https://cran.r-project.org/web/packages/PracTools/index.html].
During grad school, I tried to derive a non-model-based weighting effect, using synthetic populations and Finite Population Bayesian Bootstrap, but I never managed to make it work, unfortunately. That’s definitely an area that still needs some further research to be done!
I’m not sure if they are correspondent, I’d have to read Meng (2018) again to refresh my mind of what he means by empirical distribution on the paper.
Thanks, Raphael !
Taking a look now at Practical Tools (2nd Ed, Section 14.4 Weight Variability). :)
Neat that in grad school you were working on non-model-based derivations of weighting effects ! I think Meng (2018) equation (3.13) does this, but let me know what you think ?
I can’t find it, but one time Andrew shared one of his papers on here including a private survey and some larger government survey (that was required to get something from them, iirc).
It wasn’t noted in the paper but in the data 80% of the private survey participants responded they had power over the government vs 5% of the gov survey responders.
Maybe someone could find it based on those fuzzy details.
Point is that there is a 1%-er population of survey responders, quite possibly very skewed towards political activism. I guess still less so than media influencers making money off it though.
Anon:
Yes, it’s a well known problem that political surveys overrepresent people who are interested in politics. That came up in this comment thread, and it’s something that concerns pollsters. I wouldn’t use the term “1%-er,” though, because that expression gives an image of rich people, and survey respondents are not typically rich.
Also, the problem is real but maybe not as serious as you think, in that surveys ask whether you voted in previous elections. The assumption is that past voting is overreported–some combination of recall and social desirability bias–and, even so, many respondents report not voting in the past.
What about adding a section on survey measurement issues? We’ve often lamented the lack of serious attention paid to measurement and I think that is doubly true for surveys. Often it seems like analysts believe that carefully worded questions solve survey measurement issues, but I think these are frequently too subtle to work. The more carefully worded the question, the more it is out of sync with the attention of the respondent. So, while carefully thought out questions are essential, it is also essential to recognize the time pressure and relative inattention of survey respondents. It would be nice to include this is the blog series and it would be nice to see some references on good design practices (not just sampling design).
Great idea, Dale ! I fully agree that measurement gets too little attention. Do you happen to have favorite references on this topic ? When I taught the survey class at NYU (https://steinhardt.nyu.edu/courses/survey-research-methods) we used Groves et al. Chapter 8 https://www.wiley.com/en-us/Survey+Methodology%2C+2nd+Edition-p-9780470465462
@shira I can’t reply directly, but I believe the difference is that section 2 in Meng 2022 is about a deterministic identity for actual (realised) error in individual samples, not the asymptotic statement given in Meng 2018.
Yes, OliP, Meng 2022 sticks with expressions for the actual realized error. Meng 2018, starting with equation (2.4), takes averages over the sampling-and-response mechanism. I always thought of deff as being about averages over that mechanism, not conditional on the specific sample chosen ?