So far we’ve focused on means E[Y] as our quantity of interest (or means for subgroups like voters E[Y | V=1]). But now suppose you want to browse Y with your eyeballs. For example, maybe Y are survey openends (“describe how you feel about the candidate”), and you want to read thru a few of them. But you want these to look like draws from the population, not survey responders, who might have a different distribution of openends.
Suppose your survey openends are y_1, …, y_n. Can we sample from them to get openends that look like they were drawn from the population ?
Let Y* be samples (with replacement) from our survey openends with probabilities p_1,…,p_n. What should these probabilities be so that P[Y* = y] = P[Y = y] for all openends y ? We’ve been working with means, so let’s write this as E[f(Y*)] = E[f(Y)] where we can take f(Y) as an indicator that Y=y.
We can identify means E[f(Y)] as E[f(Y)W | R=1] where we get weights W either by a model for response R or by “equivalent weights” (see the October post and its follow-up in November). Estimated with our survey sample this is
1/n sum_i=1^n w_i f(y_i)
So if we take p_i = w_i/n, then we get Y* with the correct distribution ! In other words, we should sample from our openends with probability proportional to the weights. Has anyone done this before ? Tell me more !
I have read open-ended questions without weighting them. In our current project, participants have answered several such questions. Depending on the purpose of the question, we proceed differently.
If the question is for our own interest, such as comments on the survey methodology, I read all the answers and judge their meaning intuitively.
If the question is intended to provide meaningful information for our paper’s future readers, we assign codes from a custom-made code book. This code book is generated both deductively (using our own reasoning and literature results) and inductively (creating new codes based on participants’ responses). This produces several columns of binary variables, one for each code. These results can be poststratified. Finally, you can look at them in a bar chart.
Thanks, Raphael ! This makes a ton of sense.
So at one point I was working with I think it was ACS data to understand cost of living. If I remember the weights for different households are different, so if you want to take say 100k rows out of a few million and estimate something from it, you need to sample the households in the survey according to the weights because a person who represents say 500 households is very different importance than someone who represents 5000. So I wound up just doing a sample with respect to the weights so that I could treat each household as equally important and then estimate the functions I was interested in. I wasn’t interested in simple averages of say voting tendency etc but rather things like the fraction of income spent on rent or mortgage as a function of household size and spatial location. Doing it this way was the only way to stay sane.
(It might not have been ACS data, but in any case it was weighted sample data)
Thanks, Daniel !
Interesting use case ! Very understandable to want to focus on your analysis and not on the weights and survey design more generally. I think this was one of the aims of Thomas Lumley’s survey package, discussed here:
https://youtu.be/4xfRZp_8XFs?si=EjNk310qSsdwB3Xf&t=161
I wonder if anything in that package works for “things like the fraction of income spent on rent or mortgage as a function of household size and spatial location”, though it depends on your exact approach ?