Art Owen writes:
It is my turn to teach our sampling course again. Last time I did that, the highlight was that students could design and conduct their own online survey using Google consumer surveys. Today I learned that Google has discontinued that platform and I don’t see any successor product.
Do you know anybody offering something similar that could be used in a classroom?
Traditional sampling theory is pretty dated now and not often useful for sampling opinions. I compress the best parts of it into the first half or so of the class. Then I talk about sampling from databases, sampling from wildlife populations, and online sampling. I’ve been tempted to add something about how every time you interact with certain businesses (hotels, ride share …. ) you get nagged for a survey response either on the 5 point scale or the net-promoter 10 point scale about recommending the product. Mainly I find those things annoying, though I should probably add something about how they are or should be used.
My reply: For several years I taught a sampling class at Columbia. In the class I’d always have to spend some time discussing basic statistics and regression modeling . . . and this always was the part of the class that students found the most interesting! So I eventually just started teaching statistics and regression modeling, which led to our Regression and Other Stories book.
Also our new book, Active Statistics: Stories, Games, Problems, and Hands-on Demonstrations for Applied Regression and Causal Inference, has lots of fun material on sampling, including many in-class activities.
Regarding surveys that students could do, I like your idea of sampling from databases, biological sampling, etc. You can point out to students that a “blood sample” is indeed a sample!
Owen responded:
Your blood example reminds me that there is a whole field (now very old) on bulk sampling. People sample from production runs, from cotton samples, from coal samples and so on. Widgets might get sampled from the beginning, middle and end of the run. David Cox wrote some papers on sampling to find the quality of cotton as measured by fiber length. The process is to draw a blue line across the sample and see the length of fibers that intersect the line. This gives you a length-biased sample that you can nicely de-bias. There’s also an interesting out there about tree sampling, literally on a tree, where branches get sampled at random and fruit is counted. I’m not sure if it’s practical.
Last time I found an interesting example where people would sample ocean tracts to see if there was a whale. If they saw one, they would then sample more intensely in the neighboring tracts. Then the trick was to correct for the bias that brings. It’s in the Sampling book by S. K. Thompson. There are also good mark-recapture examples for wildlife.
I hesitate to put a lot of regression in a sampling class; It is all too easy for every class to start looking like a regression/prediction/machine learning class. We need room for the ideas about where and how data arises and it’s too easy to crowd those out by dwelling on the modeling ideas.
I’ll probably toss in some space-filling sampling plans and other ways to down size data sets as well.
The old style, from the classic book by Cochran, was: get an estimator, show it is unbiased, find an expression for its variance, find an estimate of that variance, show this estimate is unbiased and maybe even find and compare variances of several competing variance estimates. I get why he did it but it can get dry. I include some of that but I don’t let it dominate the course. Choices you can make and their costs are more interesting.
I understand the appeal of a sampling class that focuses on measurement, data collection, and inference issues specific to sampling. The challenge I’ve seen is getting enough students interested in taking such a class.
One option would be to use audit sampling as an example. I haven’t used any educational materials (although I was involved in a legal case about such issues), but there are case studies available, such as https://publications.aaahq.org/cia/article/15/1/I1/7048/Analyzing-An-Audit-Population-Via-Either-Excel. The issue would be to find a case study (with real or simulated data) that does not involve intricate accounting issues but focuses on the sampling issues instead.
The google survey stuff was paid no? There are alternatives that are similar in price (pollfish), but I don’t think making students pay to do that makes sense for a class.
Taking a public survey (like GSS), asking the same question in an ad-hoc convenience sample, and then seeing differences I always thought would be an interesting project (and would be a good use case to show off MRP). Although just grabbing online data is of course easier (check out your cities open data site, questions about public services are becoming somewhat more common).
But maybe pollsters are just now becoming obsolete, https://news.ycombinator.com/item?id=36865625. (In jest — verifying the accuracy of their results would be an interesting assignment as well, but I don’t think they will let you have a feebie simulation to do that. Bit pricey.)
“We need room for the ideas about where and how data arises and it’s too easy to crowd those out by dwelling on the modeling ideas.”
Applause!
If it’s hard to get students to take a course like that, that’s too bad. From my perspective it sounds far more interesting and important than modelling.
Regarding the incessant “feedback” surveys on the internet, I find them an intensifier for everything that’s infuriating about surveys in general. The questions are too specific and designed for researcher goals not customer feedback, a sorry effort to categorize the continuous variation along n-dimensions into five categories. No room for nuance whatsoever.
A rare opportunity to agree with you! For many purposes, I prefer qualitative research methods such as focus groups (underappreciated, in my opinion) to surveys. The feedback is much richer, though harder to analyze. But it is precisely the ease of analysis of many surveys that makes them so vulnerable to poor practices such as the ones you are describing. It is so hard to design questions (though not impossible, just rarely practiced) that don’t lead respondents to particular answers.
Everytime I am on a jury the rating sheet usually has these multiple questions on a 5 or 10 point scale. Presentation. Content. Novelty. Creativity. Blah blah blah.
While interviewing candidates for a position too.
I always find it simpler if they just asked me one question: rate the candidate on a 10 point scale.
I wonder what others think? Is it really better to elicit people’s opinions on multiple dimensions. Do people actually think that way?
I’ve always thought the detailed course evaluation forms were a mistake. I think it causes many students to just check a single number for all the questions rather than think about them individually. Of course, some forms are “well designed” and switch the order of questions (some have 1=good and others have 1=bad), but these suffer from the erroneous belief that most students read the questions carefully and try to answer accurately and carefully. As a result, I rarely pay much attention to any of the numerical questions (except perhaps the overall average). I find the open-ended questions much more useful. Even then, I’ve seen some people’s evaluation forms that contain virtually no open-ended questions filled out (what does it mean when an instructor gets perfect scores but nobody has anything to say about them?).
That makes me think of student evaluations. Seems like three quarters of the questions ask the same thing – or at least most students interpret it that way:
“Say another student needed to take a science class. Would you recommend this class?”
“Was the material presented in an interesting and accessible way?”
“Was the material easy to understand?”
equal:
“Did this class suck?”
Most students just answer the last question three times. Sometimes a few of the better students in the class recognize some subtleties in the questions and give some quality feedback but mostly it’s just either they like you or they don’t
We have an epidemic of asking a dozen questions when one would suffice ( in the context of surveys).
The misconception seems to be that more the questions the richer the insight. Unfortunately most surveys just put the respondents off and you get crap responses with mindless clicks
Dale: glad we agree! I’ve never been involved in focus groups so I can’t say for sure but I agree with your point in principle. I suppose it’s true that surveys are easier to analyze but it’s not clear if that’s a net benefit since the match to reality could be low unless the questions are very tightly constrained.
It’s fun to see this discussion come out some time after the class ended. Small correction: David Cox was studying the quality of wool not cotton. I asked him if he still had the data, but alas no, something in the 50 or 60 intervening years, had lead to their disappearance.
There is a lot of good material on sampling wildlife populations and I put a lot of that into the course. It can be done a lot less invasively now with camera gear that recognizes individual animals by their markings. The analysis goes way beyond hypergeometric distributions for how many of the fish you catch have been marked.
The enrolment was pretty small. I get that students value learning topics developed in the last handful of years and I would encourage that. However some things are only going to last a few more years before being replaced, and so making room for material that has stood the test of time is also worth it.
Sorry to put this out of place, but I wanted to share my latest distaste for surveys. Today, the AP reports (headline):
“Nearly half of Americans think the US is spending too much on Ukraine aid, an AP-NORC poll says”
The actual survey question is
“Do you think the U.S. government is spending too much, too little, or the right amount on
each of the following?”
and the response is 45% say “too much” 14% say “too little” and 38% say “about the right amount.”
It is true that the same question also listed social security, the military, education, aid to Israel, and a few other items and this was the highest saying too much out of the list. But the list is hardly complete. If I were asked, I’d say the US government spends too much on almost everything. Now, is the problem with the survey or the AP headline, or both? I see such surveys as damaging, not informative. In this polarized environment, numbers like these can affect future policies and future elections. But do they reveal anything actually useful or meaningful about what the public really wants? It is easy to say they provide some kind of indication, but I fail to see exactly what. And I think such “information” does not contribute to meaningful discussion of the issues. Nearly half of the polled think the government spends too much on aid to Ukraine – compared to what?
The more I see survey results the less I believe they are helpful. I find it distressing that so much research (and so much of politics) makes use of surveys.
Dale:
I still think that some knowledge of public opinion is better than not. The alternative would seem to me to be people speculating about opinion using even worse data.
I appreciate what you are saying. But if we use this example, what do we make of this “some knowledge of public opinion?” Does it mean the public doesn’t think we should support Ukraine? Does it mean they think that too much money is wasted? Are they expressing their support for the extreme right Republican views they hear? Will some politicians take this as evidence that they can score points (votes) by opposing aid for Ukraine? Will other politicians take this as evidence they should tone down their support for Ukraine?
In the absence of the poll, these questions will remain and you are right that other “data” will be sought that may well be “worse.” But what is missing is any meaningful civil discourse about the issues involved. And I don’t think the poll promotes such discourse. Perhaps in the absence of any good data, discourse might be necessary.