Someone who wishes to wishes to remain anonymous writes:
My son is in 11th grade. Below you’ll find a HW question which was given in my son’s statistics class. How would you answer?
Ummmm, it’s hard to read that so let me retype it:
A large university emailed a questionnaire to a randomly selected sample of 120 students in this year’s freshman class. The questionnaire included topics on student life, academics, and athletics. Thirty of the students emailed did not respond to the questionnaire. What is the best plan of action for the university?
Select one answer
A. Send another email to those 30 students who did not respond encouraging them to complete the questionnaire.
B. Start over with a new sample of 120 students from this year’s freshman class.
D. Use the 90 questionnaires that were submitted as the final sample.
E. Randomly choose 30 more students from this year’s freshman class and email them the questionnaire.
My reply:
I wonder what was option C, which already got eliminated . . .
Of the four options given above, I think option A is the best, then D is the second best, but the best choice is not completely clear. The trouble with option A is that the norespondents might not respond even after encouragement. The trouble with option D is that the nonrespondents can differ from the respondents. Options B and E are fine too, as they both involve gathering more data, but presumably the new data will still have noresponse issues.
Usual good practice would be to gather available data on the respondents and use this information to adjust the sample to the population. We discuss this in chapter 17 of Regression and Other Stories. This would go beyond what is taught in high school, though. I actually don’t like this sort of message that says that nonrandom samples are bad but without a discussion of how to fix the problems. I hope someday to design a high-school-level statistics course, but I’m not there yet.
Stepping back, it’s interesting to see a homework question where there’s no unambiguously correct answer. This is sending a message that statistics is more like social science than like math, which could be valuable in itself—as long as it is made clear to students and teachers that this is the case.
I quit giving multiple choice questions 45 years ago after my first semester teaching. Even when the question and responses are well-defined (unlike this example), they still tell me little about a student’s understanding. Sometimes understanding things in depth makes it difficult to provide the “right” answer, while a superficial understanding can produce the correct answer due to being unaware of the subtleties. Even if you carefully design the question, there is the additional problem that the question embedded in a large number of test questions with a time limit, may not permit the kind of reflection required to properly consider everything required to get it right (a problem shared with many/most surveys – even when the questions are well-designed, which they are usually not, respondents usually don’t have the time or incentive to think about them carefully enough to provide good responses).
So, it is unclear to me what such questions really test – more than anything, it would appear to test the student’s ability to guess what the tester is looking for – a usual ability, but not what was really intended. In the case of this particular question, my responses are the same as Andrew’s – but it is clear that having a correct vs incorrect response tells us nothing. If you turn the multiple choice into a short answer – multiple choice + explain your choice, it suddenly becomes useful.
I think I’ll choose C.
I’m curious if you have read Meehl’s “The Seven Sacred Cows of Academia” in particular chapter IX: exams where he discusses multiple choice formats in some detail (page 111 and onward – you can find it here: https://meehl.umn.edu/sites/meehl.umn.edu/files/files/186cows_2018_book.pdf)?
It seems to me that some of your criticisms are not endogenous to the method but rather how most examiners construct their exams. You cannot test the ability to expound ideas, sure, but most of the time you can ensure one need to think critically in order to arrive at the credited response (even if that is not the norm). And for ordinary undergraduate courses an examination that tests critical thinking (well) is good enough in most circumstances IMHO. Of course constructing an MC exam that is faithful to this aim is an easier task in the more developed sciences where the foundations are far less murkier. It gets tricky in subjects like statistics or psychology or whatever where there are a lot of rational paths of analysis, but even then it can be done well – though the effort required to do so increases rapidly.
Thanks for the Meehl reference, I had not been aware of that. Many of his comments are correct – it is easier to disguise badly constructed essay exams than poorly constructed multiple choice exams, for example. I also agree with his statement that bad multiple choice exams really reflect deficiencies in the creator than the instrument. The only place I would take issue with is his description of a good multiple choice question. He gave an example of a physics exam where the questions require careful thought and construction of a series of steps leading to a correct answer – one of 6 possible choices. This sounds like a difficult task to create such questions, one consideration in their use.
But my concern then is how such a good question is graded. Right/wrong has the same binary thinking problem we have discussed in other contexts. Shouldn’t there be a “more right” grade? If I need to follow 10 logical steps and I fail at step 7, isn’t that better than failing at step 3? And if we are to grade along these lines, then creation of these questions becomes even more difficult. I’ll admit to never having written questions that would satisfy such criteria. In the spirit of Meehl’s writing, the energy required to do so seems so much greater than the energy required to write and grade good essays or open-ended problems, that I don’t think the multiple choice exam passes muster.
Along the lines Meehl suggests, I guess we should compare real world essay questions and real world multiple choice questions rather than the idealized form of either. I won’t claim that multiple choice is generally inferior from that perspective – but real world multiple choice exams don’t really test for the skills and talents that I am looking for in my courses.
I agree that there are reasons to prefer essay or short-form response over MC. In fact, I’m currently marking an exam on concrete quantity takeoffs where I easily could have used MC but elected for written response for one of the reasons you mention. Namely that it matters where in the calculus they screwed up (if it’s a conversion error that’s better than say not accounting for loss due to placement).
But I do think that an intelligently constructed MC exam can be better than most other forms of examination in a lot of circumstances, if for no other reason than the reliability is vastly increased. And I think it’s disservice to our students to banish it to the waste bin just because most professors usually mess up their construction.
+1 Multiple choice questions make no sense for statistics or data courses. Ironically, the people who took education statistics classes are forcing instructors to turn all assignments into multiple choice, so that all such questions can be marked automatically by a robot on those crappy online course websites (this is called “taking our subjective biases out of the equation”), and also so that the questions could be stored in a secret item bank allowing an AI eventually to “create” class assignments.
After losing the debate with administrators, I was forced to make multiple-choice quizzes; I learned that the only questions that could be set were dumb, and dumber. They must be factual questions, such as Which of the following is not a statistician? (Answer: Tom Brady.) Anything else, there will be some student arguing that some other answer could be correct.
If the question is:
When pollsters compute “margin of error” for 95% confidence, what multiple of the standard error is used?
A) 1, B) 2, C) 3, D) 4
There will be one student who got it wrong but now claims that the correct answer is “None of the above” since it should be 1.96, and so I should accept all answers.
In the blog post, answer choice C is “eliminated” by the “elimination tool”. I have no idea what that means.
Answer A is most likely what is done in practice. Answer D is the KISS answer – and the challenge to the teacher is whether it’s too early to talk about nonresponse bias in a high school class. But answers B and E also bring up an issue I feel should not be taught in high school (what to do when the second sample contains someone already in the first sample).
Kaiser:
I disagree! No joke, I prefer multiple-choice tests. I can send you some of the test questions I’ve written to go with Regression and Other Stories. This will best be the subject of its own post, but just briefly: my problem with essay-style questions is that they encourage students to spew out guesses and B.S. I think it’s good to force them to commit to an answer.
The problem with them committing to an answer is that 2 students committing to the same answer may have completely different understanding of the material, yet they get the same grade. The reason I abandoned multiple choice after my first teaching experience was precisely what Kaiser says: students would explain why they chose the “wrong” answer and I was sufficiently impressed by their reasoning that I would accept the wrong answer – in some cases, their reasoning was superior to those getting the “right” answer. But why not get the best of both worlds? Multiple choice + explain your answer. The only downside I can think of is the time required to grade such exams – and frankly I have little sympathy for that concern. If teachers don’t have the time to do their jobs, then they shouldn’t be teaching.
Sure, would love to see some examples of your multiple choice questions!
Also, for essay type questions, I totally get your point about commiting to an answer – that’s a crucial part of statistics training, being able to draw a conclusion despite imperfect data, uncertainty, noise, etc. So, I state clearly to the students that (a) I will accept many different answers and (b) the highest points go to those who argue convincingly their one answer, and points will be deducted for answers along the lines of it could be A because of this and B because of that. Imagine you’re delivering the final report, not the exploratory analysis.
The biggest downside of essay questions is the time and brain cells needed to mark them :)
Addendum: also (c) if you have to make assumptions, do it and state it.
Dale –
> Even when the question and responses are well-defined (unlike this example), they still tell me little about a student’s understanding.
What is the purpose of the test?
Is it to ranks students, one against the other?
Is it to obtain criteria reference points, so that the students and the teacher both can assess what has been mastered and where focus should be next?
Seems to me that the style of testing would be different depending on the actual function of giving the test.
So much about the purpose of testing is being assumed – without explicit acknowledgement. Indeed, I’m not even sure most of the time people are examining for themselves exactly WHY they are testing students.
Unfortunately, imo, people tend to just assume and accept that ranking students is the primary purpose of education, and on that basis just accept testing as a means to that end. So in that situation, criteria referencing a students knowledge isn’t particularly important.
Joshua:
I agree that testing has many goals. For me, the most important purpose of a test is to give students something to study for, a particular aim for them during the semester. For that purpose I find it helpful to give a test that has multiple choice questions with correct answers. That works best for me. I can believe that other sorts of tests will work better for other instructors.
I agree with you. Most tests appear to be designed to measure (in some imperfect way) a student’s ability to answer questions. In my mind, this is an increasingly irrelevant ability (even counterproductive in many circumstances). I’d much rather see students able to ask good questions – an ability that is increasingly important. Of course, you could design multiple choice questions that test for that. But what I am testing for is a student’s ability to think critically and creatively. Almost never am I trying to see if they can accomplish a task in a particular way – the subjects I deal with always permit different ways to approach questions.
Of course, if I am asking them to conduct analysis of a particular data set (e.g., looking at Facebook’s movement data in relation to COVID restrictions in different areas), I want them to demonstrate appropriate use of visualizations and statistical methods. But my real focus is on how they approach the analysis – I penalize them for mistakes they make with the methods, but put much greater weight on how they approach the analysis. The narrow concern with whether they have implemented a particular technique correctly counts for little in my mind – there is plenty of software to prevent those kind of mistakes. What software cannot yet do (and isn’t likely to be capable of for some time yet) is approach an analysis in a way that leads to improved decision making. Testing for that is easier than creating good multiple choice tests, in my opinion. But as you say, the appropriate test does depend on what the function of the test is.
I would also add that how tests are graded is a significant dimension of the question of how best to test. I’ve used a simple grading system (that I’ve used for 45 years now – and one that virtually nobody else I know of uses). The A range demonstrates a “command” of the tools, “B” is control of the tools (but too many errors or omissions to be called “command”), C is for effort (but frankly the tools control the student rather than the reverse), D is for questionable effort, and F means they’ve convinced me of the lack of effort. I can grade open-ended exams/homeworks very quickly using this scheme (I do use +/-) and rarely (almost never) find any disagreement between my judgement and the student’s subjective judgement.
Dale:
My guess is that you’re just a better teacher than I am! I’d love for students to gain critical understanding, but over the years I’ve concluded that, if they can learn the basics, that would already be enough. I’ve just seen too often that I’ve tried to teach concepts but then when the course is over the students can’t do anything! So my current preference in basic courses is to try to be specific in what they should learn, and then to have seminars with more open-ended discussion.
Dale –
> Most tests appear to be designed to measure (in some imperfect way) a student’s ability to answer questions. In my mind, this is an increasingly irrelevant ability (even counterproductive in many circumstances).
Agreed (see my comment below). Testing was (perhaps) a more legitimate tool when it was more generally transparently true that the function of schooling was to sort students and train them to function in a workplace hierarchy. Arguably, a more functional skill in today’s world, is the ability to ask questions rather than an ability to answer them. I like your grading system, as I think it begins to get to the heart of what I consider to be the higher goal of education – to help students gain and understanding of, and agency within, their own learning process. It stresses the metacognitive skills that are critical for empowering students. Testing, as much as anything else, points students towards a passive, top-down model of learning, where they pass over assessing their own learning process to a black box system where teachers judge their learning for them.
I know that’s very idealistic rhetoric, and I don’t mean to say I think there’s ZERO benefit in a more standard educational paradigm. But over time, I do think that the balance between what’s productive versus counterproductive in the standard paradigm shifts, and testing is one method where over time, I think, it’s shifting in the direction of being counterproductive.
Acknowledging ahead of time that I can get a bit preachy here.
As to the utility of testing:
I think for sure it has some utility, but it’s also important to acknowledge in part, testing is a part of our educational paradigm because (1) people just do it because it’s what’s done and (2) testing takes place because teachers have to grade students, and tests makes it easier to grade them. The use of testing is a function of a general acceptance that our education-industrial complex is grounded on a model of competition (and largely serves as a social sorting system that perpetuates the status quo).
It’s not irrelevant that testing is such an important part of our educational structure yet pretty much totally disappears from the scene once people graduate to begin implementing what it was that they learned in school. Why would that be? If testing is some kind of inherent component of learning and applying what’s learned, why would graduation mark a kind of dichotomous inflection point on the use of testing?
Multiple choice:
Seems to me that multiple choice testing could certainly be a useful sort of testing, if the different choices all conveyed information. If wrong choice (a) is selected, it should show a student as misunderstanding a’. Wrong choice (b) would indicate that student has misunderstanding b’. If it’s looked only as binary information, where the wrong choice choices aren’t used to indicate anything except that the testee gave the wrong choice…well that doesn’t really seem to me to convey much information.
I don’t mean to be impractical – teachers in classrooms need to make compromises. And I don’t mean to imply, in the least, that the education-industrial complex doesn’t bring value. I’m not a binary thinker. But IMO, in an ideal environment the function of testing should be to criteria reference a student’s mastery. Ideally there’s a scope and sequence of the material being taught, and a test shows where a student is in that track. A student should then be able to retest after studying more, so that they can know they’ve mastered the underlying material and is ready to move on to material that is contingent of that mastery.
Criterion-referencing is kind of a different kettle of fish than assessing a student’s learning process. I don’t see how that could be broken down into discrete criteria referencing.
Makes me recall an experience from graduate school (one of the few meaningful lessons I learned): walking out after my last prelim, I remarked to one of my fellow students, “that’s the last test I’ll ever have to take.” One professor overheard, and said “actually the tests are only beginning, and they will all be much harder.” We are always being tested, but the real world tests don’t resemble those school tests we’ve been exposed to for 12+ years. I’ve tried to make my tests look as much like those real world tests as possible. And I do permit students to redo their work – after I’ve gone over my answers – for half credit.
I frequently find that even when I am willing to complete a survey, I abandon it before I finish if it just drags on and on and requires a lot of clicking (I am a keyboard person). The kind I hate most is asking for very precise ratings (1-10) on some subjective scale, stacked in rows and rows forever.
I understand that some topics need long surveys. I just wish they would employ someone to design their survey well. Also, if a survey can be meaningfully decomposed into subsets, it may make sense to send various parts to subgroups randomly. Don’t know how much this is done in real life.
What is the pupose of this questionaire and why is there seemingly an obsession with getting exactly 120 responses? Also, why did the university fail to anticipate a substantial non-response rate?
Thinking more, the description and answers make me suspect this survey will also have deeper issues.
The best thing to do might be just forget about it, since misleading info can be worse than no info at all. Avoid falling for the sunk cost fallacy. Maybe that was option C.
Why is answer A preferred over E? I personally find the answer A to be less optimal compared to E, because you may be introducing another level of bias (1.) maybe those who do respond second time, after encouragemet are, on average, different than the first time responders and also (2.) you can’t know for sure that even after encouragemet they would respodent. While the E gives you potantial to get another 30 responses while limiting those problematic aspects.
Anyway, this is highly context/survey dependent problem. I would choose not to answer at all.
I’d think that the 30 non-respondents are special as a group, and I’d suspect that they’d have systematically different views from the others. Currently they are not represented at all, and asking them once more will at least bring down the size of the not represented group. (I’d also be rather curious how many of them would respond; I find this interesting in its own right.)
So I like A. If B means discard the 90 you already have this is pure nonsense as there will be non-response again. D is the time pressure and tight budget choice (background information is needed to know whether this is the way to go). E is indeed strangely fixated on the sample size 120, and of course there will be further missings. It does nothing about systematic bias through lack of representation of the non-response group.
“you may be introducing another level of bias (1.) maybe those who do respond second time, after encouragemet are, on average, different than the first time responders” – this is a very strange argument against A because if they were identical to the first time respondents, what would even be the point of trying to get hold of them? We want them *because* we suspect they are different, and we want that represented in the sample as well as we can.
Wait a minute…an 11th grade statistics class??? Wow.
Or maybe they finally thought they figured out a suitable class for the football coach to teach.
First off, I’d wager that this is something that was explicitly taught in the student’s class, as part of a broader module on survey administration. When I took high school statistics in the 1990s in a California high school, the examples were coin flips, surveys, and RCTs. We also learned best practices for administering surveys and RCTs.
Second, this would be much better as a short answer question, but those take much longer to grade. It seems the teacher is overworked, which also explains the poorly-designed question with no best answer and the lack of a C option. This is a homework, not even an exam. Have none of you ever made a mistake with an exam question?
This relates to my final point, which is that it seems you’ve bounded the problem scope too narrowly. You’re asking us all to guess and argue about what the “best” answer is, absent any other information. I’d say the proper response to this problem is out-of-band for this assignment: it would be to contact the teacher for clarification.
> statistics is more like social science than like math
I generally agree. My own favorite analogy is that statistics is more like baking than math.
Even if you follow a recipe exactly (i.e., adhere to all the rituals of a typical statistical procedure like a t test or ANOVA), you won’t get the same outcome each time. This could be due to “chance”, to sampling biases, to various minor violations of assumptions, etc. These are analogous to the differences in equipment and ingredients that will vary between kitchens.
And in the “real world”, you often don’t have access to all the equipment and ingredients the recipe says you need, so you have to be creative in how to adjust. But how you adjust depends on what you want to achieve (e.g., can you substitute dark brown sugar for light brown sugar? Maybe, but then you might need to adjust the timing/temperature or you might get something with a taste/consistency you didn’t expect…).
A question like the one in the post that implies there is one best way to adjust for something like non-response bias doesn’t make much sense. I agree with Dale that a question that involved explaining your reasoning would be far superior.
I think an interesting question would be to understand the ramifications of non-reponse bias.
The problem with that “usual good practice” is that you can only “correct” for variables you can identify and already have some good information about, and only under assumptions about independence or lack thereof among various variables which may or may not actually hold. Which introduces all kinds of problems, starting with your own biases about what might be important.
And a lot of that information you’re using to do the corrections may itself have been “corrected” using the same “usual good practice”.
Keep doing that, and your literature is going to end up pretty polluted, with the pollution building on itself as time goes on. Especially when your data are already noisy, and your samples are usually too small for your effects, and the categories you’re putting the data into, and the questions you’re trying to answer, and the answers you will consider, can very easily fail to match the “natural domain structure”, and are often subject to huge amounts of cultural, political, and emotional influence.
Statistical rituals can easily be used to make randomness and bias sound respectable, and to make problems harder to see… but you can still end up with randomness and bias at the root.
The problem with the social sciences, and the reason they’re so disreputable among hard scientists, is that they’re unwilling to adopt the REAL “good practice” of admitting that they simply can’t draw any useful conclusions at all about a lot of the things they want to study. That’s not a harmless thing; there are sharks circling, waiting to intentionally overinterpret and politically abuse pretty much any result they come up with.
Sok:
Sure, but meanwhile life goes on. We do surveys with 5% response rates but they still are pretty accurate (see discussion here). Yes, our adjustments make lots of assumptions; I still think that adjustments can be useful.
In general, I recommend carefully listing the assumptions that we’re making, and then recognizing that our resulting inferences will be conditional on those assumptions.
Finally, do I care that the social sciences are “so disreputable among hard scientists.” No, I do not care. Maybe that’s because I am a “hard scientist,” and I’ve done research in physics. “Hard scientists” have their own area of expertise, and that’s fine. Although the social sciences are mostly useless, I think they still have value.
There is often attrition when doing surveys. If the correct answer is always A then that implies there can be no attrition for completing the survey. Since the list of topics asked don’t seem to correlate with non-responses (such as doing a survey about sexual harassment where non-responses may be very important), the correct answer would be likely D, to discard the subpopulation.
The answer would only be A if “student life, academics, and athletics” were correlated with non-responses. The best answer would be D.
I believe it is a trick for observation and you should display eliminated answers.
F. Hire a consultant to redo the survey.
ChatGPT says:
C. Adjust the analysis to account for the missing data from the 30 students who did not respond.
The best plan of action for the university would be to adjust the analysis to account for the missing data from the 30 students who did not respond. This will allow the university to make decisions based on the available data and avoid potential bias that could result from using only the 90 questionnaires that were submitted. The university could also consider sending another email to those 30 students who did not respond to encourage them to complete the questionnaire. This could help to improve the representativeness of the sample and provide a more complete picture of the opinions and experiences of the freshman class.
static: how did you pick c when it wasn’t listed?
ChatGPT: In the original question, option C was not listed as a possible answer. However, I believe that adjusting the analysis to account for the missing data is the best plan of action for the university in this situation. This is because it allows the university to use the available data to make decisions, while avoiding potential bias that could result from using only the 90 questionnaires that were submitted. The university could also consider sending another email to those 30 students who did not respond to encourage them to complete the questionnaire, which could help to improve the representativeness of the sample.
That is truly impressive. But I must add that I am unsure whether you really used ChatGPT or are making this up! I guess that almost passes a strange version of the Turing test.
I was impressed by the quality of the answer. Better than most students!
I tried to get it to tell me why it invented an answer C, but it was a bit evasive….
static: yeah, I guess sometimes the best answer just isn’t listed and you want to be creative, but other times they want you to pick the least bad answer
ChatGPT: Yes, that’s true. In some situations, the best answer may not be listed as a possible option, and it can be helpful to think creatively and consider other possibilities. However, in other situations, the options provided may represent the only realistic options available, and it may be necessary to choose the least bad answer. It’s important to carefully consider the specific circumstances and context of the situation in order to determine the best course of action.
“Stepping back, it’s interesting to see a homework question where there’s no unambiguously correct answer. This is sending a message that statistics is more like social science than like math, which could be valuable in itself—as long as it is made clear to students and teachers that this is the case.”
I suppose the same applies to the natural sciences, such as the earth sciences. If the system you are trying to find an answer is chaotic, then in most cases, there is no clear answer.