Here’s question 6 of our exam:

6. You are applying hierarchical logistic regression on a survey of 1500 people to estimate support for a federal jobs program. The model is fit using, as a state-level predictor, the Republican presidential vote in the state. Which of the following two statements is basically true?

(a) Adding a predictor specifically for this model (for example, state-level unemployment) could improve the estimates of state-level opinion.

(b) It would not be appropriate to add a predictor such as state-level unemployment: by adding such a predictor to the model, you would essentially be assuming what you are trying to prove.

Briefly explain your answer in one to two sentences.

And the solution to question 5:

5. You have just graded an exam with 28 questions and 15 students. You fit a logistic item-response model estimating ability, difficulty, and discrimination parameters. Which of the following statements are basically true?

(a) If a question is answered correctly by students with low ability, but is missed by students with high ability, then its discrimination parameter will be near zero.

(b) It is not possible to fit an item-response model when you have more questions than students. In order to fit the model, you either need to reduce the number of questions (for example, by discarding some questions or by putting together some questions into a combined score) or increase the number of students in the dataset.

Briefly explain your answer in one to two sentences.

(a) is false. If a question is answered correctly by students with low ability, but is missed by students with high ability, then its discrimination parameter will be negative.

(b) is false. It’s no problem at all to have more questions than students. Even in a classical regression, even without a multilevel model, this is typically no problem as long as each question is answered by a few different students.

**Common mistakes**

Most of the students had the impression that one of (a) or (b) had to be true, so a common response was to work through one of the two options, figure out that it was false, and then mistakenly conclude that the other one was true. I guess I should rephrase the question. Instead of “Which of the following statements are basically true?”, I could say, “For each of the following statements, say whether it is true or false.”

Possible typo.

Should “each question is asked by a few different students” read “each question is ANSWERED by a few different students”?

Fixed; thanks.

I thought (a) was true (reasoning that either a or b would be true). My reasoning was: since I made the exam, I have a strong prior that the discrimination parameter is not negative (no way low ability you be right and high ability will be wrong). Thus, it will be near zero.

You’re perhaps joking, because you’re reading of the question implies that the *premise* of the argument is false, and because your prior can’t outweigh the actual, non-probabilistic thing being measured, but in any case I’ll note that it can be dangerous to have high confidence that one will not write an exam question that has a negative discrimination parameter. It’s rare, but I’ve found it easier to do than one might think!

Sometimes a simple, obvious correct answer may be chosen by the middling student while the more knowledgeable student see possibly spurious shades of meaning that confuse him or her…

I remember in undergraduate engineering school (way back in a previous century!) there was an exam question on a topic that I just happened to be a total geek about. I could tell the professor meant it to be a bog-simple question with an obvious answer (he was no doubt pitching it on the “easy” end of the difficulty scale) but I knew of several real-world exceptions to the general principle he was espousing.

So I answered by penciling in a few details of two counter-examples along with what was (in retrospect) perhaps an ill-considered commentary on his over-simplistic characterisation of the topic. He rewarded me with a big, red X…the only question I missed on the exam. I took it up with him privately after class and his response was, “If you knew the answer I was looking for you should have written it down instead of arguing exceptions”.

I thought it was five points well spent.

I picked (a) as well, figuring that (a) was more true than (b). Discrimination parameter of 0 or negative both mean you have a question that should be replaced.

I’ll be honest, I had no idea how to approach Question 5 because I’ve never encountered an item-response model before. Was this a specific topic covered in the course?

Samuel:

Yes, everything on the exam was covered in the course. Item-response models and ideal-point models are important in political science.

Looking at this question yesterday, I was sure that both were false and I could not figure out where my mistake was -indeed assuming that one of both “had to be true”.

Honestly, the phrasing of the question suggests to me that you think that one is true and I do not have sufficient audacity to argue that Andrew would be wrong -given that I learned about item-reponse models from his paper with Bafumi, Park and Kaplan in the first place.

I’m curious what the discrimination parameter is for Question 5, since (as many have noted), the phrasing is easy to mis-interpret. “Which of the following is true” does, to most people, imply that one or more of the items that follow are true — the google dictionary hit for “which” is “asking for information specifying one or more people or things from a definite set.” It’s like “or,” which to most people isn’t “logical OR” but rather “XOR.” It would be better to write “Which of the following, if any, is true,” or “Evaluate whether each of the following statements is true or false.”