Steven Levitt writes, “I wish that I was teaching intermediate microeconomics this term, because this would be a perfect exam question.”
I have mixed feelings about cool exam questions. I used to put effort into making my exams really cool, but a few years ago I decided that it wasn’t always clear to the students what they were expected to learn in my classes, so I switched to writing non-clever exams that more directly addressed key points in the course. I expect that the optimal exam depends on how the course is organized. (Also, of course, different exams are good for different students. Presumably clever exams are great for the top students.)
For intro statistics, I wish we used standardized tests so we could have a better sense of what (if anything) the kids are learning during the semester. Also, pre-tests at the beginning of the semester. The whole deal.
Personally I like to save the cool questions for homework, and use the staid ones for exams.
Clever exams are *not* great for top students!
As long as final grades are on a curve, top students benefit from exams that sort better. Poor students benefit from exams that sort worse. The class benefits from exams that sort better (as long as the purpose of the exam is to reveal the ordering of the class).
Clever questions create a lot of variability; maybe you "see" the answer, maybe you don't. This makes them worse for sorting. I think top students should prefer a hard exam that covers the standard material.