I sas this in the New York TImes today: A CBS News poll asked the following question:
Should U.S. troops stay in Iraq as long as it takes to make sure Iraq has a stable democracy, even if it takes a long time, or leave as soon as possible, even if Iraq is not completely stable?
I seem to recall some advice in the sample survey literature about not asking double-barrelled questions (here, the assumption that U.S. troops will “make sure Iraq has a stable democracy,” along with the question of how long the troops should stay). In any case, it seems like a good example of a problem with question wording.
Incidentally, the Times feature on this poll (it was only a paragraph, not a full article) did not point out the problem with the question wording, and it also featured a yucky graph (as Tufte would put it, “chartjunk”).
I understand the issue of question wording and its effect on poll results…and of course I'm aware of the related issue of "push polling." It's clear that some wordings will yield a distorted picture of the target's views.
Less clear to me, though, is the question of whether there is any question wording that will _not_ yield a distorted picture of the target's views. The mental picture that I think people apply when trying to create a "perfect" question, or more usually a perfect set of questions, is that the questionee has a mental state that can be characterized by various parameters — degree of support for war in Iraq, degree of support for the President's policies, level of desire for U.S. troops to be extracted soon, etc. — and that a perfect set of questions will quantify the state of the person without changing it. But I wonder if that mental picture of the polling process is actually true, especially for complicated issues (like the stance on the war in Iraq). Maybe there is no way to probe the person's state of mind without changing it, for some or many people, and maybe even at a given instant some people don't have a self-consistent view (i.e. a consistent set of parameters to describe their position). A person might be upset about the the war, feel that the U.S. should withdraw as soon as possible, feel that we should stay until stability is achieved, feel that we are not doing any good there, feel that we can't "cut and run" because it will set a bad precedent, and on and on…all at once.
I guess I'm saying: It's obvious that some questions are bad…but are there any questions that are really good? Is there a platonic ideal that we can strive for in desiging a survey, or do we just have to realize that we live in an imperfect world and try to pick from a class of questions that, while flawed, cannot be improved?
P.S. Am I the only one who is irritated by the phrase "cut and run"? My stock response has become: "I think we should cut, but not run." This usually draws a comment such as "huh?", to which I say "You know…we go ahead and cut, but then we don't run. Maybe we just keep cutting and cutting and cutting, but never running. What do you think the insurgents would do then?"
Yes, no question is perfect. Bob Shapiro commented that no single question will elicit a full response on this issue. Two or more questions may be needed.
What interested me about this particular survey item was not so much whether it is "slanted" but rather that it contained such a big implicit assumption (that if the troops stay long enough, that Iraq will have a stable democracy), and it seemed that the survey writers, the people at CBS, and the people at the New York Times all just took the assumption for granted. At least, I didn't see any comments on it.
The "cut and run" thing reminds me of a dialogue from first grade or so:
Kid 1: Do you wanna fight?
Kid 2: Yeah.
Kid 1: Are you gonna run?
Kid 2: No!
Kid 1: Then how ya gonna catch me???
[I guess this falls within the "no soap, radio" genre of humor.]