Johannes Haushofer sends along this article with Jonathan de Quidt and Christopher Roth, “Measuring and Bounding Experimenter Demand,” which begins:
We propose a technique for assessing robustness to demand effects of findings from experiments and surveys. The core idea is that by deliberately inducing demand in a structured way we can bound its influence. We present a model in which participants respond to their beliefs about the researcher’s objectives. Bounds are obtained by manipulating those beliefs with “demand treatments.” We apply the method to 11 classic tasks, and estimate bounds averaging 0.13 standard deviations, suggesting that typical demand effects are probably modest. We also show how to compute demand-robust treatment effects and how to structurally estimate the model.
I like the idea of measuring and understanding these through experimentation. This reminds me of an idea I had awhile ago to reduce certain cognitive biases by changing the way that questions are asked to better match the way people think. Instead of hoping that a certain bias doesn’t exist (or, worse, engaging in dismissive argumentation when the possibility is suggested), you try to include it in your experiment.
One of the most important ways to reduce demand effects is to renounce repeated measurements. In other words, between-subjects measurements may provide a higher validity at the expense of less statistical power.
>This reminds me of an idea I had awhile ago to reduce certain cognitive biases by changing the way that questions are asked to better match the way people think. Instead of hoping that a certain bias doesn’t exist (or, worse, engaging in dismissive argumentation when the possibility is suggested), you try to include it in your experiment.
In some cases, if the deviation from normative answers disappears when the question is posed in a human friendly way, it can mean that it is not a _cognitive_ bias, just a bad way of asking the question. Some ways of asking questions are ambiguous, or push the limits of human sensation (not cognition), or violate conversational norms, etc.