Skip to content

Concerned about demand effects in psychology experiments? Incorporate them into the design.

Johannes Haushofer sends along this article with Jonathan de Quidt and Christopher Roth, “Measuring and Bounding Experimenter Demand,” which begins:

We propose a technique for assessing robustness to demand effects of findings from experiments and surveys. The core idea is that by deliberately inducing demand in a structured way we can bound its influence. We present a model in which participants respond to their beliefs about the researcher’s objectives. Bounds are obtained by manipulating those beliefs with “demand treatments.” We apply the method to 11 classic tasks, and estimate bounds averaging 0.13 standard deviations, suggesting that typical demand effects are probably modest. We also show how to compute demand-robust treatment effects and how to structurally estimate the model.

I like the idea of measuring and understanding these through experimentation. This reminds me of an idea I had awhile ago to reduce certain cognitive biases by changing the way that questions are asked to better match the way people think. Instead of hoping that a certain bias doesn’t exist (or, worse, engaging in dismissive argumentation when the possibility is suggested), you try to include it in your experiment.

One Comment

  1. Fritz Strack says:

    One of the most important ways to reduce demand effects is to renounce repeated measurements. In other words, between-subjects measurements may provide a higher validity at the expense of less statistical power.

Leave a Reply