This is Jessica. Occasionally on the blog the topic of “design freedoms” has come up in passing: the idea that in addition to the many degrees of freedom one can have in deciding how exactly to analyze data from some experiment, experimenters also tend to have a lot of freedom in how they design conditions, stimuli, instructions, etc, and what they choose to measure. What seems interesting is that 1) these don’t get discussed nearly as much analysis freedoms, as far as I can tell, though the problems they can pose are conceptually similar, and 2) the ability to pilot experiments repeatedly is often seen as a good thing among experimentalists. I.e., seeing something in pilot experiments then confirming it in the final experiment means you’re not reporting on your first experience observing some effect; by the time you publish it you’re replicating your own work. 

I’ve never really questioned the value of pilot experiments for identifying and trying to reduce sources of noise, such as in measurement. But I’ve also suspected for a while that given all the flexibility in design conditional on seeing results through piloting, any experienced experimenter could probably take some not-so-strong effect (maybe even non-existent) and design an experiment to register it by repeatedly tweaking aspects of the design and stimuli based on pilot results. 

Essentially, you have an interactive feedback process where the primary goal is to increase signal. As we know from the researcher degrees of freedom/garden of forking paths understanding, what constitutes the right signal is likely to be defined by the experimenter’s preconceptions of what they are looking for, such as a difference between conditions in a particular direction.  And similarly to motivated reasoning in analysis, I could imagine some researchers not even realizing they are gaming things in a way that limits the generalizability in their results, instead thinking they are just clarifying or de-noising the observational process they’ve designed.

One obvious checkpoint in catching design gaming would seem to be peer review – you have to explain the details of your experiment to publish on it. Presumably, if you gave people in one condition some stronger directive of how to act based on the difference you want to see, reviewers should catch that, just like they should be weighing how plausibly your setting seems to capture whatever the target scenario is. E.g., for an experiment on visualizations, if you want to say something about how people interpret election graphics or other visualizations of uncertainty, you should make the experimental stimuli and task feel like something someone might go through if they encountered the visualization on the web, and you should ask them for judgments and decisions in ways they are likely to understand, and you should turn in all the instructions you have people in each condition to show that there weren’t somehow priming them with answers. But while snapshots of experimental stimuli or even code for reproducing the experiment for a reviewer to try are often submitted with experimental papers, peer review is clearly not perfect, and reviewers might miss subtle asymmetries across conditions or other artifacts of the tweaking process. Or, authors might fail to document them. 

Some examples of degrees of freedom in design:

• Operationalizing the construct: In psych style studies there are often there are many ways to measure what could be construed as evidence of the same underlying construct, e.g., political attitudes, comprehension of some information, etc. 
• Suggestive instructions/prompts: If you’re trying to study some effect you think is plausible, but the first, straightforward wording of the questions for subjects isn’t getting at it, you could try to walk the line between fair question and too suggestive. More generally how you explain an experimental task to people can matter a great deal in seeing effects, but interpretations often fail to acknowledge this conditionality.
• Adding validation checks: If the results seem too noisy to see what you want, try adding some more validation questions. Multiple validation questions can give the appearance that the experimenter is simply being careful to omit people who might not be reading instructions. But you could also use it to omit participants who are using more heuristic or gist-based reasoning that would make the difference between conditions look smaller.

Then there’s the whole question of choosing stimuli, which introduces lots of generalizability issues that too often get overlooked.   

• Cherrypicking stimuli: The range of stimuli you test on is up to you as an experimenter.  Worse, often it’s not well defined, even when it could be, which may make it harder to diagnose when it seems unreasonable.

I often think about this issue with stimuli generated to vary along some quantitative measure, since this is what happens in visualization research.  For example, imagine evaluating the effect of different ways of communicating medical risks to people. How do you decide the range of risks, and how you sample from it, to create the stimuli? Ideally it’s chosen based on domain knowledge about the range of risks people face in the world in that setting (e.g., probability of some outcome if you get some disease), and sampled proportional to the expected real world distribution for whatever the target scenario is. But the chain of reasoning I see in many experimental papers (at least in my field) is weak and underspecified, more along the lines of ‘we sampled a few points along some range we think is meaningful’. An experimenter could spend some time piloting and identify some points in stimuli space that allow them to show differences, e.g., between different ways of visualizing the risk. In light of this, whether you can show an effect or even how big the effect are not at all good indicators of whether what you found was important. There are always edge cases where something counterintuitive might happen. 

The importance of ample variation in stimuli is related to this. If you want to make claims about how people react to violence in video games, but then you choose some single instance of violence of a certain type, you can’t really claim to be saying something about violence in video games at large unless you have a good argument for why variation is expected to be minimal. 

These conditions set up the potential for experimental results to overfit the specific context of study, i.e., a generalizability problem. So while piloting leads to replication, as Devezer et al. have made clear, this doesn’t necessarily mean that you’ve observed something important. But it does give you opportunities to tweak the choice of stimuli to your liking. 

One way to better characterize this could be to simulate it.  For the classic degrees of freedom in analysis, you can draw samples from the same distribution and show how you can increase your chance of seeing significance when you have flexibility around how you remove outliers, how many observations you test on, what measures that you collected end up in your final model, etc., as demonstrated in Simmons et al 2011 paper. But when piloting is used to tweak things until the experimenter is pretty sure that the “official” (perhaps pre-registered) experiment will produce results they like, issues like sampling error leading to overestimates of effects should be less of an issue. What is being exploited is some regularity induced by the experiment design, but one that isn’t really a valid proxy for the construct of interest. One way to conceptualize might be that there’s some underlying data generating process that gives rise to some moderate correlations but they can only be seen in small ranges of some total space of possible observational processes that are reasonable for studying whatever the construct of interest is. But I’m not sure that really captures it. 

There’s also the obvious question of whether there should be steps to reduce potential design freedoms, like there are steps to reduce analysis freedoms. Probably not avoiding piloting experiments. We could assume that current review processes are likely to catch any important design gaming, which seems to be the current status quo. What if experimenters were also asked to report full details of the piloting process (all the things tried, screen captures of all the iterations)?  Often these are tracked internally, but I haven’t really considered before what it would mean if they were submitted as part of the paper. But I would be interested as a reviewer in seeing the progression of measurement, instructions, etc choices.

All this also makes me think of blinding. When you’re piloting, even if your design uses blinding, e.g., so that an experimenter running an in-person lab experiment doesn’t know who is in what condition, the piloting process you use to tweak the design is full knowledge, and it’s hard to imagine how it wouldn’t be. Maybe if we’re going to pre-register analyses we should also pre-register piloting processes, to commit a priori to the kinds of things one might tweak and how. But that’s not really a satisfying answer as there are potentially lots of valid ways one adjusts one’s observational process that are hard to conceive of a priori.