Peter Bergman writes:

is it possible to “overstratify” when assigning a treatment in a randomized control trial? I [Bergman] have a sample size of roughly 400 people, and several binary variables correlate strongly with the outcome of interest and would also define interesting subgroups for analysis. The problem is, stratifying over all of these (five or six) variables leaves me with strata that have only 1 person in them. I have done some background reading on whether there is a rule of thumb for the maximum number of variables to stratify. There does not seem to be much agreement (some say there should be between N/50-N/100 strata, others say as few as possible). In economics, the paper I looked to is here, which seems to summarize literature related to clinical trials. In short, my question is: is it bad to have several strata with 1 person in them? Should I group these people in with another stratum?

P.S. In the paper I mention above, they also say it is important to include stratum indicators in the regression analysis to ensure the appropriate sized type-I error in the final analysis (i.e. regress outcome on treatment & strata indicators). They demonstrate this through simulation, but is there a reference (or intuition) that shows why these indicators are important theoretically?

My reply: I doubt it matters so much exactly how you do this. If you want, there are techniques to ensure balance over many predictors. In balanced setups, you have ideas such as latin squares, and similar methods can be developed in unbalanced scenarios. It’s ok to have strata with one person in them, but if you think people won’t like it, then you should feel free to use larger strata.

In answer to your other question about references: Yes, it’s standard advice to include all design information as regression predictors. We discuss this in chapter 7 of BDA, and I’m sure there’s some non-Bayesian discussion out there too. I think Pearl discusses this in his Causality book as well. In any case, I don’t give a damn about Type 1 error, but the idea is that the sorts of factors that you would be stratifying on are the sorts of things that can be correlated with your outcome, so if you don’t adjust for them, any imbalance in the predictors will lead to bias in your estimated treatment effects.

P.S. Bergman actually wrote “dummies,” but I couldn’t bear to see that term so I changed it to “ïndicators.”

Some care is likely worth considering

You have to actually carry out the trial with other people and more strata can seem more complicated in the minds of some.

The over stratification usually refers to the likleihood of less balance given more strata that attempted to balance more finely because of unfilled blocks that occur with randomization over time (recruitment over time).

And you do not want to confuse the systematic balancing methods Andrew is refering to with things like minimization that randomly/iteratively try to obtain some marginal type of balance (Stephen Senn has written on those problems).

K?