A colleague was telling me that people are running lots of A/B tests at the company she works at, and people are interested in interactions. If they start grabbing things that are large and conventionally “statistically significant,” they see all sorts of random things. But when she fits a multilevel model, everything goes away. Also, the interaction models she fits look worse under cross-validation than simple additive, no-interaction models. What to do?
My quick guess is that what’s going on is that the standard errors on these interactions are large and so any reasonable estimate will do tons of partial pooling. Here’s a simple example.
Regarding what to do . . . I think you need some theory or at least some prior expectations. If the client has some idea ahead of time of what might work, even something as simple as pre-scoring each interaction from -5 (expect a large negative interaction), through 0 (no expectation of any effect), through +5 (expect something large and positive), you could use this as a predictor in the multilevel model, and if something’s there, you might learn something.
This sort of thing comes up all the time in applications. On one hand, interactions are important; on the other hand, their estimates are noisy. The solution has to be some mix of (a) prior information (as in my above example) and acceptance of uncertainty (not expecting or demanding near-certainty in order to make a decision).
A quick google search turned up this article from 2016 by Kelvyn Jones, Ron Johnston, and David Manley, who try to do this sort of thing. I haven’t read the article in detail but it could be worth looking at.
I suppose the causal machine learning literature also provides a good way of tackling this problem, i.e. split your data, use a machine learning approach to learn what interactions are important, and then estimate the interactions on the other half of your data.
It’s amazing how powerful sample splitting is. This is one of the most important ideas in statistics that I wish I learned more about when I was in grad school for economics years ago.
Josh:
I agree that splitting the sample is a way of tackling this problem, but I disagree with your claim that it’s a good way. The trouble is that interactions are already difficult to estimate with all the data (see paragraph 4 of the above post). Estimating interactions with only half the data is even more difficult!
I would push back against the claim that splitting the sample makes it harder to estimate interactions. Cross-validation* would not be used to estimate the interaction, but to tune the hyperparameters. Once the hyperparameters have been tuned and a good fit of the model has been established, the model would be estimated using the full data set. Finally, the interaction effect is estimated with the model trained on the full data set. This is more work, and in particular more computationally intensive, but from where I stand at the moment I would disagree that the interaction is estimated with fewer data points than available.
*Assuming the data set is not so large that we can conveniently split it into a training and a validation set, e.g. if the data set is so large that we cannot reasonably work with the whole thing. I heard that at one point Google was storing something like 3TB per second; they only use tiny fractions of their data for research, because a tiny fraction is still a huge amount of data!
I agree that by refitting the final model on the full dataset the estimates of interactions wouldn’t suffer from using less data. But if I’m understanding this correctly one of the hyperparameters tuned using cross-validation would be a selection of interactions to include in the final model. In this case we’ll be choosing whether to include an interaction using only a subset of the data. I guess in some cases we may end up with the wrong set of interactions in the final model even if the final estimate of those interactions is less noisy.
it sounds like they’re running a conventional A/B test and then looking at a bunch of interactions ex post. wouldn’t designing stratified experiments specifically tailored to the experiments they’re interested in (i.e., samples across strata calibrated to have power to detect reasonable sample sizes) work?
This comes up a lot at places I’ve worked at. Agreed that (a) standard errors are generally large on interactions they don’t tend to be super useful and (b) they are often not pre specified. What they are great for is a mechanism for hypothesis generation and potentially future a/b tests / product changes.
Side note: a/b test interactions can actually mean two different things in this context — how treatment effects vary across groups of exposed users but also how a/b tests interfere with each other. For the latter, Microsoft posted this article based on their platform experience: https://www.microsoft.com/en-us/research/articles/a-b-interactions-a-call-to-relax/. Essentially they look at the distribution of p-values from a meta analysis that searches across the 10000s of overlapping experiments. They conclude that the distribution of pvalues looks basically uniform and therefore interactions across experiments aren’t typically a big deal. I generally tend to agree unless one can specify ex-ante the underlying mechanism that would generate some sort effect across a/b tests.
Anon:
I recommend looking at all the interactions using a multilevel model, following this general philosophy (with free link here).
I think one of my favorite things about multilevel models is how they allow for incorporating additional data/priors as predictors.
One recent example for me is how people in my department often feel surveys are a fairly useless addition to an experiment, since they often only might only use them in separate, supplemental analyses. Saddest of all, I’ve seen some economists do complex individual-level analysis like mixture models as a separate analysis per person! So much data is left unused!
Are there any good posts or articles whether including individual-level predictors can have much effect on treatment estimates?
I have once noticed an interesting thing. In biology, you often make multiple comparisons – for example, you test thousands of genes trying to understand differences between two conditions. Now, understanding what it means is hard, and one way of dealing with it is to use gene set enrichment analysis. This is basically a test to see whether the “significant” genes are somewhat related to one or other predefined set of genes (in its simplest variant, it is a hypergeometric test).
Now if you have a 2×2 design, say wildtype vs knockout and control vs treatment, interaction is what you should do. However, coupled with correcting for false discovery rate, testing for interactions is often not very satisfactory. Basically, very often you simply get nothing. What people do in such a case is to run the tests in one group (wildtype) and compare the results with the tests in the other group (knockout). And then, presto! we have N genes “specific” for the treatment in knockout.
Of course, significant and non-significant are not significantly different, but that is not even my point. The interesting thing happens when you run gene set enrichment on these genes.
You see, if a gene is significant in one, but not significant in the other comparison, then quite likely it is a false negative in the second comparison. The high throughput methods with correction for false discovery rate such as Benjamini-Hochberg have very high false negative rates. That is, this gene *is* regulated in the given context. Therefore, gene set enrichments look very convincing and can be used to weave a biological story. Some years ago I wrote a paper where I looked at two groups of people. In both of them there were some COVID patients. Following this (incorrect) methodology we found different immunological mechanisms “characterizing” both groups. Of course, in reality the groups were no different at all, and patients were simply sampled from a larger population.
I have no good advice to my experimentally working colleagues other than to collect more samples in order to get more oomph and to very carefully set up an a priori hypothesis, so we can limit our testing.
Weiner 3rd, January, Benedikt Obermayer, and Dieter Beule. “Venn diagrams may indicate erroneous statistical reasoning in transcriptomics.” Frontiers in Genetics 13 (2022): 818683.