Daniel Kassler, Ira Nichols-Barrer, and Mariel Finucane write:
Researchers often wish to test a large set of related interventions or approaches to implementation. A factorial experiment accomplishes this by examining not only basic treatment–control comparisons but also the effects of multiple implementation “factors” such as different dosages or implementation strategies and the interactions between these factor levels. However, traditional methods of statistical inference may require prohibitively large sample sizes to perform complex factorial experiments.
We present a Bayesian approach to factorial design. Through the use of hierarchical priors and partial pooling, we show how Bayesian analysis substantially increases the precision of estimates in complex experiments with many factors and factor levels, while controlling the risk of false positives from multiple comparisons.
Using an experiment we performed for the U.S. Department of Education as a motivating example, we perform power calculations for both classical and Bayesian methods. We repeatedly simulate factorial experiments with a variety of sample sizes and numbers of treatment arms to estimate the minimum detectable effect (MDE) for each combination.
The Bayesian approach yields substantially lower MDEs when compared with classical methods for complex factorial experiments. For example, to test 72 treatment arms (five factors with two or three levels each), a classical experiment requires nearly twice the sample size as a Bayesian experiment to obtain a given MDE.
They conclude:
Bayesian methods are a valuable tool for researchers interested in studying complex interventions. They make factorial experiments with many treatment arms vastly more feasible.
I love it. This is stuff that I’ve been talking about for a long time but have never actually done. These people really did it. Progress!
Andrew said:
Are you sure? It looks like NHST to me:
They give some lip service to “there is no theoretical reason why the posterior probability cannot be used directly as the outcome” and “dispensing with the intermediate significance calculation”, but do not explain to the reader what exactly they would do with this info.
The goal of the study was apparently:
Do you even needs stats to do that? This reminds me of the “riddle”:
You collected data from two groups and are interested in the difference using alpha = 0.05. Group A had a mean of 10, while Group B had a mean of 8. The p-value = 0.1. What is the average difference between group A and group B?
I’ve been exploring a similar area and trying to hash out appropriate simulations!
http://dilsherdhillon.rbind.io
This paper gives me some more ideas to think about. Thanks for posting this!
Uh oh. Here’s the post I mean to refer to.
https://dilshersinghdhillon.netlify.com/post/multiple_comparison/
“Bayesian methods are a valuable tool for researchers” that I fear would require a year or more of coursework for a classically-trained statistician like myself to become proficient at this level. Which I would actually love to do, if anyone is has a fellowship that covers my current salary and benefits, and the tuition costs to boot. :)
Michael,
I gave serious consideration to it last fall and into the winter. Decided that with most likely only 5-6 years left in my career it didn’t merit spending maybe 500-1,000 hours of my own time (in addition to work) to try and become somewhat proficient in techniques that would require major advocacy to even use in any meaningful way.
If I were 10 years younger it might a battle worth waging but then again 10 years ago it would have been even harder to attempt without the modern tools for Bayesian workflow that have become readily available of late.
If you know R or python it is not difficult at all to do something like this. If you don’t, then you will gain a very useful skill independent of anything having to do with bayesian stats.
I came across this link elsewhere on this site:
https://betanalpha.github.io/assets/case_studies/principled_bayesian_workflow.html
It seems to be worth a look. Many books and ebooks are available too, but it looks like mostly self study at this point.