Someone who knows that I hate the so-called Fisher exact test asks:
I was hoping you could point to a Bayesian counterpart or improvement to “Fisher’s exact test” – for 2 x 2 categorical, contigency tables with possibly very small numbers (too small to do a chi-square.) I see that you had a blog post on it before (1) but there are several issues i’m unclear about:
(i) What would a full applied Bayesian analysis look like of this type of problem, in general? I have seen one beta-binomial like analysis but never in practical/applied examples. Any practical examples you may have for this, e.g. papers or code examples you’ve used in teaching, would be great.
(ii) What if we add the twist that the data from the two populations for our 2 x 2 test is paired? e.g. we have several male and several female patients, and the two conditions are drug / no drug. But, each male and female are paired as they are twins (which breaks the independence of the samples obviously.) How is this modeled from a Bayesian perspective?
(iii) Less important: when in practice is it ok to use Fisher’s exact test if you’re open to Bayesian analysis? ‘Never’ is a reasonable answer but i’d like to understand practical reasons why you think this. Finally, if all of our data counts are greater then 10, do you think its legitimate to use a chi-square?
(i) The basic analysis is pretty simple, it goes like this:
y1 ~ Binomial (n1, p1)
y2 ~ Binomial (n2, p2)
We need a prior distribution on (p1,p2), and we usually assume that n1,n2 provide no information about p1,p2. (This latter point depends on the design of the study, but I’m keeping it simple here.) What’s a good prior distribution depends on the problem, but in many cases, a simple uniform distribution on (p1,p2) will be fine. Whatever your prior is, you then throw in the likelihood and you get posterior inference for (p1,p2). Draw 1000 simulations and then use these to get inference for p1-p2. That’s it. With moderate or large sample sizes, this is basically equivalent to the standard t-test.
If you have many tables, you can set up a hierarchical model for the p’s. We have an example near the end of chapter 5 of Bayesian Data Analysis.
(ii) With paired data, you can fit a logistic regression. Call the data y_ij, where i=1 or 2 and j is the index for the pairing. Then you can model Pr(y_ij=1) = invlogit (a_i + b_j), with a hierarchical model for the b_j’s, something like b_j ~ N (mu_b, sigma_b^2), with weakly informative or flat prior distributions on mu_b, sigma_b.
(iii) The only case I could even imagine using the so-called Fisher exact test is if the data were collected so that the row and column margins were both pre-specified. The only example I can think of with this design is Fisher’s tea-tasting experiment. In all cases I’ve seen, at most one margin is preset by design. Also, I’d never do a chi-squared test in this setting. See chapter 2 of ARM for an example where I thought a chi-squared test was OK.