Jay Kaufman writes:

This article by Nicholas Wade, “Genes Tied to Gap in Treatment of Hepatitis C,” notes that 55% of European Americans respond favorably to standard hepatitis C treatment, but only 25% of African-Americans. The authors of the new article in Nature assert that this is due in part to an allele they discovered, which is more common in European Americans (based on a survey of Duke University students!). The authors state that 58% of the ethnic difference is due to the differential distribution of this one allele. An interesting part of this story from the point of view of statisticians would be the basis for this 58% number. In the supplementary material available with the article, the authors explain their statistical analysis as follows:

Logistic regression does not have a direct equivalent to the R2 that is found in ordinary least squares (OLS) regression that represents the proportion of variance explained by the predictors. However, it is possible to use an analog, so-called a pseudo-R2, to mimic the OLS-R2 in evaluating the goodness-of-fit and the variability explained, which is the approach we used (ref 14). Using this approach we estimated that rs12979860 could account for 58% of the ethnicity-explained variability by estimating the difference between the expected variability if the IL28B SNP does not account for the variability explained by ethnicity at all, and the observed variability explained by both ethnicity and rs12979860.

While these details are somewhat vague, I trust that you will join me in finding this very suspicious.

The pseudo-R2 simply compares the log-likelihood from the null model (only an intercept) to the log-likelihood from the full model (all covariates included). I would call neither the R2 nor the pseudo-R2 a measure of “goodness of fit”, but at least the R2 in a linear model does mean something straightforward. The pseudo-R2 in a logistic model, however, seems to me to have no straightforward interpretation at all, and I was under the impression that no serious statistician uses this statistic.

The authors wrote that they used this statistic to ascertain that the exposure (allele) “could account for 58% of the ethnicity-explained variability” and yet the pseudo-R2 does not measure variability at all. They claim to have come up with the 58% number by “by estimating the difference between the expected variability if the IL28B SNP does not account for the variability explained by ethnicity at all, and the observed variability explained by both ethnicity and rs12979860.” I am not sure I quite follow that, but it sounds like they are computing the pseudo-R2 statistic twice, once for a model that contains ethnicity and the allele of interest, and once for the model that contains only ethnicity. Perhaps one of these numbers that results is 58% as big as the other, or something like that. If this is indeed what they did, I see no logical connection between this analysis and their claim that 58% of the ethnic disparity is due to the differential distribution of the allele. I have to assume that Nature has careful statistical review, but this doesn’t make a lot of sense to me, based on what I can glean from this description.

My reply: I’ve never used pseudo-R-squared myself, but I can’t speak for the general population of “serious statisticians” here. I know that a lot of statisticians don’t like regular R-squared, but I find it helpful sometimes (see graphs on page 42 of ARM) and even wrote a research article (with Iain Pardoe) on the topic. So I’d be wary about slamming pseudo-R-squared in general terms.

I also don’t know enough about genetics to try to interpret the 58%. But, yes, my guess is that this difference isn’t really 58% of something. Maybe it makes sense as some approximation, though. My recommendation in this sort of situation is to forget about log-likelihoods and R-squares and just attack the comparison more directly, perhaps through an ROC curve or some similar approach.