A couple people pointed me to this research article, “Facial recognition technology can expose political orientation from naturalistic facial images,” by Michal Kosinski, which reports:

A facial recognition algorithm was applied to naturalistic images of 1,085,795 individuals to predict their political orientation by comparing their similarity to faces of liberal and conservative others. Political orientation was correctly classified in 72% of liberal–conservative face pairs, remarkably better than chance (50%), human accuracy (55%), or one afforded by a 100-item personality questionnaire (66%).

The result seemed plausible to me. As Kosinski writes, a face contains lots of information; it’s not just bone structure. It seems that people of different social classes have different-looking faces, so I could imagine this to be true of political orientation as well.

I wan’t quite sure how to think about the 72% result—is that high or low?—so I asked a colleague who prefers to remain anonymous, who told me this:

Regarding the paper, I’m not exactly sure what to make of the controversy. He released the underlying data, so I ran a couple of models [see script at end of this post] and here are the results:

AUC [area under the curve, a measure of classification accuracy] of simple demographic model: 0.63
AUC of model with more observable traits: 0.65
AUC of model with observable traits + personality: 0.72

It looks like ethnicity doesn’t include “Hispanic” for some reason, which is probably hurting the performance of the demographic model. If you use demographics + directly observable traits in the dataset (e.g., smiling, facial hair, glasses, etc.), you get 65% AUC. When you throw in personality (based on a survey), you get to 72% AUC, which is the same as what is reported in the paper based on the face recognition algorithm alone.

Personality is not directly visible, so the face algorithm is picking up on something more than the obviously observable traits. But I think the gap between 65% and 72% is not that big, and likely can be closed if you have more fine-grained ethnicity (e.g., Hispanic) and account for things like wearing make-up. Perhaps throwing in a few interactions and using a non-linear model would also close the gap a bit more.

As another point of comparison, I computed performance of a simple demographic model for 2012 presidential vote choice (based on exit poll data that included Hispanic ethnicity), and got an AUC of 68%.

So (unsurprisingly) there is real information about people’s political leanings that is discernible from how they look and choose to present themselves. But I think the paper itself over-claims (e.g., only hints at the comparisons above), and strongly suggests something more subtle is going on.

That’s a somewhat critical take, but I’d like to thank Kosinksi for posting his data.

I got some more feedback on the paper from political scientist Brian Sala, who sent comments to Jeff Lax who passed them on to me. Here’s Sala:

This is really interesting. Limited, because the model is about picking the liberal (conservative) in a liberal/conservative pair. But the model should be useful for classifying individuals. I would have liked to have seen it applied via an ordered probit/logit to model self-assessed ideology (the typical, 5-pt scale from very liberal to very conservative) instead of
this binary classification.

Good point! A more precise measure should allow us to learn more.

Sala continues:

I [Sala] am a little unclear on the out-of-sample classifications here from my quick skim (the main model, I think, classifies within sample which in a pair is the liberal or conservative, by comparing facial attributes of the individuals drawn from the sample to the average lib/conservative attributes in the sample). Still, cool stuff: “In other words, a single facial image reveals more about a person’s political orientation than their responses to a fairly long personality questionnaire, including many items ostensibly related to political orientation (e.g., ‘I treat all people equally’ or ‘I believe that too much tax money goes to support artists’).”

What I don’t like is the reliance on central tendencies for the collapsed “liberal” and “conservative” groups. Because one would think the tool could be used to compare all pairs from the sample to choose the “more liberal” of the pair to get a rank ordering of the whole sample, a la Groseclose and Milyo’s approach to ordering media outlets or the power rankings of sports teams.

It seems to me (again, based on a quick skim) that this paper’s approach, by comparing individuals in a lib/conservative pair, is asking which in the pair is most like a “centrist” lib or “centrist” conservative. If the recovered geometry of facial features is multidimensional, “very liberal” and “very conservative” individuals could be closer to each other than they are to their respective ideological centroids, even as each is closer to the “right” centroid than the “wrong” one (or rather, one in the pair is closer to the reference centroid, leading to the classifications in the pair). Or worse, the “very liberal” person in the pair could be closer to the conservative centroid than the “very conservative” person AND closer to the liberal centroid (or vice-versa). Implicit here is two pairs of points in the facial characteristics space: the two individuals and the two centroids.

I’m doubtful about this claim from Sala. I’d guess there’d be more of a linear trend going on, with moderate liberals and moderate conservatives closer to the center of the distribution. But I don’t really know; this is just my guess.

Sala continues:

I [Sala] presume that the model is projecting the two individuals’ locations on to the line through the two centroids. If the model is “right”, the two individuals will be ordered the same way as the two centroids (liberal projected to the left of the conservative on the line through the centroids). The two individuals could be “outside” their respective centroids, inside, both to one side of a centroid, or straddling a centroid. If the left/right order is “correct but both are projected to one side, you get classification errors (the liberal is further from the liberal centroid than is the conservative), whereas you get correct classifications if both project in between. If they straddle one centroid, it depends on the reference category (are you comparing each to the liberal centroid or the conservative centroid? If comparing to the liberal and they straddle the liberal, I think the model will classify the “closer” projection as liberal, leading to some classification errors. If comparing to the conservative centroid, no classification errors in this case.) Again, my quick read.

Again, I’m skeptical of Sala’s conjectures but who knows? and so I thought I’d share this with you.

Finally, someone forwarded me a copy of this email that was floating around Stanford (where Kosinski works):

Dear colleagues,

Several members of the ACM US TPC on AI and Algorithms are always concerned on the unethical use of AI, even more when it comes from top universities like Stanford, given its reputation and influence. The last example, but not the only one, is more modern phrenology:

Facial recognition technology can expose political orientation from naturalistic facial images, M. Kosinski, Scientific Reports, 2021. (and is not facial recognition, is facial biometrics).

The argument that this is to warn about the bad use of ML, is not a valid one for publishing this type of research as this is pseudo-science (the backslash was worse when the same approach was used for sexual orientation). Surprisingly, Stanford’s IRB approved this research, so they are not taken care of the ethical aspect. Of course Nature also shares the responsibility. For this reason we have already contacted all the stakeholders involved to avoid more of this in the future.

However, I believe that we as computer scientists also should be concerned about unethical science and bad press in AI (and for you, the same for Stanford, especially with the HAI initiative). So this personal action is just to make sure you are aware of this, in case you were not.

I disagree with this “Dear colleagues” letter on many levels. OK, let me be clear. I fully support the freedom of the author of this letter to send it around; I’m not saying such letters should not be allowed; I’m just saying I disagree with the substance.

First, I don’t think it’s helpful to refer to this as “phrenology.” It’s a statistical analysis of photos. Similarly, I don’t see why they call it “pseudo-science.” The work of that ESP guy at Cornell, or the Pizzagate guy, or the beauty-and-sex-ratio research that we’ve discussed on this blog . . . I could see calling all of that pseudoscience, or, at least, really really bad science. But this facial recognition paper seems legit. I don’t like the idea of calling a paper “pseudoscience” just because you find it annoying.

Second is the ethical question. I have mixed feelings on this one. Maybe the analysis shouldn’t be done, as there’s something Big Brotherish about it—but as Kosinski points out, companies and governments are already doing such things, so it’s not clear that academics shouldn’t be doing it too. On the other hand, if it’s really something that shouldn’t be done, then no point in academics leading the way. Overall I don’t think I’d consider this to be unethical research, but that’s a matter of opinion; I can’t really say that the letter writer is right or wrong on this one.

Third is the statement, “Surprisingly, Stanford’s IRB approved this research.” I’ll agree that Stanford researchers can have ethical problems; for example there was this mailer they sent to voters in Montana, and Stanford also employs the business school professor who notoriously told 240 different restaurants that “Our special romantic evening became reduced to my wife watching me curl up in a fetal position on the tiled floor of our bathroom between rounds of throwing up.” So, sure, there are some studies that never should be approved. For this facial recognition study, though, on what ground would Stanford not approve it? Because someone thinks it’s evil? I really don’t like the idea of the IRB being used as some sort of political correctness filter, and I’m bothered that these people think the job of the IRB is to stop research from being done, just because somebody finds it politically objectionable.

P.S. Here’s my colleague’s R script:

library(tidyverse)
library(ROCR)

auc <- function(model) {
  pred_ROCR <- prediction(predict(model), model$y)
  auc_ROCR <- performance(pred_ROCR, measure = "auc")@y.values[[1]]
  auc_ROCR
}

round_any = function(x, accuracy, f=round){f(x/ accuracy) * accuracy}

# load the original face data and restrict to americans
# downloaded from https://drive.google.com/file/d/1I3QMFzb12-i6Mu9lSD1xxymm5nmQyqm9/view?usp=sharing
# note that race/ethnicity is classified as 'asian', 'black', 'india', 'white'
# in particular, hispanic ethnicity is not included
faces <- load('faces.RData')
faces <- tibble(d) %>% 
  filter(country == 'united states') %>%
  select(-userid, -starts_with('pol_'), -database, -age) %>% 
  mutate(age_bin = factor(round_any(age.value, 10))) %>%
  drop_na()

# exit poll data from 2012 presidential election
survey <- read_tsv('https://5harad.com/mse125/assets/hw6/survey.tsv')

# fit a simple logistic regression based on sex, race, and age bin
model_demo <- glm(pol == 'liberal' ~ 
                      gender + ethnicity.value + age_bin, 
                    data = faces, family = 'binomial')

# fit a model with more observables
model_obs <- glm(pol == 'liberal' ~ 
                    gender + ethnicity.value + age_bin +
                    facial_hair +
                    emotion.sadness + emotion.neutral + emotion.disgust + 
                    emotion.anger + emotion.surprise + emotion.fear + emotion.happiness +
                    headpose.yaw_angle + headpose.pitch_angle + headpose.roll_angle + 
                    smile.value +
                    left_eye_status.normal_glass_eye_open + left_eye_status.no_glass_eye_close +
                    left_eye_status.occlusion + left_eye_status.no_glass_eye_open +    
                    left_eye_status.normal_glass_eye_close + right_eye_status.dark_glasses +
                    right_eye_status.normal_glass_eye_open + right_eye_status.no_glass_eye_close +
                    right_eye_status.occlusion + right_eye_status.no_glass_eye_open +    
                    right_eye_status.normal_glass_eye_close + right_eye_status.dark_glasses,
                  data = faces, family = 'binomial')

# fit a model with observables and personality (openness)
model_personality <- glm(pol == 'liberal' ~ 
                           gender + ethnicity.value + age_bin +
                           facial_hair +
                           emotion.sadness + emotion.neutral + emotion.disgust + 
                           emotion.anger + emotion.surprise + emotion.fear + emotion.happiness +
                           headpose.yaw_angle + headpose.pitch_angle + headpose.roll_angle + 
                           smile.value +
                           left_eye_status.normal_glass_eye_open + left_eye_status.no_glass_eye_close +
                           left_eye_status.occlusion + left_eye_status.no_glass_eye_open +    
                           left_eye_status.normal_glass_eye_close + right_eye_status.dark_glasses +
                           right_eye_status.normal_glass_eye_open + right_eye_status.no_glass_eye_close +
                           right_eye_status.occlusion + right_eye_status.no_glass_eye_open +    
                           right_eye_status.normal_glass_eye_close + right_eye_status.dark_glasses +
                           ext + neu + ope + agr + con,
                         data = faces, family = 'binomial')

# fit a simple logistic regression based on sex, race, and age
model_exit_poll <- glm(vote == 'A' ~ sex + race + age, data = survey, family = 'binomial')

# compute AUC of the various models
cat('AUC of simple demographic model: ', auc(model_demo), '\n')
cat('AUC of model with more observable traits: ', auc(model_obs), '\n')
cat('AUC of model with observable traits + personality: ', auc(model_personality), '\n')
cat('AUC of demographic model based on survey data: ', auc(model_exit_poll), '\n')