Many years ago, a research psychologist whose judgment I greatly respect told me that the characterization of personality by the so-called Big Five traits (extraversion, etc.) was old-fashioned. So I’m always surprised to see that the Big Five keeps cropping up. I guess not everyone agrees that it’s a bad idea.
For example, Hamdan Azhar wrote to me:
I was wondering if you’d seen this recent paper (De Young et al. 2010) that finds significant correlations between brain volume in selected regions and personality trait measures (from the Big Five). This is quite a ground-breaking finding and it was covered extensively in the mainstream media. I think readers of your blog would be interested in your thoughts, statistically speaking, on their methodology and findings.
My reply: I’d be interested in my thoughts on this too! But I don’t know enough to say anything useful.
From the abstract of the paper under discussion:
Controlling for age, sex, and whole-brain volume, results from structural magnetic resonance imaging of 116 healthy adults supported our hypotheses for four of the five traits: Extraversion, Neuroticism, Agreeableness, and Conscientiousness. Extraversion covaried with volume of medial orbitofrontal cortex, a brain region involved in processing reward information. Neuroticism covaried with volume of brain regions associated with threat, punishment, and negative affect. Agreeableness covaried with volume in regions that process information about the intentions and mental states of other individuals. Conscientiousness covaried with volume in lateral prefrontal cortex, a region involved in planning and the voluntary control of behavior.
I could be glib and call this the new phrenology etc etc, but really maybe this stuff is great, I just don’t know. I guess the next step is to study how these traits interact with situations.
I think Walter Mischel is the guy who shot down the Big Five, decades ago, but I don’t see any reference to Mischel in the DeYoung et al. paper. Probably the field’s moved on since then.
On to the statistics
I sent Azhar the above and he replied:
My [Azhar’s] concern was more of a statistical nature actually. For example, the key finding here is that there are certain brain regions in which relative local volume correlates with a Big Five trait. The paper provides beta estimates and number of voxels for each of these regions and indicates that an overall threshold of p<.05 corrected was implemented. I [Azhar] want to get a better sense of how large this effect is and whether it is indeed physiologically and functionally significant. For example, it would help to know, according to the authors' computations, what would be the difference in medial temporal lobe volume between someone at the 75th percentile in Neuroticism and someone at the 25th percentile. And more importantly, how this "between-groups" variance compared to the mean variance within the Low Neuroticism or the High Neuroticism group. In other words, I'll accept that brain regions vary in size based on personality traits - but I want to know how this variance compares to the naturally occurring variance in brain region volume between people with similar personality traits. Or rather, some notion of what percent of the variation in relative volume of various brain regions can be accounted for by knowledge of personality traits. I [Azhar] can't seem to find any data along these lines in the paper, and without this, the findings don't seem very interpretable to me. My question for you as a statistician is whether this type of information that I've referred to should routinely be included in data analysis of this sort. Does it constitute an oversight or a flaw in either methodology or presentation to not include any of this information? Or am I missing some other statistical information provided in the paper that does address my concerns?
I agree completely. You always want to have some scale to understand variation. For example, in education they will often express effects as fractions of the standard deviation of the population of test scores. Or in political science we can interpret changes in voting and opinion as compared to previous changes. Statistical significance is not enough.
A couple things were encouraging
From their conclusions “If results in this field are to accumulate systematically”
and an earlier a reference to other work they claim to have replicated.
Also a link to supplementary data (not up yet?).
But the real risk here is that with say 20 groups working around the world on this topic – if we only get reports (and even raw data) from the one that had the most promising looking results – most of us have better things to read/think about.
K?
Walter Mischel's famous critique predated the Big Five. His critique was of the concept of a personality trait more broadly. If you ask around at Mischel's home department they'll probably tell you that Mischel won the argument, but that's not the mainstream view among personality psychologists elsewhere. In fact, I don't think there's a single mainstream view on traits or the Big Five, but I'd guess that many personality psychologists would endorse, at a minimum, "useful enough until a better model comes along." Some might go a lot farther. (Some of my own views on the Big Five are in this paper, if you're curious.)
As for the brain structure paper, I agree with Azhar. This is a case where there the null hypothesis is meaningless: any stable individual difference in behavior (regardless of whether that individual difference comes from the Big Five or some other model) is going to have some basis in something stable about the brain. As far as I can tell the associations are statistically significant (not voodoo) but the effect sizes are inflated by non-independence error. I know that some people have gotten pretty excited about these findings as validating the Big Five. But without an unbiased effect size estimate I'm not sure what to make of them.
I'm fairly certain that mischel couldn't shoot down the big five as that particular trait theory began to develop after mischel's key works regarding traits vs. situations in behavior. on that ground, trait theorists would likely stipulate that Mischel's wound was not fatal, as even key situationist theorists agreed that there exists some level of behavioral consistency across situations, even if at a level lower than earlier trait theories had stipulated (with the big five being the leading if not sole theory explaining that consistency). But I am only an interested observer, so an actual researcher on that field may have a more nuanced take on the matter.
A quick comment about the Big-5. A couple of years ago, I did some consulting with a psychologist who was developing new personality measures. The standard practice for validating the new measure was to give it as part of a battery to a sample of the target population, along with the Big 5, and other known measure that were similar to the target. I got the impression that it wasn't that the Big 5 were thought of as the answer to everything, but that it was a starting point that most people in the field understood. The burden of proof was to show that your proposed measures was something other than a composite of the 5 factors in the Big 5.
As far as the other stuff goes, I couldn't agree more. In my intro stat class, I'm teaching that one always needs to provide both a p-value and an effect size, as the two are measuring important but often orthogonal properties of the data (indeed, APA style requires both). I'd blame the reviewers for failing to hold their feet to the fire on that one.
Russell:
No, don't blame the reviewers! Reviewers are volunteers who are working for free, and I think it's a bit much to ask anybody to hold someone's feet to the fire without compensation.
The main complaint one hears about the big five is that there's an absence of theory explaining what's driving the different dimensions. It's all descriptive, or at best "finger in the wind" theorizing. At least that's (my perception of) the general perception of the problem (usual disclaimers apply).
That there's a relationship between personality and brain structures is unsurprising. That it can be detected is nice, though. That it can be detected with such a crude measure (bigger is better function – except for one bit of the brain and agreeableness) is perhaps surprising, but has been spotted before in (part of) taxi drivers' hippocampi in the context of spatial navigation.
This work belongs to the genre of trying to work out what cognitive processes are driving personality. The geography is not particularly interesting in itself. But with a spot of detective work linking to other studies, the geography gives clues about what might be going on there.
Concerning the reporting, yes, effects given in interpretable dimensions of volume would be nice! Even nicer would be brain imaging that exposes the neural representations and processes beyond just size – but early days…
The Big Five are the opposite of, say, Freud's theory. They weren't brought down from the mountain by one genius theorist, they just sort of emerged out of decades of research and argumentation. They tend to be kind of useful (but not earthshaking) and a little dull. They don't arouse much political controversy the way IQ does.
I certainly agree that it's difficult to know what to make of this. It's not generally possible to settle these issues in a single study, given current funding levels, the cost of imaging, etc. I have some minor concerns about the image analysis methods as well, although on a quick skim they didn't do anything grossly unreasonable. But I will disagree that the null hypothesis here is meaningless. Conceivably personality traits could have nothing to do with the shape/size of cortical structures at the scale imaged, and could have much more to do with patterns of connectivity or neurochemical differences. I'm sure someone more familiar with the area could come up with many legitimate alternative views.