When there’s a lot of variation, it can be a mistake to make statements about “typical” attitudes

This story has two points:

1. There’s a tendency for scientific results to be framed in absolute terms (in psychology, this corresponds to general claims about the population) but that can be a mistake in that sometimes the most important part of the story is variation; and

2. Before getting to the comparisons, it can make sense to just look at the data.

Here’s the background. I came across a post by Leif Nelson, who wrote:

Recently Science published a paper [by Timothy Wilson, David Reinhard, Erin Westgate, Daniel Gilbert, Nicole Ellerbeck, Cheryl Hahn, Casey Brown, and Adi Shaked] concluding that people do not like sitting quietly by themselves. . . .

The reason I [Nelson] write this post is that upon analyzing the data for those studies, I arrived at an inference opposite the authors’. They write things like:

Participants typically did not enjoy spending 6 to 15 minutes in a room by themselves with nothing to do but think. (abstract)

It is surprisingly difficult to think in enjoyable ways even in the absence of competing external demands. (p.75, 2nd column)

The untutored mind does not like to be alone with itself (last phrase).

But the raw data point in the opposite direction: people reported to enjoy thinking. . . .

In the studies, people sit in a room for a while and then answer a few questions when they leave, including how enjoyable, how boring, and how entertaining the thinking period was, in 1-9 scales (anchored at 1 = “not at all”, 5 = “somewhat”, 9 = “extremely”). Across the nine studies, 663 people rated the experience of thinking, the overall mean for these three variables was M=4.94, SD=1.83 . . . Which is to say, people endorse the midpoint of the scale composite: “somewhat boring, somewhat entertaining, and somewhat enjoyable.”

Five studies had means below the midpoint, four had means above it.

I see no empirical support for the core claim that “participants typically did not enjoy spending 6 to 15 minutes in a room by themselves.”

Here are the data:

Figure-2

Nelson writes:

Out of 663 participants, MOST (69.6%) [or, as we would say in statistics, “70%” — ed.] said that the experience was somewhat enjoyable or better.

If I were trying out a new manipulation and wanted to ensure that participants typically DID enjoy it, I would be satisfied with the distribution above. I would infer people typically enjoy being alone in a room with nothing to do but think.

Nelson concludes:

If readers think that the electric shock finding is interesting conditional on the (I think, erroneous) belief that it is not enjoyable to be alone in thought, then the finding is surely even more interesting if we instead take the data at face value: Some people choose to self-administer an electric shock despite enjoying sitting alone with their thoughts.

He also asked the authors if they had any comments on his reaction that their paper showed a finding opposite to what they’d claimed, and the authors sent him a reply in which they wrote that they “were continually surprised by these results” which reminded me of our earlier discussion of how to interpret surprising results.

Reconciling the article and the criticism

It’s a challenge to go back and forth reading the original article, Nelson’s comments, and Wilson and Gilbert’s reply. I agree with Nelson that it seems incorrect to state that people did not enjoy being alone with their thoughts, given that more than two-thirds of the people in the study reported the experience to be “somewhat enjoyable” or better. On the other hand, Wilson and Gilbert point out that “The percentage who admitted cheating [doing other activities beyond just sitting and thinking] ranged from 32% to 54% . . . 67% of men and 25% of women opted to shock themselves rather than ‘just think’ . . .”

The resolution, I think, is that we have to avoid the tendency to think deterministically. There’s variation! As shown in the above histogram, some people reported thinking to be “not at all enjoyable,” some reported it to be “somewhat enjoyable,” and there were a lot of people in the middle. Given this, it’s not so helpful to make statements about what people “typically” enjoy (as in the abstract of the paper).

Finally, let me return to my original point about respecting the data. In their reply, Wilson and Gilbert write, “we believe the preponderance of the evidence does not favor Professor Nelson’s claim that most people in our studies enjoyed thinking.” Looking at the above graph, it all seems to depend on how you categorize the “somewhat enjoyable” response.

Perhaps it’s most accurate to say that (a) two-thirds of respondents find thinking to be at least somewhat enjoyable, and, at the same time, (b) two-thirds of respondents find thinking to be no more than somewhat enjoyable! The glass is both two-thirds empty (according to Wilson et al.) and two-thirds full (according to Nelson).

P.S. Nelson credits the paper to Science, the journal where it is published. I think it’s more appropriate to credit the authors, so I’ve done it that way (see brackets in the first paragraph of quoted material above). The authors are the ones who do the work; the journal is just a vessel where it is published.

P.P.S. Zach wins the thread with this comment:

I enjoy thinking, but I can do that any time. Put me in a room with a way to safely shock myself and I’ll take the opportunity to experiment.

15 thoughts on “When there’s a lot of variation, it can be a mistake to make statements about “typical” attitudes

  1. One of the things I don’t know is what sort of positivity bias there is.

    Consider a thought experiment: suppose people answered this “enjoyable” question every waking minute of every day for a year. And further suppose these people were distributed across the globe, so some fraction of them were, in objective terms, pretty miserable. Would this distribution still have most of the heft on the enjoyable side?

    Is this a “Lake Wogebon” effect, in which nearly all our moments are going to be rated above average in enjoyment, so that it’s appropriate to look at mediocre enjoyment results and infer that they really didn’t enjoy it?

    This sort of effect occurs in product testing. It’s unusual to find the average of any tested product below the middle of the distribution.

  2. The authors’ interpretation is wrong for a more basic reason: their scale is unipolar. Anything above “not at all enjoyable” means there was some positive amount of enjoyment. “Somewhat enjoyable” cannot be interpreted to mean “unenjoyable” as it was measured. The only way in which one can claim that most people did not enjoy the activity is if most people circled the response indicating “not at all enjoyable.”

    • Yes, it would have been good if they’d also done some comparisons:

      which do you think you would have preferred: “the thinking experience” or “watching an OK movie for the second time”, “the thinking experience” or “sitting on a crowded bus”… etc

      4 or 5 of these comparisons would have anchored the responses to things that people have a specific feeling for.

    • Brandon:

      I thought so, too. At first. But, at least, you need to consider the tendency of some people not to tick off extremes. I’d also doubt that all people would interpret the scale as unipolar. Your criticism is still valid but I don’t think the operationalization measures the underlying construct very well. The other Daniel’s suggesions might be good options to make the measurement more interpretable.

    • It’s the opposite for me. I don’t like to sit in a room with nothing to do, but think, but I would enjoy being shocked even less. I might rather do the thinking and enjoy it at least somewhat.

  3. This is instance of a common form of error in social sciences: covariance partialings generate “effects” that lack any real-world referent, & researcher then proclaims this disembodied essence to be what is “really driving” this or that real world phenomenon….

  4. Pingback: Briefly | Stats Chat

  5. Pingback: On Worse Reviews and Exaggerating Small Truths | datatante

Leave a Reply

Your email address will not be published. Required fields are marked *