“How We’re Duped by Data” and how we can do better

Richard Juster points us to this press release by Laura Counts from the business school of the University of California, promoting the work of Leif Nelson, one of the authors of the modern classic paper on “false-positive psychology” and “researcher degrees of freedom.”

It’s great to see this sort of work get positive publicity. I actually would’ve liked to see some skepticism about the skepticism in the press release, maybe a quote by someone saying that Nelson et al. have gone too far, but given that this sort of scientist-as-hero reporting is out there, I’m glad that it’s being wielded in the service of what I consider to be good science.

Just to be clear, I’m fully in support of Nelson’s work, and I suspect that I’d strongly disagree with any criticism that would take the this-science-reform-business-has-gone-too-far position, but just on general principles it’s my advice for reporters and publicists to present a more rounded view.

Just by analogy, if you were writing an article about someone who’s promoting evolution, the appropriate alternative perspective might be, not to interview a creationist, but to interview someone who works on evolution and can talk about open problems.

I’m not quite sure what’s the right alternative perspective to offer alongside of Nelson here. I wouldn’t recommend they interview Brian Wansink or Robert Sternberg! Maybe step back and consider someone whose research ideas are orthogonal to the open-science movement, someone who’s not opposed to open science but thinks there are more important issues to consider. For example, someone like Nancy Cartwright or Angus Deaton, who have argued that experimental methods are overrated and that social science needs stronger theory. That could be a good countervailing perspective that Nelson could then react to, and it would make it a stronger article.

Don’t get me wrong: I like this press release, and I also know that people who write these things are busy. It’s just interesting to think about this going forward, about how to offer multiple perspectives when writing about a controversial topic.

“Data 101 for Leaders: Avoid Cherry-Picking”

In a sidebar, the article offers advice from business-school professors Leif Nelson and Don Moore, “questions when someone presents you with data they claim proves something”:

1. How did you decide how much data you would collect?

2. Did any other analyses yield different results?

3. Did you measure any other variables worth discussing?

4. Did these results surprise you or were they expected at the outset?

This is fine general advice, but I also have some concerns with which regular blog readers will be familiar.

First, my problem with the advice to “avoid cherry picking” is is that it’s possible to distort your results by following forking paths without ever consciously “cherry picking” or “p-hacking.” The other problem with this advice is that it can instill a false sense of security, if a researcher thinks, “I didn’t do any cherry picking, therefore I’m ok.” Remember, honesty and transparency are not enuf. I fear that talk of “cherry picking” can make people think the problem is with other, bad researchers.

Going on to the other questions:

1. It’s ok to ask how you decided how much data you would collect. But I think in general there’s been way too much emphasis on sample size and significance levels, and not enough on design and data collection. So I’d like question #1 to be: How did you decide what you were going to measure, and how you would measure it?

2. Did any other analyses yield different results? Yes, definitely. Different analyses will always yield different results. We have to be careful (a) not to dichotomize results into “same” and “different” as this can be a way of adding huge amounts of noise to our results, and (b) not to expect that, if a result is real, that all analyses will show it, as that gets us into Armstrong territory, as has been discussed by Gregory Francis and others.

3, 4. Overall, I think it’s a good idea for any research team to situate their work within the literature. The question is not just: Did you perform any other analyses of your data? or Did you measure other variables? or Did these results surprise you? It’s also: What sorts of analyses are performed in other studies in this literature?, What other variables have been studied?, What other claims have been made? Again, forking paths is not just what you did, it’s what you could’ve done.

8 thoughts on ““How We’re Duped by Data” and how we can do better

  1. I used to tell my students to”smell the data”, that is, to see how the data was collected, to ask the people who collected it what they knew about its value and problems, to see if the same standard was used by everyone who was involved in the data collection process.

  2. This issue is entirely due people trying to replace replication with p-values and predictive skill with peer review.

    It doesn’t work, and it would have been very surprising if it did work.

  3. All four points are good ones. I especially appreciate, “So I’d like question #1 to be: How did you decide what you were going to measure, and how you would measure it?” Measures are often chosen “because that’s the way we’ve always done it”, but that’s not a good answer. Choosing measures needs to be tailored to each situation individually.

  4. These are good questions that editors should ask reviewers to ask. Consider an example from the Jan. 8 issue of Science, at p. 292. Here is the editor’s summary:

    “Humans often focus on how different we are from other animals. Certainly, there are some important differences, but more and more we are learning that we differ by degree rather than kind. We see these similarities most clearly when we look at human populations that live a more traditional, foraging lifestyle. Barsbai et al. compared more than 300 such foraging human populations with mammal and bird species living in the same environment across a wide array of environmental conditions (see the Perspective by Hill and Boyd). They found that all three groups converged with regard to foraging, social, and reproductive behaviors. Thus, adaptation to environmental selection shapes similar responses across a wide diversity of life forms.”

    As a biologist of sorts, I think social scientists should be more conscious that humans are herd animals than most seem to be, so I’m basically sympathetic to this line of inquiry. However, this article leaves me cold; some of the findings are hardly remarkable (e.g., humans, other mammals, and bird eat more fish when they live where there are a lot of fish, like along coasts), and the authors fail to explain how they picked the 15 behaviors that they did, or to disclose whether other behaviors don’t fit their pattern (they say they explain the choices in the supplemental data, but really just comment on the variables they chose).

Leave a Reply to Martha (Smith) Cancel reply

Your email address will not be published. Required fields are marked *