Kevin Lewis pointed me to an article that said, “We use panel survey data from the National Longitudinal Study of Adolescent to Adult Health in the United States to examine the relationship between interviewer-rated in-person physical attractiveness and actual cardiometabolic risk.”
My response: If someone were to sneak into Add Health HQ and replace the data there with random numbers and not tell anyone, . . . the numbers of papers publishing from Ad Health data would not change.
Just to be clear, I’m not saying that Add Health is useless, nor am I saying that its data are equivalent to random numbers. I’m just saying that, at this point, people are gonna mine the data indefinitely, and the limiting factor is not the data, it’s the creativity of researchers and the patience of readers. If someone were to sneak into Add Health HQ and replace the data there with random numbers and not tell anyone, I think the numbers of papers publishing from Add Health data would not change—but I do think the average quality of these papers would be lower! It’s a good thing that Add Health has real data.
This highlights that sometimes the best way to create a public good by sharing data is not to share it all at once, but to share some of it, then let people think carefully about how to analyze it (and, e.g., preregister some analyses), then share the rest.
The researchers who managed the Upworthy Research Archive did exactly this and I think it was a great idea: https://upworthy.natematias.com/
(Of course, some of this already implicitly happened with AddHealth because of the multiple waves etc.)
Is this a general problem which all easily available big survey data sets have (like PSID, GSS, ANES)? At first, the questions are examined for which the data was initially surveyed for, whereas after while the data are mined for other patterns. So, would you recommend to avoid to collect large general purpose surveys in favor of smaller surveys designed for specific questions. I use the term survey here, but it should be interchangeable with other data sources, i guess.