Here’s his idea:
If I were a student, I’d do a study on how Science Talent Search judges are biased. That way, they can’t reject it, otherwise it’s self-confirming.
That’s a great idea! Maybe it’s possible to go meta on this one by adding some sort of game-theoretic model or simulation of talent search submission and judging?
The background was that Lewis wrote this to me:
The Science Talent Search needs more social science. I could only find one such study in this year’s 40 finalists: “Evaluation of Gender Bias in Social Media Using Artificial Intelligence.”
I replied that I’ve actually advised a few local high school students on social science projects for this competition, and one of them made it into the final round, a few years ago! I think it was based on a survey she did; I don’t remember, as it was entirely her idea, and my role was only to supply some feedback. Each year, some high school students come to me asking to be advised on a socisl science project. I sometimes have an idea that I suggest but usually they do their own thing. It’s surprisingly difficult to come up with good ideas! The best ideas I’ve had involve collecting data, and collecting data is hard. For example, a couple years ago after that article about evictions (later made into a book) appeared in the New Yorker, I suggested to a student that he track down some actual data on evictions: time series on the number of evictions in the U.S., or in some states, or somewhere. But that wasn’t so easy to do. What students want to do is collect a survey or run a regression. I guess that the easiest way to make progress would be to run a computer simulation.
And then Lewis responded with his suggestion given above. We’ll see what happens.
My first thought was that it is not easy to do social or behavioural scientific research, at least with live humans. The logistics can be daunting even for seasoned investigators (or desperate graduate students).
I am less familiar with work using existing data bases but it has seemed that a data base that is designed, say, for financial control is often unfriendly when one wants to answer a social question or even aggregate the data in an unexpected manner.
> That way, they can’t reject it, otherwise it’s self-confirming.
Well I put in a grant app to clinical research funder in 2004/5 requesting funds to assess the resources required to reproduce a random sample of randomized trial published results that they had funded (and so could request the raw data though may not force access) as well as estimate the success rate.
Decision: Not funded, given preliminary review along with an accusation by one reviewer of having an ulterior motive to support a junior statistician to do other or additional work.
These logical encouragements for those who have the gold to give some to projects that may threaten their authority – usually don’t work.
Actually a preliminary meta-analysis might work will – search for related studies, find at least a few, extract the treatment effect estimates and standard errors and make some plots.
And they will likely learn what a mess published literature is.
(Did this for rehab students in 1988 and they jumped on it doing searches on their faculty’s publication and learning how bad their faculty were at research. After that class, they found someone else to teach their research methods course.)
Good idea!