Jennifer Hill writes:
The Society for Causal Inference is excited to announce the launch of the 2022 ACIC Data Challenge! This is an annual (modulo covid) causal inference competition, and a team at Mathematica led by Mariel Finucane and Dan Thal is running the show this year.
Here’s how it works. The Mathematica team generated thousands of policy-evaluation-like datasets and baked in hidden causal impacts. Participants will compete to estimate those effects, and in so doing will help to identify those cutting-edge methods best suited for evaluating which social policies move the needle for the individuals and communities they serve. There’s lots more info about the competition here.
This is great stuff. Nothing magic about a competition, but it can help focus things, and the people involved are interested in applied outcomes for real problems. I also like the collaborative model, which I think is much better than the usual story of research teams hyping their pet ideas.
Do you see how your policies might possibly negatively impact an outlier such as myself, when you arbitrarily reward contestants for uncovering effects you baked in? How do you know winners just haven’t figured out how you think about manipulating data to find effects? How far removed from my personal, actual, non-ergodic life are your statistical stories, and what policies that impede me unintentionally are you contributing to?
Rsm:
That’s a funny way of putting things! I’d say that if you don’t buy the premise of this competition, then you don’t have to play. Kinda like if you aren’t interested in winter sports you don’t need to watch the olympics right now. I guess you might reply that our tax money is (indirectly) funding this competition, but then again our tax money funds the olympics too.
Getting to the topic at hand: No, I don’t know that the research resulting from this sort of competition will ultimately improve education policy. Or, even if does, it presumably won’t improve everyone’s education, and it could be that students who are similar to you in some ways will be among those who end up with worse outcomes. All I can say is that this sort of question—variation in treatment effects, looking at effects on individuals, not just on averages—is a central topic of modern causal inference and has been so for awhile. So, to the extent that you’re interested in evaluating policies in this way, I think this sort of competition is going in the right direction.
Regarding specifics: I think that after the competition is over, the team that constructed it will publicly release the details of what they did. So at that point in the not-so-distant future, you can take a look, and, if you see problems with it, you can publish your criticisms. That could be useful.
My daughter just sent me this meme today:
“I’m just sayin’, everyone that confuses correlation with causation eventually ends up dead.”
It’s pretty amazing when kids grow up! Kinda sad, but still one of life’s joys.