This is Jessica. Today I start teaching a new seminar course I created to CS grad students at Northwestern, called Explanation and Reproducibility in Data-Driven Science.
Here’s the description:
In this seminar course, we will consider what it means to provide reproducible explanations in data-driven science. As the complexity and size of available data and state-of-the-art models increase, intuitive explanations of what has been learned from data are in high demand. However, events such as the so-called replication crisis in social science and medicine suggest that conventional approaches to modeling can be widely misapplied even at the highest levels of science. What does it mean for an explanation to be accurate and reproducible, and how do threats to validity of data-driven inferences differ depending on the goals of statistical modeling? The readings of the course will be drawn from recent and classic literature pertaining to reproducibility, replication, and explanation in data-driven inference published in computer science, psychology, statistics, and related fields. We will examine recent evidence of problems of reproducibility, replicability and robustness in data-driven science; theories and evidence related to causes of these problems; and solutions and open questions. Topics include: ML reproducibility, interpretability, the social science replication crisis, adaptive data analysis, causal inference, generalizability, and uncertainty communication.
The high level goal is to expose more CS PhD students to results and theories related to blind spots in conventional use of statistical inference in research. My hope is that reading a bunch of papers related to this (ambitious) topic but from different angles will naturally encourage thinking beyond the specific results to make observations about how overinterpreting results and overtrusting certain procedures (randomized experiments, test-train splits, etc) can become conventional in a field.
Putting together the reading list (below) was fun, but I’m open to any suggestions of what I missed or better alternatives for some of the topics. The biggest challenge I suspect will be having these discussions without being able to assume a certain series of stats courses (prerequisites call for exposure to both explanatory and predictive modeling, but I left it kind of loose). I am doing a few lectures early on to review key assumptions and methods but there’s no way I can do it all justice.
In developing it, I consuilted syllabi from a few related courses: Duncan Watts’ Explaining Explanation course at Wharton (which my course overlaps with the most), Matt Salganik and Arvind Narayanan’s Limits of Prediction at Princeton, and Jake Hofman’s Modeling Social Data and Data Science Summer School courses.
Schedule of readings
1. Course introduction
- Hullman, Kapoor, Nanayakkara, Gelman, Narayanan (2022). The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning.
- Breiman (2001). Statistical Modeling: The Two Cultures (with comments and rejoinder).
Optional:
- Efron (2020). Prediction, estimation, and attribution.
- Mullainathan and Spiess (2017). Machine learning: An applied econometric approach.
2. Review: Statistical Modeling in Social Science
Note: These references are for your benefit, and can be consulted as needed to fill gaps in your prior exposure.
- Simple linear regression, multiple regression
- Lecture Notes 3-7, 13-14, others as needed by Shalizi from Wasserman’s Modern Regression course
- Null Hypothesis Significance Testing.
- Hubbard and Bayarri (2003). Confusion Over Measures of Evidence (p’s) Versus Errors (alpha’s) in Classical Statistical Testing.
- Effect size and statistical power.
- Coe (2002). It’s the effect size, stupid.
- Cohen (1992). Statistical power analysis.
3. Review: Statistical Modeling in Machine Learning
Note: These references are for your benefit, and can be consulted as needed to fill gaps in your prior exposure.
- Definition of learning problem, theoretical underpinnings
- Shaley-Schwartz and Ben-David (2014). Understanding Machine Learning: From Theory to Algorithms. Chapters 2-6.
- Supervised learning, deep learning, stochastic gradient descent, backpropagation, loss function, etc.
- Andrew Ng’s ML course notes and lectures.
- Goodfellow, Bengio, Courville (2016). Deep Learning. Chapters 6-8, 11
- Cross-validation, model selection
- Common task framework
- Donoho (2017). 50 years of data science (Section 6 only).
PROBLEMS AND DEFINITIONS
4. What does it mean to explain?
- Pearl (2013). Epilogue. The Art and Science of Cause and Effect.
- Watts et al. (2018). Explanation, prediction, and causality: Three sides of same coin?
- Hedström and Ylikoski (2010). Causal Mechanisms in the Social Sciences.
5. What does it mean to reproduce?
- Bollen et al. (2015). Social, Behavioral, and Economic Sciences Perspectives on Robust and Reliable Science.
- Committee on Reproducibility and Replicability in Science, National Academies of Sciences, Engineering, and Medicine. (2019) Reproducibility and replicability in science. Chapters 3-5.
Optional:
- Nosek and Arrington (2020). What is replication?
- Stroebe and Strack (2014). The Alleged Crisis and the Illusion of Exact Replication
6. Evidence of reproducibility in social science and ML
- Open Science Collaboration (2015). Estimating the reproducibility of published science.
- Gunderson and Kjensmo. State of the art: Reproducibility in artificial intelligence.
- Liao et al. (2022). Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning.
Optional:
- Gelman (2016). “What has happened down here is the winds have changed” (blog post).
- Agarwal et al. (2021) Deep Reinforcement Learning at the Edge of the Statistical Precipice.
PROPOSED CAUSES
7. Adaptive overfitting: social science
- Simmons, Nelson, Simohnson (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant.
- Button et al. (2013). Power failure: Why small sample size undermines the reliability of neuroscience.
- Gelman and Loken (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time.
Optional:
- Yarkoni and Westfall (2017). Choosing prediction over explanation in psychology: Lessons from machine learning.
- Ioannadis (2005). Why Most Published Research Findings Are False, and Goodman and Greenland (2007). Why Most Published Research Findings Are False: Problems in the Analysis
- van Zwet, Schwab, and Greenland (2021). Addressing exaggeration of effects from single RCTs.
8. Adaptive overfitting: ML
- Henderson et al. (2018). Deep Reinforcement Learning that Matters.
- Cawley and Talbot (2010). On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation.
Optional:
- Hosseini et al. (2020). I tried a bunch of things: The dangers of unexpected overfitting in classification of brain data.
- Roelofs et al. (2019). A Meta-Analysis of Overfitting in Machine Learning
9. Generalizability
- Yarkoni (2022). The generalizability crisis, and commentaries: (Gigerenzer) We need to think more about how we conduct research; (Bonifay) Increasing generalizability via the principle of minimum description length. Please also choose two others to read.
- Recht et al. (2019). Do ImageNet Classifiers Generalize to ImageNet?
- Raji et al. (2021). AI and the Everything in the Whole Wide World Benchmark.
Optional:
- Hand (2006). “Classifier Technology and the Illusion of Progress”, Statistical Science. (Read article, skim comments and rejoinder)
- Clark (1973). The Language-as-Fixed-Effect Fallacy: A Critique of Language Statistics in Psychological Research
- Geirhos et al. (2020). Shortcut learning in deep neural networks.
10. Causal inference
- Pearl (2019). The Seven Tools of Causal Inference, with Reflections on Machine Learning.
- Gelman and Imbens (2013). Why ask why? Forward causal inference and reverse causal questions.
Optional:
- Feder, Keith, Manzoor … Stewart, Veitch, Yang (2022). Causal inference in natural language processing: Estimation, prediction, interpretation and beyond.
- Hu and Kohler-Haussman (2020). What’s Sex Got to Do With Fair Machine Learning?
11. Misspecification & multiplicity
- Bernardo & Smith (1994). Bayesian Theory. Chapter 6: Remodelling. (focus on 6.1.2 and 6.1.3; it may be helpful to read alongside Betancourt (2019). Probabilistic modeling and statistical inference).
- D’Amour et al. (2020). Underspecification Presents Challenges for Credibility in Modern Machine Learning.
Optional:
- Montgomery et al. (2018). How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It.
- Semanova, Rudin, and Parr (2022). On the Existence of Simpler Machine Learning Models.
- D’Amour (2021). Revisiting Rashomon: A Comment on “The Two Cultures.”
- Ilyas et al. (2019). Adversarial examples are not bugs, they are features
- Tosh et al. (2021). The piranha problem: Large effects swimming in a small pond.
12. Interpretability: ML
- Lipton (2018). The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery.
- Lakkaraju and Bastani (2020). “How do I fool you?”: Manipulating User Trust
- via Misleading Black Box Explanations
- Adebayo et al. (2018). Sanity Checks for Salience Maps.
Optional:
- Rudin (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.
- Fisher, Rudin, Dominici (2019). All Models are Wrong, but Many are Useful: Learning a Variable’s Importance by Studying an Entire Class of Prediction Models Simultaneously
- Slack et al. (2020). How can we fool LIME and SHAP? Adversarial Attacks on Post hoc Explanation Methods
- Denain and Steinhardt (2022). Auditing Visualizations: Transparency Methods Struggle to Detect Anomalous Behavior.
- Ding, Denain, Steinhardt (2021). Grounding Representation Similarity with Statistical Testing.
13. Interpretability: Social Science
- Meehl (1990). Why summaries of research on psychological theories are often uninterpretable.
- Greenland et al. (2016). Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations.
Optional:
- Gelman and Stern (2006). The difference between “significant” and “not significant” is not itself statistically significant.
- Hirschman (2016). Stylized facts in the social sciences.
- DeJesus et al. (2019). Generic language in science communication.
SOLUTIONS AND OPEN QUESTIONS
14. Limiting degrees of freedom
- Nosek, Ebersole, DeHaven, Mellor (2018). The preregistration revolution.
- Chambers and Szavella (2022). The past, present, and future of registered reports.
- Dwork, Feldman, Hardt, Pitassi, Reingold, Roth (2015). The reusable holdout: Preserving validity in adaptive data analysis.
Optional:
- Gunderson (2021). The case against registered reports.
- Szollosi, Kellen, Navasrro, Shiffrin, van Rooij, Van Zandt, Donkin (2019). Is preregistration worthwhile?
- Nosek, Beck, Campbell … Mellor, van t’Veer, Vazire (2019). Preregistration is hard, and worthwhile
15. Integrative methods
- Hofman et al. (2021). Integrating explanation and prediction in computational social science.
- Veitch, D’Amour, Yadlowsky, and Eisenstein. (2021). Counterfactual Invariance to Spurious Correlations.
Optional:
- Yu and Kumbler (2020). Veridical data science.
- Athey (2017). Beyond prediction: Using big data for policy problems.
- Agarwal, Peterson, Griffiths (2020). Scaling up psychology via Scientific Regret Minimization.
- Fudenberg et al. (2021). Measuring the completeness of economic models.
16. Better theory
- Oberauer and Lewandowsky (2019). Addressing the theory crisis in psychology.
- Lundberg, Johnson, Stewart (2021). What is Your Estimand? Defining the Target Quantity Connects Statistical Evidence to Theory.
Optional:
- Devezer et al. (2020). The case for formal methodology in science reform.
- van Rooij and Baggio (2021). Theory before the test: How to build high-verisimilitude explanatory theories in psychological science.
- Guest and Martin (2021). How Computational Modeling Can Force Theory Building in Psychological Science
- Szollosi and Donkin (2021). Arrested Theory Development: The Misguided Distinction Between Exploratory and Confirmatory Research.
17. Better communication of uncertainty
- Dodge, Card, Gururangan, Schwartz, Smith (2019). Show your work: Improved reporting of experimental results.
- Steegen, Tuerlinckx, Gelman, and Vanpaemel (2016). Increasing transparency through a multiverse analysis
- Rafi and Greenland (2020). Semantic and cognitive tools to aid statistical science: replace confidence and significance by compatibility and surprise.
Optional:
- Simons, Shoda, and Lindsay (2017). Constraints on generality (COG): A proposed addition to all empirical papers.
- Drummond (2006). Machine Learning as an Experimental Science (revisited).
- Demsar (2006). Statistical comparisons of classifiers over multiple datasets.
- Pineau, Vincent-Lamarre, Sinha … Fox, Larochelle (2020). Improving reproducibility in machine learning research (see Appendix, Figure 9, for checklist and section 5 for report on uncertainty communication).
Our host’s “winds have changed” post (optional reading here) honestly belongs in an anthology of the greatest posts of the blog era IMO.
I agree!
I also agree. That post is required reading on my stats syllabus.
This class looks amazing and has several resources and perspectives beyond the ones that typically get covered that I appreciate the links to learn more about.
On the topic of forking paths and adaptive overfitting, following up on the Dwork et al analysis, I would maybe mention something about Littlestone dimension, which is quite literally, as a sequential version of shatter dimension over trees, a way of quantifying forking, and its effects on generalizability and prediction, viz, that worst case prediction errors can fail to converge for classes with unbounded Littlestone dimension, which include many classes (eg, VC) for which classical learning theory suggests overfitting should not be a problem. For an introduction I like Shalev-Shwartz Ch. 3
The reason I think it’s worth bringing up is that equivalence results show that many proposed solutions to preventing adaptive overfitting are equivalent to bounding this dimension, either through explicit online learning approaches (which have a flavor of accounting for nonstationarity) or by recent results showing equivalence to differential privacy https://t.co/IpmNF7nXzW, which can then be generated in a variety of ways, including holdouts, noise, Bayesian sampling (https://www.jmlr.org/papers/volume18/15-257/15-257.pdf), etc.
I think that strand of work is only a piece of the puzzle on those topics (where do model classes come from? What kinds of adaptivity are actually common beyond the worst case and how do we address them scientifically, socially, etc?), but the rest of the syllabus seems to cover what’s known about those quite well.
Thanks, very useful to know! The relationship between differential privacy and preventing overfitting is intriguing … I have been meaning to spend more time with papers in that space (starting with https://dl.acm.org/doi/pdf/10.1145/2746539.2746580)
Great title for the course too. Titles are important. I like the idea of generalization (poststratification, etc.) being part of reproducibility. It’s a good framework. Thinking in very general terms, we can even think of the idea of a “parameter” in a parametric model as representing reproducibility, in that a parameter is what persists from one problem to the next. This is related to the point that Bois, Jiang, and I made in our paper from 1996 that substantively-motivated (rather than merely phenomenological) models play well with Bayes, for two reasons:
1. In a substantively-motivated model, the parameters have a meaning external to the data at hand, which makes it more reasonable to think we can come up with good priors for them.
2. If a model is parameterized well, the parameters will be roughly constant from person to person, or from scenario to scenario, so that a hierarchical model allowing variation will do lots of partial pooling, giving stable estimates.
Interesting, I like that way of thinking about it. Reminds me of this recent paper calling for motiving your theoretical and empirical estimands outside the context of a statistical model: https://osf.io/preprints/socarxiv/ba67n/
This is a great course! So many great papers to comb through and mull over.
As a practitioner of ML in healthcare, I do wish there was a better source of material for “OMG don’t do that”, good anti-patterns for models. One anti-pattern that would seem to fit into the lack of generalizability in your course is models that are built on poorly chosen surrogate variables as found in Obermeyer, https://www.science.org/doi/10.1126/science.aax2342 .
But to me this is not just a lack of generalizability of the original model, but a profound miss-specification due to the challenge of collecting data for the real outcome variable. The model fit something other than the use case required, but most of the testing and other checks yielded false positives because they also checked against the poor surrogate outcome.
“Profound model misspecification” is definitely going to be a theme. Thanks for the suggestion, I wasn’t aware of this paper.
Thanks for sharing your reading list! It looks very comprehensive, and I intend to check it out in greater detail.
One thing that I noticed on a cursory look over the reading list is that it has a lot of content about statistical methods and theory, but I didn’t find anything about the more mundane aspects of reproducibility from the software perspective. Things like how running the same analysis with different versions of libraries or software packages can yield different results, or how multiple runs stochastic algorithms can produce different outputs, etc.
I would be interested to hear why you chose to leave that out of the course. Are these topics things that your audience is already familiar with and do not need to be covered again? Or is there a different reason for not covering them?
Thanks!
It’s a good question. I suspect its of interest to some of the students. It comes up in a few of the readings on ML reproducibility, but more could definitely be said.
What a terrific looking course. Bookmarking a lot!
I don’t know how much computer science PhD students really understand about where data come from or how social science data are collected. I might want to suggest some “classics” including maybe Campbell and Stanley https://wagner.nyu.edu/files/doctoral/Campbell_and_Stanley_Chapter_5.pdf
and maybe Fisher since basically almost all social science statistics books still follow the basic structure of Statistical Methods for Research Methods and Design of Experiments. I feel like if you don’t understand how these works have shaped social science research some of the other works are decontextualized.
You might want a general research methods text as a reference. Or maybe just some resources on specific methods. A lot of people think it takes 10 minutes to write a survey
I just was poking around and saw this https://theoryandscience.icaap.org/content/vol7.1/denis.html which has a nice bibliography and an overview of one strain of work.
I think you’re right… I was thinking I could not say much about experiment design but after two class sessions not so sure. And the history is so useful for putting things in context.
I should go back to school!
Hi Jessica, That’s a great collection of topics and references.
I’d like to mention some work I did with Masayoshi Mase (Hitachi, Ltd.) and Ben Seiler (Stanford stat PhD student).
https://arxiv.org/abs/2205.15750
We discuss the variable importance part of interpretability as a problem of `causes of effects’ vs `effects of causes’. So it remains hard even if the causal knowledge is complete and correct. We compare methods by properties, such as whether the cost is exponential, whether they have to use previously unobserved variable combinations, whether it is possible to attribute importance to variables that the model does even use, whether the method requires user input, and so on. We do an extended worked example on the COMPAS data to illustrate our proposed cohort Shapley method.
Ultimately, setting variable importance is going to require interacting with the problem domain. We don’t think of it as something that can or should be done automatically.
Art:
Regarding “causes of effects” vs. “effects of causes,” I recommend this paper with Guido, “Why ask why? Forward causal inference and reverse causal questions.” It never got published on its own, but I stuck it in as Section 21.5 of Regression and Other Stories.
I like that you can use those attribution issues to focus the next effects of causes investigation. So then as you and Guido say, it is not cocktail party chatter.
However there are settings where the `next thing’ is not pertinent. For instance, there could be a lot of interest in why a given person had a loan declined, or was predicted to reoffend, or was deemed appropriate for the ICU, in each case by an algorithm that could be rerun with different values for a subset of the person’s variables.
I found this article useful:
Dawid AP, Musio M. Effects of causes and causes of effects. Annual Review of Statistics and Its Application. 2022 Mar 7;9:261-87.
The above anonymous reply is by me. -Art. I thought the interface might save my ID.