This was originally going to happen today, 8 Mar 2024, but it got postponed to some unspecified future date, I don’t know why. In the meantime, here’s the title and abstract:
Statistical practice as scientific exploration
Andrew Gelman, Department of Statistics and Department of Political Science, Columbia University
Much has been written on the philosophy of statistics: How can noisy data, mediated by probabilistic models, inform our understanding of the world? After a brief review of that topic (in short, I am a Bayesian but not an inductivist), I discuss the ways in which researchers when using and developing statistical methods are acting as scientists, forming, evaluating, and elaborating provisional theories about the data and processes they are modeling. This perspective has the conceptual value of pointing toward ways that statistical theory can be expanded to incorporate aspects of workflow that were formally tacit or informal aspects of good practice, and the practical value of motivating tools for improved statistical workflow, as described in part in this article: https://www.stat.columbia.edu/~gelman/research/unpublished/Bayesian_Workflow_article.pdf
The whole thing is kind of mysterious to me. In the email invitation it was called the UPenn Philosophy of Computation and Data Workshop, but then they sent me a flyer where it was called the Philosophy of A.I., Data Science, & Society Workshop in the Quantitative Theory and Methods Department at Emory University. It was going to be on zoom so I guess the particular university affiliation didn’t matter.
In any case, the topic is important, and I’m always interested in speaking with people on the philosophy of statistics. So I hope they get around to rescheduling this one.
I hope they reschedule it too ! (I also hope they record it.)
When you say “acting as scientists” what do you mean ? How does one use and develop statistical methods without acting as a scientist ?
Shira:
By “acting as a scientist,” I mean taking the steps of formulating hypotheses and gathering data to evaluate, compare, improve, and reject these hypotheses.
An example of using statistics without acting as a scientist would be: I have some poll data, I run MRP, and I use it to create a map of state-level opinions. A example of acting as a scientist in that situation is: I have a hypothesis that MRP should work well for my data; I run MRP on my data, create a map, and see if the results make sense; I simulate some fake data and check to see if my MRP procedure approximately recovers the assumed underlying truth; etc. In conducting these experiments, I’m looking for ways to refute or modify my hypothesis; i.e., I’m trying to break my method or, to put it more positively, to determine the domain of its useful applicability.
An example of developing a statistical method without acting as a scientist would be: I implement a new MRP procedure in Stan (for example, using a robust logistic regression model where probabilities go from 0.01 to 0.09) and check it with simulation-based calibration. An example of acting as a scientist in that situation is: I have a hypothesis that this robust model will give me better answers in certain problems, and I conduct some combination of mathematical analyses, simulation studies, and empirical analyses to assess the conditions under which my new method works better.
As the above examples perhaps illustrate, it’s not always a bad thing to “not act as a scientist.” As a cook, I often use recipes as is, without trying to understand or modify them. If the food tastes fine, that’s enough! When things start to go wrong or when I don’t quite have the specified ingredients or when I want to use my resources more efficiently, then it can make sense to do some research.
I wrote the above abstract, and plan to speak on the topic, because it struck me that statistics textbooks (including my own) are super-clear on the idea that statistics is in the service of substantive research, but then we tend to present statistical practice as more of a cookbook thing: Do steps A, B, C, check X, Y, Z, etc. That’s all fine, but, upon reflection, I realized that when I’m doing statistics—developing new methods or working on applications or even just setting up a little simulation study for my next class—I’m acting as a kind of baby scientist, making little hypotheses (“Yeah, I think boostrapping the errors on this will work” or “Missingness in this dataset is minor, and I think I can get away with complete-case analysis” or “If I set the assumed treatment effect here to 0.1, the result will be clean enough that it will show up clearly in the graph, and students will get the point”) and then checking my hypotheses with various auxiliary analyses. I’m applying the hypothetico-deductive method within my statistical analysis, not merely using statistics as a tool within a substantive study.
Again, it’s not necessary or even advisable to be “acting as a scientist” at all times. It’s just that I recognized that I was often acting in this way, even though my writings on workflow had not been reflecting that.
Thanks, Andrew !
Your workflow paper says:
“the process of development of scientific theories is not the same as that of statistical models (Navarro, 2020).”
Is this what you mean by “my writings on workflow had not been reflecting” that you’re often acting as a scientist ?
If someone is following many of the workflow steps in that paper (not taking 100% shortcuts to full cookbook recipe following), are they acting as a scientist ?
Shira:
I’ll have to look back at that particular quote. In general, I’d say that I mostly like the presentation of statistical workflow in our books on Bayesian data analysis, multilevel models, and regression modeling, along with the Bayesian workflow article—but I don’t think we put enough emphasis on the way in which data analysis involves acting like a scientist. I’d like to put some of that in our forthcoming book on Bayesian workflow, the idea that when doing statistics we are constantly forming and testing little hypotheses.
Very interesting topic! As a statistical, I have been interacting with clinicians in medical research. My feeling is that we help them to generate noisy data in a way that we can consider the study a scientific experiment. When researchers analyze noisy data, statistical modeling helps them to quantify the uncertainty of a scientific statement. This practice usually points out that next time you should call a statistician before you run your experiment.