Tian points us to this article by David Donoho, which argues that some of the rapid progress in data science and AI research in recent years has come from “frictionless reproducibility,” which he identifies with “data sharing, code sharing, and competitive challenges.” This makes sense: the flip side of the unreplicable research that has destroyed much of social psychology, policy analysis, and related fields is that when we can replicate an analysis with a press of a button using open-source software, it’s much easier to move forward.
Frictionless reproducibility
Frictionless reproducibility is a useful goal in research. It can take a while between the development of a statistical idea and its implementation in a reproducible way, and that’s ok. But it’s good to aim for that stage. The effort it takes to make a research idea reproducible is often worth it, in that getting to reproducibility typically requires a level of care and rigor beyond what is necessary just to get a paper published. One thing I’ve learned with Stan is that much is learned in the process of developing a general tool that will be used by strangers.
I think that statisticians have a special perspective for thinking about these issues, for the following reason:
Methods as proto-algorithms
As statisticians, we’re always working with “methods.” Sometimes we develop new methods or extend existing methods; sometimes we place existing methods into a larger theoretical framework; sometimes we study the properties of methods; sometimes we apply methods. Donoho and I are typical of statistics professors in having done all these things in our work.
A “method” is a sort of proto-algorithm, not quite fully algorithmic (for example, it could require choices of inputs, tuning parameters, expert inputs at certain points) but it follows some series of steps. The essence of a method is that it can be applied by others. In that sense, any method is a bridge between different humans; it’s a sort of communication among groups of people who may never meet or even directly correspond. Fisher invented logistic regression and decades later some psychometrician uses it; the method is a sort of message in a bottle.
Division of labor as a characteristic of statistical methods
There are different ways to take this perspective. One direction is to recognize that almost all statistical methods involve a division of labor. In Bayes, one agent creates the likelihood model and another agent creates the prior model. In bootstrap, one agent comes up with the estimator and another agent comes up with the bootstrapping procedure. In classical statistics, one agent creates the measurement protocol, another agent designs the experiment, and a third agent performs the analysis. in machine learning, there’s the training and test sets. With public surveys, one group conducts the survey and computes weights; other groups analyze the data using the weights. Etc. We discussed this general idea a few years ago here.
But that’s not the direction I want to go right here. Instead I want to consider something else, which is the way that a “method” is an establishment of a default; see here and also here.
Statistics as the science of defaults
The relevance to the current discussion is that, to the extent that defaults are a move toward automatic behavior, statisticians are in the business of automating science. That is, our methods are “successes” to the extent that they enable automatic behavior on the part of users. As we have discussed, automatic behavior is not a bad thing! When we make things automatic, users can think at the next level of abstraction. For example, push-button linear regression allows researchers to focus on the model rather than on how to solve a matrix equation, and it can even take them to the next level of abstraction and think about prediction without even thinking about the model. As teachers and users of research, we then are (rightly) concerned that lack of understanding can be a problem, but it’s hard to go back. We might as well complain that the vast majority of people drive their cars with no understanding of how those little explosions inside the engine make the car go round.
Statisticians well prepared to think about issues raised by AI
To get back to the AI issue: I think that we as statisticians are particularly well prepared to think about the issues that AI brings, because the essence of statistics is the development of tools designed to automate human thinking about models and data. Statistical methods are a sort of slow-moving AI, and it’s kind of always been our dream to automate as much of the statistics process as possible, while recognizing that for Cantorian reasons (see section 7 here) we will never be there. Given that we’re trying, to a large extent, to turn humans into machines or to routinize what has traditionally been a human behavior that has required care, knowledge, and creativity, we should have some insight into computer programs that do such things.
In some ways, we statisticians are even more qualified to think about this than computer scientists are, in that the paradigmatic action of a computer scientist is to solve a problem, whereas the paradigmatic action of a statistician is to come up with a method that will allow other people to solve their problems.
I sent the above to Jessica, who wrote:
I like the emphasis on frictionless reproducibility as a critical driver of the success in ML. Empirical ML has clearly emphasized methods for ensuring the validity of predictive performance estimates (hold out sets, common task framework etc) compared to fields that use statistical modeling to generate explanations, like social sciences, and it does seem like that has paid off.
From my perspective, there’s something else that’s been very successful though as well – post-2015ish there’s been a heavy emphasis on making models robust to adversarial attack. Being able to take an arbitrary evaluation metric and incorporate it into your loss function so you’re explicitly training for it is also likely to improve things fast. We comment on this a bit in a paper we wrote last year reflecting on what, if anything, recent concerns about ML reproducibility and replicability have in common with the so-called replication crisis in social science.
I do think we are about at max hype currently in terms of perceived success of ML though, and it can be hard to tell sometimes how much the emerging evidence of success from ML research is overfit to the standard benchmarks. Obviously there have been huge improvements on certain test suites, but just this morning for instance I saw an ML researcher present a pretty compelling graph showing that the “certified robustness” of the top LLMs (GPT-3.5, GPT 4, llambda 2, etc), when trained on the common datasets (imagenet, mnist, etc), has not really improved much at all in the past 7-8 years. This was a line graph where each line denoted changes in robustness for different benchmarks (imagenet, mnist, etc) with new methodological advances. Each point in a line represented the robustness of a deep net on that particular benchmark given whatever was considered the state of the art in robust ML at that time. The x-axis was related to time, but each tick represented a particular paper that advanced SOTA. It’s still very easy to trick LLMs into generating toxic text, leaking private data they trained on, or changing their mind based on what should be an inconsequential change to the wording of a prompt, for example.
Andrew says, “In some ways, we statisticians are even more qualified to think about this than computer scientists are, in that the paradigmatic action of a computer scientist is to solve a problem, whereas the paradigmatic action of a statistician is to come up with a method that will allow other people to solve their problems.”
What? This is a gross misrepresentation of my field! Academic computer science is just as methodology-oriented as statistics.
Let’s look at ML. Computer scientists have developed general tools (convolutional neural nets, autoencoders, normalizing flows, diffusions, attention and transformers) that can be widely applied to solve problems like natural language understanding, image recognition, protein folding, or even cosmic microwave background source separation. They then developed general tools for managing workflow (GitHub, test frameworks, deployment frameworks), sharing analyses (Jupyter inside of Google Colab, for example), and actually coding neural networks (TensorFlow, PyTorch, JAX). They also designed and built the lower-level foundations from hardware up through programming languages.
To give you some sense of how reproducible these ML tools are, the source separation problem was implemented and evaluated at scale by a postdoc (Bruno Régaldo-Saint Blancard) and colleagues using wavelets and diffusion models on our in-house compute cluster. Lots of the experiments were carried out by an undergrad intern (David Heurtel-Depeiges), though I should point out he’s an exceptional undergrad.
Bob:
I agree completely with you that computer scientists have been better than statisticians at developing reproducible tools. Indeed, that’s kind of the point of Donoho’s article.
What I’m saying is that statistics is the science of defaults and thus are well situated to think about aspects of AI that automate human analytical processes.
In addition to agreeing with Bob’s objection to your characterization of computer science, I would also disagree with your own characterization of Statistics, or at least what statistics should be rather than maybe what it is.
First you say statistics is the science of defaults, but then when you actually describe what that means you’re talking about Computer Science! Specifically, things like linear regression abstracted away from its matrix algebra implementation, that’s classic Interface vs Implementation abstraction.
I really don’t think the goal of statistics should be to automate data processing decisions. It’s more like to build a toolkit of higher level abstractions so that scientists can think about their scientific problem at a level that’s appropriate, but we absolutely can’t automate that abstract level. Doing so leads to unthinking inappropriate application of methods that are either inappropriate or have meanings that are different from the ones attributed by the scientists. We see this every day here on the blog.
I’m not sure I see the ways in which statisticians are particularly well prepared to think about AI issues. As a whole statisticians seem entrenched in their camps, slow to adapt. and not particularly pragmatic. Certainly data sharing, code sharing, and competitive challenges are important, but IMO the key drivers of the success in ML/AI/DS are the increases in computing power, the increases in data set size, and the nature of the problems themselves.
Thanks for expressing your opinion in re: Drivers. Many people believe exactly these same things! You perform a service bringing this up! I specifically identify this viewpoint and give it a label — ‘brutal scaling’ — and discuss this narrative in my paper. I don’t deny that ‘brutal scaling’ describes retrospectively what some organizations think they were doing in the last decade and what they think will help them own the future. I point to Rich Sutton’s “The Bitter Lesson” and discuss it. I specifically engage with the clashing narratives in a section called ‘The Logarithm and the Exponential’. If I were CEO of an internet hegemon I would love to uphold the ‘brutal scaling’ position, as I would own a near monopoly on ‘brutal scaling’. Contra this, there is plenty of evidence that challenges have delivered again and again across decades and not through brutal scaling. Also, there is plenty of evidence that intellect delivers.
Thanks for expressing your opinion in re statisticians. This is a very important set of two sentences you put forward. I wish more statisticians would read what you say and think about them. The second sentence in your message is particularly important for statisticians to think about. Possibly many statisticians are not really following what’s going on. I have learned a lot by interacting with young people outside of statistics departments.
The Donoho paper is truly remarkable. I buy the advantages of frictionless open services and their role in accelerating advances in data science. But I can’t help but think he is being incredibly naive. “Not all friction is yet gone, but the trend is clear: what frictions still remain will vanish soon enough.” This technological determinism seems to ignore all the sociopolitical and socioeconomic forces that humans have created that prosper because of friction. Not long ago, many people said that democracy was an unstoppable force – it, too, was a form of openness that would win out over the alternatives. I fear that this too was naive. I have no doubt that rapid advances in data science will result from increasing acceptance of frictionless open services – but there is much power and wealth to be gained from bottling up these advances as they occur. I just don’t see the natural evolution that will make frictionless open services win out over these mundane human desires for power and wealth.
The role of statisticians vis a vis computer scientists or data scientists strikes me as a fairly unimportant aspect of the changes Donoho is talking about. His vision of the “Reproducibility Singularity” is both inspiring and worrying. Ever faster advances in data science and AI offer tremendous opportunities – but dangers as well. Donoho seems to believe the forces for good will win out over such dangers – but if history is any guide, I don’t see that. Perhaps I am misreading his paper – it is aimed at the evolution of the data science field and I can buy his vision about how research is likely to evolve. But research is but a part of human evolution, and subservient to political and economic institutions.
One thing to note is the role private industry has played in this “frictionless reproducibility”, which I think would imply the opposite of “there is much power and wealth to be gained from bottling up these advances as they occur.”
As a working practitioner in data science, as the term is used in the tech industry rather than as an academic, I think this paper is beyond overoptimistic. I see a lot of frictionless yes, and virtually no reproducibility. My view is lots of data scientists sliding around on a frictionless surface bumping into walls and each other and on average going nowhere at all. Data science is accelerating nowhere.
To elaborate a little bit, what I see is that almost everyone chooses a methodology based primarily on how “frictionless” the interface is with no consideration towards correctness of assumptions or appropriateness for a final business goal. The drivers of success are entirely how easy it is to import a package and make the relevant method calls. Examples of the kind of snake oil I’m talking about:
* prophet is a timeseries forecasting package that has absolutely no concept of autocorrelation and fails horribly in all empirical accuracy comparisons
https://www.microprediction.com/blog/prophet
Its essentially terrible accuracy has been acknowledged by its original author. It is also easily the most used timeseries forecasting method in data science
17020 stars on pypi
https://pypi.org/project/prophet/
* SHAP values, a purported “unified approach to interpreting model predictions” is an “explainable AI” method that makes literally no reference to the values of the target variable in the dataset. I cannot emphasize this enough, it does not explain the data, it “explains the model” for a particular datapoint. Its explanation of the model is a ranked list of input features, where the rankings for feature A are given by the difference between the prediction for a datapoint and the average prediction for datapoints when feature A is sampled from its marginal distribution in the training set (oversimplified a bit). Despite being allegedly a “unified approach” it fails completely on models that are strongly non-additive (z = x * y). The marginalization of particular features also means it completely ignores relationships and collinearities in the data distribution. But setting aside all these technical issues, nobody can explain why anyone should care about those values, or how they constitutes a general “explanation” (which is an impossible task). Furthermore, it can only possibly explain the behavior of a particular fitted model, which I posit is rarely what people actually want. Nonetheless you see stuff like this
https://towardsdatascience.com/the-shapley-value-for-ml-models-f1100bff78d1
People primarily use shapley values to answer questions it cannot possibly answer, about why a prediction was made. The reason is simple. It has a very easy to use API for producing graphs with a very pleasing color scheme. It is frictionless; whether it can answer questions you actually have is immaterial.
https://shap.readthedocs.io/en/latest/api.html#plots
https://pypi.org/project/shap/
20245 stars on pypi
* For some reason, people will absolutely not stop coming up with new ways to create 2D projections of high dimensional data with no sense of any kind of structure preservation. Here’s a breakdown of t-SNE and UMAP, 2 fad techniques from a couple years ago
https://nitter.net/lpachter/status/1431325969411821572#m
But of course https://pypi.org/project/umap-learn/
with 6486 stars on pypi, they say
The whole practice of medium-high dimensional unsupervised learning is its own can of worms.
My point here is that the “frictionless reproducibility” here is an illusion. People do a great job of making it frictionless to do the literal code operations, sure. But that’s only part of the friction of doing data science; a large part of the friction of data science is, or rather, should be, learning what exactly it is you are doing. The interface can be frictionless, but it is an illusion propped up by people’s willingness to not read the paper, know what they are doing, and validate whether or not what they are doing actually works for the problem they want to solve.
I love what you say here! In the passage you quote, I say “… divorced from other considerations”, I think your examples reinforce that. My feeling (I thought I say this in the paper) is that where you have a defined task and challenge structure, reproducibility delivers steady improvements, but where we are just doing “reproducible computational science” with no precisely defined task, task performance, etc. we don’t get the improvements. This is why I specifically more than once say [FR-3 Challenges] is essential. SO I think we are aligned.
I agree with a risk of making the wrong things frictionless. (I especially like the digs on Shapley Values, which have no chance to answer the questions they are typically used for but are so easy that people still run them). In some ways it’s inevitable. It’s always going to be harder to pick an appropriate method for your problem than to just throw a garbage method at something.
This whole discussion reminds me of a lecture on driverless cars. If the AI systems can only handle certain situations, how do you switch back to human control without creating an even worse safety issue?
It’s that way with any AI assist. We need humans involved enough and aware enough of what is going on to be ready to engage appropriately. It’s not as much frictionless we need but appropriately shaped to enable human engagement.
I think that “statistics is the science of defaults” and the drive to automatise (aspects of) data analysis is one side of a coin and the need for human understanding and judgement is the other. I think that the tension between these really is fundamental for statistics (this is actually elaborated to some extent in our “Beyond subjective and objective in statistics” paper https://academic.oup.com/jrsssa/article/180/4/967/7068392 ).
We need automation (for enabling people to run analyses, for reproducibility, for systematic investigation…), and at the same time we should always be skeptical about everything that is automated, as whatever is automated will ignore the specifics of the situation where it’s applied, and of the people who deal with it, and furthermore it will encourage doing things that are not understood properly. Optimally we always want a human in the loop who knows better (obviously not always available…).
Well said. I would change one thing: while it would be nice to have a “human in the loop who knows better,” I think it is more important to have a human in the loop who is responsible and accountable for any decisions – even if they don’t know better. I am not advocating for superiority of human judgement over algorithms, nor the reverse, but instead want to ensure that humans are ultimately responsible for any decisions.