This is Jessica. Lydia Liu, Deb Raji, Angela Zhou, and a whole bunch of other people (one of which is me) write:
Many automated decision systems (ADS) are designed to solve prediction problems— where the goal is to learn patterns from a sample of the population and apply them to individuals from the same population. In reality, these prediction systems operationalize holistic policy interventions in deployment. Once deployed, ADS can shape impacted population outcomes through an effective policy change in how decision-makers operate, while also being defined by past and present interactions between stakeholders and the limitations of existing organizational, as well as societal, infrastructure and context. In this work, we consider the ways in which we must shift from a prediction-focused paradigm to an interventionist paradigm when considering the impact of ADS within social systems. We argue this requires a new default problem setup for ADS beyond prediction, to instead consider predictions as decision support, final decisions, and outcomes. We highlight how this perspective unifies modern statistical frameworks and other tools to study the design, implementation, and evaluation of ADS systems, and point to the research directions necessary to operationalize this paradigm shift. Using these tools, we characterize the limitations of focusing on isolated prediction tasks, and lay the foundation for a more intervention-oriented approach to developing and deploying ADS.
This paper is still in working paper format, but I’m posting because it does a good job synthesizing problems that arise if you fixate on optimizing prediction accuracy when deploying models to inform decisions about people, rather than taking a more holistic perspective. I’ll probably assign it as a course text next time I teach my Prediction for Decision-making class. The perspective the paper advances is that introducing such systems is a policy change and should be implemented and evaluated as such (e.g., in comparison to “bureaucratic counterfactuals” representing decision processes in the absence of the new system). It summarizes decision theoretic and causal inference formulations and includes case studies to illustrate.
This came out of a workshop led by Lydia, Deb, and Angela last summer, which will be followed up with another workshop at UC Berkeley’s Simon’s Institute next January.
How does this differ from past efforts to combine systems like utility theory with predictive models, e.g. Bayesian networks? Is there a richer history that is more complex than I am appreciating? Thanks.
The emphasis here is not so much on the technical tools, it’s on the difficulty of specifying prediction-for-decision problems and evaluating predictive systems that are embedded in social systems. Much of the technical literature assumes that the decision problem is well understood, the measurements are good, etc, in which case producing a good predictive model becomes easy. But in reality the formulation of a prediction problem and model is like a policy intervention on a complex system and requires a more holistic perspective to design and evaluation.