If you’re not careful, you’ll spend most of your time on computations that don’t work.

Here’s how we put it in the Bayesian workflow article:

An important intermediate goal is to be able to fail fast when fitting bad models. This can be considered as a shortcut that avoids spending a lot of time for (near) perfect inference for a bad model. There is a large literature on approximate algorithms to fit the desired model fast, but little on algorithms designed to waste as little time as possible on the models that we will ultimately abandon. We believe it is important to evaluate methods on this criterion, especially because inappropriate and poorly fitting models can often be more difficult to fit.

For a simple idealized example, suppose you are an astronomer several centuries ago fitting ellipses to a planetary orbit based on 10 data points measured with error. Figure 6a shows the sort of data that might arise, and just about any algorithm will fit reasonably well. For example, you could take various sets of five points and fit the exact ellipse to each, and then take the average of these fits. Or you could fit an ellipse to the first five points, then perturb it slightly to fit the sixth point, then perturb that slightly to fit the seventh, and so forth. Or you could implement some sort of least squares algorithm.

Now suppose some Death Star comes along and alters the orbit—in this case, we are purposely choosing an unrealistic example to create a gross discrepancy between model and data—so that your 10 data points look like Figure 6b. In this case, convergence will be much harder to attain. If you start with the ellipse fit to the first five points, it will be difficult to take any set of small perturbations that will allow the curve to fit the later points in the series. But, more than that, even if you could obtain a least squares solution, any ellipse would be a terrible fit to the data. It’s just an inappropriate model. If you fit an ellipse to these data, you should want the fit to fail fast so you can quickly move on to something more reasonable.

This example has, in extreme form, a common pattern of difficult statistical computations, that fitting to different subsets of the data yields much different parameter estimates.

The “fail fast” principle leads to three recommendations:

1. Recognize when your computation is failing. Don’t just run it blind.

2. When computation fails, don’t just brute-force it by running it overnight, setting max treedepth to 20, or whatever. Instead, use the fit from a simpler model as a starting point and try to figure out what’s going wrong. In some settings, you can also try fitting different models to different subsets of the data.

3. Thinking about this from a system level, set up your workflow so that failures will show up as soon as possible. This is not as easy as it sounds! It’s something that we’re working on. My dream is to put this all into a generalized expectation propagation framework, unifying the computational idea of federated learning with statistical ideas of multilevel modeling.

P.S. See here for some interesting history of the “fail fast” principle in engineering.