In a comment on our post, Using black-box machine learning predictions as inputs to a Bayesian analysis, Allan Cousins writes:

I find this combination of techniques exceedingly useful when I have a lot of data on an indicator that informs me about the outcome of interest but where I have relatively sparse data about the outcome itself. A prime example of this is estimating constructions costs where you have 10x-20x as many estimates as you do actual project costs.

So far this is all about postprocessing: the idea that the black-box predictions represent a shortcut that allows us to avoid full Bayesian modeling of the data so we can focus on improving the predictions of interest.

But then, as happens sometimes in comments, the discussion took an interesting and unexpected turn, when Daniel Lakeland replied:

My experience with estimating construction costs was that it was rarely about getting the right answer, or making good decisions, and mostly about some kind of negotiation tactics combined with lawsuit protection.

Cousins responded:

You need to separate cost estimation from bid preparation. They are fundamentally different tasks.

When estimating construction costs your goal should always be to estimate as closely as possible the future costs (as they occur so that financing can be properly incorporated into the bid) for the project as specified.

And then they were off, with the discussion touching on the value of subjective human inputs, the need to calibrate such inputs, and how they fit into statistical modeling.

What I wanted to point out was how several concepts arose in different places of the discussion:

– Combining information from different sources, which first arose in the context of postprocessing machine learning estimates, then later when considering how to integrate human predictions with data-based statistical models.

– Calibration, which is relevant for when taking predictions from a statistical or machine learning model and applying them in new cases (I recommend hierarchical modeling) and also, as Cousins wrote, for working with expert judgments. I’ve long held the view that, instead of trying to elicit subjective priors, you should eliciting estimates and then construct a statistical model to get probabilistic forecasts from these estimates by calibrating them on data. In any given case, the probabilistic forecast can be used a prior (a “subjective prior,” if you like) for the decision problem at hand. We demonstrate some of these steps in this article on decision making for radon measurement and remediation.

– The different goals of statistical modeling, decision theory, and game theory, which came up all over. The need to go back and forth between methods and goals.

One of the fascinating things about statistics is how these and other ideas keep reappearing in different places.

P.S. Title from here; cat picture from Diana Senechal.