Using forecasts to estimate individual variances

Someone who would like to remain anonymous writes:

I’m a student during the school year, but am working in industry this summer. I am currently attempting to overhaul my company’s model of retail demand. We advise suppliers to national retailers, our customers are suppliers. Right now, for each of our customers, our demand model outputs a point estimate of how much of their product will be consumed at one of roughly a hundred locations. This allows our customers to decide how much to send to each location.

However, because we are issuing point estimates of mean demand, we are *not* modeling risk directly, and I want to change that, as understanding risk is critical to making good decisions about inventory management – the entire point of excess inventory is to provide a buffer against surprises.

Additionally, the model currently operates on a per-day basis, so that predictions for a month from now are obtained by chaining together thirty predictions about what day N+1 will look like. I want to change that too, because it seems to be causing a lot of problems with errors in the model propagating across time, to the point that predictions over even moderate time intervals are not reliable.

I already know how to do both of these in an abstract way.

I’m willing to bite the bullet of assuming that the underlying distribution of the PDF should be multivariate Gaussian. From there, arriving at the parameters of that PDF just requires max likelihood estimation. For the other change, without going into a lot of tedious detail, Neural ODE models are flexible with respect to time such that you can use the same model to predict the net demand accumulated over t=10 days as you would to predict the net demand accumulated over t=90 days, just by changing the time parameter that you query the model with.

The problem is, although I know how to build a model that will do this, I want the estimated variance for each customer’s product to be individualized. Yet frustratingly, in a one-shot scenario, the maximum likelihood estimator of variance is zero. The only datapoint I’ll have to use to train the model to estimate the mean aggregate demand for, say, cowboy hats in Seattle at time t=T (hereafter (c,S,T)) is the actual demand for that instance, so the difference between the mean outcome and the actual outcome will be zero.

It’s clear to me that if I want to arrive at a good target for variance or covariance in order to conduct risk assessment, I need to do some kind of aggregation over the outcomes, but most of the obvious options don’t seem appealing.

– If I obtain an estimate of variance by thinking about the difference between (c,S,T) and (c,Country,T), aggregating over space, I’m assuming that each location shares the same mean demand, which I know is false.

– If I obtain one by thinking about the difference between (c,S,T) and (c,S,tbar), aggregating over time, I am assuming there’s a stationary covariance matrix for how demand accumulates at that location over time, which I know is false. This will fail especially badly if issuing predictions across major seasonal events, such as holidays or large temperature changes.

– If I aggregate across customers by thinking about the difference between (c,S,T) and (cbar,S,T), I’ll be assuming that the demand for cowboy hats at S,T should obey similar patterns as the demand for other products, such as ice cream or underwear sales, which seems obviously false.

I have thought of an alternative to these, but I don’t know if it’s even remotely sensible, because I’ve never seen anything like it done before. I would love your thoughts and criticisms on the possible approach. Alternatively, if I need to bite the bullet and go with one of the above aggregation strategies instead, it would benefit me a lot to have someone authoritative tell me so, so that I stop messing around with bad ideas.

My thought was that instead of asking the model to use the single input vector associated with t=0 to predict a single output vector at t=T, I could instead ask the model to make one prediction per input vector for many different input vectors from the neighborhood of time around t=0 in order to predict outcomes at a neighborhood of time around t=T. For example, I’d want one prediction for t=-5 to t=T, another prediction for t=-3 to t=T+4, and so on.

I would then judge the “true” target variance for the model relative to the difference between (c,S,T)’s predicted demand and the average of the model’s predicted demands for those nearby time slices. The hope is that this would reasonably correspond to the risks that customers should consider when optimizing their inventory management, by describing the sensitivity of the model to small changes in the input features and target dates it’s queried on. The model’s estimate of its own uncertainty wouldn’t do a good job of representing out-of-model error, of course, but the hope is that it’d at least give customers *something*.

Does this make any sense at all as a possible approach, or am I fooling myself?

My reply: I haven’t followed all the details, but my guess is that your general approach is sound. It should be possible to just fit a big Bayesian model in Stan, but maybe that would be too slow, I don’t really know how big the problem is. The sort of approach described above, where different models are fit and compared, can be thought of as a kind of computational approximation to a more structured hierarchical model, in the same way that cross-validation can be thought of as an approximation to an error model, or smoothing can be thought of as an approximation to a time-series model.

5 thoughts on “Using forecasts to estimate individual variances”

  1. I’m gonna disagree with Andrew and say that the general approach is not sound. It looks like you’re planning to examine variance in the mean predictions in some neighborhood of the time domain. But the behavior of the mean function doesn’t necessarily have any particular relationship with the noise or residual variance. Consider the dead simple case of the stochastic difference equation

    y_0 = 0

    y_{t+1} = y_t + a + N(0, sigma)

    Assuming your model gets a decent estimation of a and sigma, as a gets large and sigma goes to 0, your method would overstate variance since the mean function changes rapidly over time. As a goes to 0 and sigma gets large, your method would understate variance since the mean function doesn’t change despite there being lots of noise.

    Visually, look at

    https://miro.medium.com/v2/resize:fit:668/format:webp/1*yeNqtXXdLvh82LKOHmsWZA.png

    If you take a small epsilon in the domain, the blue line changes very little relative to the actual noisiness in the data.

    Some questions about your thing:

    I’m willing to bite the bullet of assuming that the underlying distribution of the PDF should be multivariate Gaussian. From there, arriving at the parameters of that PDF just requires max likelihood estimation.
    …
    The problem is, although I know how to build a model that will do this, I want the estimated variance for each customer’s product to be individualized. Yet frustratingly, in a one-shot scenario, the maximum likelihood estimator of variance is zero.

    Am I to assume you’re using squared loss, i.e. assuming a constant diagonal covariance matrix that drops out of optimization? And it seems like you’re saying model is trained until it reproduces all the training data exactly? If you have zero in-sample error, it seems like you’re probably overfitting and want to introduce some regularization.

    The most natural choice to me for uncertainty quantification is to use a bayesian model like a gaussian process instead. Fully or variational bayesian methods. Averaging over the posterior doesn’t collapse the residual variances, but instead examines what the data support by assumption.

    If you want to keep using the neural ODE, the next most natural choice to me, assuming you have enough data, is to use some kind of hold-out based uncertainty calibration method like conformal prediction. The nonexchangeability over time makes this kind of tricky, but it looks like it’s been done https://arxiv.org/pdf/2212.03463.pdf

    Alternatively, instead of using squared loss, you can use koenker’s quantile/ping pong loss at several alpha levels, then give an uncertainty interval as the inter-quantile range. The calibration properties of this are kind of arbitrary though, and, again, you’ll have to introduce some kind of regularization.

Leave a Reply

Your email address will not be published. Required fields are marked *