Alex Malinowski has a question about evaluating the calibration of interval forecasts:
We publish interval forecasts under a pre-registration scheme: each forecast is serialised, hashed and timestamped into a Bitcoin block before publication, so the stated interval cannot be adjusted after the outcome is known. Across 56 resolved forecasts our stated 50% intervals contained 47 outcomes; at the 24-hour horizon, 36 of 44. Under honest 50% intervals, P(>=36 of 44) is about 1.3e-05.
The diagnosis, and the part I would most like criticised: walk-forward across 20,058 observations showed the miscalibration was conditional rather than uniform – 55.6% coverage across all days, 67.1% restricted to the calmest fifth. Our first hypothesis, that the sample carried too much old high-volatility history, was tested and rejected. The surviving explanation is that interval width ignored the current volatility regime. Conditioning the historical sample on regime measured at each window’s open (not its close, which leaks the outcome) brings coverage to 50.8%.
Two things I am unsure about. The instruments are correlated, so effective sample size is well below 56 and I have not done that properly. And at a 30-day horizon the conditional interval comes out wider than the unconditional one, which I have kept but cannot fully account for.
I asked him what was the application, and he replied:
Crypto prices. 24-hour and 7-day intervals on eight pairs (BTC, ETH, SOL, BNB, XRP, DOGE, ADA, LINK), scored against Binance closes.
I chose it as the substrate rather than the subject. Outcomes resolve within a day, the reference price is unambiguous, and there is no data-vintage or revision problem, so a coverage check accumulates evidence quickly and cheaply. The same construction is what we use for sports and prediction-market questions, but those resolve slowly and the sample is thin.
I realise crypto invites a certain reaction, and the reaction is mostly deserved. The calibration question doesn’t depend on it – the same test applies to any published range.
He adds:
If it helps frame it, the two things I’m least confident about are the ones I’d want readers to attack:
1. The eight instruments are correlated, so the effective sample size is well below 56 and I have not handled that properly. The binomial p-value I quoted assumes independence it doesn’t have.
2. At a 30-day horizon the regime-conditional interval comes out wider than the unconditional one. I kept the result because it’s inconvenient, but I can’t fully account for it beyond “calm periods have historically preceded larger monthly moves”, which feels like a description rather than an explanation.
Data and outcomes are CC-BY, one row per sealed forecast with its hash:
https://huggingface.co/datasets/neuportal/neuportal-sealed-crypto-forecasts
My only quick thought here is that it’s indeed a good idea to look at conditional calibration, but you have to be careful only to condition on things in the forecast, not on the outcome. So when he writes, “67.1% restricted to the calmest fifth,” this is relevant as long as “calmest fifth” is something that can be determined before the outcome occurs.
More generally, yeah, it’s hard to evaluate the calibration of forecasts for correlated outcomes–this comes up in election prediction too. My only answer is to try to model intermediate outcomes as much as possible to avoid getting stuck in a purely empirical mode of evaluations.
Maybe you in comments have other ideas?
Author here. On the first point, which is the right one to press: yes, the conditioning variable is fixed before the window opens.
Concretely, for a window covering [t, t+h] the conditioning quantity is realised volatility over the 14 bars ENDING at t – the 14 daily log returns up to and including the forecast date. In code it is a 14-bar rolling std shifted forward by h, so the value attached to each historical window is the one an observer standing at t would have had. Nothing from inside [t, t+h] enters it.
The same applies to the “calmest fifth” split in the walk-forward: the ranking is on that same pre-window quantity, not on realised volatility during the window. Measuring at the close instead would select windows partly on their own outcome, and would produce a beautifully calibrated backtest that means nothing out of sample. That was the first thing I checked when the number came out looking good.
On correlated outcomes – agreed, and I have not solved it. The scoreboard reports distinct forecast dates alongside forecast count for exactly this reason (56 forecasts, 7 distinct days), but reporting the problem is not the same as handling it. Modelling intermediate outcomes is a better answer than my current one, which is to treat the binomial p-value as directional rather than exact.
Every sealed forecast, its hash, and the method flag are in the CC-BY dataset if anyone wants to check the conditioning claim rather than take my word for it.
The parts of this post attributed to Malinowski were in fact generated by AI. A nice test of Pangram’s new feature that detects not just AI writing patterns but also where text switches from human to AI. It could not detect Andrew’s brief sections at the top (I gave it the raw text, not the formatted post).