This is Jessica. As part of teaching prep this week, I was re-reading this paper by Gneiting et al on the usefulness of maximizing sharpness subject to calibration (over prioritizing calibration alone). It got me thinking about how certain convenient-to-assume statistical myths can induce a kind of moral dismay, even as you acknowledge their practical value.
The paper applies methods for checking calibration to various hypothetical forecaster types, using weather as an example domain. One of the types of forecasters they consider is the climatological forecaster, who predicts based on an understanding that the climate will be stable. Assume that for each of t rounds (which might be points in time, space, or different subjects), nature chooses draws some random number mu_t from the standard normal distribution N(0,1) then defines the outcome-generating distribution as a normal distribution centered on mu_t with variance of 1. Meanwhile the forecaster provides a predictive CDF as their forecast. The climatological forecaster predicts the unconditional normal distribution centered on 0 with variance of 2. For this example, caring about marginal calibration, which asserts that in the limit, the probabilities assigned by the forecast and the generating distribution chosen by nature are equivalent, is like proposing the existence of a stable climate.
This analogy asserts nothing new about calibration. But the stable climate interpretation helps emphasize how certain statistical properties or procedures can seem to deny basic truths about the world. When I first read it, I thought, wait, is there really anywhere on earth that truly has a stable climate? Looking at it this way puts it in the same class as other strange myths that we sometimes act like we truly believe, like the point null hypothesis that some intervention could feasibly have exactly zero effect. Which is more unrealistic, believing in processes in the world that never change or believing that two different situations could be perfectly equal? Both are sort of unsettling if you imagine really taking them seriously. They imagine a different, less dynamic world.
This reminds me how as much as I respect a kind of basic research that sets out to explore what can be proven once we have certain properties, it’s not always easy to like certain results. I’m tired of blogging about calibration (and probably some of you are tired of my posts on it!) but this is part of why I get preoccupied with such topics. It’s like there’s something in it that I’m morally or psychological predisposed to resist.
That kneejerk reactions to certain concepts can go beyond the purely intellectual or practical realm is interesting to me partly because it’s in tension with the pragmatic attitude that I tend to gravitate to (which Andrew has espoused many times on this blog). I’m wary of getting too caught up in philosophical arguments about statistics, or rejecting certain useful methods or conceptualizations because they cannot be perfectly realized. I would not refuse to ever use a hypothesis test, or declare certain methods useless (polling, for example), though sometimes people do. Obviously we don’t have to fully embrace certain myths to get some value out of acting as if they are true. This is part of the beauty of statistics.
But at the same time, it seems important to acknowledge that there often is some role of personal values or predispositions about what’s valid, or what is attractive as an object of study, even if we try to pretend there’s not. Why, when pressed, do I want to argue that Bayesian inference is more honest? Why do seemingly simple concepts like exchangeability sometimes appear engimatic? Maybe it’s the irrational predilections that draw some of us toward statistics in the first place. Maybe we should just get used to the idea that statistics is also the territory of the heart.
Perhaps I misunderstood, but in one case you are making a simplifying assumption to make the problem more tractable. In the other you are testing a simplifying assumption.
Yes, agreed. My point is just that taking the results you get from either as foolproof can seem like a denial of certain expectations we should have of real world processes. Overall though, I’d consider acting as if you believe there are truly zero effects more counterproductive than caring about calibration. It’s more a question of how much effort to put in achieving it over other things.
I came here to say the same thing as Anoneuoid. A better example might have been using the pool of all humans for height and guessing everyone’s height probabilistically with the population distribution. It’s well calibrated by construction, but not very sharp compared to making predictions with more covariate information like sex, age, ethnicity, country of birth, etc. I think they were just using the “predict the population statistics” as an example of calibrated prediction that is not sharp.
We never know the population statistics going forward unless we’re in a stationary process. Maybe we just need an out-of-sample danger warning label for all statistical analyses.
The thing I found most frustrating about that paper is the lack of definition of “sharp”. I imagine they’re talking about entropy, but they never come out and say that as far as I recall.
Yes, agreed. I just liked the stable climate example as a way of getting someone to start thinking about whether marginal calibration is ever really enough to make us confident in a forecast (my experience is that its sometimes surprisingly hard to get people to acknowledge this). My comparison to NHST was meant at the higher level that sometimes methods can seem to imply a different kind of world than the one we live in.
I also wondered about the vagueness of how they present sharpness. At one point they say they are being deliberately loose in stating their belief that ideal forecasts and maximizing sharpness subject to calibration are equivalent. Maybe they avoided getting too precise because it would somehow complicate making this point.
>Maybe we just need an out-of-sample danger warning label for all statistical analyses.
I suppose this is what numerous qualitative “risk-of-bias” (RoB) assessment tools in causal, predictive and descriptive inference try to get analysts to declare clearly in papers. In health research there are even related initiatives to put RoB warning labels on plots (Google “ROBVIS”).
The problem with labeling schemes for things like this is that the absence of a warning starts to be seen as evidence that one was not needed. I recall seeing a paper on this happening when misinformation vgets labeled on social media… suddenly anything that isn’t labeled as potentially being misinformation is perceived as having higher truth value than it would be without the warning system. So there’s an incentive to hold back information on possible problems or uncertainty.
Jessica,
With non-stationary processes, there isn’t even a well defined notion called “calibration” right?
Just as an example, suppose there’s a web service that you can call to get data. you put in some covariates a,b,c,d and it outputs cryptographic random numbers from Normal(f(a,b,c,d,t),1) where f is a deterministic nonlinear oscillating trending function and t is an integer days since 2000-01-01, furthermore suppose that it generates one value for each hour of the day, caches it, and always gives the same value to anyone that asks for it in that hour. So, there are 24 random numbers generated in any given day.
Suppose you don’t download these numbers for a given week, and they’re lost forever, you want to know what numbers were used in that time-period because they’re used in driving some automated process or other.
There’s no “long term limit of infinite samples calibration” that you can fall back on. It’s just there were 24 x 7 unknown numbers generated in some non-stationary way during this week. You either know them, or you don’t. Looking deep in the past and seeing the histogram of past samples won’t help you because f() is a function of time.
Still, you have *information* about them, in some bayesian sense… you know f(a,b,c,d,t) and you known the normal has 1 sd. You just don’t know the cryptographic RNG output.
There’s no calibration (in the sense of the correct long term distribution) because it’s not an infinite process, it’s just 24 numbers, but there is information about those numbers.
now… shhhh… Secretly, this is all of science.
Can you say more about terminology?
If I understand the Gneitling paper, its examples seem to come from weather forecasting, although they call one kind of forecast “climatological”, which seems odd, since climate modeling (boundary-value problems) is usually considered different from shorter-term weather forecasting (initial value problem.)
AFAIK, (long-term) climate forecasting certainly doesn’t assume stationarity, although I suppose short-term weather forecasting might.
The example they use in the paper assumes a generating process where in each round, some random number mu_t is drawn from a standard normal distribution, then an outcome is sampled from N(mu_t, 1). The climatological forecaster just predicts N(0, 2) each time, i.e., their forecasts are unconditional/non-informative/good only over the long run. In practice, this means assuming that the future outcomes will be generated fron a consistent process with what we can infer from historical data. The paper is from 2007, which seems like a time when talking about the weather as generated under a static climate would already seem questionable, but I don’t really recall how broadly accepted climate change was. Today talking about forecasts that assume a stable climate just seems weird, unless you are only interested in the short term, but then are you really likely to have enough data to get a good read on the expectation of mu_t?
Jessica:
Yeah, back in 2009 the sages of Freakonomics promoted the claims of some nut who said we were “virtually assured of about 30 years of global cooling” (see here for background). So they certainly weren’t expecting stability!
To the best of my knowledge, they never apologized for or addressed this erroneous prediction–not that it’s stopped them from continuing to get continuous media coverage. But I guess it’s not in the nature of some economists to address their past forecasts. Recall the Nobel prize winners Paul Samuelson and William Nordhaus who wrote, “The Soviet economy is proof that, contrary to what many skeptics had earlier believed, a socialist command economy can function and even thrive.” They wrote that in 1989!
But I guess all that should be the least of our worries, given that when I go on twitter every day to post the link for this blog, I keep being shown videos featuring Alex Jones! How is this even possible? It’s hard to rate the relative harm of that sort of thing, compared to smug econ professors touting the mighty Soviet economy or NPR media personalities getting cute about environmental threats. At a personal level, Jones is by far the most repulsive of these, but arguably more damage has been done by the command-economy sympathies of the authors of a prominent economics textbook and by the spreading of climate misinformation by trusted science journalists.
> arguably more damage has been done by the command-economy sympathies of the authors of a prominent economics textbook
If this is about the 1961 edition of Samuelson, the interesting thing is that even within the confines of the basic models (Solow, Ramsey with exogenous tech growth) more investment does not lead to higher growth in the long run, just a different (and not even necessarily higher) level. In those models, the long-run slope of log GDP is unaffected by policy or choices, it is purely mechanical.
The growth rate is only affected in endogenous growth models, which started in the mid-1980s. So how textbooks made that prediction about the Soviet economy in the 1960s is a complete mystery to me, it does not follow from any of the models at the time.
Getting back on topic for the original post: long-run economic growth is the most important factor affecting welfare (1% difference in growth accumulates to 50% in 40 years), but we cannot measure it directly, only via a persistent noisy process (the business cycle). Extracting causal claims about growth (TFP) changes from even decades of macro data is very challenging. It is very likely that if there is an underlying “growth”, it is time-varying, but most of the time this is assumed away as a simplification.
When estimating forecasting skill, it is necessary to compare forecasts to a reference set of forecasts deemed to be non-skillful but nonetheless useful. Two such forecasts are “climatology” and “persistence”.
“Climatology” predicts the PDF of a weather variable on a given day as equal to the PDF based on the historical record of that weather variable. Standard would be the historical PDF for that time of year. A deterministic forecast would be the climatological average or normal for that day.
“Persistence” predicts the PDF of a weather variable on a given day as equal to the PDF of the historical changes of that weather variable from one day to the next plus the value of that variable on the previous day. A deterministic forecast would be the previous day’s value.
I have two comments:
1. I see the problem with point null hypotheses when using ”naturally formed” (e.g. registry) data but I’d like to hear your thoughts (Jessica’s, but also anyone who would be willing) on whether they are more reasonable in laboratory experiments. In a tightly controlled world, I think many of the increasingly proximal routes a distal cause might take are cut off.
My dishwasher being on probably has non-zero correlations with almost anything, but experimentally manipulating whether my dishwasher is on probably has zero effect on any concurrent experiment.
2. I am a huge fan of this blog, but I believe juxtaposing Paul Samuelson and William Nordhaus to Alex Jones is missing a difference in kind. Not very long ago everything seemed to be going well for Assad, and then suddenly it didn’t. Many Europeans were predicting that it suddenly would start going badly for Putin, but now things are suddenly going tremendously for him (but not for Europeans, for whom things are going much worse than we were predicting).
One could argue that academics have a particular responsibility to calibrate the certainty they communicate (I have argued so). Maybe Samuelson and Nordhaus failed in that regard but failing at something genuinely difficult is different in kind to what Freakonomics and Alex Jones have been doing.
>1. I see the problem with point null hypotheses when using ”naturally formed” (e.g. registry) data but I’d like to hear your thoughts (Jessica’s, but also anyone who would be willing) on whether they are more reasonable in laboratory experiments.
I find point null hypotheses annoying mostly in how they encourage people to interpret experiment results. Thinking about the things as either the same or different leads to conclusions that we’ve either learned of an effect or we haven’t, and shifts focus away from more pragmatic questions like how much variance there was in the measurements or how big a difference would have to be for us to consider it practically meaningful.
Take the famous prediction about starlight deflection during a solar eclipse. Newtonian mechanics predicted 0.8 arcsec, general relativity predicted 1.7 arcsec. Both are point predictions.
If they had both predicted “not exactly zero arcsec”, no one would have cared about it.
The important point is not observational vs experimental. It is whether you test *your prediction* (vs some arbitrary null hypothesis), and how well it fits compared to the other explanations that have been proposed.
According to today’s physics, that dishwasher is emitting extra IR at the speed of light in all directions when on. Thus it is already expected to have non-zero (but very possibly negligible) influence on any other measurements having the event in their past light cone. If you described the circumstances in more detail I am sure I can devise more ways it can influence the results.
I doubt it was truly possible to derive point predictions (other than in the sense that the authors say: “if we assume the speed of light is exactly this” etc.), as these predictions depend on uncertainties in the speed of light, the mass of the objects, and so on.
You could take (0.8+1.7) / 2 = 1.25 arcsec as the null, and examine whether evidence against it is provided in either direction, and then additionally whether confidence or credible intervals contain the point prediction. Results in either direction suggest the prediction in that direction better captures the data. To the degree that the estimate corresponds to the prediction, this further suggests that there is little problem with the theory or other simplifying assumptions. If the former holds but not the latter, then the question is whether the simplifications can be responsible for the deviations or not, but it will likely be harder to square with the prediction in the other direction of the null.
It is worth looking into further. This is the first paper that came up in my search and considers a three-possibility null hypothesis:
https://royalsocietypublishing.org/doi/10.1098/rsnr.2020.0040
I’m no expert, but note that paper is kind of weird and I didn’t see it mention that Einstein’s original prediction in 1911 was the same as for Newton. The earlier attempts at observations failed due to weather and WW1. It does look like a good intro to the controversies though.
But yea, definitely plenty of room for “p-hacking” and systematic error. Now imagine the trouble if you do not even make a precise prediction.
Thanks for these replies!
Sorry, I realised that I failed to provide some important context: as a cognitive psychologist, I was thinking specifically about behavioural laboratory experiments.
I couldn’t begin to comment on the physics, but according to today’s (and yesterday’s) psychology we should expect discontinuities in and thresholds for effects of stimuli due to how we believe perception, attention, memory, judgement/decision and, eventually, response selection works. At various stages, information is filtered out and doesn’t become available to systems “further downstream”. Even if the information makes it all the way through and affects the covert judgement/decision, the response selection process could (potentially consistently) overrule it so that the effect is never expressed as overt behaviour.
Because of these discontinuities and thresholds, I’m not convinced that null hypotheses *of zero* (I was indeed sloppy here, thanks) should be considered myths. (If you are a faculty psychologist à la Fodor, I suppose it is actually a strong prediction of your theory that they should exist in language, vision, and other candidates for modules).
So I’m not convinced that it is obvious that effects of zero would be a myth in a (tightly controlled) behavioural context.
Myth was perhaps not the right word. What I mean with the post is more that some methods build in expectations about the world that seem farfetched if you stop to think about them.
Why frame entire papers around whether or not we can reject a hypothesis that an effect was exactly zero when there are alteratives ways to go about it that don’t involve setting up a straw-man expectation.
Sounds like you’ve got a quantitative model then. Next steps are working out the details of your model, then testing it.
Hi Mattias (not Berggren),
On my end, I think the more interesting case for when to test a null (of zero or otherwise) is when different theories make predictions in different directions of that null, or when there is at least some good reason to think that results could be in either direction (or, of course, even more-so when the null is a prediction from a theory). Because then results in either direction of the null will challenge at least one of those theories – experiments are primarily good in limiting confounds for those results. In many cases in the social sciences, a quantifiable prediction that a result will be in a certain direction, or in a fairly broad range, is probably as good as one can do (although I often think it should be fairly safe, and encouraged, to produce a smaller range – when authors calculate the power of a test based on a Pearson correlation of .10, say, then they seem to implicitly be stating a belief that the correlation will be of at least that size – and the goal should be to produce theories that can explicitly make more precise such predictions, as well as this is currently possible). Otherwise, I don’t really have anything against doing the test, I know reviewers will expect it, but then I think it is not that interesting, and that more focus should be put on, say, just estimating the effect size (this should be of interest in the former case as well though).
I think the problem with testing the null is the binary nature of the conclusion. Even if theories predict contradictory directions for an effect, there is no way a NHST proves either theory right or wrong. It does lead me to change my beliefs about the relative likelihood of the theories, but the uncertainty about truth will remain. I don’t share the concerns that NHST tests a hypothesis we know is not true – that seems superficial/trivial compared with the fact that it leads to a conclusion which sounds like uncertainty has been eliminated.
Dale,
I don’t like interpreting the test in terms of “rejecting” or “not rejecting” a hypothesis either, but just calculating the results of the test does not require one to interpret it in that way.
I was reading about this:
https://en.wikipedia.org/wiki/Substitute_king_ritual
I wondered, what distinguishes the NHST-based methods here from that ritual? Ie, some esteemed experts decide something.
Ie, are the premises more plausible? Is the logic more sound? Is there more evidence it works?
I would say no to all three. Pretend you are explaining to a king what makes your method better.
Hi Mathias (very Berggren),
I absolutely agree with this and also with Jessica’s response above – in (very) many settings an effect of exactly zero is indeed farfetched. I suppose I was focusing more narrowly on the position that Dale is hinting at in his comment: that all null hypotheses (of zero) are non-starters. I think that claim is sometimes accepted a bit to easily.
Thanks for a great post and interesting discussion!