I spoke at a session with the above title at the American Economic Association meeting a few months ago. It was organized by Serena Ng and Elie Tamer, and the other talks were given by Patrick Kline, James Powell, Jeffrey Wooldridge, and Bin Yu. In addition to speaking, Bin and I wrote short papers that will appear in the Journal of Econometrics. Here’s mine:
What is a standard error?
In statistics, the standard error has a clear technical definition: it is the estimated standard deviation of a parameter estimate. In practice, though, challenges arise when we go beyond the simple balls-in-urn model to consider generalizations beyond the population from which the data were sampled. This is important because generalization is nearly always the goal of quantitative studies. In this brief paper we consider three examples.
What is the standard error when the bias is unknown and changing (my bathroom scale)?
I recently bought a cheap bathroom scale. I took the scale home and zeroed it—there’s a little gear in front to turn. I tapped my foot on the scale, it went to -1 kg, I turned the gear a bit, then it went up to +2, then I turned a bit back to get it exactly to zero, and tapped again . . . it was back at -1. That was frustrating, but I still wanted to estimate my weight. So I got on and off the scale multiple times. The first few measurements were 66 kg, 65.5 kg, 68 kg, and 67 kg. A lot of variation! To get a good estimate in the presence of variation, it is recommended to take multiple measurements. So I did so. After 46 measurements, I got bored and stopped. The resulting measurements had mean 67.1 with standard deviation 0.7, hence a standard error of 0.7/sqrt(46) = 0.1.
Would I want to use the resulting 95% confidence interval, 67.1 +/- 0.2? Of course not! The whole scale is off by some unknown amount. What, then, to do? One approach would be to calibrate, either using a known object that weighs in the neighborhood of 67 kg or else my own weight measured on an accurate instrument. If that is not possible, then I would want a wider uncertainty interval to account for the uncertainty in the scale’s bias. The usual purpose of a standard error is to attach uncertainty to an estimate, and for that purpose, the usual standard error formula is inappropriate.
How do you interpret standard errors from a regression fit to the entire population (all 50 states)?
Sometimes we can all agree that if you have a whole population, your standard error is zero. This is basic finite population inference from survey sampling theory, if your goal is to estimate the population average or total. Consider a regression fit to data on all 50 states in the United States. This gives you an estimate and a standard error. Maybe the estimated coefficient of interest is only one standard error from zero, so it’s not “statistically significant.” But what does that mean, if you have the whole population? You might say that the standard error doesn’t matter, but the internal variation in the data still seems relevant, no?
One way to think about this is to imagine the regression being used for prediction. For example, you have all 50 states, but you might use the model to understand these states in a different year. So you can think of the data you have from the 50 states as being a sample from a larger population of state-years. It’s not a random or representative sample, though, in that it’s data from just one year. So to get the right uncertainty you’ll need to use a multilevel model or clustered standard errors. With data from only one cluster, some external assumptions will be needed to compute the standard error. Alternatively, one could just use the standard error that pops out of the regression, which would correspond to an implicit model of equal variation between and within years. Just because you have an exhaustive sample, that does not mean that the standard error is undefined or meaningless.
How should we account for nonsampling error when reporting uncertainties (election polls)?
In an analysis of state-level pre-election polls, we have found the standard deviation of empirical errors—the difference between the poll estimate and the election outcome—to be about twice as large as would be expected from the reported standard errors of the individual surveys. This sort of nonsampling error is usual in polling; what is special about election forecasting is that here we can observe the outcome and thus measure the total error directly. The question then arises: what standard error should a pollster report? The usual formula based on sampling balls from an urn (with some correction for weighting or survey adjustment) gives an internal measure of uncertainty but does not address the forecasting question. It would seem better to augment the standard error based on past levels of nonsampling error, but then the question arises of what to do in other sampling settings where no past calibration is available. In election polling we have some sense of that extra uncertainty; it seems wrong to implicitly set it to zero when we don’t know what to do about it.
How, then, should we interpret the standard error from textbook formulas or when fitting a regression? We can think of this as a lower-bound standard error or, more precisely, as a measure of variation corresponding to a particular model.
Summary
The appropriate standard error depends not just on the data and sampling model but also on the generalization of interest, and the model of variation across units and over time corresponding to the uses to which the estimate will be put. Deciding on a generalization of interest in a sampling or regression problem is similar to the problem of focusing on a particular average treatment effect in causal inference: thinking seriously about your replications (for the goal of getting the right standard error) and inferential goals, you might well get a better understanding of what you’re trying to do with your model.
Would I want to use the resulting 95% confidence interval, 67.1 +/- 0.2? Of course not! The whole scale is off by some unknown amount.
Is this last sentence true in principle or was there something in the measurements that lead you to believe that it is true?
Garnett,
The problem was that the position of the needle on the scale kept changing when I got on and off. Taking a large number of measurements gives me a sense of the variance of the scale at the time of measurement (which was several months ago; since then the scale has settled down a bit and its measurements have become much less variable), but there’s no reason to think the measurements are unbiased. It all depends on where I turn that gear.
Andrew – do you think the measurements are independent, or do you think that if you made a second measurement soon after the first the errors would be correlated? That would also affect the SE estimate.
Robin:
Good question. The data are public and are posted right here, so I can check:
So, yes, the observations are autocorrelated. The quick correction would be to increase the standard error by a factor of 1/(1 – 0.34) = 1.5, so the conf interval becomes 67.1 +/- 0.3. Better, but still too narrow because still does not account for possible bias.
“After 46 (!) measurements”
This short note has more original data than the typical econometric/stat paper.
Dean:
To paraphrase a now-famous researcher, I “relied on the help of research assistants on any given project to help prepare IRB applications, conduct laboratory studies, clean the data, prepare it for analyses, and often conduct preliminary analyses on the data.” In addition, I “did not run the studies or sort or handle the data.” And my “practice in this regard was consistent with other behavioral scientists in the field.”
So everything’s cool. And if there weren’t really 46 observations, well, “there is no evidence that the culture of my lab [ok, my bathroom] incentivized or motivated research assistants to manipulate data.” So nothing to see here. And if you claim otherwise, you can expect to hear from my lawyer, Alan Dershowitz.
I do not see what is gained by using the term sample in “[Y]ou can think of the data you have from the 50 states as being a sample from a larger population of state-years. It’s not a random or representative sample, though, in that it’s data from just one year.” If something is not randomized, then calling it a sample encourages people to apply what they learned about estimators under random sampling of datasets. Suppose you want to say something Frequentist about the data conditional on the population of states but marginal over the years. Having one year of data or the most recent T years of data does not allow you to do that.
Entire textbooks are written on the sampling implications for estimators of fixing the number of consecutive time points (T) but randomizing the starting point, which no researcher ever does. No textbooks are written about the implications for estimators for what researchers actually do, which is to fix the end point as close to the present as possible and take all the time points that are available up until then. So, researchers apply the sampling theory from the textbooks to situations that are almost the opposite of what the textbooks are assuming.
The role of randomness in making the magic happen is really really poorly understood and taught. What even *is* randomness? Most people can’t really give an answer to that. (though there is a perfectly good though non-computable answer, a sequence of numbers is random if it has high Kolmogorov Complexity)
If you inject 5 mice with a drug and 5 mice with a different drug is that “a random sample of the effects of each drug?” Hell no. Even if you used a random number generator to select which drug the mice get. This is just a complete census of the effects of the drug on *those mice*. In order to extrapolate to a broader population you must be taking the mice from the broader population. There’s nothing to say that your mice in your lab are similar to other mice in other labs. Strain differences can be huge. For example 20 years ago or so my wife had mice that would just suddenly seize up and die in your hands while transporting cages out of the mouse facility for experiments. After a long night of testing a bunch of different plausible causes, we discovered the reason… An ultrasound motion detector on the lights in the gowning and de-gowning vestibule of the facility was triggering audiogenic siezures in young mice of a particular strain that they were using in that facility.
The vast majority of frequentist p values are just mis-used even at the mathematical level. Without a high complexity sequence inducing sampling from a well defined population p values are actually meaningless.
Well that’s depressing.
I mean, it is and it isn’t. It’s depressing that so much science has been built up from so much superstition… But it’s freeing in that there’s a perfectly reasonable alternative… use probability not in terms of the properties of high complexity sequences, but in terms of the properties of plausibility numbers that obey Cox’s Axioms. Plausibility does not in **any way** rely on high complexity sequences / random sampling. Except in doing computing such as MCMC, where pre-verified pseudo-random algorithms are used to map between plausibilities, and frequency in computed quantities.
Can you explain further why you think using bayesian would be superior to frequentist methods in understanding results from these ultrasound-sensitive mice?
I mean, if you didn’t know about that issue I don’t see it making much of a difference. If the researcher did and failed to tell anyone, well that is a different problem.
My position is this whole bayesian vs. frequentist problem is negligible compared to testing your hypothesis rather than some strawman. So it is not helpful to frame it that way. If you want to test some non-toy hypothesis, then the bayesian tools are there while frequentist are not.
Anoneuoid,
First of all I agree that testing the wrong model is at the heart of what’s wrong with NHST. Let’s agree to that and not bother discussing it here. Instead let’s talk about what’s problematic with p values vs Bayesian methods with regards to extrapolation to a larger population even when you’re trying to test your own model instead of a “null”. The issue with the audiogenic siezures was just to show that there can be large differences between the outcomes in various mice without you knowing why and without it being “random” in the sense of high complexity sequences. Those mice weren’t “randomly” dying of seizures, it was a deterministic thing. And in general, you could have some much less noticeable deterministic thing going on, which affects your results in some way. For example your mouse might have somewhat extra activity in the kidneys that clears a drug more quickly than other kinds of mice, or be subject to higher risk of cancer due to excess metabolic oxidative stress.
Suppose you know you are testing a specific strain of mouse only, the ones you have available at your facility. Suppose you need to extrapolate to a different population than the ones you have available to test.
In a Bayesian analysis you can do something like:
strain_bias2 ~ SomeDistro()
parameter_difference ~ SomeDistro2()
# a likelihood
measured_outcome ~ AnotherDistribution(some_formula(measured_covariates),some_parameters)
# a prediction for future data in a different mouse strain
predicted_outcome_strain_2 ~ AnotherDistribution(some_formula(assumed_covariates) + strain_bias2,some_parameters+parameter_difference)
That is, you can build a model which is *self aware* that when predicting for a different strain, you should add (or in general, modify) the parameters of the probability distribution within some bounds or in some approximate way.
With a Frequentist analysis you are just SOL, there’s no observations of strain2 so there’s literally no outcomes to test against, there is no p value to calculate, no language in which you can express “the second strain should be similar in some ways to the first strain but with some modifications which are within some range of possibilities”.
By expressing these models carefully you can avoid fooling yourself, if that’s what you actually want to do. I think the biggest problem with the bizarro world science isn’t even NHST, it’s that a lot of scientists **want to be fooled** so long as it’s in a way that enhances their career, creates notoriety, etc.
>I think the biggest problem with the bizarro world science isn’t even NHST, it’s that a lot of scientists **want to be fooled** so long as it’s in a way that enhances their career, creates notoriety, etc.
Yes, I have wondered the same thing, but that is a very strong claim, because you are actually claiming that they are knowingly and willfully stealing large sums of public and private funding. That’s a pretty strong accusation, and while that certainly happens, I’m not sure that describes the majority… *want to believe in pet theory* and ignorant of the ramifications of their ‘methods’ might be more like it. I haven’t personally encountered a scientist wanting to be fooled, but I have encountered many (almost all) ignorant of the ramifications of poor analysis methods.
jd, I don’t mean they are intentionally fraudulent. That wouldn’t be “being fooled”, what I mean is they like using methods which result in “easy discoveries” and they don’t want to hear about how to do serious data analysis and intentionally inflate the uncertainty in their results just in the same way that maybe business people like selling expensive drugs and they don’t want to hear about how the average benefit for the population they are selling to is marginal or maybe even negative when you take side effects and economic costs into account. People still sell Tamiflu for example and can probably convince themselves that it has some benefit to the customer. I don’t think the Tamiflu people think “gee what a load of suckers we are fleecing” but they don’t want to know the truth either and will studiously avoid understanding it.
The Frequentist vs Bayesian controversies in stats help in that sense, some other expert people have looked into it and there’s controversy. A little like global warming but without the heat waves and monsoons to slap them in the face a little.
> what I mean is they like using methods which result in “easy discoveries” and they don’t want to hear about how to do serious data analysis and intentionally inflate the uncertainty in their results
Gotcha. I misunderstood your view. The head-in-the-sand sort of thing. Yes, maybe so… especially since the entire system of promotion and funding seems to reward this. And ironically, the ‘easy discoveries’ might be evidence to them that their methods ‘work’.