This one from 2017 is good; I want to share it with all of you again:
Our recent discussion with mathematician Russ Lyons on confidence intervals reminded me of a famous logic paradox, in which equality is not as simple as it seems.
The classic example goes as follows: Abraham Lincoln is the 16th president of the United States, but this does not mean that one can substitute the two expressions “Abraham Lincoln” and “the 16th president of the United States” at will. For example, consider the statement, “If things had gone a bit differently in 1860, Stephen Douglas could have become the 16th president of the United States.” This becomes flat-out false if we do the substitution: “If things had gone a bit differently in 1860, Stephen Douglas could have become Abraham Lincoln.”
Now to confidence intervals. I agree with Rink Hoekstra, Richard Morey, Jeff Rouder, and Eric-Jan Wagenmakers that the following sort of statement, “We can be 95% confident that the true mean lies between 0.1 and 0.4,” is not in general a correct way to describe a classical confidence interval. Classical confidence intervals represent statements that are correct under repeated sampling based on some model; thus the correct statement (as we see it) is something like, “Under repeated sampling, the true mean will be inside the confidence interval 95% of the time” or even “Averaging over repeated samples, we can be 95% confident that the true mean lies between the lower and upper endpoints of the confidence interval.” Russ Lyons, however, felt the statement “We can be 95% confident that the true mean lies between 0.1 and 0.4,” was just fine. In his view, “this is the very meaning of “confidence.'”
This is where Abraham Lincoln comes in. We can all agree on the following summary:
A. Averaging over repeated samples, we can be 95% confident that the true mean lies between the lower and upper endpoints of the confidence interval.
And we could even perhaps feel that the phrase “confidence interval” implies “averaging over repeated samples,” and thus the following statement is reasonable:
B. “We can be 95% confident that the true mean lies between the lower and upper endpoints of the confidence interval.”
Now consider the other statement that caused so much trouble:
C. “We can be 95% confident that the true mean lies between 0.1 and 0.4.”
In a problem where the confidence interval is [0.1, 0.4], “the lower and upper endpoints of the confidence interval” is just “0.1 and 0.4.” So B and C are the same, no? No. Abraham Lincoln, meet the 16th president of the United States.
In statistical terms, once you supply numbers on the interval, you’re conditioning on it. You’re no longer implicitly averaging over repeated samples. Just as, once you supply a name to the president, you’re no longer implicitly averaging over possible elections.
So here’s what happened. We can all agree on statement A. Statement B is a briefer version of A, eliminating the explicit mention of replications because they are implicit in the reference to a confidence interval. Statement C does a seemingly innocuous switch but, as a result, implies conditioning on the interval, thus resulting in a much stronger statement that is not necessarily true (that is, in mathematical terms, is not in general true).
None of this is an argument over statistical practice. One might feel that classical confidence statements are a worthy goal for statistical procedures, or maybe not. But, like it or not, confidence statements are all about repeated sampling and are not in general true about any particular interval that you might see.
P.S. More here.
Here we go again. Every time I see this discussion, I reach the same conclusions.
I can agree on several things: the correct definition of the confidence interval, the misuse of confidence intervals to make binary statements (just NHST in another form), and the easily misinterpreted use of the word “confidence.” But I think this insistence of using “repeated sampling” is counterproductive in practice. When someone has a single sample and the estimated confidence interval is [0.1, 0.4], if you insist on the correct definition, then what use can someone have for those particular numbers, 0.1 and 0.4? If you really want to say “no use” then why bother doing any analysis? It’s fine to point out the importance of repeated sampling and the correct definition, but we often don’t have more than the one sample (which, as you often point out, is rarely random). As I see it, we have two choices then: either use the incorrect interpretation of this particular interval, or say nothing at all. If your objection is that we shouldn’t use the words “confidence interval,” but instead use “uncertainty interval” or some other terminology, I’m fine with that. But I can’t see the usefulness of concluding from the study that if we did repeated random sampling, then 95% of the intervals derived in this way would contain the true value. I can reach that conclusion by not having any data at all.
Now, Anon may insist (perhaps I’m wrong, but this is what I’d guess) it is better to conclude nothing that use the incorrect interpretation. But when I use this in courses, I provide the correct definition but then pose the question: is this particular [0.1, 0.4] interval one of the “lucky” 95% that include the true value or one of the “unlucky 5%” that do not? What is your best assessment of that probability?
Dale:
In the above post I’m just taking the position that confidence intervals are a method that people use, so we should try as best as we can to understand what they mean.
Its just a computationally efficient approximation of the credible interval, in particular using a uniform prior.
This is how everyone uses them except a small group of math-centric people whose minds are apparently overtrained in precise deductive logic to the point they lack intuition for the role of systematic error and the general messiness of reality.
Heres Fisher on this topic: https://www.york.ac.uk/depts/maths/histstat/fisher272.pdf
Anon:
In many cases, yes, I agree that this is what the confidence interval is, and it has all the problems that we’ve discussed with the uniform prior. But developers of statistical methods and applied researchers will sometimes take that Neyman-Pearson logic literally and define and use confidence intervals that are constructed by inverting hypothesis tests. This can create all sorts of problems such as this situation. So, again, I think it’s useful to identify the Abraham-Lincoln-style logic fallacy that leads to much confusion among statisticians as well as practitioners. See here for further discussion of this point.
+1
I recommend reading that last paragraph of Fisher’s missive. In general, Anoneuoid’s reference illustrates Fisher’s famous forensic abilities. Here is perhaps the most well-known critique of Fisher’s way with words:
https://orsj.org/wp-content/or-archives50/pdf/e_mag/Vol.03_04_145.pdf
Paul:
I hate that “Sir Ronald” crap. As an American, I don’t like titles of nobility. If you can talk some government into calling you a lord or a duke or a knight, then, sure, go for it, put it on your business card, whatever. But the hell I’m gonna start calling you Lord Worcestershire or Sir Karl or whatever.
So according to Neyman, if you calculate the CI *from the data* then you are doing it wrong (and in fact you need the credible interval instead). The CI is supposed to be derived from your theory then compared to data you collected later.
I hadn’t gathered this from his other writings, but makes perfect sense to me. I’d even say that should go in the lede of the CI wikipedia page. I doubt it would last though.
Anon, to reinterpret that statement.
There’s no way to calculate an interval before getting data. What there is, is a way to calculate the frequency with which your interval generating scheme will generate an interval that contains the correct interval (conditional on your model of the random variable being true, which is always false except in purely computational schemes with RNGs).
That’s [Theta1(X),Theta2(X)], where the capital X is a random variable, and therefore this is a whole family of intervals. All the future intervals you might get put into a bag so to speak.
[Theta1(x),Theta2(x)] on the other hand is the particular interval you get from the particular data you got in your experiment. It’s one draw from the bag so to speak.
He says you can only calculate the frequency with which the intervals in the bag contain the parameter, he says you can’t get a probability that [Theta1(x),Theta2(x)] contains the correct interval “without a prior” because you’d need a Bayesian interpretation to give that specific interval some probability.
If you flip a fair coin 3 times how many heads do you expect to get?
You work out the combinations:
HHH
HHT
HTH
THH
HTT
THT
TTH
TTT
Aggregating by number of heads:
Zero: 1
One: 3
Two : 3
Three : 1
That gives us a 6/8 = 75% confidence interval of [1, 2]. No data required, its derived from the model.
If we repeatedly flip this coin 3 times, our interval will contain the observed value 75% of the time. That is 75% coverage, correct?
There’s no inference to do in your example. You assume a “fair” coinflip with p = 0.5 known exactly. Nothing is unknown except the data.
Flip a coin *with an unknown p*, you can make a 100% interval, since the parameter is on a compact set. You have 100% confidence it’ll be in [0,1]. But you can’t make a 75% interval for p without data.
Don’t use them in most cases. It’s ring toss https://twitter.com/epiellie/status/1073385427317465089
Nice graphic analogy, but I don’t follow your conclusion (“don’t use them in most cases”). I thought the issue was along the lines of what is the probability that the ring toss will fall over the target. The correct definition would say that upon repeated ring tosses, 95% of them would capture the target. I would say that the probability that any one toss will capture the target is 95% – of course, that probability is either 1 or 0, but before the toss is thrown I would say the probability is 95%.
When you say “don’t use them in most cases,” then what do you use?
Just build the joint model in Stan and work with HMC samples from the posterior distribution for parameters or quantities of interest. That is actually the practically simplest implementation. In most cases, I don’t see that the convoluted notion of ‘confidence intervals’ provides much that is helpful. In addition, from the actual implementation standpoint, determining confidence intervals is far from trivial without a load of baked in assumptions! For example, try obtaining confidence intervals for a basic hierarchical GLM. It takes longer to run all the bootstrapping than to simply fit the same model in brms or rstanarm. Finally, a narratively generative model built as a full Bayesian model is far better (in my opinion) to implement than defaulting to a regression or glm, and most statistical software that gives you confidence intervals relies on heuristic approaches like regression.
In summary, confidence intervals have major drawbacks: 1) the definition seems of limited usefulness, 2) unless you have the mathematical prowess to pull it off, you are stuck with obtaining them from heuristic approaches and asymptotics.
I think Michael Betancourt has some nice thoughts on this in sections 2.2, 2.3, and 4 here https://betanalpha.github.io/assets/case_studies/modeling_and_inference.html#22_set_estimators
If you are suggesting using credible intervals in place of confidence intervals, that I don’t disagree – despite my lack of Bayesian training. And, if you prefer the posterior distribution to any particular credible interval derived from it, I don’t have any disagreement with that either. If neither of these apply to your comment, then I’m not understanding what you mean.
Given that I was not trained in Bayesian analysis, but have read enough on this blog to appreciate it, I have two inconsistent thoughts about Bayesian methods. First, I completely agree with the position that Andrew and others have often stated that all analyses require assumptions, and that the Bayesian specification of priors is explicit and superior to either completely uninformative priors or making implicit assumptions. On the other hand, I still find the idea of specifying priors daunting. Many of the things I model have no clear prior distributions without imposing a sort of momentum to past research. If a few studies have suggested a particular size of an effect, then I am bothered by imposing some constraint on a new study, based on those prior efforts. This is somewhat analogous to the phenomenon of the first published study receiving undue influence over subsequent efforts.
As an example, I have done some studies of telecommunications investment as functions of regulatory interventions. Previous studies exist – but there is no consensus about the size of the effects (or even their direction), and most studies are prone to bias due to funding sources. I have no idea what prior to use unless I want to align myself with one of those prior efforts. But I recognize the value in explicitly stating my prior distribution as opposed to pretending that nothing is known from the past. I still don’t know how to resolve this tension.
>On the other hand, I still find the idea of specifying priors daunting.
Sure, it can be, but so is specifying the functional form of the relationship between regulatory interventions and telecommunication investments, the likelihood, and every other part of the model. It’s assumptions all the way down.
One of the nice reasons to use the full Bayesian modeling approach by building the joint model in Stan (or something similar) is the ability to interrogate the assumptions that you build into the model both before you fit it to the observed data (prior predictive checks) and after (posterior predictive checks). Let’s say you have some model where telecom investment is some particular function of regulatory interventions. Maybe you also have a model for regulatory interventions as well, so it’s a joint model. You’re interest is a generated quantity from the model, ‘$ invested’, under some regulations vs without those regulations. You can actually use HMC to sample from your joint model, ignoring the likelihood with observed data, and generate simulations of the quantity ‘$ invested’ from your priors. While the parameters in your model might be difficult to reason about in terms of prior distributions, you probably have some concept of what values for ‘$ invested’ are even plausible. For example, ‘$ invested’ is likely not as large as the GDP of the US, but it could possibly be next to nothing. That’s a broad range, but the GDP of the US isn’t infinity, either, so you do have some information. I find that you pretty much always have at least some information. Oftentimes, the key to setting good priors is getting them into a form that is interpretable to you, so that you can actually apply the information that you have. If you can’t reason about them in terms of each parameter in the model, then (and even if you can reason about them, this is still a good idea) you can in terms of some familiar quantity. Setting priors, simulating data, checking, and re-setting them is often an iterative process.
Dale. Priors can sometimes be hard to specify, mostly in cases of very high dimensional models. But often they’re not that bad. If there are a number of prior expert opinons for a number “a”, just specify a prior that includes in its high probability region the entirety of all the values the previous expert opinions have given for “a”. Then plug in your data and get a posterior.
Daniel, JD:
Why not show us what you mean in practice? The current hot case in Minneapolis, the controversy over the “right” wage for “ride share” drivers, seems highly analogous to Dale’s regulatory interventions in airline pricing. You can solve it! Show us how Big Data can bring prosperity to the world! Make those long gone members of the Office of Pricing Everything from the former USSR slap themselves in the forehead and say “oh, darn, if we had only had Stan and built hierarchical models!”
Chipmunk, nowhere in this thread will you see anyone suggesting they could do a statistical analysis and use it to run a command economy.
In the statement “We can be 95% confident that …”, I don’t know what “confident” means, I don’t think it’s a well defined term. It *seems* to be used interchangeably with “probability”, but then why not say “there is a 95% probability that …”? I think the term is being used as a kind of shorthand, but a shorthand for what isn’t usually defined. We have to guess.
Suppose we could somehow know the true mean of a distribution – I mean one that we have sampled. One example would be if we measured the length of the standard meter bar. By definition it is one meter. There is no probability attached to its value. (Let’s ignore that that bar in Paris isn’t the official standard anymore, having bee superseded by some number of wavelengths of a light emitted by a certain element ). One could say “When I measure the meter bar a number of times and get a sample mean, 95% of the time the true mean, which I happen to know, will lie within the 95% confidence limits that I construct from the measured data”.
Whew, quite a mouthful! No wonder someone would want to use a shorthand expression. Too bad it’s not the only statement the shorthand could mean. In my view, using the word “confident” suggests a higher degree of certainty than is warranted.
I would take improperly understood CI over the binary nonsense of NHST, any day.
The way CIs are graphically represented (along with SD or SE error bars) doesn’t help either. Instead of a hard stop at the end of the CI line, they should be getting wider and fainter, fading into oblivion. Uncertainty would be easier presented that way, so the reader doesn’t think there is some magic ‘force’ causing the CI to stop at the end of the line.
Also, the term ‘confidence’ which we use so casually, really only pairs well with number 100% and any other number leaves some wiggle room. What does it mean to be 60% confident? Isn’t it really saying that one is 40% NOT confident?
Navigator:
I prefer the term “uncertainty interval” or “compatability interval.” Also, yes on not emphasizing the endpoints of the 95% interval. This comes up in statistical computing as well: sometimes people want to have enough posterior simulation draws to compute the endpoints of the interval to 2 digits of accuracy, and I’m like, Who cares about the endpoints of that interval? The quantity of interest is the underlying parameter itself, not its 95% interval.
Andrew, with all due respect, I here stand with Russ Lyons (whom I don’t know) and Freedman, Pisani, Purves (whose Italian translation I appreciated as a teacher): the statement “We can be 95% confident that the true mean lies between 0.1 and 0.4” is a technical one, which means precisely that 0.1 and 0.4 were obtained as the left and right endpoints of an interval estimator that works (covers the true mean) with probability 95%. When you plug in the numerical values, you choose to feel as you are not 5% unlucky, and you should have studied enough statistics to know what your choice means.
Then, I am aware of the fallacy of placing confidence in confidence intervals (https://link.springer.com/article/10.3758/s13423-015-0947-8) but I also think that no one would like to use a method that works 49% of the time. Would you?
Ideally, I would like to use an interval estimator that is principled (formed by high likelihood or high posterior density values) and has a good coverage probability, so that I can have confidence in a set of values supported by the data. Then, of course, I am ready to discount that confidence value, because it is conditional on the chosen model, and I don’t like overconfidence, but that’s another story.
I should have written “I don’t want to confuse the model with reality” rather than “I don’t like overconfidence” to be more specific about my viewpoint.
Luca:
Ultimately, it’s just words. If you are willing to define “confident” in the way that you say, then that’s all fine. The difficulty is that the word “confident” as used otherwise in the English language implies a degree of certainty in a particular outcome. Technical meanings will never line up exactly with common usage, but I think it’s a problem when the two meanings differ by a lot.
Again, this statement about usage of the language is not a statement that confidence intervals, or uncertainty intervals, or compatibility intervals, are a bad thing, or that they should not be used in applied statistics; it’s just a statement about what is the meaning of the procedure and how it is often understood. As I wrote in the above post, none of this is an argument over statistical practice.
I see your point, as far as a non-native English speaker can, and yet I think a word for that property is needed, and I can’t see a better one (at the present stage). Thank you for this discussion.
95% confident is technical bullshit for “the interval generating procedure output [a,b]”. If that’s what you mean, say it.
There are CI procedures and datasets that put together will output an interval [a,b] that is 100% excluded from containing any real parameter by basic logic (for example a CI which outputs negative numbers for a logically positive value).
Would you be “95% confident” that the average mas of an acorn is in the interval [-3,-1] grams? Yet, it’s possible under some conditions for that to be the result you get.
“95% confident” is a meaningless phrase used to lie.
Daniel
Your dislike of the word “confident” is fine – I don’t like it either (although none of the substitutes are particularly good either). But pointing out that it is possible to get a 95% confidence interval that completely excludes possible values is a weak argument against using these intervals. How common is that case? Does it always occur? From my experience, these “impossible” confidence intervals are relatively rare – and when they occur, usually clear thinking will alert you to their impossibility. Because some of these intervals are impossible, doesn’t make them all so.
Dale, this is simply a device to make it clear that there is no *logical* connection between “confidence” and “the interval” though. We need not have any kind of “confidence” at all for any given interval. We can be anything between 0% and 100% sure (Bayesian probability) that the “right” value is in the interval, all values on that spectrum are possible regardless of the 95% name on the tin.
Daniel
Again. “We can be anything between 0% and 100% sure…” Are you saying that all of these values are equally likely? In other words, because there are cases where the confidence interval contains impossible values, there is no value whatsoever in the particular confidence interval obtained from the sample data you have? That seems unduly harsh to me. Follow up question: would a credibility interval obtained from a Bayesian analysis be ok to use but not the confidence interval? If so, then what about the rare cases where the credibility interval produces absurd results – or is that impossible?
Much like some of the other threads lately, I think the words “confidence” and “guaranteed” cause lots of problems. It would be nice if we could avoid their use and find more humble words to use. But if we had such words would you still maintain that the intervals that are derived are meaningless to look at?
Dale, a Bayesian Credible interval can not produce absurd results unless you’re failing to fully use your prior information. If the prior has zero probability in a region the posterior does too.
CIs are useful precisely when they are decent and quick approximations of Bayesian Credible intervals.
Basically do Bayes, or use CIs when they’re convenient and computationally quick approximation of Bayes.
Daniel: I’m not sure I understand how this would happen with confidence intervals. It would be interesting to have an example where this happens to see how much of a problem it is in practice. Otherwise, isn’t the problem with the [-3,-1] interval, or intervals that contain impossible values (how you get an interval from -3 to -1 with something measured in grams, which should logically all be above 0, I don’t understand), that you are failing to “fully use your likelihood information”? (To paraphrase what you say about prior information for the necessary conditions for the credible interval to never produce absurd results.) An example would be when you are assuming a normal distribution for a binomial variable.
Anon:
The only restriction on Confidence intervals is that 95% of the time the 95% confidence interval contains the true value if the data really does come from the assumed model. There is no requirement that confidence intervals come from likelihoods, and indeed people like Deborah Mayo argue strongly *against* using likelihoods or at least limiting yourself to likelihood based intervals.
But even a likelihood based interval can be bad. Consider the case where you have a noisy scale which measures accurately on average but with normal(0,3) noise. You measure something that’s 0.5 grams 4 times, you get unlucky and all 4 measurements are negative.
> rnorm(4,0.5,3)
[1] -2.64869631 -1.66590938 -2.78412416 -0.08657093
What’s the likelihood based confidence interval for the mean? Remember the likelihood is based *purely* on your knowledge of the **measurement apparatus** which has normal(0,3) distribution around the correct answer. Any information you use about the fact that objects can only have positive weights is **prior** information which is off-the-table according to Frequentist inference.
Also, a legitimate confidence procedure is 95% of the time to return (-inf,inf) and 5% of the time return {}. According to the requirements of a confidence procedure this satisfies those requirements. That no-one would use such a procedure is no excuse, because someone might use another procedure which is less obviously wrong but still can return something absurd. Even the likelihood based inference for the above 4 measurements example is fairly absurd:
> t.test(stuff)
One Sample t-test
data: stuff
t = -2.888, df = 3, p-value = 0.06312
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
-3.7757947 0.1831443
sample estimates:
mean of x
-1.796325
The section of the interval -3.776 to 0 is impossible, the region 0 to 0.183 should be our new interval?
The right way to handle this is Bayesian Inference. The fact that people rarely see these really stupid CIs is basically because in the real world mostly people only use Frequentist inference when it gives an approximation of the Bayesian result for a stupid simple Bayesian model.
Daniel says:
“ Any information you use about the fact that objects can only have positive weights is **prior** information which is off-the-table according to Frequentist inference.”
I’m not a frequentist, but I don’t think this statement is accurate.
It does not seem to violate any principles of frequentist statistics for someone to assume the model X~N(mu, 3), with mu>0 based on physical constraints.
Here’s an example of frequentists calculating interval estimates in the presence of physical constraints on the parameters:
https://www.slac.stanford.edu/econf/C030908/papers/MOAT004.pdf
JYD:
So, their “minimax expected size procedure” requires the logic of using the likelihood. They essentially truncate the region of interest by multiplying by the prior 1/(2 tau) on the region [-tau,tau]. You can obviously generalize that to other finite intervals. But in essence it works because it’s a modification of Bayes with a flat prior on a finite interval.
In my experience, whenever a Frequentist procedure makes no sense, the solution is always to make it closer to Bayes. That’s what they’ve done here. Or, you could just use Bayes.
I’ve always thought that at least part of the difficulties folks create for themselves when communicating what the 95% corresponds to is that they’ve overloaded the term ‘confidence interval’ to mean both the ‘interval estimator’ and the ‘interval estimate’. Obviously the probability statement is a property of the interval estimator and not any particular interval estimate.
I think a fair number of introductory mathematical statistics texts are very clear about this, and go to the trouble of using the extra word to be precise.
I am also opposed to suggesting that the word ‘confidence’ has a special technical meaning that makes the statement (B) clear when the interval being discussed is an interval estimator. We already have the precise language of probability theory to describe this. We don’t need to use the problematic word confidence in these types of statements at all.
I like this point. We have unbiased estimators but not unbiased estimates. “Unbiased” describes an estimator just as “95 percent” describes an (interval) estimator, and these properties simply don’t apply to estimates.
All of this fussing over wanting to apply some confidence or compatibility or what have you to an obtained interval is just ridiculous. Where are so many scientists needing to write about being 95% confident when using a CI?
When someone writes about a statistical test they don’t feel the need to explain the probability of a Type I error. The result obtained is described as statistically significant or not. That’s it. All of this discussion of it being hard to write about CIs is because people want to provide some kind of statistics lesson in their writing or convey in some way how they feel about it (subjective meaning of confident). If they just did the same thing as with a test, make a decision, then they wouldn’t have this problem. (And yes, I’m advocating a binary decision but a more useful one than NHST and the one that defines the CI)
Consider how the 95% probability is obtained. Through repeated sampling, calculation, and *decision making* one obtains a 95% probability of capturing the population value. It is this procedure that the probability is attached to and the procedure, just as with a significance test, requires a decision. The decision is that the population value is in the interval. That’s it. Once you know that writing about them is easy because you don’t have the choice to talk about confidence in your obtained interval. There’s no 95% unless you decide the population value is in the CI.
For example, you would just write, “The mean effect is between 1 and 4.” That’s the entire sentence. As long as somewhere, parenthetically, comma separated, at the beginning of the results writing, wherever, it is made clear that the kind of interval that is being discussed is a 95% confidence interval, there is nothing more to be said than to substantively interpret just what 1 to 4 means. It’s redundant and shows a complete lack of understanding to call it significant. But it is useful to say things like, the effect is between a small and moderate size, or the effect is rather large, or whatever 1 and 4 mean in the context.
Perhaps a little focus on interpreting actual values, something useful that can come from using a CI (whether confident or credible), and less on trying to find a short hand for a technical term description that one should avoid unless they’re writing a book on introductory statistics, would lead to more productive discussion.
I don’t get your suggestion at all. “The mean effect is between 1 and 4” sounds worse than almost any other badly worded interpretation of the interval. At least the 95% “confidence” admits that other values are possible. It seems like your position is that, because all of these words are subject to misinterpretation, you would prefer we hide between jargon. “Significant” or “not significant” and “the interval is (1,4)” all presume that the readers understand what these terms mean. So, we can avoid the ambiguities and misunderstandings by simply assuming that these terms are well understood and thereby avoid providing any interpretation.
Unless I am misunderstanding you, this doesn’t seem to avoid the issues with confidence intervals.
“Mean effects between 1 and 4 are highly compatible with the observed data.”
So, the question becomes whether “highly compatible” is less subject to abuse and misunderstanding than “confidence.” I suppose we could make a list of all the potential terms to describe uncertainty of results and conduct some research on which ones are less prone to misunderstanding. Of course, we then need to consider heterogeneity; some words are less prone to abuse by some groups and not others. The only problem is that we will need to find a way to describe the results of such a study that avoid binary thinking and overconfidence.
How shall we rate the veracity of historical claims?
Is it more likely that Caesar was in Gaul than — say — Germanus of Auxerre made a second visit to Britain in the 420s?
Let us not split hairs too finely in treating that word “likely” as if it must (by virtue of being uttered) carry with it the same content never mind the context.
We must remember I think Aristotle’s dictum (which he never failed to repeat at the start of his lecture … on every conceivable topic too): We must remember to only seek so much precision as the subject matter proper will admit.
“We can be 95% confident that the true mean lies between the lower and upper endpoints of the confidence interval.” I don’t think that a true mean exists (unless we’re talking about a well defined finite population), so I don’t feel confident at all about any statement that has a “true mean” anywhere. Furthermore I strongly dislike an interpretation of a statistical result that starts with “we” (assuming that it is meant to include me as the reader) as I don’t want a prescription of what I should feel from statistical analyses done by other people, thank you very much. Other than that, as mentioned before, as “x% confident” doesn’t have a well defined meaning outside the confidence interval context, it is not in fact an interpretation (as it pretends to be), rather than introducing new and superfluous terminology for not saying anything else than “the 95% CI is (…)”
I’m a pluralist and rarely out for banning anything, but if I could ban any “interpretation” that starts with “we can be 95% confident”, I would. Please stop using this!
Nicely said.
> I don’t think that a true mean exists (unless we’re talking about a well defined finite population), so I don’t feel confident at all about any statement that has a “true mean” anywhere.
Would you feel confident about any other statement of the form “95% confident that X lies between the lower and upper endpoints of the confidence interval”? It seems that any X we make statements about would need to be in some sense “true” and “well defined”. Can you give an example of any value that exists and we can make statements about?
What a confidence interval does is well defined within the model. The model is a tool to think about reality. If we’re thinking about reality in terms of the model, we can think about the effect we’re interested in in terms of the corresponding model parameter, and the CI gives an indication of uncertainty based on this. I’m all fine with that (of course potentially questioning whether the model is a good one in a specific situation). This doesn’t mean I’m confident that the truth in reality behaves as the model prescribes. Reality is infinitely more complex than any statistical model, and we simplify to get a handle on it. I don’t like the term “confident” as far as it suggests that I should feel confident about reality behaving as a formal model (at best I may be confident that in certain situations we don’t have better options than model-based analysis). I don’t think science is about what I should feel. One may go on and say it shouldn’t have been called confidence interval in the first place, and I’d have a hard time arguing against that, but at least the term “confidence interval” doesn’t come with an “I” or “we” that is supposedly confident.
> .. a famous logic paradox, in which equality is not as simple as it seems.
> The classic example goes as follows: Abraham Lincoln is the 16th president of the United States, but this does not mean that one can substitute the two expressions “Abraham Lincoln” and “the 16th president of the United States” at will.
Fun. I used to teach this material every year. It’s only a paradox if you start with the assumption that you should be able to freely substitute two expressions that have the same concrete referent. This is patently false in natural language, so we tend not to think of it as a paradox per se.
Nouns or other expressions where you can’t substitute another expression with an equivalent reference are called “intensional” in linguistics and philosophy. At the risk of sounding like Andrew, you can read chapter 11 of my semantics book :-). Half of my dissertation was an analysis of intensional adverbs. Intensionality is what motivated Frege to create an intensional logic, which Russell then showed was inconsistent due to its overly general comprehension axiom. This all got fixed in the well-founded/stratified ZF set theory.
Another example is the de dicto / de re distinction, as you see in “I’m hoping to find a unicorn.” We can’t just use the reference here because there are no unicorns! But if you say “I’m looking for a restaurant”, it can go either way—there might be a specific one, or you might just have a general requirement. The place this is most apparent is with propositional attitudes, where we talk about people’s attitudes toward facts/concepts/propositions, as in “John believes the earth is flat”. This doesn’t entail that John believes every false thing as it would in a coherent logic.
Yes, if you then collect data you can compare it to the model prediction and get p(D|H). If we only consider the single fair coin then p(H|D) = p(H)*p(D|H)/p(H)*p(D|H) = 1
In other words, p = 0.5. There *is* no further inference to made about p.
To do that we need to consider other models. Perhaps (assume) we have two coins in our pocket, (assume we) pull one out with equal chance, flip it, and put it back. Do this three times.
Lets further assume one coin is fair and the other is double-heads. Then you have four possible outcomes for each trial:
[H1, T1, H2, H3]
I worked this out in julia: https://pastebin.com/5esbVQWw
Counts:
0 => 1
1 => 9
2 => 27
3 => 27
Proportions:
0 => 0.015625
1 => 0.140625
2 => 0.421875
3 => 0.421875
So first of all now this makes clear that p is an average value. In the single fair coin model half the possible outcomes correspond to heads, ie p = 0.5. Now we are considering a second model where p = 3/4 = 0.75 correspond to heads. Further, the single fair coin is indistinguishable from a set of one double-headed and one double-tailed coin.
Given these two models, a priori we can say p may be either 0.5 or 0.75. Say we flip three times and observe zero heads. Further, as far as we know either model is equally likely:
Priors:
p(H_0) = p(H_1) = 0.5
Likelihoods:
p(D|H_0) = 0.125
p(D|H_1) = 0.0156
Posteriors (priors cancel out):
p(H_0|D) = 0.125/(0.125 + 0.0156) = 0.89
p(H_1|D) = 0.0156/(0.125 + 0.0156) = 0.11
Now we can make some inference about p, ie that it is ~8x more likely to be 0.5 than 0.75. Then we can consider maybe there are three coins in our pocket, two fair and one double headed. Ie, p=4/6=0.67
Then we would get:
0 => 8
1 => 48
2 => 96
3 => 64
Which works out to:
p(H_0|D) = 0.125/(0.125 + 0.0156 + 0.037) = 0.703
p(H_1|D) = 0.0156/(0.125 + 0.0156+ 0.037) = 0.087
p(H_2|D) = 0.037/(0.125 + 0.0156+ 0.037) = 0.208
Keep considering more variations on sets of fair and double-sided coins, and eventually we’ll get a distribution where it makes sense to summarize with an interval for p since there will be so many different values.
We can even approximate it as a continuous distribution and come up with a dozen different ways to calculate binomial confidence intervals since none of them are really correct (they allow for irrational values of p).
I didn’t read through the whole thing in detail, but you’re talking about posterior probabilities for unknown p, not confidence intervals. You can calculate posterior probabilities before data, that’s called a prior. You can’t calculate confidence intervals before data, because a confidence interval tells you which interval contains values that are all “compatible with the data” in the sense that a hypothesis test of that value of the parameter using the data as input doesn’t reject that value of the parameter.
Andrew likes to say something about not generating confidence intervals by inversion of hypothesis tests, but all confidence intervals are inversions of *some* hypothesis test.
You already agreed we can calculate one derived from the model… for the outcomes. You can also use an empirical distribution from prior data. Check 100 known good widgets, 95% weigh out at x to y grams.
Then compare your new data to those values (eg, QA for manufacturing tolerances). That is the valid usecase.
What you can’t do is invert this and calculate (a valid) one for the model parameters from the sample. Of course that is what everyone wants to do, but it doesn’t actually work.
Starting from the simplest coin flip scenario, it doesn’t actually work. It only “works” as an approximation insofar as:
1) The different parameter values are equally likely a priori.
2) Your choice of continuous distribution approximates the (actual) discrete one for your problem.
The latter issue applies to bayesian credible intervals too.
I don’t think what you’re describing is a confidence interval. Confidence intervals are explicitly about parameters. If you know all the parameters, and want to ask questions about data you don’t need “confidence” you can just talk about probability, because the probability of a given set of outcomes given fixed parameters is well defined by the parameters and the specification of the distribution.
From wikipedia: https://en.wikipedia.org/wiki/Confidence_interval#Definition
“Let X {\displaystyle X} be a random sample from a probability distribution with statistical parameter θ {\displaystyle \theta }, which is a quantity to be estimated, and φ {\displaystyle \varphi }, representing quantities that are not of immediate interest. A confidence interval for the parameter θ {\displaystyle \theta }, with confidence level or coefficient γ {\displaystyle \gamma }, is an interval ( u ( X ) , v ( X ) ) {\displaystyle (u(X),v(X))} determined by random variables u ( X ) {\displaystyle u(X)} and v ( X ) {\displaystyle v(X)} with the property:
P ( u ( X ) < θ < v ( X ) ) = γ for every ( θ , φ ) . {\displaystyle P(u(X)<\theta <v(X))=\gamma \quad {\text{ for every }}(\theta ,\varphi ).}"
In any case I think Carlos is maybe doing a better job of explaining the confusion than I am.
All that is fine, but has nothing to do with your claim that “according to Neyman, if you calculate the CI *from the data* then you are doing it wrong (and in fact you need the credible interval instead). The CI is supposed to be derived from your theory then compared to data you collected later.”
You were quoting a paragraph from Neyman about “the process of estimation is that by interval [θ1(X), θ2(X)], the confidence interval, appropriately defined before making the observations on X.”
That confidence interval is _defined_ – as a function of X – before observing X but it cannot be _calculated_ until you observe X.
Neyman is not saying that calculating the CI from the data is wrong – the data is used to determine the limits of the interval.
What he says is that one shouldn’t misunderstand the probability regarding the rule used to calculate intervals for different observations as a probability regarding the interval calculated for some particular observation.
That’s right. the capital X is a big bag of possible outcomes… the lower case x is one data set drawn from the bag.
You can’t calculate theta1(X) as a number until you’ve drawn the lower case x from the bag X.
Exactly. Where does this bag of possible outcomes come from? Either its prior data or you assume it. I think its explained quite clearly by that coinflip model, but I guess not.
I don’t think Neyman realized what he was saying, he was forced to just explain it in a revealing way when explain what was wrong with Fisher’s “fiducial probability”.
A confidence interval is an answer the following question:
ASSUMING a data generating model of the form p(x; q) (this is the big bag X, an assumed RNG with unknown q parameter, we don’t use a conditioning bar here because the only probabilities involved are on the x outcomes) and a set of observations {x_i} which values of q (the unknown parameter) are compatible with the data {x_i} in the sense that the test statistic t({x_i}; q) has tail probability greater than 0.05 (or whatever cutoff you choose) for all those values of q in the interval.
This is the “inversion of the hypothesis test” formulation.
So the big bag X is definitely an assumed data generating process. But note that the interval is in parameter space q. If you just tell me a value of q, it’s true we can calculate probabilities for data x but this is just from the definition of the model, it’s not any kind of inference, in that case it’s assumed, no testing required.
I would say that Neyman understood perfectly that a confidence interval cannot be calculated until you “reach within the bag” for an observation – nothing in what he wrote suggests that calculating the CI from the data is wrong.
> Where does this bag of possible outcomes come from? Either its prior data or you assume it.
What I think that you don’t understand is that in Neyman’s discussion the content of the bag is not well defined: it depends on an unknown parameter. There is a model that says that for any value of the parameter it will follow the corresponding distribution. But the parameter is just unknown: there is no prior data or assumption about it.
There is a definition of ghosts too. That doesn’t mean they exist.
Go back to the simplest statistical/probability problem:
https://projecteuclid.org/journals/statistical-science/volume-16/issue-2/Interval-Estimation-for-a-Binomial-Proportion/10.1214/ss/1009213286.full
No confidence interval exists! Just lots of calculations that fail to achieve the defined properties. Isn’t that strange?
CIs only appear to exist for more complex problems if you smooth/abstract away the actual probabilities. Just like blurring a photo can create “ghosts”.
Reread Neyman’s quote more carefully, paying attention to what he wrote rather than what (at least we think) he meant.
Anon
Thank you for the link to that interesting article – which I was not aware of. I would note that they say “The oscillation in the coverage probability is caused by the discreteness of the binomial distribution, more precisely, the lattice structure of the binomial distribution.” You appear to be suggesting that this is grounds for ignoring all confidence intervals. That seems an overly strong conclusion in relation to this particular demonstration. Are there similar results for continuous distributions? I would also note that there are a number of alternative confidence intervals for the binomial case with better coverage behavior. So, rather than this reference being a justification for rejecting all confidence intervals, I see it as a caution against one particular case (albeit an important one), with better performing confidence intervals presented as options. Are confidence intervals “ghosts” or merely “eccentric and troublesome relatives?”
There is no such thing as an infinitely precise measurement, ie they are discrete (regardless of whether the underlying reality is or not).
Probabilities are calculated from these discrete measurements, therefore they must also be discrete.
Also, the original definition of probability was combinatoric. Ie, if you pull one card from a 52 card deck there are 4 ways for it to be an ace, thus the probability of ace is 4/52. Note that is a ratio of two integers.
This concept was quickly extended to more complex scenarios where it was infeasible to delineate all the possibilities (eg, throwing dice thousands of times). Thus continuous approximations were adopted.
For other cases, eg probability an athelete wins a round, guestimated numbers were plugged in that also violated this discreteness property.
Ever since, there has been endless discussion amongst experts over the meaning of probability and so on. While people trying to use it are mired in hopeless confusion.
So yes, afaict this should hold true for all confidence intervals. But the magnitude of the problem depends on how closely the approximation/estimation follows reality.
There is nothing wrong with approximations, but if you start thinking they are reality you get lots of confusion.
Daniel (or anyone with a strong math background), I have a question.
Is it correct to say that there are infinitely more irrational numbers between 0-1 than rational? So for cases where probabilities must be rational numbers, integrating over that interval (or any sub-interval) is almost entirely summing up values that are not actually probabilities?
Eg:
https://math.stackexchange.com/questions/508217/showing-that-rationals-have-lebesgue-measure-zero
Obviously it still largely works as an approximation, but if that is the case shouldn’t we question any proofs about “exactness” and so on based on those continuous calculations.
The rationals are “dense” in the reals meaning that you can get as close to any real number as you’d like using rationals, but at the same time the rationals are strictly smaller than the reals in the sense that there is no 1-1 and onto correspondence between rationals and reals. There’s a thing called the “continuum hypothesis” which says roughly that the “next largest” size after countably infinite (which is the size of the rationals) is the size of the reals. This is not decidable without an extra axiom. It is “independent of the axioms of ZFC set theory” https://en.wikipedia.org/wiki/Continuum_hypothesis
Whether an irrational real number “is a probability” is not uniquely specified. If you say that only rationals can be probabilities, then this may have some implications for some foundational questions. Most people would say the uniform(0,1) distribution is a well posed probability distribution based on a density function and wouldn’t have any problem with it. Me included.
I have a personal preference for founding analysis and probability on the hyperreal numbers. In that context you could say that rather than the uniform(0,1) distribution we actually have the discrete distribution on {0/N,1/N,…N/N} for N a nonstandard integer. There are N elements and probability 1/N assigned to each. You actually **can not** get any result other than one of these nonstandard rationals. We could define a function which is equal to 1 over the interval [0,1] and define a nonstandard integral = standard_part (sum(1 * 1/N, 0 … N*X)), let 1/N = dx then this is standard_part( sum(1 * dx,0 … N*X)) = integral(1 dx, 0, X)
All of the nonstandard grid points are rational numbers, so this doesn’t violate your concern about rationality, but the integral from this construct is the same number as you’d get from more “standard” approaches to integration, and so there isn’t any disagreement between the result you’d get from a standard “measure theoretic integral” compared to a construct that uses only nonstandard rationals. This is a way to formalize the notion that continuous mathematics is an approximation of a very ultra-fine discreteness… basically because the “continuous result” gives you an answer that is “infinitesimally close” to the correct answer (which is the nonstandard sum).
There is no way to distinguish between a truly irrational number say pi and a nonstandard rational infinitesimally close to pi without the nonstandard machinery. Also, there’s no such thing as infinitely precise measurements, so there’s no way to know whether pi is a possible value of any experiment etc.
Where things go wonky foundationally is with really strange functions. So long as you don’t work with “density functions” that are a countable collection of point masses with a strange weights assigned to them (like the “derivative” of the Cantor function https://en.wikipedia.org/wiki/Cantor_function ) you are not going to have problems.
Nonstandard constructs like “grid functions” https://www.sciencedirect.com/science/article/pii/S0001870819300490 form an alternative way to discuss the really odd “distributions” of Schwartz. Essentially, there’s no advantage to having irrational numbers compared to nonstandard rationals.
Furthermore, almost every irrational number is “uncomputable” and “indescribable” https://blog.ram.rachum.com/post/54747783932/indescribable-numbers-the-theorem-that-made-me
All of which is to say that there’s an interesting rabbit hole to go down in foundations of mathematics, but in the end it doesn’t result in disqualifying the measure theoretic version of probability, even though I personally have a strong aesthetic preference for the nonstandard analysis version (and it’s how I’m writing my book on data analysis and mathematical modeling)
You have n double-sided coins in your pocket. These can be any combination of double-headed and double-tailed. (We can include the only other physically-possible type of coin, fair coins, but it reduces to the same thing.)
Pull one out, flip it, put it back. Repeat for m flips.
In this way you can generate any possible value for p = probability of heads (which is now a property of the collection of coins). For any combination of n and m, the result will be a rational number.
If you say N = infinity and do the continuous calculation, now you are making a contradictory assumption, or am I wrong?
Here is what I mean about the possible sets of coins in julia:
https://pastebin.com/nuc8UWnz
I can add the code to get the likelihood and posterior, but I’m trying to keep it simple…
Another thing that looks interesting to me from this example (including fair coins now) is that something like a prior probability shows up.
Eg, two double-tailed plus one fair results in the same value for p as one double-tailed and two fair coins:
# “dTdTdH”
# “dTFF”
But there is only one way to to have a set of coins where p = 0, etc.
For anyone interested, here is discussion about this topic I found:
https://www.researchgate.net/post/Is-there-any-use-in-constructing-defining-integration-over-some-subset-of-the-rationals
What I got out of it is maybe reality is an edge case of mathematics. At least as currently formulated.
Afaict you almost immediately get contradictions when trying to apply it to the simplest statistical/probability questions. Eg, the “timeless problem” of the binomial proportion confidence interval.
Nah, there’s no contradiction, you just have to learn what convergence is
The problem is lack of convergence. Here is another paper that sees the same for poisson and negative binomial:
INTERVAL ESTIMATION IN EXPONENTIAL FAMILIES. Statistica Sinica Vol. 13, No. 1 (January 2003), pp. 19-49 (31 pages)
https://faculty.wharton.upenn.edu/wp-content/uploads/2012/04/Interval-estimation.pdf
Seems there is nothing that actually meets the definition of confidence interval for discrete distributions. Only various approximations.
It isn’t some philosophical argument.
This is the same as a theory making a prediction that disagrees with the observations. If you predict some ice cubes will sink in my glass of water but we see them float, then some assumption you made is wrong. We don’t even need any understanding of the details.
Anon:
It’s interesting at the foundations of mathematics to ask “what is a number?”. In Von Neumann’s construction, the natural numbers 0 means {}, 1 means {{}} or what’s the same.. {0}, 2 means {0,1} or {{},{{}}}… This lets us use sets as the foundation of numbers.
rational numbers we can define as the set of all pairs of natural numbers (a,b) such that there are “no prime factors that cancel”. We can define a pair of objects (a,b) as the set {a,{a,b}}… at each stage we build up from sets to more complex objects.
Now, what is the irrational sqrt(2)? we define an irrational real number as the set of all rational numbers less than the quantity of interest. we can clearly check whether q*q = 2, if we check any rational number and find that its square is less than 2 then it’s a member of the set that defines sqrt(2) because it must be less than the sqrt(2) since x^2 is an increasing function.
So, anyway, you can get as close as you like to sqrt(2) using a rational number a/b but you can never get “the closest rational to sqrt(2)” because the set of rationals less than sqrt(2) is an infinite set.
Next we can move to the nonstandard hyperreal numbers… This can be thought of informally as all the quantities that are the “last” element of infinite sequences of real numbers… (that’s the construct used in “Alpha theory” https://www.sciencedirect.com/science/article/pii/S0723086903800385 ).
In this construct the value at infinity of the sequence (0,1,2,3,4…) is called alpha. It is an infinite hyperreal integer.
Using this construct we can construct infinitesimal numbers. For example the sequence (1/1,1/2,1/3,1/4,1/5,…) has as its “value at infinity” the value 1/alpha, an infinitesimal positive number. A hyperreal epsilon is infinitesimal if -r < epsilon < r for all *real* r. Note that 0 is an infinitesimal real number, but also 1/alpha is an infinitesimal hyperreal and it's NOT a real number.
So far so good. Now, suppose we work in the hyperreals of the Alpha theory. You say to yourself, probabilities must be rational numbers like 17/32 or 1999/2000 or etc. Fine, this means sqrt(2)/2 is not a legitimate probability. But how far away is it from a legitimate probability? What is the maximum error you commit by using such a thing as a probability? Well, we can construct a sequence of rationals that approaches sqrt(2)/2, like (0.7,0.70,0.707,0.7071…) the error we commit must be less than each of these quantities. Let's call it the value at infinity of the error… but that's 0 in this case, and infinitesimal. So the error we commit is infinitesimal.
If we had taken our sequence and subtracted (1/1,1/2,1/3,1/4…) each of which are also rationals, we'd be subtracting 1/alpha… and we'd still have the value at infinity of the result as (sqrt(2)/2 – 1/alpha) which is not equal to sqrt(2)/2, but *is* infinitesimally close to sqrt(2)/2… In this way we prove that the error from utilizing an irrational number must be less than some infinitesimal.
So, we've got that by allowing sqrt(2)/2 as a probability we commit at most an infinitesimal error compared to if we only allowed rationals. So, tell me where the scientific relevance is of such infinitesimal errors? What calculation will go wrong when we substitute into the formula for the calculation a value which differs from a "legitimate" value by at most an infinitesimal quantity?
Typo… when I said “for all *real* r” what I should have said is “for all *positive real* r”
I played around with the methods in those Brown et al papers. Eg, here is what I got using clopper-pearson interval for p = 0.2 and n = 10,000 – 10,100:
https://i.ibb.co/TgNvVhH/clopper-pearson.png
That is not random variation, and such patterns are the norm rather than exceptional. To me this shows that you cannot actually calculate a confidence interval as it was defined, only approximate it. The actual coverage varies according to n and p.
Are you saying this structure disappears when n = infinity and it becomes a flat line at 95%?
My code: https://pastebin.com/bMBLLqy9
As far as I know, most Frequentists only demand that a 95% coverage interval have coverage of **at least** 95% and they often basically look at the worst case to figure this out.
there may be some confidence procedures that have 95% exactly for any parameter value and any n, but it’s common to have only “at least” coverage.
I haven’t looked into your particular example, but I don’t think it has anything to do with p being potentially an irrational.
Well the main difference between discrete and continuous is exactly that. So I don’t follow why you would assume otherwise.
But the main point is your claim to prove:
If some function is oscillating in some strange way for all observable outcomes, but doesn’t when you assume infinity, that is a qualitative difference.
So actually the “error” you may be making can be infinitely large since there would be something wrong with your premises or deduction.
But I want to ask the question again:
Are you saying this structure disappears when n = infinity and it becomes a flat line at 95%?
In your example you use p = 0.2 which is represented in a computer as something about 0.20000000000000001110 which is nevertheless a rational number (hint, every floating point number has an exact representation as a rational). I wasn’t saying anything about n at all. Nothing about the weirdness of the coverage curve you created is because of an irrational frequency in the binomial distribution for example Binomial(p=pi) or p=sqrt(2) since you used a rational value for the binomial p.
Can things be qualitatively different in a pure math context when some parameter is a nonstandard unlimited quantity? sure.
inverse_logit(N*x) = 1/(1+exp(-Nx)) is a function that is 0.5 at x=0 goes to 0 for large negative x and 1 for large positive x… The bigger N is the faster that transition. Let N be a nonstandard integer, and the standardization of that function is the function which takes on only the values in {-1,0.5,1} for the cases x less than 0, x = 0, x greater than 0. You can only see the difference between that function and inverse_logit(N*x) within the hyperreal system, if you “project it back” into standard mathematics it’s a discontinuous function. Here’s a whole family of curves where all the standard N result in an *infinitely smooth* function, and for nonstandard N it’s a discontinuous function.
Another weird example is the normal density pdf_normal(0,1/N). For N a large standard number this is a sharply peaked function that’s otherwise totally sane and integrates to 1. For N nonstandard this becomes the dirac delta distribution which is not a function but rather a generalized measure. So in this case the *type* of object changes from a function mapping reals to reals, to a measure mapping sets to numbers.
So, basically you *can* see the consequences of infinitesimal errors precisely when you have a discontinuous function. The simple function y = 0 for x less than or equal to 0 and y = 1 for x greater than 0 will have a difference in output of 1 for infinitesimal positive values. There are some things that we for convenience model as discontinuous, but basically nothing in science is truly discontinuous. Consider that an infinitesimal distance is “infinitely” smaller than 10^-1000000000000000000000000000000000000000000000000000000000 meters
What I’m trying to say though is that if you allow any rational value in [0,1] to be a probability but NOT irrational values, then you can construct some mathematical constructs where that will result in very weird results, but I disbelieve that those mathematical constructs correspond to any real world system of interest.
As Richard Hamming said https://www.goodreads.com/quotes/1010626-does-anyone-believe-that-the-difference-between-the-author-lebesgue-2403467-and
“Does anyone believe that the difference between the Lebesgue and Riemann integrals can have physical significance, and that whether say, an airplane would or would not fly could depend on this difference? If such were claimed, I should not care to fly in that plane.”
― Richard W. Hamming
The purpose of things like Lebesgue integration is to make it easier to *prove things* about mathematical ideas, not to make calculations come out right. Almost all our calculations are done in computers with 64 bit floats. Almost every irrational number is non-describable (ie. you can’t write down a formula that defines it)
As to the direct question: “Are you saying this structure disappears when n = infinity and it becomes a flat line at 95%?” I honestly don’t know enough about the math of the clopper-pearson interval to know what happens (and I have no real motive to care, I hate confidence intervals). I do know that many Frequentist intervals have coverage that *exceeds* the nominal coverage. your curve shows that the coverage is always greater than 0.9500 the only thing that would “disprove” the coverage of the Clopper-Pearson interval is if you could show it had coverage reliably lower than 0.95 for certain parameter values or certain N.
https://en.wikipedia.org/wiki/Confidence_interval#Approximate_confidence_intervals
says that some authors require that the interval contains the true parameter greater than or equal to the specified fraction of the time (ie. ≥ 0.95)
Go back to the earlier discussion:
https://statmodeling.stat.columbia.edu/2024/03/14/abraham-lincoln-and-confidence-intervals/#comment-2340762
There is an interval with 75% coverage for this scenario, but no 95% interval. Do you agree with that?
How about this?
It was “common knowledge” ten years ago that a single layer neural network could approximate any function [given infinite time and resources]. This was taken to mean its probably not worth looking into multiple layers.
Now everyone acknowledges this was an entirely correct but misleading proof about the finite reality we live in.
There are logically sound proofs that mislead anyone operating in the real world where the assumptions do not hold. The magnitude of the error from extrapolating from infinity to the real world can essentially be infinite.
The problem of non-existent confidence intervals is a lesser version of that.
Here is the same for n = 10k – 11k
https://i.ibb.co/Pm5gRKh/clopper-pearson1000.png
I look at it and wonder what are these things really? Everyone only cares about whether they get close enough to 95% I guess.
Here is for the wilson interval:
https://i.ibb.co/wpnbWpS/wilson1000.png
It is similar but different. I’m starting to think maybe a website showing all these would help people. Essentially going back to the tables in the before-computer days.