Relating t-statistics and the relative width of confidence intervals

Posted on by

How much does a statistically significant estimate tell us quantitatively? If you have an estimate that’s statistically distinguishable from zero with some t-statistic, what does that say about your confidence interval?

Perhaps most simply, with a t-statistic of 2, your 95% confidence intervals will nearly touch 0. That is, they’re just about 100% wide in each direction. So they cover everything from nothing (0%) to around double your estimate (200%).

More generally, for a 95% confidence interval (CI), 1.96/t — or let’s say 2/t — gives the relative half-width of the CI. So for an estimate with t=4, then everything from half your estimate to 150% of your estimate is in the 95% CI.

For other commonly-used nominal coverage rates, the confidence intervals have a width that is less conducive to a rule of thumb, since the critical value isn’t something nice like ~2. (For example, with 99% CIs, the Gaussian critical value is 2.58.) Let’s look at 90, 95, and 99% confidence intervals for t = 1.96, 3, 4, 5, and 6:

Confidence intervals on the scale of the estimate

You can see, for example, that even at t=5, the halved point estimate is still inside the 99% CI. Perhaps this helpfully highlights how much more precision you need to confidently state the size of an effect than just to reject the null.

These “relative” confidence intervals are just this smooth function of t (and thus the p-value), as displayed here:

confidence intervals on scale of the estimate by p-value and t-statistic

It is only when the statistical evidence against the null is overwhelming — “six sigma” overwhelming or more —that you’re also getting tight confidence intervals in relative terms. Among other things, this highlights that if you need to use your estimates quantitatively, rather than just to reject the null, default power analysis is going to be overoptimistic.

A caveat: All of this just considers standard confidence intervals based on normal theory labeled by their nominal coverage. Of course, many p < 0.05 estimates may have been arrived at by wandering through a garden of forking paths, or precisely because it passed a statistical significance filter. Then these CIs are not going to conditionally have their advertised coverage.

29 thoughts on “Relating t-statistics and the relative width of confidence intervals”

  1. As Jacob Cohen pointed out:

    “…imperfectly understood confidence intervals are more useful and less dangerous than incorrectly understood P values and hypothesis tests…”

    Reply ↓

  2. I like that this helps people better relate what they see in T-values to CIs. It’s good to know that just significant means the CI nearly touches 0 for some people.

    It’s unfortunate that it continues to place emphasis on using an interval as a test when, once used this way, it’s questionable it has value as a CI anymore. As Andrew has lamented in the past it seems incompatible to use it as an inverted t-test. If it was estimated going in assuming the null was true then it can’t be an estimate for where the value is when it’s false. And it can’t be used as a test unless it was estimated on an assumption that the null was true.

    I’d love to see more articles like this focusing on the idea that a CI is an estimate of where a parameter is and less reinforcement of the idea of it as a test. Some expansion of the advice of Amrhein et al 2019 would be interesting.

    Reply ↓

  3. Is this a new observation? I felt that all this is well-understood. Or is this a different way to talk about Type M error?

    Also, I always feel a bit confused about the meaning of the confidence interval. On the one hand, we are not talking about a particular interval (the definition of the confidence interval references the distribution of hypothetical CIs under hypothetical repeated sampling–we don’t have access to these hypothetical CIs because thet are hypothetical); and on the other hand, we are using a particular interval like a Bayesian credible interval to talk about our uncertainty of the *estimate* we got from our particular sample.

    It feels like that rabbit-duck illusion: https://en.wikipedia.org/wiki/Rabbit%E2%80%93duck_illusion

    It can’t be both things, but it is, depending on how you think about it. Super confusing.

    Reply ↓

    • Yeah, this surely isn’t a new observation. t_{alpha/2} / t_observed is pretty straightforward. But I hadn’t really seen this pointed out as a rule of thumb, though it was something I was finding myself using.

      I wouldn’t say that this accounts for some of the biggest ingredients to Type M errors, since those are often made terrible by selection on statistical significance. Here I’m not considering some data generating process with a true effect. Rather this is just a relationship between a given t-statistic and the associated CIs.

      Reply ↓

    • I agree with the super confusing. I continually find the insistence on the proper definition of a confidence interval (as a property of repeated sampling) to leave me wondering what to do next. If I only have one sample, technically that means I shouldn’t look at the confidence interval at all. But if I have a 95% confidence interval and don’t know if this particular interval is one of the “lucky” 95% that cover the true value or one of the 5% unlucky ones that does not, then my best guess is that this is one of the 95%. What else am I supposed to do with that particular interval I have? What the post demonstrates to me is that the width of the confidence interval does convey useful information – we should be interested in more than the point estimate, and the wide range of sizes even with “significant” t statistics conveys something about the degree of uncertainty (of course, it does not reflect the uncertainties due to measurement issues, selection bias, or forked paths, so it is certainly a lower bound on the degree of uncertainty).

      If the response is that the confidence interval is meaningless, and I already accept that the usual hypothesis test is meaningless, then exactly what am I supposed to do with any analysis that is not an experimental setup with repeated sampling? I’m all for skepticism, but not to the point of ignoring the data I have.

      Reply ↓

      • If we are willing to accept that there is not one correct statistical philosophy but many, and to accept the frequentist theory as described, then I don’t see the problem.

        ‘But if I have a 95% confidence interval and don’t know if this particular interval is one of the “lucky” 95% that cover the true value or one of the 5% unlucky ones that does not, then my best guess is that this is one of the 95%.’

        Exactly that. In the real world, I can usually only observe one sample ever and I see that statement above as the frequentists’ way of describing the risk/uncertainty of treating the observed sample as normal. Effectively, if we must make decisions based on one sample, we have to assume that it is not from the tails.

        I don’t think there is any statistical philosophy that can get around this issue. If we believe that the one sample we observe can be the 1% chance sample (we’re just unlucky), then under what basis can we make decisions?

        I think confusion on this issue is really not about the interpretation but reflects disagreement with the fundamental frequentist concepts (counterfactual repeated samples, etc.). That’s ok as there isn’t one correct philosophy.

        Reply ↓

    • Well why don’t we make things even more confusing!

      Run this simulation a zillion times (quickly thrown-together R code at the end of this message):

      1: Draw y ~ N(0, 1)
      2: Compute confidence interval for a one-sample t-test
      3: Record the length of the CI and if it contains the population mean or not

      After zillion simulations, bin the lengths of confidence intervals and for each bin calculate the proportion of intervals containing the population mean. Now you can take a look at how the probability of the population mean being inside the interval varies as a function of the length of the CI.

      You should see that the probability that the interval containts population mean is lower for short intervals, approaches 0.95 and then goes beyond that.

      So the fact that the interval contains the population mean is true only if we discard information about the length of the interval. If we want to somehow implicitly condition on the length of it (“shorter = more info”) then the frequentist properties of the CI (ie that it contains the population mean with some pre-defined probability) don’t apply anymore.

      And if we care about estimating the “true value” of some population parameter… it would seem that shorter intervals are actually worse?!

      —

      The quickly thrown-toget R code. Not optimized for efficiency (there are easy optimizations), but I tried to make this somewhat understandable.

      nsim = 10000

      lengthOfCI = rep(NaN, nsim)
      containsPM = rep(NaN, nsim)

      for(i in 1:nsim){
      y = rnorm(10, 0, 1)
      ci = t.test(y)$conf.int

      lengthOfCI[i] = diff(range(ci))

      if(ci[1] 0){
      containsPM[i] = 1
      } else {
      containsPM[i] = 0
      }
      }

      binLims = seq(min(lengthOfCI) – 0.01, max(lengthOfCI) + 0.01, length.out = 20)
      p = c()
      nPerBin = c()

      for(i in 1:(length(binLims) – 1)){
      inds = intersect(which(lengthOfCI >= binLims[i]),
      which(lengthOfCI < binLims[i + 1]))
      p[i] = sum(containsPM[inds]) / length(inds)
      nPerBin[i] = length(inds)
      }

      plot(p, ylim = c(0, 1))
      # Expected probability is about 0.95:
      sum(p * (nPerBin / sum(nPerBin)))

      Reply ↓

        • Shravan… the Fediverse is the answer here I think. Much better to subscribe to this blog’s output, and then comment on it through Mastodon or another fediverse related server, where you can upload images, videos, code, etc. It’s not quite there yet… but it’s getting there.

        • Daniel, you wrote:

          > Shravan… the Fediverse is the answer here I think. Much better to subscribe to this blog’s output, and then comment on it through Mastodon or another fediverse related server, where you can upload images, videos, code, etc. It’s not quite there yet… but it’s getting there.

          We are talking about allowing html tags on a web page, in 2024. It can’t be that hard.

        • Aw, thanks man! I knew about these sorts of issues but somehow I ended up thinking that this simple snippet would be free from problematic bits. Should’ve stopped to think for a second.

      • Without looking at your code or running it, and just from your summary of the procedure you wrote down, isn’t this fact (that the smaller the interval width, the poorer the coverage) simply due to the misestimation of the standard deviation in each simulation run? The CI width will be a function of the sd estimate. I would expect that if we plotted the widths against the sd estimates, we would see a straight line there. Yup:

        https://gist.github.com/vasishth/f313df5b50f8902a4e18e965253178e5

        Reply ↓

        • Well of course, that’s the point of a t-test, Raghu isn’t it :)

          If we knew the population standard deviation, we could use that to construct a constant interval with 95 percent coverage. In this case the length of the CI to one direction would be 1 / sqrt(10) * 1.959

          In the long run these CI’s would have a 95 percent coverage.

          But since we don’t know the population sd (or pretend not to in a simulation) it has to be estimated from the sample and becomes a random variable. But as such the length of the CI indeed is a function of the sample sd.

          Now, because the sample sd is indeed a random variable with chisq distribution, it becomes more and more skewed when n is small. That’s why the quantile from the t distribution is used when calculating the length of the CI.

          To me a surprising consequence of this was that this also leads the length of the CI to affect the probability of the population mean being contained in the interval. Maybe and probably this is obvious to others but for some reason to me that was very strange and unintuitive. Or at least something I had not thought about.

          P.S. I just noticed (when testing the above idea with my earlier simulation code) that in the main simulation loop the test should of course be that the lower edge of the CI is smaller than zero AND the higher edge is greater. Damn you, disappearing letters!

        • His point is – I think – that everything else being equal you may get a narrow interval when the sample is relatively concentrated and such a narrow interval is “less informative” than a wide interval produced by a more disperse sample.

          Imagine that you make a couple of measurements of something with a machine that gives either the true value or adds/substracts 1 unit. If you get {41, 43} you know the answer is 42. If you get two values one unit apart you know the answer is one of those. If you get the same value twice there are three possible values for the answer.

          Note that one could fix the interval width independently of the data and the issue would still be present. In that case the “informativeness” classification can’t be made depending on the fixed width but one could look at low/high sample standard deviation for example – and the “shorter intervals are actually worse” irony would be lost.

        • > Note that one could fix the interval width independently of the data and the issue would still be present.

          That thing that I wrote before is of course wrong in the context of a normal model – the mean is a suficient statistic for the normal(mu,1) model and the sample standard deviation is irrelevant.

          However the claim can be saved to some extent if we remember that the normal model is just an approximation. The concentration of the sample is correlated with the fixed with interval coverage if what we really have is a truncated normal for example.

          By the way, there is an error in Daniel’s reconstruction of the mangled script, line 15 should read

          if(ci[1] _less_than_ 0 && ci[2] _greater_than_ 0){

        • Ah, damn, sory Shravan. That’s what happens when I’m writing before having my coffee. Bad coffee is better than no coffee at all, as David Lynch mused.

      • Andrew,

        my complaint is not specifically about this blog being unable to allow html tags in general. I have a more general dissatisfaction with the extreme low level at which everything digital functions, or doesn’t function, given that we are now in a dramatic AI revolution. Maybe it’s just Germany, but the internet doesn’t always work, and when it works it is slow, the train schedules cannot be updated in real time and often mislead one badly. The pharmacy recently announced, very proudly, in my dialysis center that they would deliver any medication needed to the patient, but the payment for it has to be done laboriously by issuing a physical invoice that is sent by mail. Germany made a big hoohaa about creating digital id cards, but they are completely useless. When parking tickets are issued, one has to hope to find the small piece of paper stuck on the windscreen. In my dialysis center, they need to see the patient’s health card every three months otherwise the insurance company has no way to confirm that I am still insured with them; I guess they have an excel sheet sitting somewhere that records my monthly payments to the insurance, but probably nobody knows where it is. I the Charite hospital, which is supposed to be a top 10 hospital in the world, the eye clinic on the ground floor does not have access to my medical records in the kidney transplant center on the first floor; reason: they forgot the password in the eye clinic to the records from the transplant center (I assume that going up one flight of stairs to ask someone is out of the question). So I have to explain my medical history in real time and verbally to the eye doctor.

        And whoever designs the software for this blog can’t think to and/or can’t figure out how to allow html functionality in comments.

        It’s hard to believe it’s 2024.

        Reply ↓

    • +1 to Shravan. In practice, we always seem to revert to treating the frequentist CI as an approximate Bayes estimate. It is ironic that, as explored below, these intervals can even have several well known issues with their advertised coverage properties…you know, the hill that frequentism dies on. This doesn’t invalidate utility of frequency analyses, but does highlight that the ^practice^ of statistics often violates the philosophical justifications given. Bayesian do this too all the time, eg using cop-out priors, but it’s easier for me at least to spot and assess the possible importance of the assumptions

      Reply ↓

  4. This reminds me of related discussions about how much overlap the CIs for two comparable measurements (e.g. control vs treatment group, or two independent measurements by different means of the same observable) can have while being statistically significant in their difference, see e.g. here and here.

    Reply ↓

Leave a Reply

Your email address will not be published. Required fields are marked *