This post is by Aki.
I recently blogged about the term effective sample size and also commented “A further important point is that the effective sample size depends also on which expectation is estimated.” In the comments Visruth asked “Why does E(X^2) net a different ESS than E(X)?” and Kyurae Kim provided correct short answer. I decided to provide additional illustration
The No-U-Turn-Sampling (NUTS) variant of Hamiltonian Monte Carlo (HMC) aims to maximize the expected jump distance. The jump distance is not exactly maximized as there is randomness to keep the Markov chain reversible (which is a useful property to prove that the stationary distribution is the desired target distribution) and due to some algorithm efficiency choices. In some cases, this maximization of jump distance can lead to odd lag negative autocorrelations and higher effective sample size than the total number of draws. Let’s consider a case with theta being normally distributed. When the Markov chain is in a tail, a large jump distance will tend to take the Markov chain on the opposite side of the distribution, and if the next jump has also a large jump distance we go again to the opposite side and close to the first point. Then lag 1 and other odd lag autocorrelations are negative and lag 2 and other even lag correlations are positive. This will make effective sample size for E[theta] to be larger than the number of draws. Next it’s easiest to consider a normal distribution with mean 0 (vector of 0’s for multivariate normal) but this generalizes to non-zero means. If we now consider the absolute value of abs(theta) when we jump from tail to tail, the jump distance is likely to small and both odd and even lag autocorrelations are positive. For abs(theta) to maximize the jump distance, it would be better to jump between the tail and near mode, but NUTS is not designed to do that. theta^2 behaves as abs(theta).
We can test this with Stan and sampling from a multivariate normal. I sample from 16-dimensional unit normal, as I know that with 16 dimensions the algorithm details on how the Hamiltonian simulation is extended and U-turn decided, happen to be such that we get strong negative autocorrelations for theta.
data {
int<lower=0> D; // number of dimensions
}
parameters {
vector[D] theta;
}
model {
theta ~ normal(0, 1);
}
And we run NUTS with CmdStanR
library(cmdstanr)
mod <- cmdstan_model("normal.stan")
fit <- mod$sample(data=list(D=16), refresh=0)
In the following, I examine just the first dimension of theta[1]
The autocorrelations for theta and theta^2 clearly show what I described above.

Negative autocorrelations lead to super-efficiency and the estimated ESS for theta is 10457, which is 2.6 times bigger than the total number of draws (4000). The estimated ESS for theta^2 is 1534, which is just 38% of the total number of draws! Even if we are not interested in theta^2 directly, we may be interested in sd of the posterior of theta, and computation of sd uses theta^2.
The following plot shows the ESS for the quantiles 0.05,…,0.95 (see Rank-normalization, folding, and localization: An improved Rhat for assessing convergence of MCMC for the details)

We see that if we want to report e.g. 90% posterior interval for theta, the accuracy of the interval end-points are based on about ESS of 2900. Thus, if we want to report posterior sd or posterior interval, there is not much benefit from having very high ESS for theta. In fact, as maximizing the jump distance in theta space is decreasing ESS for theta^2 and quantiles, we are wasting some computation time. However, multivariate isotropic normal is not that common posterior for most interesting models and data, and most of the time we don’t see negative autocorrelations and superefficient ESS, and thus there is no big need to change the NUTS algorithm related to this.
ESS is used to estimate Monte Carlo standard error and for any quantity of interest when MCSE is computed using, e.g., posterior package, the appropriate ESS is used. For convenience, posterior package report also Bulk-ESS and Tail-ESS which provide two summary values on the sampling efficiency in general (see the Rhat paper for the details). These are useful as quick summaries as they are scale free unlike MCSE which needs to be interpreted in the context of the scale of the quantity of interest (see more in the Digits case study).
EDIT: added 0 break for y axis
Aki:
Great post. Just one thing. Why does the y-axis on the ESS graph go below zero? ESS has to be nonnegative, no?
Sorry for omitting 0. I edited the post so that the figure has now 0. The range of plots is from 0 to 9000 and thus nonnegative. The bayesplot theme default is to have a small gap between the plot area and the axis line (which I assume you have approved at some point).
Funny thing is that we did actually have negative ESS *estimates* at some point, but it turned out to be a numerical problem in case of extreme negative correlations so that instead of getting extremely high ESS the computed valued flipped to be negative. To avoid negative ESS *estimates* posterior package is capping the reported ESS as S*log10(S), e.g. with Stan’s default 4000 draws, the highest possible reported ESS is 14408. It is really rare to see such high ESS values with Stan, but in the testing we did test the ESS computation to the limits.
This is a problem that I run into every time I try to compare the efficiency of samplers.
I usually compute the bulk-ess of the mean for each continuous variable (whether generated or not), and then compare the minimum across all variables between samplers, either directly, or in terms of ess/grad_eval or ess/time.
But I really don’t know if this is a good approach or not.
In general, for continuous variables, I would use minimum of Bulk-ESS and Tail-ESS. Using only Bull-ESS would be fine if you want to compare efficiency only for central quantities (mean / median).
I don’t have good suggestion what to do with multiple variables. As the posterior geometry can be quite different for different variables (and combinations of variables) it is possible that the sampling efficiency for one or some is much worse. On the other hand, if the posterior geometry is such that actual efficiency is the same for all variables, using minimum of ESSs based on finite computation increases variance in comparing the methods, especially when the number of variables is very high.
Thanks for the full post, it’s quite clear how super efficiency works now. I had seen it before but kinda took it for granted.
Can you always construct an (not necessarily useful) estimand which can be estimated super efficiently? (If you have a fixed posterior and are using NUTS)
Many non-normal posteriors, like multimodal and narrow banana shape, make NUTS to have high autocorrelation for all quantities