Boot

Joshua Hartshorne writes:

I ran several large-N experiments (separate participants) and looked at performance against age. What we want to do is compare age-of-peak-performance across the different tasks (again, different participants).

We bootstrapped age-of-peak-performance. On each iteration, we sampled (with replacement) the X scores at each age, where X=num of participants at that age, and recorded the age at which performance peaked on that task. We then recorded the age at which performance was at peak and repeated. Once we had distributions of age-of-peak-performance, we used the means and SDs to calculate t-statistics to compare the results across different tasks. For graphical presentation, we used medians, interquartile ranges, and 95% confidence intervals (based on the distributions: the range within which 75% and 95% of the bootstrapped peaks appeared).

While a number of people we consulted with thought this made a lot of sense, one reviewer of the paper insists that this is no good, writing that “constructing the confidence intervals directly from the corresponding percentiles of the bootstrap distribution [has been] shown to be relatively poor in asymmetric cases” and that alternative methods have been developed. No citations are given.

My reply: I’m not really sure but here’s what I think. My instinct is that it would be better to fit a curve to each dataset rather than to just pick the age at which the raw data average is highest. You could, for example, fit a Gaussian process or even a lowess and find the age at which the fitted curve is maximized. I’m guessing that will be more accurate than taking the max of the raw data. Whatever data summary you use, though, getting standard errors via bootstrap seems reasonable to me. With large N, the statistic should have an approximately symmetric sampling distribution if it’s not near the boundary, so you can use estimates and standard errors, shouldn’t need to bother with quantiles.

5 thoughts on “Boot

  1. While I think bootstrapping does make sense in this situation, the reviewer has a point if the distribution really is decidedly asymmetric. You should stay with the bootstrapping approach and take a look at the BC-intervals or other alternative methods to construct confidence intervals via bootstrapping in a somewhat more robust way.

  2. It’s true that “structing the confidence intervals directly from the corresponding percentiles of the bootstrap distribution [has been] shown to be relatively poor”. This can be seen from Efron’s Book on Bootstrap. Several other methods to construct confidence interval has been developed. For example, biased corrected intervel could be an alternative method to the percentitle based method. Further details can be found in Efron’s Book on Bootstrap. There are three or four chapters to discuss different kinds to construct confidence interval.

    • Wenhao:

      Just to be clear, I suggested using the bootstrap to get standard errors, not to use the percentiles of the bootstrap distribution.

      Also, I think my more important suggestion was to fit curves to the data rather than using raw maxima. In bootstrap discussions, I think there’s often too much discussion of the bootstrapping and not enough discussion of the estimator that is getting bootstrapped. The action’s mostly in the estimator, not the procedure used to get probabilistic inferences from it.

      • Dear Andrew! Thanks for your reply! I agree with you to fit curves to data rather simply using raw maximal.It seems more reasonable. And my reply above just want to explain why the reviewer suggested Joshua to use alternative methods. Joshua can find several alternative methods in Efron’s Book.

  3. > You could, for example, fit a Gaussian process or even a lowess and find the age at which the fitted curve is maximized.

    Yes. Generate a pdf for the fit coeffs by bootstrapping the data you fit. Look for systematic residuals to assess whether your model is right. If it is then (hopefully) the fit coeffs and associated uncertainties tell a story – depends on the fit model obviously but if you formulate it in terms of cause and effect then hopefully so.

    > No citations are given.

    I hate that. It should be common courtesy to provide a citation.

Comments are closed.