Chris Harrison writes:
I have just come across your paper in the 2009 American Scientist. Another problem that I frequently come across is when people do power spectral analyses of signals. If one has 1200 points (fairly modest in this day and age) then there are 600 power spectral estimates. People will then determine the 95% confidence limits and pick out any spectral estimate that sticks up above this, claiming that it is significant. But there will be on average 30 estimates that stick up too high or too low. So in general there will be 15 spectral estimates which are higher than the 95% confidence limit which could happen just by chance. I suppose that this means that you have to set a much higher confidence limit, which would depend on the number of data in your signal.
I would also like your opinion about a paper in the Proceedings of the National Academy of Science, “The causality analysis of climate change and large-scale
human crisis” by David D. Zhang, Harry F. Lee, Cong Wang, Baosheng Li, Qing Pei, Jane Zhang, and Yulun An.These authors take whole series of annual data from 1500 to 1800, giving 301 data in all and do linear correlations between pairs of data sets them. But some of the data sets only have data at longer intervals, such as 25 years. So the authors linearly interpolate the data to give an annual signal and then assume that they still have 301 data. Is this legitimate?
My reply:
1. For your spectral estimation problem, I think it would best to fit some sort of hierarchical model for the 600 parameters.
2. I didn’t actually read the paper, but from your description I’d think it might be a good idea for them to bootstrap their data to get standard errors.
@ Chris: if someone is doing a naive periodogram spectrum estimate, and then applying 95% confidence intervals to it, they are silly. That technique has been obsolete for decades if not longer: there are comments by Rayleigh and Schuster pre-1900 concerning the problems with this approach!
Serious power spectrum work will begin with a multiple-taper quadratic estimate of the spectrum, and use all of the excellent tools that come from this approach, including harmonic F-test statistics for significance, multiple series- or frequency-coherence, and so on. A good reference if you’re interested in learning more about these techniques is to refer to the classic paper in the area: Thomson’s “Spectrum Estimation and Harmonic Analysis”, http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=1456701. If you don’t have IEEExplore access, Google Scholar will lead you to a link on the web.
There are also excellent analyses done in the climate area using modern, robust techniques. From a brief reading of this paper, it’s not really clear that it has much to do with spectrum estimation at all. Given the crudeness of the data the authors were using, perhaps the interpolation in order to retain the trend is permissible. It certainly seems that there are more sophisticated techniques they could have used in exploring the relationships, as Andrew indicated.
@ Andrew: I’m not sure what you are referring to wrt a hierarchical model for the 600 parameters of a *spectrum*? I don’t think I’ve ever seen anyone apply hierarchical techniques to spectrum estimation before.
I’ve seen this exact problem repeated in many, many, many papers in several different areas of science.
I guess stats experts would be amazed just how standard this is in many fields.
A few examples: In geology the usual ‘cyclostratigraphy’ analysis uses peridograms (even after MTM-type smoothing) and trawls them for any point rising above the “95% confidence” level (calculated for a single, one-sided test). See the book by Weedon (2003). In climatology and meteorology methods like Torrence & Compo (1998) or Mann & Lees (1996) routinely use 0.05 tests with no accounting for the large number of frequencies examined. Various subfields of astronomy also allow this practice – see Benlloch, Wilms et al. (2001) for discussion. These aren’t isolated examples of flakey papers – they are the norm for their field.
There seems to be a huge gap between stats/time series experts to whom this the problem is obvious, and practitioners or spectral analysis in geology, climatology, astronomy, … who don’t seem to know differently.