John Cook has a fun and thoughtful post on Zipf’s law, which “says that the frequency of the nth word in a language is proportional to n^(−s),” linking to an earlier post of his on Heaps’s law, which “says that the number of unique words in a text of n words is approximated by Kn^β, where K is a positive constant and β is between 0 and 1. According to the Wikipedia article on Heaps’ law, K is often between 10 and 100 and β is often between 0.4 and 0.6.” Unsurprisingly, you can derive one of these laws from the other; see links on the aforementioned wikipedia page.
In his post on Zipf, Cook discusses the idea that setting large numbers to infinity can work in some settings but not in others. In some way this should already be clear to you—for example, if a = 3.4 + 1/x, and x is very large, then if you’re interested in a, then for most purposes you can just say a = 3.4, but if you care about x, you can’t just call it infinity. If you can use infinity, that simplifies your life. As Cook puts it, “Infinite is easier than big.” Another way of saying this is that, if you can use infinity, you can use some number like 10^8, thus avoiding literal infinities but getting many of the benefits in simplicity and computation.
Cook continues:
Whether it is possible to model N [the number of words in the language] as infinite depends on s [the exponent in the Zipf formula]. The value of s that models word frequency in many languages is close to 1. . . . When s = 1, we don’t have a probability distribution because the sum of 1/n from 1 to ∞ diverges. And so no, we cannot assume N = ∞. Now you may object that all we have to do is set s to be slightly larger than 1. If s = 1.0000001, then the sum of n−s converges. Problem solved. But not really.
When s = 1 the series diverges, but when s is slightly larger than 1 the sum is very large. Practically speaking this assigns too much probability to rare words.
I like how he translates the model error into a real-world issue.
This all reminds me of a confusion that sometimes arises in statistical inference. As Cook says, if you have problems with infinity, you’ll often also have problems with large finite numbers. For example, it’s not good to have an estimate that has an infinite variance, but if it has a very large variance, you’ll still have instability. Convergence conditions aren’t just about yes or no, they’re also about how close you are. Similarly with all that crap in time series about unit roots. The right question is not, Is there a unit root? It’s, What are you trying to model?
Thanks for this. As I started reading, I was all set to do a diatribe on unit roots, and then you did it calmlyu in the last two sentences,
Happy Thanksgiving, everyone.
I like the issues discussed here, it’s very interesting to see the distinction being made between infinite and nearly very large. Not something that I’ve confronted in practice, but I do appreciate the issue at hand.
That said, I am a bit curious about the aside on unit roots. I think I have an idea of what you, Andrew, and Jonathan, the other commenter above mean, but I’m not completely sure and I’d be delighted to know as I’m about to teach time series in one of my courses; zero pressure of course.
My guess is that both of you are thinking that testing for unit roots means focusing on a test instead of the fundamental issue of whether the thing you’re trying to model and estimate can be so (i.e., non differentiated random walks are problematic to say the least), is that more or less it?
Without speaking for Andrew. IMO if it makes a big difference whether a parameter in the world is 0.9999 or 1.0000 or 1.000001 then you’re letting theory get in the way of reality. If an ocular test can’t tell you which regime you’re in, statistical techniques can’t save you.
I can create physics examples in which a movement of 10 angstroms take you from standing firmly on the cliff to being in free fall off the cliff. But in practice, you either want to to stand a foot away or jump off.
Jonathan:
Also, nothing is stationary in the human sciences. Any parameter will change over time. So the idea that there is some true underlying process which either has a “unit root” or not, is just silly.
To put it more technically, suppose your model is y(t) = mu(t|theta) + e(t|phi), where mu is the mean function, e is the error term, theta represents the vector of parameters characterizing the mean function, and phi is the parameters characterizing the variance/correlation function. The “unit root” thing is all about the e(t|phi) process, but this is all relative to the mu(t|theta) function. If you have a time series with a linear trend that you don’t include in mu, it will show up as a high autocorrelation in e.
Fair point about the distinction of high autocorrelation vs unit root being essentially meaningless in practice, especially with the limited time series that a lot of social sciences uses.
And I think I see both your points Andrew, but if I do, I’m not sure I agree with them.
Your first point, that parameters change all the time, is probably correct and leads to the dictum that estimates/models can be locally useful, but are dubious globally. I don’t see how that relates to the issue of unit roots, though, if a “local” time series has high autocorrelation or a unit root, yes, it might change, but isn’t it interesting, from a statistical perspective, to identify this?
And I feel very lost with your second point. I agree that functional forms and trends can significantly affect tests for the presence of unit roots. The absence of a linear trend could lead to misidentification of a trend as a unit root, no question. So I certainly agree that omitted variables and incorrect functional form specification can easily lead the series that look like they have these issues when they don’t.
But are you claiming that essentially no cases of pure unit roots/high autocorrelation? That these reflect our ignorance more than anything else? I’d be willing to agree that’s likely the case for the majority of cases, but I’m not sure it’d be all.
Take a simple habit formation model, which suggests that people explicitly look back at past consumption to decide their current consumption: C(t)=alphaC(t-1)+(bunch of other things). Alpha could, in theory, be quite high and even if it changes over time, we should be interested in the properties that the series in that case, no?
Anyway, I feel like I’ve dramatically missed the point, if so, apologies. And thanks for the response.
Pedro:
I care about long-range dependence and effective nonstationarity, not about “unit roots.” A test for unit root misses the point, because you can have long-range dependence and effective nonstationarity without a unit root; conversely you can have a unit root without long-range dependence and effective nonstationarity. In short, I think that, as with many problems in hypothesis testing, the precise mathematical concept being tested does not line up well with the scientific or applied goal.
For example, in your habit-formation model: Yes, I’m interested in alpha; No, I’m not interested in testing alpha=1.
Can’t seem reply to your post below, but got it now Andrew! Thank you for the further clarification. And yes, I already mention that to my students, but I think it’s worth putting more emphasis; if you don’t mind, I think I’ll paraphrase what you wrote to them (with attribution, of course).
Many thanks again to both of you.
Thanks for this. I have a corpus of Japanese text (actually three corpuses) and a Japanese-English dictionary that I’ve annotated with the frequencies of occurrence of each word, and I was wondering if there was something statistical to do with the data…
But, Hey! It’s a national holiday here, too: Kinrou Kansha no Hi, so happy “Give Thanks to Labor Day”!
And, if you were wondering, for Kinrou (labor), my program gives:
勤労 = (244 61:15:23) (勤:6 労:4 )
and for Kansha (thanks) it gives:
感謝 = (2599 25:19:56) (感:3 謝:5 )
So the Japanese are into giving thanks an order of magnitude (2599 occurences) more than they’re into working (244 occurences)**.
(The ii:jj:kk numbers are the percentage of those occurences that occur in each of my corpuses of 1900-1955 literary writing, 1900-1955 popular writing, and 2011-2065 popular writing. (Not much material from 1956-2010 is available in digital form yet, and most of that third 55-year period hasn’t happened yet.) But you can see that Kinrou is a rather dated literary sort of word (since the recent writing corpus is about half the total text*, a “uniformly used word” would be 25:25:50.)
And you really don’t want to know what the (勤:6 労:4 ) bits are about…
*: Or will be after I’ve added this year’s text, when they’ll be about 100,000, 100,000, and 200,000 pages of text, respectively.
**: Not really: they’re workaholics, and there a plethora of words for labor, work, employment, drudgery and the like, some way more common that Kinrou or even Kansha.
This made me think: could there be a language with no repeated words, ie every word is unique?
I’d think no, but maybe someone has worked out exactly why and that can be used to derive the ~sqrt(n) law.
Or look at it in reverse. Why would we see:
n ~ C*u^2
If given u unique words, will people eventually produce a corpus of text of length proportional to u^2 words? As if using every pairwise permutation:
aa ba ca
ab bb cb
ac bc cc
Whilst at high school (more than 50 years ago) I recall going to a talk by one of my local university’s mathematics faculty who claimed that the largest number you ever needed was 10^80, the number of protons in the universe (known as Eddington’s number).
I could appreciate his point at the time, but thought it was likely used as a throwaway line for (thought-provoking) effect. I suppose that the fact it stayed in my memory all this time proved it’s effectiveness!