Jake Humphries writes:
I for many years wanted to pursue medicine but after recently completing a master of public health, I caught the statistics bug. I need to complete the usual minimum prerequisites for graduate study in statistics (calculus through multivariable calculus plus linear algebra) but want to take additional math courses as highly competitive stats and biostats programs either require or highly recommend more than the minimum. I could of course end up earning a whole other bachelor degree in math if I tried to take all the recommended courses. Could you please rank the following courses according to importance/practical utility in working in statistics and in applying for a competitive stats PhD program? This would greatly assist me in prioritising which courses to complete.
1. Mathematical modeling
2. Real analysis
3. Complex analysis
4. Numerical analysis
My quick advice:
– “Mathematical modeling”: I don’t know what’s in this class. But, from the title, it seems very relevant to statistics.
– “Real analysis”: Not so relevant to real-world statistics but important for PhD applications because it’s a way to demonstrate that you understand math. And understanding math _is_ important to real-world statistics. Thus, the point of a “real analysis” class for a statistician is not so much that you learn real analysis, which is pretty irrelevant for most things, but that it demonstrates that you can do real analysis.
– “Complex analysis”: A fun topic but you’ll probably never ever need it, so no need to take this one.
– “Numerical analysis”: I don’t know what’s in this class. You could take it but it’s not really necessary.
Take complex analysis, because it will be one of the most interesting things you’ll ever study, and if you don’t take it now then its unlikely that you’ll ever take it. The other classes you will eventually end up covering in grad school, in some way or another.
In terms of utility to statistics, Mathematical Modeling will _probably_ be a course on differential equations/physics, so unlikely to be too useful. Real analysis will have the most direct utility since it will give you basic training in how to read/think about mathematics which will be important for reading methodological papers.
In the coursework at my university (not that this is necessarily universal, of course), math modeling at the undergraduate level is basically undergraduate linear optimization-based models solved via the simplex algorithm—the beginnings of operations research—while numerical analysis is where we primarily teach numerical solutions to ODEs/PDEs.
I really enjoyed complex analysis and believe it or not have occasionally actually found it useful (e.g., in evaluating integrals using residues).
I second the view that complex analysis is interesting.
It seems to me that an important factor in the choice among courses is the quality of the teacher. I took two numerical analysis courses—one much more complete than the other. But, the less complete course was much more valuable. Early on, the professor posed a number of problems but did not tell us the algorithm to use to solve them. Those exercises left me with a profound distrust of numbers that are the output of machine calculations. That paranoia has served me well over time.
For a good exposition of the basis for this paranoia see:
http://www.maa.org/sites/default/files/pdf/upload_library/22/Ford/GeorgeForsythe.pdf
A quote from that paper: “The conclusion of this example is that one should not follow a statistics book blindly here.”
Bob
I once looked into the source code for a relatively well used R package for some sort of SIR algorithm (in my mind, this package was called ISIS, but that’s unlikely) when my friend kept getting “divide by zero” errors, when things couldn’t logically be zero. It turned out that when Normalising the likelihood values to compute the weights, the function summed them on the absolute scale (rather than the log scale), which lead to underflow.
In this scenario, underflow makes baby Turing cry.
Mmmmm hmmmmm. Although I’d been working with statisticians for long enough at that point to realise that a large number of them thought computers were magic (due to, in part, a lack for useful numerical analysis in their education.)
(Ideally this should be day one of the first stats class. “how many decimals can you trust?” Etc
IMIS.
Aha! You have stumbled on the dark underbelly of R packages: many (most?) of them are written by people who have no knowledge of rounding error or how to bound it. Further, they usually provide no test cases to make sure that the package really does what it claims to do.
Second the taking of complex analysis. I’m a math phd student who has an amateurs interest in data science (just keeping my bases covered and all that) so if I had to suggest just one math topic to a non-mathematician, it’d have to be complex analysis. It’s ideas foreshadow two of the most beautiful subjects of modern mathematics – algebraic topology and algebraic geometry. (a little known historical fact is that so many central ideas in these subjects come out from Riemann’s paper on abelian functions (and his notebooks))
Actually, I think that Complex Analysis is more useful than Real Analysis. It’s useful in the sense that it gives you a bit of insight into how problems are often solved in statistics and mathematics through taking a complex problem and mapping it to another domain to make it easier to solve. Useful in computer science too (feature maps in SVMs and Neural Networks.)
Numerical analysis is the study of algorithms for numerical approximation. Courses typically focus on algorithms for optimization, regression, numerical integration, linear algebra, etc.
Personally, I found that class to be one of the most valuable classes when preparing for grad school in statistics, especially if you want to focus on computational statistics. Far more relevant than real analysis, at least for me.
What are some good books in this area? Numerical analysis is a big gap in my understanding, and I plan to do some reading this summer.
Probably not the kind of rigor or depth taught in a Math Course, but from a practical perspective I like the classic by Teukolsky / Flannery “Numerical Recipies in C / Fortran”
I have been planning to read this book for forever now. I think this summer it’s finally going to happen.
“Classic”. Heh. Some us consider the C version the newfangled one.
Ah yes. Punch card fan club? :)
I came of (computing) age with 5-1/4 floppies. About two weeks before graduation I lost the one which had my undergrad thesis on it – a Word Perfect file, a slightly earlier version of WP than this https://www.youtube.com/watch?v=yYWPHkCPhe4. That was not a happy day.
Ah, numerical recipes. First investigate the bona fides of the authors. Are they trained in computing or numerical analysis? Let’s see:
Press: astrophysics
Flannery: astrophysics
Teukolsky: astronomy
Vetterling: Ph. D. in condensed matter physics
Answer: No. They are self-taught. They might be bright men, but they are not experts in computing. To paraphrase the best opinion of the book I ever read, “Ask any expert in [insert topic, e.g., solutions of PDEs] about Numerical Recipes, and he’ll tell you that the book is ok, except for the chapter on [PDEs].”
I do not know of one specialist in computing who recommends the book.
For statistics, try: “Numerical Methods of Statistics, 2e” by Monahan.
Anything by GW Stewart.
“Accuracy and Stability of Numerical Algorithms, 2e” by Higham.
It all comes down to handling floating point numbers, and the best place to start (after reading the Forsyth article mentioned above” is “What Every Computer Scientist Should Know About Floating-Point Arithmetic” by Goldberg.
If you can check it out from your library, I recommend the old “Elements of Statistical Computing” by Thisted. I have yet to encounter a book with better explanations and examples. The ideas that he talks about with respect to linear algebra, dealing with error, numerical optimization, etc. haven’t changed, and will therefore provide you with the foundation you need.
Another book, and one with more recent material, is “Numerical Methods of Statistics” by Monahan. It provides solid explanations of the concepts you should know. The focus is on statistics rather than numerical methods.
I do recommend a book for statisticians. Others will usually emphasize engineering and physics applications, or just the mathematical side of things with no practical applications, and are thus not effective.
Thanks! I will try to get hold of these.
Thisted! How could I have omitted it. Yes, I second this recommendation. Do not bother with Kennedy and Gentle, it is far too old (although some Microsoft Excel programmers could definitely benefit from reading it, but that’s another story).
I agree with this. A statistician is going to do a lot of number-crunching, and a solid course in numerical analysis is the foundation for doing it right.
Indeed; numerical issues are important to consider even for things as simple as calculating means and variances.
Note that it depends on what you want to do in statistics!
If you’re interested in applied problems then “mathematical modeling” is definitely the best choice provided that they’re talking about statistical modeling.
“Real analysis” is relevant if you ever want to do any statistical theory, as “[p]robability is just analysis in a tuxedo, and statistics is just probability after several beers”.
If you ever want to get into the efficient software implementation of statistical algorithms, however, then complex analysis and numerical analysis can be incredibly helpful. Complex analysis may seem obscure, but I can speak from direct experience that analytic continuation is critical for efficiently implementing lots of transcendental functions. We wouldn’t have most of the CDFs in Stan without it!
If one does indeed want to do real-world applied stat, is Grad School in Stat a good choice?
The courses I saw friends in the Stat PhD program take seemed kinda distant from applications.
I agree that it depends on what you want to do in statistics.
That is why you shouldn’t take math courses beyond the minimum you mention, and instead learn more statistics, and do as much statistical research and/or applications are you are able to at your level of knowledge. Once you know more about what kind of statistics you are interested in (and for that matter, whether statistics actually is your real interest), you’ll know more about what (if any) math you need.
You should also learn a minimum about programming, if you don’t have that already.
Take a programming course. I recommend Python, but any modern language would do. R would work, too.
The anonymous comment above about Numerical Analysis gives an accurate picture. It was required for my BS in Computer Science. Unless the course available to you is more specialized, it won’t focus on statistics. It will probably cover numerical methods for what you’ve learned in your calculus classes. However, it will also cover the sources of error that are inherent in numerical methods, including the propagation of errors through multiple operations. It may also cover how sensitive some methods are so choices of initial conditions, step size, etc.
Numerical analysis is crucial. It should cover all the matrix decompositions (QR, LU, cholesky, svd etc) crucial for stats. And Optimization methods needed for MLE (sorry I know there are Bayesians here). And maybe even some MCMC which is just probabilistic numerical analysis.
Real analysis is needed for any large sample theory probably. And for probability (ie not statistics).
Complex is only for solving funny integrals you’ll probably never need.
Math modelling could be flakey.
Mathematical modeling: Extremely important to have command of the subject but a class? Maybe, but you might be better off spending a year in a lab where modeling is a focus.
Real analysis: Would be easier to judge if the syllabus were available
Complex analysis: I dropped it halfway through my first semester of grad school. In hindsight, I really wished I’d completed it. Definitely take it. It’ll dividends in the long run.
Numerical analysis: Importance depends upon whether you intend to focus on developing tools for solving problems or to focus on solving problems with the tools available to you. If your interest is in developing new tools then definitely take it.
Chris:
Spending a year in a good lab is great, but I think good labs are harder to find than good classes.
True enough. I’ll rephrase: “If you can find a position in a good lab with a modeling focus and it’s feasible for you to spend a year there – even if it’s just a few days a week – then that could be a more beneficial experience than taking a class. You might not be able to find such a situation but it’s worth exploring.”
Apropos of mathematical modeling, a column in today’s NY Times – http://www.nytimes.com/2015/05/03/opinion/sunday/manil-suri-mathematicians-and-blue-crabs.html
A bit frustrating to me that they call this “math” rather than “statistics.” By the way, Bayesian methods are huge in ecology.
I think this is both “math” and “statistics”. This is the type of modeling – describing dynamics of a harvested population – that could be done from an entirely mathematical (theoretical) perspective or one well-balanced by data and statistics. So it could really be a bit of math underlying this. It’s clear that there is data involved in this problem, but it’s unclear based on the discussion or my quick read of it, the balance of modeling mechanisms and linking to data or statistical analyses.
If the modeling class were about developing models of systems like this, it could be really interesting and useful – sort of like that old discussion of using a mechanistic model as the backbone for lynx hare dynamics and then using statistical techniques to learn more, as opposed to throwing non-mechanistic models at the data.
Otherwise the class could be about learning hard math techniques and notation applied to classic physics problems – probably not so useful.
Back to the post, I’d say numerical in the lead, followed by modeling (depending on the focus), complex, and lagging behind real.
I vote for real analysis. I started and floundered in an MS in applied stats program. Similar situation–didn’t major in math as an undergraduate but caught the statistics bug. I would just call myself a data analyst now, because I never finished the MS, and (to the point) I’ve since realized that the difficult material in the grad stats courses were mostly real-analysis-type stuff.
For what it’s worth, I massively disagree with you in the numerical analysis front. A decent numerics class teaches how an algorithm works, why it works, how to implement it, and (critically) how well you can expect a computer to do on a problem. I think all of these things are *at least* as vital as real analysis.
I think it’s very important to take at least one rigorous proofs class, e.g. real analysis, because doing mathematical proofs gets you into the habit of criticising your own work, which I think is a very important and under-valued skill for the statistician.
I second this. At least in my experience, real analysis is valuable not just because you will learn aspects of measure theory that are vital for advanced probability, but also because it is the best class for learning how to do a proof. And the aspect that Floundunded mentions of checking and criticizing your own work, to find corner cases and exceptions, is at the heart of both proofs and lots of good statistics.
If it wasn’t for application hurdle I would suggest the following based on my experience of being pretty much in the same state many years ago – entering a Msc in Biostats with only one previous math course (1st year calculus but with proofs) though a few serious statistics courses.
You probably have to get the multivariable calculus and linear algebra up front but try to do other math training concurrently with statistical courses. I found it useful to sit in on math courses (more breadth with less commitment though always less learning). I found it best to take math courses from mathematicians training math students and the higher level courses were best.
Part of the challenge in statistics is that you have to understand math more than you have to do it proficiently – a lot of it (e.g. measure theory) once you get enough of a grasp of it you can forget the details/manipulations.
And it depends on what you want to do and your aptitude for learning on your own versus in courses, for me Complex analysis/Fourier theory ended up being very helpful.
4,1,3,2
1. Mathematical modeling – first applied math class. super word problems: ‘calculate the amount of fuel to get to the moon on a rocket…’ very good for translating real world problems in to mathematical statements. not much statistics in the class but good for thinking about how to transfer theory to the real world
2. Real analysis – rigorous theory of limits and integration. proofs of handwaving done in intro calculus
3. Complex analysis – (not at all an extension of real analysis) – rigorous theory of functions of complex variables (I know that doesn’t really add anything, just look it up!). one small part is special functions, of which one small part are functions that popup in probability/stats (normal, Gamma, F, Poisson), but these will probably be left off to a small corner of a sequel class ‘Special Functions’
4. Numerical analysis – computer science of actually calculating in floating point: approximating weird functions/integrals, linear algebra (solving systems numerically, SVD). All of it very directly related to implementing statistical computations (which is informative for interpreting numerical results.
2 and 3 are totally theoretical and very, very far away from stats. A class in mathematical statistics would cover all that is remotely needed from those two classes and so much more; math stats will answer all your ‘why’ questions from stats. These classes are like studying ancient Akkadian vocabulary when you’re just writing an essay on child care (there may be -one- word that you use that you wonder where it came from, but you’re still going to use that word).
Of course if you’re interested, take them all, but I’m answering for efficiency from those 4 choices.
I really enjoyed learning 2-4 as an undergraduate, so I will comment on them and their relation to statistics.
1. Real analysis: I was fascinated by this course when I was more interested in pure mathematics and the axiomatic approach. It is useful for better understanding convergence concepts in probability theory and issues like how “minimax” should really be “infisup”. It was also useful for gaining practice in writing and reading proofs.
2. Complex analysis: I was completely mesmerized by complex analysis because of the residue theorem for evaluating contour integrals; it seems magical that complicated integrals can be solved simply by finding residues at poles of the function one is integrating! I vaguely recall solving one problem from BDA with a contour integral and an ML-bound argument, I think involving a (ironically) Cauchy distribution. Also, complex analysis may be useful for going from a characteristic function to a density function representation of a random variable, which involves the inverse Fourier transform.
3. Numerical analysis: Typically covers numerical linear algebra and algorithms to solve for odes and pdes. (E.g., the Runge-Kutta method.) Numerical linear algebra can be very useful for efficiently sampling from distributions. For instance, a multivariate normal distribution can be simulated by finding the Cholesky decomposition of its covariance matrix and multiplying it with a vector of standard unit normals. Unfortunately this can be very computationally costly, scaling with O(N^3) where N is the dimension of the MVN. (This can get problematic fast! A problem that takes 1 grad student year suddenly takes 1000 grad student years with a single order of magnitude more of data!) Numerical analysis can provide slick ways to circumvent this computational bottleneck; e.g., if the covariance matrix is Toeplitz (as in a GP with evenly spaced grid points) methods due to Trench and Durbin can be used, which operate in O(N^2), a full order of magnitude less. Some other modern methods attempt to exploit “sparsity” of the covariance matrix for this problem.
Sorry to belabor the point, but I am not happy with the way I wrote the last sentence of numeral 2. To be clearer, the inverse Fourier transform allows you to go from the density representation to the characteristic function representation of a random variable.
I’ve taken all 4 of those classes and recommend Numerical Analysis first. Depending on the syllabus, Math Modeling could be good. Real analysis helps motivate numerical analysis and is usually taken first by math majors. But, if you aren’t a math major, and don’t need to see rigorous proofs before learning how to use/apply ideas, then it’s not necessary.
Complex analysis is the lowest priority. Sure, it is cool. But, do you really need the calculus of residues IRL?