Here’s the paradox: Analytically, derivatives are easier than integrals. You can take a messy function and differentiate using the chain rule, no problem. But lots of simple expressions have no analytical integrals. They have to define new things like Bessel functions just because of all indefinite integrals that can’t be evaluated in closed form.
So, fine. Differentiation is easier than integration. Everybody knows that; it’s why when you learn calculus they teach you derivatives first and integrals second.
But . . . computationally, it’s the other way. Computationally, derivatives are unstable and integrals are stable. The derivative is the slope of a curve, and if the curve is noisy (as can occur with computed quantities), the derivative will amplify that noise. Differentiation is a high-pass filter. In contract, integrals are stable: you’re averaging a bunch of little pieces, and even if they’re variable, the average can be smooth.
To put it another way, derivatives are differences; integrals are sums. Differences are less stable than sums. Just think of examples like 379124.32948293 – 379124.32948287, where you need to keep a huge number of digits in the original values in order to calculate the difference with any precision.
OK, in practice that doesn’t happen when we compute derivatives, but that’s because we use analytical differentiation (nowadays, we’ll use autodiff). Or if we compute numerical derivatives, we still do some smoothing or local approximation. Noisy data can be stably integrated but can’t be stably differentiated.
From that perspective, we’re damn lucky that derivatives are easy to compute analytically. Conversely, it’s not such a problem that we can’t in general do analytic integrals, because, hey, we can just evaluate them numerically, no problem.
So is that it–we’re just lucky? I feel like there’s something deeper than that going on.
And it’s not just an issue in math and statistics. The same concerns arise in economics, say. If you have causal identification, it’s easy to estimate aggregates–in econometrics jargon, “average treatment effects.” But it’s hard to estimate effects on the margin (where I’m using the term “marginal” in its economics sense, not its statistics sense). The trouble is, you only have a little bit of data on the knife edge that is the margin. To estimate a marginal effect at all, you need to do some modeling–that is, some averaging. That’s not an issue when you’re estimating an aggregate effect, because there you want to average. An aggregate effect is a sum, an integral. A marginal effect is a difference, a derivative, and it’s inherently harder to estimate.
Again, to estimate that differential effect, you need some modeling, and from there we can take advantage of the ease of analytical differentiation.
So there you have it. An interesting paradox, I think.
P.S. People offer some good thoughts on this in the comments.
Queue the XKCD comparison in 3, 2, 1. ;)
FWIW, I don’t see it as that ironic. A little, maybe. More like nature of the beast. And I don’t obsess on theoretical calculus.
Maybe one similar example is following the course of the war in Ukraine. I see people get all bent out of shape on which mapper has what bias. But it only matters if you zero in on the day’s change. As you zoom out to a month or a year or the like, gray zone debates become silly. It’s pretty clear that the Russians are advancing. But at a half snail pace (literally). I.e. pretty close to a stalemate.
I think you see the same thing in time series or econ on finance, with people obsessing on recent variations (which have a lot of noise anyhow) and not considering the long run (OK before the “dead” part) enough.
Thought-provoking. Thanks. I think “contract” should be “contrast” in the third paragraph.
A cool thought about something I’ve sort of known for a long time, but never appreciated as you have here. In my thesis (long ago) I calculated the number of electrons in water using the oscillator strength sum rule and the Kramers-Kronig relation to integrate the spectra of water, liquid and vapor. The spectra are complete with respect to frequency but not so good, very noisy, but I got 10.1 +/- 1. Yeah, I know that the 0.1 isn’t significant but it made me feel good. I was really going for the Kerr effect in the visible, which depends on the orientation of the transitions with respect to the dipole moment so calculating the oscillator strength sum rule was just a check. It’s good to know that water has 10 electrons experimentally.
The point I’m making I guess is that the integral calculation allows a reasonably robust estimate of the error even given noisy data. I think i used a variable step width, finite difference method to do the integral using APL, it was fun and no derivatives were killed in this project,,,,
Numerical integrals are easy in low dimensions. In higher dimensions, there are too many points to sample.
That’s why we have Hamiltonian Monte Carlo!
And hence use the derivative….
It feels circular to suggest integrals are “easier” if you rely on autodiff gradients to make the computation feasible at scale
Taking this line of thought a step farther (although in which direction I do not know), special functions (such as `lgamma`) often have integral representations, so differentiating them with respect to their primary argument is not too hard, although you often end up with another special function (such as `polygamma`). Differentiating a special function with respect to a parameter is often much harder to accomplish analytically; see
https://blog.wolfram.com/2016/05/16/new-derivatives-of-the-bessel-functions-have-been-discovered-with-the-help-of-the-wolfram-language/
Autodifferentiating an expression involving a special function that depends on an unknown parameter is not hard, but it is quite slow. I have been trying (intermittently) for like a decade to get the logarithm of the modified Bessel function of the second kind into the Stan Math library, which is useful for avoiding overflow and underflow but difficult to differentiate with respect to the unknown order. Other people have also worked on this problem in recent years, but it has not culminated in a Stan Math implementation.
This is a good example of a post you write where I’m just like “well said” and don’t need to make any meaningful comment.
Is one way out = don’t differentiate the data, differentiate a model of the data? I tried that here: https://github.com/finite-sample/incline
“Differentiation is easier than integration. Everybody knows that; it’s why when you learn calculus they teach you derivatives first and integrals second”
And yet, when you learn analysis, it’s the other way around. It’s trivial to define a function that is integrable but nowhere differentiable. As you mentioned, this is because integrability is global but differentiability is local property of the function ar each point.
I feel like the paradox has more to do with the universe of functions we tend to deal with. For general functions, I think, integration is just a much simpler concept. We just tend to like to live in a world of functions where differentiation is more likely to be computable in a closed form.
“integration is just a much simpler concept”: This post being on Pi Day, it might also be amusing to note that more than two millennia ago Archimedes computed pi essentially by integration, but it wasn’t until the 17th century that Leibniz computed pi using differentiation.
Integrals were discovered first! The Babylonians were doing trapezoid rule calculations 1400 years before they were rediscovered in Europe (https://www.science.org/content/article/math-whizzes-ancient-babylon-figured-out-forerunner-calculus-2).
Archimedes’ method of exhaustion is also a precursor to integration. They had no concept of the slope of a curve, that was discovered later.
See also David Bressoud, Calculus Reordered. Maybe we should teach integrals first, as he suggests.
Analytical differentiation: create a new function regarding a property, the slope, of the original function at any single point.
Analytical integration: create a new function regarding a property, the sum, of the original function over any arbitrary range.
The integral encapsulates more information about the original function.
Numerical differentiation: taking the ratio of two numerical integrals.
2>1
https://mathoverflow.net/questions/66377/why-is-differentiating-mechanics-and-integration-art
Here’s a discussion of the mathematical reasons for thus discrepancy
In analysis, integration is usually viewed as being much easier than differentiation because you can (basically) always integrate, but you need assumptions on your function in order to differentiate. If you integrate, then the function becomes nicer (it has more derivatives). There are many extremely useful theorems of the form “if a function is differentiable k times and its derivatives are appropriately bounded, then it is in fact differentiable k + 1 times.” One reason why this is useful is because it is sometimes possible to show that a solution to some problem is unique if you know that the solution is sufficiently differentiable.
> Computationally, derivatives are unstable and integrals are stable.
I think this is a key point! And there is a strong connection between computation and continuity: integration is continuous while differentiation is not.
I think that differentiation “falsely” seems easy as an “analytical” exercise because we when we say that derivatives are easy, we are referring to taking the derivatives of analytic functions. But by definition, analytic functions basically have all of their derivatives specified up front — at least, that’s how I think about it from a computational perspective. If you have a programming language that supports autodiff and want to support an externally defined function that’s a black box, you need to supply not only the code for calculation the function’s value applied to a point, but also the infinite tower of all of its derivatives. When you view these analytic functions basically as these derivative towers, i.e., the collection of f, f’, f”,…, then the differentiation function becomes a trivially continuous (and hence computable!) operation, since the derivatives are already there.
What happens in real life? Ever try to catch a ball using changes in the ball’s speed? Or do you try to figure out where it will land based on its trajectory?
Difference equations vs. Summation.