Stock, flow, and two smoking regressions

In a comment on our recent discussion of stock and flow, Tom Fiddaman writes:

Here’s an egregious example of statistical stock-flow confusion that got published.

Fiddaman is pointing to a post of his from 2011 discussing a paper that “examines the relationship between CO2 concentration and flooding in the US, and finds no significant impact.”

Here’s the title and abstract of the paper in question:

Has the magnitude of floods across the USA changed with global CO2 levels?

R. M. Hirsch & K. R. Ryberg

Abstract

Statistical relationships between annual floods at 200 long-term (85–127 years of record) streamgauges in the coterminous United States and the global mean carbon dioxide concentration (GMCO2) record are explored. The streamgauge locations are limited to those with little or no regulation or urban development. The coterminous US is divided into four large regions and stationary bootstrapping is used to evaluate if the patterns of these statistical associations are significantly different from what would be expected under the null hypothesis that flood magnitudes are independent of GMCO2. In none of the four regions defined in this study is there strong statistical evidence for flood magnitudes increasing with increasing GMCO2. One region, the southwest, showed a statistically significant negative relationship between GMCO2 and flood magnitudes. The statistical methods applied compensate both for the inter-site correlation of flood magnitudes and the shorter-term (up to a few decades) serial correlation of floods.

And here’s Fiddaman’s takedown:

There are several serious problems here.

First, it ignores bathtub dynamics. The authors describe causality from CO2 -> energy balance -> temperature & precipitation -> flooding. But they regress:

ln(peak streamflow) = beta0 + beta1 × global mean CO2 + error

That alone is a fatal gaffe, because temperature and precipitation depend on the integration of the global energy balance. Integration renders simple pattern matching of cause and effect invalid. For example, if A influences B, with B as the integral of A, and A grows linearly with time, B will grow quadratically with time.

This sort of thing comes up a lot in political science, where the right thing to do is not so clear. For example, suppose we’re comparing economic outcomes under Democratic and Republican presidents. The standard thing to look at is economic growth. But maybe it is changes in growth that should matter? As Jim Campbell points out, if you run a regression using economic growth as an outcome, you’re implicitly assuming that these effects on growth persist indefinitely, and that’s a strong assumption.

Anyway, back to Fiddaman’s critique of that climate-change regression:

The situation is actually worse than that for climate, because the system is not first order; you need at least a second-order model to do a decent job of approximating the global dynamics, and much higher order models to even think about simulating regional effects. At the very least, the authors might have explored the usual approach of taking first differences to undo the integration, though it seems likely that the data are too noisy for this to reveal much.

Second, it ignores a lot of other influences. The global energy balance, temperature and precipitation are influenced by a lot of natural and anthropogenic forcings in addition to CO2. Aerosols are particularly problematic since they offset the warming effect of CO2 and influence cloud formation directly. Since data for total GHG loads (CO2eq), total forcing and temperature, which are more proximate in the causal chain to precipitation, are readily available, using CO2 alone seems like willful ignorance. The authors also discuss issues “downstream” in the causal chain, with difficult-to-assess changes due to human disturbance of watersheds; while these seem plausible (not my area), they are not a good argument for the use of CO2. The authors also test other factors by including oscillatory climate indices, the AMO, PDO and ENSO, but these don’t address the problem either. . . .

I’ll skip a bit, but there’s one more point I wanted to pick up on:

Fourth, the treatment of nonlinearity and distributions is a bit fishy. The relationship between CO2 and forcing is logarithmic, which is captured in the regression equation, but I’m surprised that there aren’t other important nonlinearities or nonnormalities. Isn’t flooding heavy-tailed, for example? I’d like to see just a bit more physics in the model to handle such issues.

If there’s a monotonic pattern, it should show up even if the functional form is wrong. But in this case Fiddaman has a point, in that the paper he’s criticizing makes a big deal about not finding a pattern, in which case, yes, using a less efficient model could be a problem.

Similarly with this point:

Fifth, I question the approach of estimating each watershed individually, then examining the distribution of results. The signal to noise ratio on any individual watershed is probably pretty horrible, so one ought to be able to do a lot better with some spatial pooling of the betas (which would also help with issue three above).

Fiddaman concludes:

I think that it’s actually interesting to hold your nose and use linear regression as a simple screening tool, in spite of violated assumptions. If a relationship is strong, you may still find it. If you don’t find it, that may not tell you much, other than that you need better methods. The authors seem to hold to this philosophy in the conclusion, though it doesn’t come across that way in the abstract.

11 thoughts on “Stock, flow, and two smoking regressions

  1. “””
    First, it ignores bathtub dynamics. The authors describe causality from CO2 -> energy balance -> temperature & precipitation -> flooding. But they regress:

    ln(peak streamflow) = beta0 + beta1 × global mean CO2 + error

    That alone is a fatal gaffe, because temperature and precipitation depend on the integration of the global energy balance. Integration renders simple pattern matching of cause and effect invalid. For example, if A influences B, with B as the integral of A, and A grows linearly with time, B will grow quadratically with time.

    “””

    The global energy balance is composed of incoming energy from the sun, and outgoing energy from radiation. CO2 concentrations change the balance, and hence the equilibrium point. If the dynamics are fast enough, then CO2 will track energy level, like if the damping is fast enough in a spring, length of the spring will track weight of the load. I suspect that’s the motivation for this regression. Though, I doubt that equilibrium in the climate occurs on a fast enough timescale to make it valid.

      • Maybe, but there are a large number of cases where the dynamics occur rapidly compared to the timescale of a measurement. So, for example, if you read the temperature out of a thermocouple, the voltage is related to the temperature at the junction between the two metals, but it takes time for the thermal energy to reach the junction, so the actual temperature at the surface of the thermocouple probe is probably a little higher… in this context, most of the time the answer to this dynamics problem is “who cares” and you can treat the two temperatures as the same, but if you need to know the surface temperature extremely exactly and with high time-resolution… you need to take the dynamics into account.

        Usually a regression of a dynamically changing thingamajig is more or less an implicit asymptotic model where the equilibrium occurs much faster than the sampling rate.

    • Correct. I think it rises from violated assumption to gaffe because it’s really well known that equilibrium is not fast enough to make the assumption valid. It’s astonishing that the reviewers didn’t catch it. Hydrologists ought to be familiar with integration.

      The dynamics critique ought to be pondered in any regression, but often the equilibrium assumption holds, so it won’t be an issue.

      • I see regression used aggressively in many social / political science studies. Is the fast dynamics assumption good here?

        Do opinions, IQ, voting preference etc. reach equilibrium faster than these studies sample?

  2. This is getting away from the stocks versus flow discussion, but the model here has an underlying assumption that tends to be in most climate related analyses, whether implicitly or explicitly. And that is that there is a global mean behavior. This assumption is implicit in the above model, because the assumption is that increases in CO2 will cause all temperatures to rise, which will increase rainfall everywhere. As Fiddaman notes on his web page, the results appear awfully similar to the spatial distribution of predicted precipitation anomalies. And in fact, the above paper finds a weak negative correlation in the SW. United States, which is consistent, because most climate models predict that this region will have drier, not wet conditions.

    The paleo record suggests that in periods of warming or cooling, different parts of the world will have different responses, most notably the poles vis a vis the polar regions. But there is good reason to believe for example that upwelling will increase along the eastern boundary regions with global warming, which would mean cooler ocean temperatures and cooler coastal land temperatures in those regions. As long as the statistical models try to fit the data into a framework of a consistent global response, we will keep seeing incorrect and physically unlikely results.

    • Title of the post is a play on a Guy Ritchie film, “Lock, Stock, and Two Smoking Barrels.” Madonna was once married to Guy Ritchie, hence the image, I suppose. It’s a long way to travel for a small payoff, don’t you think?

        • Gary: I’ve just spent 120 quid on me hair. If you think I’m puttin a stockin over me head you’re very much mistaken.

      • Aaaaaaaahhhh… Well, I guess this is pretty hilarious if one is well-versed pop-culturally. But for me… your comment contains at least two pieces of information that I had not been aware of – – – and others still that I had to find out, among them the fact that Guy Ritchie apparently is not a film genre…

Leave a Reply to jrc Cancel reply

Your email address will not be published. Required fields are marked *