Selection bias, or, How you can think the experts don’t check their models, if you simply don’t look at what the experts actually are doing

My friend Seth, whom I know from Berkeley (we taught a course together on left-handedness), has a blog on topics ranging from thoughtful discussions of scientific evidence, to experiences with his unconventional weight-loss scheme, offbeat self-experimentation, and advocacy of fringe scientific theories, leavened with occasional dollops of cynicism and political extremism. I agree with Seth on some things but not others. (Here’s Seth’s reason for not attempting a clinical trial of his diet.)

Recently I was disturbed (but, I’m sorry to say, not surprised) to see Seth post the following:

Predictions of climate models versus reality. I [Seth] have only seen careful prediction-vs-reality comparisons made by AGW [anthropogenic global warming] skeptics. Those who believe humans are dangerously warming the planet appear to be silent on this subject.

In response, Phil commented:

Funny, on the day you [Seth] made your post saying that you haven’t seen comparisons between models and predictions except by skeptics, the top entry on RealClimate, the single most prominent global-warming-related blog that is not run by skeptics, was “Evaluating a 1981 temperature projection.”

Pretty amazing, huh? On its face it would seem surprising to claim that the majority of leading climate scientists don’t do “careful prediction-vs-reality comparisons,” and indeed on the very day of Seth’s post, there is such a comparison right there on the first place you might look for what the climate scientists are doing!

How did Seth miss it?

A clue comes from the sources on which Seth relies. His link above points to the webpage of the Von Mises Institute, a political advocacy organization. Other Seth links to global warming stories have come from an climate change skeptic blog, a right-leaning politics blog, another climate change skeptic blog, another advocacy organization, one more climate change skeptic blog, a letter to the Wall Street Journal, yet another climate change skeptic blog, an op-ed in Forbes magazine, a lecture by science writer and political activist Matt Ridley, another blog that seems to specialize in climate change skepticism, conservative political columnist Jeff Jacoby, a Wall Street Journal op-ed, still one more climate change skeptic blog, a conservative religious magazine, . . . ummm, you get the idea.

If these are your sources, you can get a distorted view of the opinions and arguments of “those who believe humans are dangerously warming the planet”! Any one or two or three of the above sources might be informative, but it doesn’t make sense to only look there. (Quick: glance again at the list of sources in the above paragraph.)

I’m not saying that Seth has any sort of duty to read the scientific literature—he’s trained as a psychologist, not as a physicist—nor does he need to read RealClimate (even if only as a supplement to the Von Mises Institute page, Wall Street journal op-eds, and so on), but it seems pretty silly for him to be so sure of himself on the science, given that his selection of politically-loaded sources.

I’m not an expert on climate science either (I’m currently involved in a research project using tree-ring data to estimate historical climate, but my expertise in this project is in the statistics, not the biology or physics), so I asked Phil what his thoughts were on the particular article that Seth linked to (by David M. W. Evans, described as “a mathematician and engineer, with six university degrees including a PhD from Stanford University in electrical engineering”).

Phil responded:

If you look at the temperature anomaly was higher in 1988 than in any other year until 1997…and was lower in the past couple of years than at most times in the past decade. The paper Seth blogged about looks at the change in temperatures since 1988 and says hey, there hasn’t been any increase whereas this modeler guy said the temperature would be way higher by now. If he had chosen 1985 or 1989 as the starting point instead, it would look different. Of course he’d say he chose 1988 because that’s when Hansen testified in Congress, so fair enough in a way…but this is a bit like the housing data that you blogged about, why not show the earlier data too? And the reason is the same as in the housing data: it’s because if you show the earlier data it undermines the story you are trying to tell. 1988 was an unusually hot year, by a lot.

Phil then turns to the RealClimate post (by Geert Jan van Oldenborgh and Rein Haarsma) mentioned earlier:

They compare a forecast from 1981 to data. This uses a land-ocean index rather than tropospheric temperatures…perhaps that’s an example of this guy cherry-picking too (the land-ocean temperature increase has been more rapid than the satellite-derived tropospheric temperature increase for reasons someone else might know but I don’t). Anyway this one shows a totally different story of course. Actually this guy cheats a little by starting his baseline too high I think, though that’s just based on a visual impression.

One thing that people who do these things don’t seem to understand is that the uncertainties in climate models (like low/medium/high curves) are supposed to represent the uncertainty in the trend, not the uncertainty for an individual year. It’s sort of like showing uncertainties in a regression model with one of those bow-tie curves: that doesn’t mean that you’re predicting that the individual data points will fall within the upper and lower curve, it means that the true line is probably somewhere in there. Similarly, people say “the past five years (or whatever) have been below the predicted trend line so the trend line must be too high,” apparently not realizing (or deliberately ignoring) that there is a lot of year-to-year variation, plus autocorrelation on the scale of a decade or so. It’s a true statement that global mean temperature was as high in 1988 as it was last year; it’s false to say that there’s not an upward trend in temperatures.

I think, though, that if Seth continues to search for information from his usual sources, he will remain convinced that there is no man-made global warming and that vast majority of scientists who study these problems are not interested in checking their models, have not ever thought about the ice age, etc etc.

I think this is important not only for the followers of Seth Roberts but more generally in that it illustrates the traps that people can fall into when seeking out confirmation of their beliefs. Seth is better-equipped than most people to read about scientific evidence, yet he is stuck, not only in holding a scientific view which I find implausible (after all, I might be wrong) but in not understanding that Geert Jan van Oldenborgh, Rein Haarsma, etc etc etc are doing serious science. It’s sad, and it’s scary.

P.S. Phil provides a good summary in this comment.

54 thoughts on “Selection bias, or, How you can think the experts don’t check their models, if you simply don’t look at what the experts actually are doing

  1. I think this is important not only for the followers of Seth Roberts but more generally in that it illustrates the traps that people can fall into when seeking out confirmation of their beliefs.

    As a psychologist I’d argue he *does* have a professional obligation to know about this basic human bias, and try to correct it in himself where possible. (Then again, you’ve pointed out how difficult it is to apply your statistical knowledge in real-world contexts like investment and teaching, and of course sometimes I find myself thinking unpsychologically and unstatistically too; so perhaps I should be more humble here and acknowledge that this kind of thing is probably hard for everyone, in every profession!)

    • I’m sure Seth is aware of confirmation bias and other similar biases. However, the whole point of the literature on these biases is that they also occur for experts (certainly in everyday life and outside their domain of expertise – but its well known they can also occur – perhaps less frequently – if you are aware of them and in areas where you are expert).

  2. Correct me if I’m wrong here, but it looks like the actual values all lie outside the forecast range of scenarios (on the high side). How is this supposed to make me more comfortable that climate models are accurate? Couldn’t this be considered evidence that they did not identify the important predictors of warming?

    • It also appears, by eye, that the smoothed GISS broke through the “last century’s temperature” 2-sigma about a decade earlier than projected.

      An interesting quote from the original paper is: “Models do not yet accurately simulate many parts of the climate system, especially the ocean, clouds, polar sea ice, and ice sheets.” Which seems to mean that a majority of the system was not accurately simulated, even in isolation.

      I’m certainly not an expert in the field. My skeptical leanings are fueled by the papers I’ve read. To learn more about statistics, I’ve read hundreds of papers in statistics, economics, etc, by folks like Andrew. I’m still learning and may not understand much of what I’ve read, but I do recognize that the papers tend to have a similar “feel” to them. I don’t know what to call it except perhaps “rigorous”, “statistical”, or “doctoral”. Several of the key global warming papers I’ve read don’t have this feel, but rather feel like undergraduate work.

      For example, one of the recently-trumpeted papers by Foster and Rahmstorf (2011) doesn’t bother with standardized coefficients, assumes that solar irradiance is the only possible influence of the sun on the earth (as does the original Hanson paper), and in its summary extrapolates their time series linear regression for decades into the future.

      • I know what you mean; my background is in econometrics\statistics and many of the climate science papers I’ve read do not seem as professional or rigorous as those you would see in other fields, though I’m not an academic and my ability to judge such things is limited to the statistical aspects of the analysis.

        The bottom line to me is that if these models were so good as to make the evidence “overwhelming”, then we would see people making public predictions, and then publicizing the fact that they made accurate predictions. I think that was Seth’s point, and a good one, in no way refuted by linking to a blog post that shows a very poor and incomplete forecast.

    • The thing to bear in mind about all the early predictions (Hansen 81 & 88, IPCC 1990) is that there are multiple moving parts:
      1. Inputs (emissions)
      2. Core biogeophysics, like GHG cycles, radiation and convection, etc.
      3. Peripheral biogeophysics, like ice sheets and land/biomass feedbacks, which may be important, but were (and remain somewhat) poorly understood
      4. Endogenous variability with long correlation times, that can’t exactly replicate reality even in principle

      So, to decide whether you’re comfortable or not, you’d have to go a step further, decompose the errors as much as possible, and then evaluate the results subject to #4. I think you’d find that a lot of the error/uncertainty boils down to #1. The core science (#2) has held up well, in that models today have about the same climate sensitivity as they did then, in spite of much refinement and huge improvements in computing. More importantly, specific predictions other than the global temp trend, like water vapor dynamics, have worked out well. And several model-data disagreements, like the satellite troposphere temp trend and WWII-era sea surface temperatures, have been resolved in favor of the models.

      The other thing you have to ask yourself is, how are the models doing relative to alternative hypotheses and contemporaneous skeptic predictions? This is tough, because there are few if any that are explicit enough to test (I’m not aware of any, and I’ve looked). So, you’re stuck with reconstructing naive forecasts (e.g., temperature is a mean-reverting random walk). Or, perhaps you could plug Lindzen’s assertion that climate sensitivity is < 1C into a simple energy balance model with some noise thrown in. Neither approach would have predicted warming, so with that as backdrop, the early models look even better.

  3. Pingback: Evaluating a 1981 Temperature Projection | The Lign Above

  4. Beautifully articulated.

    This kind of bias has broad implications for how the public at large responds to all sorts of issues, scientific and otherwise. It makes it hard to have coherent discussions with people on complex topics. I speak from experience here.

    Reminds me of an article I read a while back called, ‘How to cheat with citations’, or something similar.

  5. Reminds me a bit of the J. Scott Armstrong – Al Gore bet (to the extent one call call it that, since Gore hasn’t accepted it).

    Based on what I read IN THE PRESS, all sides of this issue seem massively overconfident that they are correct. It’s quite likely that in more technical publications there is more humility.

    • Zbicyclist: what level of credibility do you assign to theclimatebet page you mentioned? And, in climate, to J. Scott Armstrong?

      • 1. They seem to be reporting the factual data accurately.

        In terms of credibility, I can’t give a simple rating:

        2a. Armstrong is a forecasting expert, and as such would be expected to know about the perils of extrapolating trend, the tendency of doom forecasts not to happen, the tendency to overconfidence, etc.

        2b. So far as I know, he knows little about climate science per se. In the 850 pages of his “Principles of Forecasting” the index does not have listings for “Climate” or “temperature” or “warming” or “global warming”. Worth noting is that the book is dedicated to Julian Simon, who won the famous wager with Paul Ehrlich.

        • How would you assess this?

          “To provide some perspective, climatologists sometimes use seven years as the duration of a climate period. Over the last seven years, the UAH global temperature anomaly series has trended upwards at a rate of 0.008C per year. The solar magnetic activity cycle has a period of about 11 year. Over the last 11 years, the temperature series has had a trend of +0.019C per year. The former trend is much closer to Prof Armstrong’s no-change forecast than it is to Mr Gore’s extrapolation, but the latter is somewhat closer to Mr Gore’s extrapolation. The trend for the entire 33 year period of the UAH temperature series, at +0.0138C per annum, marginally favors Prof Armstrong’s forecasting method and suggests that there is no reason for alarm.”

    • The funny thing about this bet is that Armstrong’s naive forecast is exactly the sort of thing that would perform poorly against the early model forecasts of warming, if it were implemented rationally.

      However, Armstrong stacks the weighting of the bet to favor the very near term (i.e. favor noise over signal), and obscure global trends with local variability.

      In spite of the fact that he claims forecasting as his domain, he fails to identify an optimal naive forecast, so that you could beat him with a better naive model.


      • Thanks Tom.
        Yes, but more.

        1) That page is marketing of confusion to the unwary.

        a) The full page alludes to longer intervals needed, then focuses on 7 years.
        ““To provide some perspective, climatologists sometimes use seven years as the duration of a climate period.”
        This is either ignorant or purposeful.

        Seven years is seven years, just as 5 years is 5 years and a day is a day. This is written to confuse people who do not understand the interval (17-20 years called by Gavin Schmidt elsewhere) needed for decent signal extraction here.

        b) Likewise, 11 years is too short, but the solar stuff sounds impressive.

        c) They use UAH satellite numbers, despite the long history of errors and issues, generally being low, but as errors have been fixed, they are getting closer to the others. See Tamino, again.

        2) All this markeitng is unsurprising as J. Scott Armstrong is:
        a) A A Professor of Marketing at Wharton.

        b) A Heartland Expert.
        Heartland is a not a science entity, but a confusion-marketing and advocacy one, starting with tobacco.

        c) A Heartland Global Warming Expert, as presented to legislators.
        See PDF @ p.52.

        d) A contributor to the NIPCC reports by Fred Singer, Craig Idso, etc. See pp.34-36, where Armstrong row on p.36 lists various activities, talks at Heartland, signing CATO advertisement, etc.

        e) Heartland has misled people with the UAH satellite numbers for at least a decade, see pp.100-101.
        There are plenty of examples of “how to lie with graphs” and also “how to ignore the last line of a data set that contradicts your conclusion.”

        f) Heartland’s Environment & Climate News has often marketed dumb bets or challenges for debates, not ways to make progress in science, … but then this is marketing/advocacy.

        3) Back to the original topic: if somebody *wants* to believe something, that’s that. Forget about weighing evidence or assessing credibility of sources.

        Appendix P, pp.78-81 shows how the American Institute of Professional Geologists (AIPG), who ought to know better, rejected mainstream science in favor of Heartland sources.

        In this case, I’m afraid Seth Roberts seems to have *wanted* to assign credibility to David Evans, who:
        -is not a climate modeller,

        -has not published any peer-reviewed papers on climate,
        -who rehashes much of the same debunked material and
        -is NOT a “rocket scientist,”

        He is also married to Jo Nova, who wrote The Skeptics Handbook, which Heartland sent to 14,000 school board presidents, as part of its Fakeducation effort. Both have been involved with Heartland, and it seems rather likely Heartland paid for some of this (pp.63-64).

        4) Many people pride themselves on being skeptics, in the classical good sense. Some read CFI’s Skeptical Inquirer, which has a long history of debunking silly beliefs and fakery.

        Retired NASA scientist Stuart Jordan published a straightforward article on global warming in 2007, followed by others. E-i-C Kendrick Frazier was surprised by the firestorm of “cancel my subscription for printing this junk” letters, from avowed “skeptics” who were absolutely and desperately certain Stuart was wrong.

        Skepticism can be quite selective, sad to say.

        • By odd coincidence, Heartland just got itself in the news for comparing believers of global warming to the Unabmomber, Charles Manson, Fidel Castro, via billboards around Chicago and a PR campaign.
          See this, for example. This is a good example of intensity of belief … although it rather breaks the IRS rules for tax-exempt 501(c)(3) public charities…

          ‘The people who still believe in man-made global warming are mostly on the radical fringe of society. This is why the most prominent advocates of global warming aren’t scientists. They are murderers, tyrants, and madmen.’

  6. Most evaluations of past climate model performance are problematic because they don’t disentangle the model’s physical assumptions from their socioeconomic assumptions. It could be that the model got the climate physics right, but made wrong assumptions about future industrial emissions of greenhouse gases, sulfate aerosols, etc. Unfortunately, it’s often hard to go back to those very early models and disentangle these factors, because the model code and forcing scenario data used in various published studies aren’t archived.

    • RealClimate does a look-back each year, and this is for 2012. See section on Hansen et al, 1988.

      Of course, for those interested in accessible, but serious, non-cherry-picking trend analysis by credible people, see post by tamino, who in real life is a very good time-series guy.

      Just recently, John Nielsen-Gammon, TX State Climatologist at TAMU, offered a really fine analysis and especially, a great graphic. Most people get confused by the noise of ENSO (El Nino / La Nina), which makes it hard to see trends. John separates years into El Nino (hot surface air temp), Neutral, La Nina (cool), and then shows regression lines for each, yielding one of the best graphics I’ve seen in this. That’s a good example of the fact that with the same data, there is still room for creativity in achieving good presentation.

      • RealClimate’s look-backs have the same problem I mentioned above: they don’t separate climate physics from emissions scenarios. They can’t, really, not without re-running all the GCMs with updated forcing data every year. You could try to build a regression or other reduced model to predict what a GCM “would have” predicted under different forcings, but this is rarely done (or done well). Nor do the data-model comparisons generally account for forcing uncertainty (how well should we “expect” the GCMs to do, if we’re not completely sure of their inputs?).

  7. See Catalog of reasons for anti-science opinions, a few specific to climate issues, but general. From experience, these occur in different combinations, often having been acquired in different orders, even when the end result is similar. Generally, when someone with little domain-specific expertise takes a clear position against any strong consensus by the experts, it might be considered a form of PSY5 there (Dunning Kruger), found often. [One of the nice things about working at Bell Labs was the calibration provided for one’s own expertise and training to go find real experts.]

    But, I’d guess from Seth’s links, that IDE2 (ideology) might be a strong influence, which in fact I’d speculate is one of the most common.

    • John:

      I’d be interested to hear what Seth thinks is behind his strong belief that most climate scientists are wrong. I’m guessing he’s generalizing from his experience as a psychology professor, where he felt that many highly-credentialed experts—researchers who were indeed authorities in their subfields—were doing pointless work and ignoring the big issues. Meanwhile, he felt that on many problems that were important to people (ranging from weight gain to mental health), progress was being made by the people who Seth refers to as insider-outsiders: researchers with a small enough stake in the current system to be willing to entertain new ideas, but with enough scientific background to make progress, to construct and test interesting hypotheses.

      As I discuss in my blog above, whether or not Seth is right on the substance of climate change, there’s no way he’s going to get a good perspective on climate research by limiting his reading diet as he does.

  8. If so, that might fit TEC8 (over-generalization), although that is more common amongst people with computer-modeling experience in specific fields, but without the breadth of experience to understand the differences in kinds of models.

    See comments on specific ways people from different disciplines get confused and RC FAQ on models, which starts by explaining the differences between physics models and statistics models.

    In this case, though, it might well be indicative of the inherent differences between social sciences and physics. The latter’s conservation laws tend to impose ferocious constraints on hypotheses.

  9. You write “How did Seth miss it?”

    Uh, because I wrote my post before it came out?

    Your overall point seems to be that, due to a bias in searching for evidence, I have missed many many comparisons of actual global temperature with model predictions of global temperature. Okay, where are they?

    • Seth:

      I didn’t mean that you missed that particular post. I meant that you missed all such comparisons. But, as noted in my post above, this is not such a surprise, given your apparent sources of news. Phil writes:

      Here’s an earlier post, also on RealClimate:

      And here’s another:

      The answer to why I [Phil] choose the one I chose is that when I did a google search for something like [compare global warming model data] it was the first one that came up that wasn’t from a skeptic site.

      • I have looked at those two links. They are not persuasive support for climate models.

        The 2009 link (isn’t that a long way back): Look at figure 1. When did the model make those predictions? You can’t tell.

        The 2011 link: Now at least it is clear (in Figure 1) when the model (or suite of models) made those predictions. It is also clear that the variation in prediction is so great that it is highly implausible that the actual data could have fallen outside the predictions (the grey area). Meaning this verification provides essentially no support for the model. Later in the post the fact that Hansen’s 1981 model (Scenario B) predicts too much warming is shrugged off.

        My belief about climate scientists — they are claiming the evidence for AGW is much stronger than it actually is — is reinforced by these two posts. They do not support your claim that I am skeptical of AGW because I am fed biased information by my biased sources. In this particular case your information (from Phil, a less or differently biased source) led me to the same conclusion.

        • I have “no idea”? Zero? Are you sure about that? As for the link you give, so what? My claim is that AGW advocates have made up their mind too soon. AGW is indisputable, they say. And now you point to something that suggests stuff isn’t nearly so settled. That supports my point of view — that the AGW advocates have overstated the strength of the evidence.

        • 2009 is a long way back for evaluating a forecast made in the 80s?

          Anyway, with a little digging you could easily discover the submission deadline for the AR4 runs in fig 1.

          But what you should be looking at is fig 3, published in 88, on the basis of data and models that were probably locked in a year or two prior to that.

          But the bigger point is that global temperature time series are not the whole database – you also have to look at ocean heat (Schneider & Thompson’s 0D energy balance model from the 80s does pretty well, even though the data were not available until much later), precipitation, spatial patterns, etc.

  10. Andrew writes: “I’d be interested to hear what Seth thinks is behind his strong belief that most climate scientists are wrong.”

    Let’s make it clear what I think they are wrong about: They overestimate the strength of the evidence for man-made warming. They think the evidence is very strong, I don’t think it is very strong. You wrongly describe my views when you say I am “convinced there is no man-made global warming.” What I am convinced of is that many climate scientists, such as Michael Mann, have vastly overstated the strength of the evidence for their claims. There is a big difference between what you say I believe and what I actually believe.

    What is behind this belief of mine (that climate scientists have vastly overstated how strong the evidence is): I believe that the only good test of a theory is whether it makes correct predictions. Hal Pashler and I wrote a paper about how for 50 years psychological theorists had claimed their models deserved support because the models could be fitted to data. We pointed out the absence of verified predictions. Perhaps the models were flexible enough to fit any plausible data set. They certainly had several adjustable parameters.

    I wondered if climate scientists were doing the same thing — passing off fitting past data as persuasive evidence that a model deserves belief. Their models had plenty of adjustable parameters. As far as I could tell, that is what has happened. I haven’t looked lately but I have looked in the past — including at a chapter that Phil pointed me to (something about “why we should believe climate models”) — and I kept failing to find the only evidence I find persuasive: accurate and surprising predictions of global temperature. Surely the climate scientists pushing AGW understand how persuasive such predictions would be. So I took their absence from my search among AGW advocates to reflect their actual absence.

    • Seth:

      Your comment illustrates the selection bias problem. You write, “I don’t think it [the strength of the evidence for man-made warming] is very strong.” But your sources are the webpage of a political advocacy organization, a climate change skeptic blog, a right-leaning politics blog, another climate change skeptic blog, another advocacy organization, one more climate change skeptic blog, a letter to the Wall Street Journal, yet another climate change skeptic blog, an op-ed in Forbes magazine, a lecture by science writer and political activist Matt Ridley, another blog that seems to specialize in climate change skepticism, conservative political columnist Jeff Jacoby, a Wall Street Journal op-ed, still one more climate change skeptic blog, a conservative religious magazine, . . .

      These are good places to confirm your existing beliefs.

    • Reliance on correct predictions is certainly reasonable, but you seem to be disregarding predictions of anything other than the global temperature time series. (Gavin provides examples below, including stratospheric cooling, which ought to be very surprising for anyone who thinks the temperature trend is of solar origin). For global temperature, you conclude that signal can not yet be discerned from the noise, though you don’t state your model for the noise or a null forecast. Convincing you of anything would seem to be an unwinnable game, or at least a very long waiting game.

  11. I have looked at far more stuff than I have linked to. In particular I have looked extensively at pro-AGW writing. Of course it’s silly to form an opinion based on what only one side says. That’s why I looked at both sides. I ended up siding with people you disagree with, apparently. So be it. That doesn’t mean I did a biased search.

    • Seth:

      You have been very clear about lack of evidence versus lack of effect.

      It would be great if you had evidence of your search (and it unbiased nature).

      In clinical research, various groups suggest the search be well enough documented so that it can be re-run. Also best if an initial version is posted before the search and modifications tracked.

      Very difficult to discern bias in the (just the) outcome of a process.

      • It is entirely possible that I have missed something. If you think I have missed something where climate scientists persuasively show that their models can predict climate 30 years from now, please, tell me what it is.

        • Seth: I have no idea.

          I was just trying to make the point that having a reproducible documented search is the _easiest_ way to dispel criticisms of having done a biased search.

  12. Skeptical Science has a nice list of well-debunked climate anti-science memes, which are repeated endlessly. Each entry explains the meme, gives an accessible debunk, and points at credible backup literature. It isn’t worth anyone’s time to re-debunk these things again and again, just reference them there. That list varies by currently popularity, so I usually cite the fixed list. That is especially useful when faced with a dozen or so memes “Gish galloped” into a paragraph, as one can just list the numbers.

    But this is easier.

    Models are unreliable.
    Actually, they are pretty good and getting better. Climate models are not weather models (even when they use some of the same code), and they are certainly not statistical models merely tweaked with free parameters to hindcast well.

    Earlier, I mentioned the RC FAQ on models, which started with physics versus statistics. It also addresses “What is tuning.”

    There is also FAQ II, whose first topic is parameterizations.

    Models make successively better predictions, but they are dealing with very noisy natural time series, overlaid with multiple human effects.

    a) They cannot predict the timing of major volcanoes. However, they were already good enough in 1991 to predict the likely effects of Pinatubo.

    b) They cannot predict the timing of ENSO (El Nino/La Nina) events … but over longer term, those tend to average out.

    c) In 1988, Jim Hansen failed to predict the massive industrialization of China in the late 1990s and 2000s (and thus the attendant aerosol effects that reduce the warming in a big chunk fo the world, just as they did in the US before the Clean Air Act.)

    Anyway, one more time, RC reviews model to data comparisons every year.

    Finally, it is odd to name Mike Mann for predictions and models. As noted in the recent Oeschger Medal award, his own research has been primarily focused on paleoclimate.

    For modeling, one talks NASA GISS, GFDL, NCAR, etc, etc or to Ben Santer @ LLNL, recently inducted into National Academy of Sciences for his many years of work on model comparisons.

    Of course, one can go read IPCC AR4 WG I, of which the relevant 70-page chapter is Climate Models and Their Evaluation. Of course, that is a few years old and progress has been made since.

    • Long ago I read “Climate Models and Their Evaluation”. It is one reason I believe what I do.

      As for your list of reasons that models do not predict correctly — volcanos, El Nino, China — sure, those are three possible explanations for wrong predictions. Another is that the model is wrong some other way. If it turns out to be so hard to test model predictions, maybe we don’t know if the models are right or wrong? In which case we shouldn’t assume they are right. Which is what I’m saying.

  13. Given the seemingly hard limits in the rate at which global temperatures can change (absent a catastrophe), any real life data over a period of 20 years could be overlaid on that predictions graph and be claimed as “reasonable agreement”. Absolutely any. So, it’s a phony argument. It’s even more phony when omitted are the 100% of comparisons that clearly do show the completely non-surprising deviation between predictions and data.

    • More seriously, this appears to be more a case of trusted reference bias then selection bias, aka the Fox effect. People don’t know everything and they rely on sources they trust. If those sources are engaged in ax grinding, well, you get the Fox effect.

    • Rubbish.

      If temperature had gone down by as much as it actually went up (which most naive forecasts would have predicted was equally likely), there wouldn’t be ‘reasonable agreement’ at all.

  14. Seth,

    The pattern of warming matches heat-trapping (warming faster at night, during winter, and toward the poles) and we can all easily see that heat is increasing in the climate system (ocean, surface, air). At its simplest level, either more heat is entering the planet or less heat is leaving. Clearly there is not more heat entering the system from the sun in the past few decades, and especially in the 2000s. The fit to GHGs as the culprit is the only explanation and it matches well-understood physics and historical evidence. Even without models these things would still be true.

    Playing devil’s advocate: Try to hypothesize a forcing that works just like CO2 while also masking the effects of massive increases in CO2 to describe the reason the planet is gaining heat.

    • My point is simple. The AGW advocates say they can see 30 years into the future. Where is the evidence of that? Discussion of heat-trapping does not provide such evidence.

      • Let’s break down your claim here. “The AGW advocates say they can see 30 years into the future”. Really? Have these ‘advocates’ claimed that they know exactly how China’s economy will behave? Whether India will reach European levels of development, what the iphone10 will bring? the technological pace of bio-fuel or solar PV development? I think not. Since all of those things (more or less) will impact the trajectory of emissions over the next 30 years, it should be clear that ‘AGW advocates’ are not claiming any clairvoyant abilities at all. As an aside, none of the people you are referring to (including me) are ‘advocates’ of AGW – it’s pretty much precisely the opposite – they don’t want to see a continuing signal of anthropogenic effects!

        So what do we claim? It is something much more reasonable – that given our understanding of how the climate system works, how it has responded to climate forcings in the past (including volcanoes, solar effects, orbital forcing, lake outbursts, natural variations in greenhouse gases) and how it appears to be responding now, then – all other things being equal – the trajectory of greenhouse gas emissions (most particularly CO2) is likely to lead to climate changes that are signficant enough to cause large scale disruption. Furthermore, if we are unlucky in that we are underestimating emissions growth, or climate sensitivity, the disruption could be very large indeed – particularly for coastal infrastructure and areas that are already water stressed (to pick the two impacts I worry about most).

        You will note that the basis of my concern is not predicated on some impossible level of accuracy in climate models.

        You rightly note above that short term trends in temperatures are not a particularly useful test of climate models – because of the relative signal to noise ratio when dealing with the dynamic nature of weather on these timescales (including ENSO etc.). The graphs I have made on RC (mentioned above) demonstrate that the models own assessment of the unpredictability of the weather means that signals (given the current trends in the forcings) will only emerge over periods of time long enough to average over the noise (estimates are around 17 to 20 years, depending a little on the observational target).

        Furthermore, climate models have been skillfully predicting future trends for decades – stratospheric cooling was first predicted in the 1960s before there were any stratospheric temperature measurements at all, water vapour increases were predicted a century ago, the response to volcanoes (radiatively, in water vapour, dynamically) was predicted in 1991 before any of the impacts of Pinatubo had been felt. Models run in the 1980s predicted that the last glacial maximum sea surface temperature assessment at that time was inconsistent with other information – that assessment was subsequently redone and ended up confirming the model result. In the 1990s, models suggested that the MSU temperature cooling trend derived by Spencer and Chrsity could not possibly be consistent with the surface observations – and indeed they found multiple errors in their analysis which confirmed the models suggestion.

        But more than this, the models discussed in AR4 (which were run in 2004) all predicted that *if* their explanation for the warming trends was correct (i.e. it was driven mainly by increasing GHGs) then we should we able to detect the change in ocean heat content implied by a significant radiative imblanace. When those models were run, the state of the art on this observation was not very high, and the trends were obscure. Subsequent developments (in particular Domingues et al, 2008; Levitus et al 2012; Lyman et al, 2010) all show much more coherent trends, and moreover trends that the models predicted beforehand. For instance:

        There is no better explanation for this than the models have basically got the story right, and for the right reasons. Which makes their projections out to 2040 worth paying attention to in a risk-management context.

        Are models perfect? no. Are their remaining uncertainties? of course. Is everything important about climate in 30 years time known and agreed about? Of course not. But dismissing sources of information based on strawman arguments and ideological bias is not wise. Though it is, of course, your prerogative.

        There is a lot more to say about models, modelling and modellers and the tests that they undergo, and if you want to have a serious discussion about it, feel free to contact me. Or not.

        • Okay, the models have predicted some things correctly. What they are claimed to predict correctly is global temperatures 30 years from now. I have seen no evidence that they can do that. Have you?

          As I have said, I have looked far more widely than the sources Andrew cites. It is entirely possible that the correct predictions you cite reveal the models are partly correct but unfortunately to make 30-year predictions additional assumptions are needed that the models get badly wrong.

      • If discussion of heat trapping (i.e. radiative properties of GHGs and so forth) has no bearing on the validity of climate predictions, by analogy would discussion of the laws of gravity and motion have no bearing on your evaluation of a prediction of the trajectory of a tennis ball?

  15. Hansen, 1988, and not the version where Pat Michaels erased Hansen’s most probable prediction. Also, there was more to that paper than just the prediction of the change in the global surface temperature.

  16. Some of this discussion is getting pretty far off-topic.

    Seth said “I [Seth] have only seen careful prediction-vs-reality comparisons made by AGW [anthropogenic global warming] skeptics. Those who believe humans are dangerously warming the planet appear to be silent on this subject.”

    As can be seen in this comment thread, there are indeed prediction-vs-reality comparisons that are as “careful” as the ones from the skeptics; they are not hard to find. There are two possibilities:
    1. Seth’s statement that he had not seen such comparisons is true.
    2. Seth’s statement that he had not seen such comparisons is false.

    Andrew took Seth at his word and went with option 1. This makes sense because all of the sources Seth cites for his AGW information are skeptics. Andrew assumed that Seth had not seen these comparisons because he was getting all of his information from a biased set of sources that wouldn’t include such things.

    Seth says above that in fact he is very familiar with the non-skeptic side too. This creates a puzzle. I do not think Seth is lying but I do not know what to think.

    In my opinion, too much of this comment thread is focusing on whether global warming is occurring, and what is the evidence, and how well do the measurements agree with the predictions, and so on. All of those are immensely interesting questions but they really have nothing to do with the issues Andrew raised in his post, which relate to why someone of good will would make a statement like “Those who believe humans are dangerously warming the planet appear to be silent on [the subject of model-measurement comparisons.”

    • Nicely framed.

      What I find odd is that Seth tacitly endorses the linked Von Mises article, which in fact does a bad job of comparing models to data. Specifically, Fig. 3 normalizes trajectories to the first data point, which projects noise in 1988 into the future trajectories – obviously bad practice for noisy time series. Fig. 5 makes the same mistake, plus superimposing a noiseless long term model trend on noisy data and censoring decades of pre-Argo data. Neither gives any indication of measurement errors or model ensemble ranges. The last figure is from Lindzen & Choi (2009), which was shown to have serious flaws.

      Since there are examples of model-data comparisons cited in this thread that are more careful than the VM article, including Fig. 3 in the RealClimate evaluation that Seth appeared to reject, one would have to conclude that either (a) the VM article is not an example of the careful comparisons that Seth has seen, or (b) Seth judges model-data comparison quality by criteria that are at best mysterious to some of us.

      I think (b) reconciles the puzzle.

  17. “As can be seen in this comment thread, there are indeed prediction-vs-reality comparisons that are as “careful” as the ones from the skeptics; they are not hard to find.” …says Phil


    There is NO objective definition of : “careful prediction-vs-reality comparisons”

    Hence the basic communication problem here, and the wandering thread & subject.

    • Maybe not, but to the extent that we can agree that there is some underlying reality, usefully describable by mathematical models, there are certain practices that are objectively wrong. I would assert that several of them are present in the Von Mises article.

  18. Fascinating discussion. I am impressed with the skill of the model projections in the linked material. Even if it is technically “off topic”.

Comments are closed.