The Commissar for Traffic presents the latest Five-Year Plan

What do Paul Samuelson and the U.S. Department of Transportation have in common?

Phil Price points us to this news article by Clark Williams-Derry:

As the State Smart Transportation Initiative at the University of Wisconsin points out, the US Department of Transportation has been making the virtually identical vehicle travel forecasts for well over a decade. All of those forecasts project rapid and incessant growth in vehicle travel for as far as the eye can see. Meanwhile, actual traffic volumes have flattened out, and may actually be falling.

VMT-C-P-chart-big1-541x550

Each of the rising colored lines represents a forecast from a different year. The black line represents actual traffic trends on US roads—which never rose as quickly as the forecasters had predicted, and actually started a modest decline in 2007.

I’d like to see a label on the y-axis, and I’d recommend labeling the x-axis at 5-year intervals rather than every year, but the point seems pretty clear.

Williams-Derry continues:

In this entirely USDOT’s fault, though. These forecasts are a “roll-up” of forecasts made by state DOTs. The US agency just collects the forecasts and reports them to the public: garbage in, garbage out.

But in a way, that’s even more sobering than if the fault were localized in USDOT, since it provides clear and compelling evidence that the nation’s entire transportation forecasting apparatus is completely broken. In the aggregate, all of those hard working forecasters in all of those state DOTs are just making up numbers.

Indeed. I’m reminded of this notorious graph from the legendary Paul Samuelson textbook (from 1961):

samuelson.png

As Alex Tabarrok points out, the graph is even worse than it looks: “in subsequent editions Samuelson presented the same analysis again and again except the overtaking time was always pushed further into the future so by 1980 the dates were 2002 to 2012. In subsequent editions, Samuelson provided no acknowledgment of his past failure to predict and little commentary beyond remarks about ‘bad weather’ in the Soviet Union.”

The bit about the bad weather is funny. If you’ve had bad weather in the past, maybe the possibility of future bad weather should be incorporated into the forecast, no?

As Tabarrok and his commenters point out, this mistake can’t simply be attributed to socialist sympathies of the center-left Samuelson: For one thing, various other leftist economists did not think that the Soviets were catching up to us; and, for another, political commentators on the right at the time were all telling us that the communists were about to overwhelm us militarily.

As with the Commissar for Traffic, the problem was the linear model, the assumption that what has changed in the past will continue to change in the same way in the future.

What’s unforgivable, both in the case of Samuelson in the 1960s and the Commissars today, is not in making the mistake (surely, a linear extrapolation is quite reasonable in many settings) but in not recognizing the problem, even after the forecast has been completely destroyed by the data.

52 thoughts on “The Commissar for Traffic presents the latest Five-Year Plan

    • Because the DoT planners showed how their increase budget “reduced traffic” 25% or so. Say you need money because the sky is falling, and claimed you solved the issue.

    • And how does the military justify spending if the communists are not about to overtake us?

      And how do economists justify paying attention to economics if another country is not about to be richer than us?

      As always what looks like irrationality is just self-serving rational bias.

      • Rick,

        If you think that “what looks like irrationality” is always “just self-serving rational bias,” you have to get out more! Believe me, people do stupid things all the time.

  1. Although these examples both present absurd projections from linear models, I think there’s something of a disconnect or logical error in what you say or seem to imply by “the problem was the linear model, the assumption that what has changed in the past will continue to change in the same way in the future.” All prediction models, whether linear or not, seem to make that assumption to some extent, even if they allow for unprecedented change in some aspects of the problem they must assume constancy in others. Justifiable? Probably less so than we think, or so argue Taleb.

    • Mark:

      Just to be clear, I think the key problem is not the bad forecast, it’s the continuing to make the bad forecast even after it’s been destroyed by data, which is what happened in both the examples above. See the very final paragraph in my post.

        • > … it wasn’t linearity that did them in (or should have), it was utter error blindness.

          I’d say it’s even worse than that – willful ignorance of data being far worse than inability to adapt to modeling errors, in my opinion.

      • > they must assume constancy in others

        The early destruction by data points to how to get over this must assume – to some short turn constancy in some (not always the same) others via new forecasts.

  2. Reminds me of the quarterly reports from our CFO when I worked at Speechworks (a speech recognition startup with great programmers and a lousy business model). Each quarter, the company’s expenses were about $2 for every $1 of revenue, but with strong revenue growth (this was right before the dot-com bubble burst). And each quarter, the CFO projected costs to be roughly flat going forward, leading to a kink in projections of profit that always coincided with the day of the presentation. I watched this every quarter for two years until the company sold itself on the brink of bankruptcy. And each quarter, I said, “Who believes these projections? How can he say this with a straight face?”. Turns out that wasn’t the only funny business going on; the SEC settled a fraud allegation relating to inflating revenue; at least they banned him from ever being a CFO again.

    • Bob Carpenter’s saga sounds all too real to me (except the fraud part). The upshot was also interesting where I worked. Naturally, in order to meet client commitments the headcount budget had to be exceeded at some level. So, to meet profit commitments, salaries would be frozen or cut, bonuses would be zeroed out, full time employees would be converted to contractors, and the best “rainmaker” salesmen would move on to greener pa$tures.

      Unrealistic forecasts have real consequences.

  3. It would be neat to see the original article, and the data on which the colored-line linear projections are based. It seems odd not just that the projections are wrong in the face of data, but also that they all have (roughly) the same slope! Is there some particular period, years past, that gave a 0.075 whatevers/year growth rate that’s being mindlessly applied to the present and future?

  4. The mile traveled number was, for decades, tightly correlated with GDP, and GDP growth has been extremely smooth. When the business cycle wobbles so does the miles traveled, but both have tended to fall back to the trend line. So these estimates reflect that conventional wisdom, and the flattening out shows that the GDP growth has turned lousy. It is not irrational for planners to assume that recessions will end and the catchup with trend will happen yet again. That said, if we subtract away the correlation with GDP there does appear to be some deviation form the GDP::Miles-traveled correlation.

    Planners have a hell of a time telling weather form climate change :). And, only recently have people started talking about stagnation, i.e. a real phase change in the economy.

    The russian and us economy were a lot more independent systems, and hence not correlated, as compared to the miles traveled and the GDP.

    So I don’t know, maybe this isnt’ as mindless and example of an institution unable to admit things are changing as it seems.

    • Ben:

      Again, see the last paragraph of my post above, and see the graph which is flat since 2004. At some point these people should be learning, not just repeating last year’s error. Same goes for Samuelson.

        • I’m not sure how that wouldn’t make it a critique of traffic planning. Suppose my method of traffic planning were to ask Jimmy the Shoeshine Boy how much traffic he thought there would be in ten years, and that Jimmy does OK for while but at some point he has a terrible track record and steers me wrong year after year after year, always in the same direction. I think it would be fair to say “you have a lousy method of predicting traffic volumes.” Sure, I could point my finger at Jimmy and say “it’s not me, it’s HIM”, but I think if it’s my name on the prediction then it’s my responsibility to not have those figures be wrong in the same direction for two decades.

        • It’s cute but a poor cartoon to suggest that taking the consensus GDP growth numbers as an input to the traffic planning model is no better than asking Shoeshine Boy.

          Yes each planner owns his predictions, but no it is entirely appropriate for him to say: “Here’s my model, here’s where it went pear shape, and this is the input that caused that.” It’s perfectly fair for Andrew to point out that something seems broken here, all I’m suggesting is (a) the breakage might be in the input not the model. And (b) that you don’t change a model that has worked well for decades impulsively.

          And yes, I agree that 20 years is a long time to be tabling changes to the model; but I’m more sympathetic as to why the motion to table carried in any given time frame over those 20 years.

        • Ben: the problem with using the graph to which you link as support for travel modeling assumptions is that it cuts off in 2005, right as changes in VMT start to seriously diverge from changes in GDP. The real critique of transportation modelers is that they’ve by-in-large been really bad at recognizing this divergence and the various mechanisms behind it.

          For discussion of the divergence the mechanisms, David Levinson is a great source: http://transportationist.org/2013/12/26/structural-errors-in-forecasting/

        • You list of what I enjoy calling “insta-theories(tm)” for the change are fun. None are particularly satisfying are then, which is frustrating. My favorite insta-theory has been regulatory (both governmental and insurance industry) changes that have suppressed various classes of drivers. It’s a lot harder to get a license these days and the cost of insurance for young drivers is onerous. There is some evidence for that here http://www.junctionatl.org/wp-content/uploads/2012/07/2012-06-28-DL400gendertrendchart400x355.jpg but I don’t think it’s sufficient to explain what look like two phase changes in the underlying system.

        • For kicks, I quickly looked up data on GDP and vehicle-miles traveled, plotting the ratio over the past few decades (http://eighteenthelephant.wordpress.com/2014/01/22/you-can-use-any-model-you-want-as-long-as-its-linear-and-has-a-positive-slope/). It seems to show a more pronounced and steady change in its trends that isn’t evident in the “oildrum” graph above (which goes to 2002).
          It’s possible the data aren’t correct — “quickly” being a key word above — but in all it seems like the reality of the past ~decade is pretty obviously different than that of 1960-1990, and it seems absurd not to account for this in a prediction.

        • Those charts are great! Everybody who’s interested should go look at them. The sharp corners in the last two are fascinating. The bend in the miles/person looks to my eye as it’s the tepid economy followed by the recession. The longer term decline in miles/$-GDP? I got nothing to explain that. Well I can make up insta-theories(tm) but I don’t like ’em much. For example it makes me I wonder if there was a change in how GDP or miles traveled was measured in 2000.

  5. I loosely follow traffic forecasts. What I see is much better: they don’t know how to model what’s been happening so they report actual and don’t hide their error. That paints Samuelson as much worse.

    I know that in the years since miles dived, there were expectations about rebounds, etc., and a belief that the curve had bent but maybe that was short term – and thus maybe the dive was short-term, etc. I’m not sure if it would be better to trash the model for “better” or to rely more on actuals.

    Another thing is that these numbers become meaningful mostly at local levels where actual counts are more the decision driver – sorry about the pun. I’ve done a bunch of traffic improvement projects and each relied mostly on current conditions plus expectations with much more emphasis on solving current issues. So states and localities with lower counts are shifting resources around – in part, because they’ve been cut so much they don’t have enough anyway and in part because they always have problems that need money. And some states and localities see growth and can’t keep up. I’m less familiar with those areas, but I assume they continue, to the extent they have money, to plan for future capacity because one time building is much less expensive than retrofitting.

  6. Pingback: Traffic Forecasts

  7. To paraphrase one former IMF official: “The problem is not the forecast. The problem is road traffic has yet to catch up with our forecast.”

    Maybe USDOT should outsource their traffic forecasts to investors with money riding on the forecast, or simply create a Kaggle competition.

    They ought to have Chinese Walls separating forecast generation from forecast use, and even then…

  8. Looks like Indiana nailed the window to do a long term lease of their toll road in 2006! (I-80/I-90). Taxpayers win; caveat emptor.

    Source: I was part of the State of Indiana deal team.

  9. Pingback: Traffic Projections and Extrapolation Failures | therhinorceros

  10. I started my career in the electric utility business in New England. We had forecasts that looked very much like the DOT after the first OPEC embargo, when households and businesses started conserving because of price.

    Utilities kept right on assuming that electric consumers would return the old trend line, after people got tired of turning off the lights after the left a room.

    We used to call those DOT like graphs, fan graphs.

  11. Pingback: You can use any model you want, as long as it’s linear and has a positive slope « The Eighteenth Elephant

  12. Criticism seems justified in these cases, however at least for some public entities, not revealing market-sensitive information may be an objective in itself. A simple linear trend is a particularly straightforward (sorry) way of achieving that. Maybe then it should not be called a “forecast”, but it could still be useful in 1) prompting thoughts on what underlying factors may be different in the future, and 2) giving perspective on what the historical development was actually like.

  13. The first graph looks a lot like a graph (the NERC fan) presumably in http://www.jstor.org/discover/10.2307/1391589?uid=2&uid=4&sid=21103313095957 showing NERC forecasted electricity consumption in the USA, except that the forecasts rose exponentially, not linearly. I haven’t read that article; I’ve seen the graph on p. 642 of John Sterman’s /Business Dynamics/. Chapter 16 of that text is a good one to read when thinking about forecasting.

  14. fair enough, but you’d have a field day with forecasts from National Association of Realtors, S&P, Gartner Group.

    even the crowdsourced forecast known as the financial market is often spectacularly wrong.

    when anyone makes a forecast, you need to understand, the context, incentives, etc.

    human fallibility is not a left/right commissar/capitalist pig issue.

  15. Pingback: Onward and upward | Stats Chat

  16. Could we infer from this that the DOT’s (and Samuelson) did not have a crystal ball? Without a crystal ball, most projections are nearly worthless, even those done more carefully than the traffic projections. The exceptions are when things have actually been constant for decades.

    • Skeptonomist:

      1. Organizations make forecasts because they need to make decisions. Good forecasts are better than dumb forecasts. If you think most projections are nearly worthless, that’s fine, you can do something else, they’ll hire someone else to do forecasts who is actually interested in the job.

      2. Actually, that “crystal ball” thing is just a story. Crystal balls don’t really tell the future. Don’t believe everything you see in the movies.

      3. See the last paragraph in my above post. To make a mistake is fine. To repeat a mistake over and over after it’s been contradicted by the data, that’s not so fine.

      • Andrew:

        “Organizations make forecasts because they need to make decisions”

        OR

        “Organizations make forecasts because they need funding”

        OR, more generally

        “Organizations make forecasts because they need [X]”

        Depending on X, USDOT’s forecasts my have been very useful indeed.

  17. Pingback: Linkfest: PIMCO shuffle; Bank reform bites swaps market, LBOs; China princelings’ assets | Inside Investing

  18. Pingback: The U.S. government keeps predicting we’ll drive more than we actually do

  19. The Department of Transportation was hardly alone in making this forecasting error. During the 2000s, the health of the economy rested upon the assumption that housing in the exurbs was a great investment because people would commute ever further to work. The collapse of some subprime firms in 2007 raised the first questions about this assumption. The gasoline price rise in the first half of 2008 finally killed that fantasy, leading to the collapse of Lehman Brothers in the late summer of 2008 and the ensuing deep recession.

    • In other words, up through more or less September 14, 2008, the Big Money Boys were betting overwhelmingly on more miles being driven. Isn’t it asking a little much to demand that the lowly bureaucrats at the Department of Transportation have contradicted the Wisdom of the Markets?

  20. Pingback: The value store | Counterparties

  21. In Utah, I’ve worked as a consultant for opponents of building a water pipeline from Lake Powell to St. George.

    I have no personal preference in this case, but it’s relatively easy to support the opposition with arguments based on data.

    State population forecasts are largely linear extrapolations. The local water authority has calculated the maximum population that can be supported by all available water resources in the area (including Lake Powell). The state’s population forecast rises smoothly to that limit, and then plateaus.

    It never seems to occur to anyone in the water authority, the county, or the state, that the plateau is determined by the yes/no answer to whether or not a pipeline will be built. In fact, the position is along the line that the pipeline must be built because the population estimates (based on the assumption that the pipeline will be built) can’t be supported unless the pipeline is built.

  22. Pingback: Updates of bad forecasts: Let’s follow them up and see what happened! « Statistical Modeling, Causal Inference, and Social Science

  23. Pingback: Greek statistician is in trouble for . . . telling the truth! « Statistical Modeling, Causal Inference, and Social Science

Comments are closed.