How effective are football coaches?

Dave Berri writes:

A recent study published in the Social Science Quarterly suggests that these moves may not lead to the happiness the fans envision (HT: the Sports Economist). E. Scott Adler, Michael J. Berry, and David Doherty looked at coaching changes from 1997 to 2010. What they found should give pause to people who demanded a coaching change (or still hope for one). Here is how these authors summarize their findings:

. . . we use matching techniques to compare the performance of football programs that replaced their head coach to those where the coach was retained. The analysis has two major innovations over existing literature. First, we consider how entry conditions moderate the effects of coaching replacements. Second, we examine team performance for several years following the replacement to assess its effects.

We find that for particularly poorly performing teams, coach replacements have little effect on team performance as measured against comparable teams that did not replace their coach. However, for teams with middling records—that is, teams where entry conditions for a new coach appear to be more favorable—replacing the head coach appears to result in worse performance over subsequent years than comparable teams who retained their coach.

Berri points to other studies from hockey, soccer, and basketball that come to similar conclusions: replacing a head coach does not, on average, lead to improvement. He concludes:

What these studies did is look at teams or players with different coaches and failed to find much of a difference. That suggests that coaches in sports are not very different from each other. It may be true (and more than likely very true) that you are better off with a professional coach than with a random person grabbed from the stands (or no one at all). But it doesn’t appear that the choice of professional coach matters much.

Berri continues:

And that means, if it costs a small fortune to fire your coach – and often it does – then a team is probably better off just keeping who they have on the sideline. Yes, this may not make the fans of the losers very happy today. But it doesn’t make sense for universities to make decisions that cost the school money and don’t systematically change the outcomes we see on the field.

I have a few thoughts on this. Let me start by saying that it’s good that Berri is pointing to research rather than just speculating (as I’m about to do). That said, I wonder if he is going too far in his interpretation of those findings.

I have not read the linked studies but, from Berri’s description, they found that a replacement coach is no better, on average, than the coach that came before. But how do you get from there to “it doesn’t appear that the choice of professional coach matters much”? Nearly all coaches change jobs at some point, good coaches as well as bad coaches. Several blog commenters make relevant points along these lines, but unfortunately I didn’t see a response from Berri. (Our blog here is unusual in that I apparently have nothing better to do than to read (and often respond to) your comments!)

The other thing that struck me, given that Berri’s post appeared in the Freakonomics blog, was how his story fits paradigm 2 of microeconomics.

Recall that economists are often making one of two arguments:

1. People are rational and respond to incentives. Behavior that looks irrational is actually completely rational once you think like an economist.

2. People are irrational and they need economists, with their open minds, to show them how to be rational and efficient.

Argument 1 is associated with “why do they do that?” sorts of puzzles. Why do they charge so much for candy at the movie theater, why are airline ticket prices such a mess, why are people drug addicts, etc. The usual answer is that there’s some rational reason for what seems like silly or self-destructive behavior.

Argument 2 is associated with “we can do better” claims such as why we should fire 80% of public-schools teachers or Moneyball-style stories about how some clever entrepreneur has made a zillion dollars by exploiting some inefficiency in the market.

Recently I remarked on a Freakonomics article that had a delightful mix of the two stories: Stephen Dubner and Steven Levitt wrote that “there is one betting strategy that will routinely beat a bookie, and you don’t even have to be smart to use it” (argument 2: with careful data analysis, you can cleverly beat the odds) but then went on to mock some (unnamed) “academics” for using this fact to underestimate the intelligence of street-smart bookies (that’s argument 1).

Berri’s example seems more like a pure case of argument 2: sports teams are spending millions of dollars on top coaches, and that’s a waste of money because professional coaches are essentially all the same. With some effort you can make an “argument 1” story here—the idea being that teams fire coaches because it makes the fans happy—but I don’t buy it. My impression is that fans want wins, they want championships, and to a lesser extent they want star players and exciting action. There’s a lot better ways of spending millions of dollars to get all these things, than to blow it on coaches with no ability.

I should also point out that, in the best form of a social-science story, arguments 1 and 2 go together. Argument 2 identifies the $20 bill on the sidewalk, and argument 1 explains why nobody was picking it up. In practice, though, I feel like we typically see one side of the story without the other. When argument 1 is uncorked, there is a tendency to treat the identified behavior as rational and normative, while with argument 2 there is typically an assumption that the newly discovered strategy is simply better.

P.S. To return to sports, I also wanted to bring up one of my pet ideas on why coaches get fired when they do:

Dan [Goldstein] has another question that I think I have the answer to. He writes that he “has always wondered why teams are so eager to fire their coaches after they lose a few big games. Don’t they realize that their desired state of having won those same few big games would have been mostly due to luck?”

My guess is that these are situations where the management has already decided they want to fire the coach, and they’re just waiting for a convenient time to do it so as not to antagonize the fans.

P.P.S. Also this on coaching.

28 thoughts on “How effective are football coaches?

  1. I would guess that the cannier team owners know that a bad record is often just the luck of the draw, but they want to show their (less statistically-minded) fan base that they’re trying to do something about the losing season. And a new coach will keep the fans interested and involved, and get more newspaper column inches than not doing anything, or than just supporting the current coach.

    • So you think the coach role is more scapegoat than actual coaching?

      It of course depends from coach to coach, but it must be a hard job where luck has such a huge role as well in your career.

  2. The question is not really how effective *all* football coaches are. The question is how effective are those few *elite* coaches that can actually make a difference? Those are the ones (like QB’s) that teams are continually struggling to find. The 49ers franchise swung around in one season based on a coaching change. You might miss a move like that in a study looking at dozens of coaches, but every subject matter expert will tell you the move from Singletary to Harbaugh made a huge, huge difference.

  3. My take is, in part, that better coaches are identified by success and they move into opportunities that fit best. A coach will take a job where opportunity is exists. That can mean a good program with issues – like Michigan football after Rich Rodriguez – or it may mean something else, like a new stadium or other commitment of resources that make a program more competitive. That is a way of saying the quality we estimate as “coaching” depends a great deal on the nature of the program support and the opportunity that presents. Maybe Nick Saban is a better coach but he’s also found the niche where he has extraordinary program advantages.

    We see this in pro sports as well. For example, the Boston Bruins were a no-win situation for a few generations because ownership didn’t put itself behind winning. They won a Cup recently because they hired a GM who hired a coach and who brought in players that fit a specific system the GM and the coach and the players did well.

    I can’t imagine that hiring a new coach would matter much when there is no temporary loss of fortune. When there is a temporary loss of fortune, the program can change and the coaching change is a signal of that. The coach gets the credit.

  4. Isn’t it possible that the very good coaches don’t tend to be available to be hired as often because they’re staying put in the successful situations they created?

  5. Given that sports are, by design, a zero-sum game where any ‘improvement’ caused by one coach must be gained by causing a disimprovement in other teams’ scores or win record, and any innovations can be copied easily, why would we necessarily expect to see any consistent effects at all?

  6. The coach is stepping into a system that has long lag times for things like: current players, incoming players, player recruiting, team spirit, player expectations, loyal and excited fans, play systems, facilities, money, etc. In terms of what can be changed quickly, that requires either a stellar coach that can demand autonomy or a desperate team that will give it.

  7. Hiring/firing coaches is not only about finding a high-quality coach (“selection”), it is also about making your coach spend the most time and effort to improving his team (“moral hazard”). Teams may be doing the rational thing (by firing coaches) from this second perspective.

    Suppose all coaches in the world are of the same quality, that is, they read the game equally well, they are equally good at scouting young talent and so on. So, if the coach works hard, a team will win the same number of games regardless of the coach. If he does not work hard, the team will lose most games. Suppose teams cannot observe the coach’s effort; but they can observe the outcome (win/lose) which is closely related to effort.

    Now, just by chance, even a hard-working coach will every now and then have a string of losses. What should the team do? If the team doesn’t punish the coach after bad results, then the coach doesn’t have an incentive to work hard. By firing the old coach, the team motivates the new coach to work hard as much as he can to delay that inevitable string of losses as long as possible.

    Note that since the old and new coaches are of the same quality and the string of losses were due to chance, *if* the old coach had been retained and *if* he continued to work hard, he would have achieved the same results as the new coach, which is what the matching method will capture. But, if the team did not fire the old coach, he would not work hard, which would delay the team’s recovery.

    Therefore the team should fire its coach after bad results even if the new one is not inherently more talented.

    • Kerokan:

      I completely disagree with your reasoning, and I think Deming and other quality control experts would as well. Chasing noise is generally considered to be a terrible strategy for quality improvement.

      • So what would you do in his model? Never fire a coach no matter how long the string of losses, on the grounds ‘well, he could be being lazy, but you know, it could just be bad luck on our part’? I think obviously you *would* fire the coach at some point – and as the joke about the prostitute goes, now we’ve established that you agree with kerokan and all we’re discussing is the exact price.

        (As stated, kerokan’s model is unrealistic: if replacing coaches were free and new coaches started off hard-working, then the optimal approach would probably be to fire the coach after every game to avoid even the smallest probability of him getting lazy, and replace him with an eager young beaver who is guaranteed to maximize the win-probability of the next game! Add in more realistic assumptions like switching costs which are significant compared to win probabilities, and so on, you’ll start getting more realistic results like ‘coaches are fired only after enough losses to raise the probability of the ‘he’s gotten lazy’ hypothesis over the ‘he’s been unlucky’ hypothesis enough to make the expected value of a replacement coach positive, that sort of thing.)

        • Gwern:

          Of course I would fire coaches sometimes. I was never saying otherwise. In fact, if you look at my post, I was specifically expressing skepticism at the claim that the coach doesn’t make a difference. If you look at the very end of my post, you’ll also see that I argue that it could make sense to fire a coach you already wanted to fire, just waiting for that string of losses to do what you were already planning to do.

          In my comment immediately above, I was disagreeing with commenter Kerokan’s claim that it would be a good idea to fire coaches after random strings of poor performance, even if all coaches are of the same quality, just as a way to motivate coaches to try harder. I don’t find that at all plausible, and there’s lots of literature on quality control that specifically recommends against this sort of chasing-the-noise behavior on the part of management.

        • Hi Andrew. You might be right about the coaches. I have no idea about baseball; I just wanted to point out an alternative way to interpret the data.

          Anyway, what you said about the quality control literature is interesting to me, but I am completely unaware of it. I googled a few things (“quality control chase noise”, “management chase noise” etc, but did not find anything relevant. Could you give one or two references for me to look up? I’d like to read more on this.


        • Kerokan:

          I’ve only actually heard of the quality control literature secondhand. Deming is the most famous quality control statistician, and the WIkipedia entry on Deming is a good starting point, I think. For example, here’s #10 of his 14 principles of management: “Eliminate slogans, exhortations, and targets for the work force asking for zero defects and new levels of productivity. Such exhortations only create adversarial relationships, as the bulk of the causes of low quality and low productivity belong to the system and thus lie beyond the power of the work force.” In general, I associate the quality control movement (starting with Shewhart in the 1920s) with the goal of separating signal from noise, and with a focus on improving the industrial process rather than trying to locally optimize everything.

        • Pretty much the first thing I was taught in the 1980’s Management of Information Systems course was that management almost always over reacts to new sources of information (i.e. reacts to the noise). Maybe loook for course readings at the local MBA school?

          Likely the same thing happens whenever someone is learning a new skill that requires reacting to feedback – initially over reacting…

        • One of the major ways to separate signal from noise is to reduce variability – ie improve consistency of production. This is equally applicable in sports performance and also resonates with the whole Vince Lombardi ‘winning is a habit’ philosophy.

        • Right. Football coaches typically work insanely hard. I recall an article about the assistant coaches at a successful L.A. public high school program who worked closed to 90 hours per week during the season for something like $1,800 a year. The current system of fairly frequent turnover acts as both carrot and stick that succeeds in incentivizing coaches to work hard.

  8. Another aspect of this is that the ‘bad’ coaches were probably mostly not that bad. That is, they must have at least had a somewhat distinguished playing history at an appropriate level in the sport, and most will have had assistant coaching positions, and recommendations from a successful head coach. So it’s not like the bad coaches were random slobs found in sports bars. So a potential way of looking at it is that as long as a coach has reached a threshold of competence, they’ll do about the same as the other coaches who have reached that threshold. Of course, no one who has reached that threshold is available for the minimum wage…

    Another factor limiting the impact of coaches is that nowadays there is a coaching organisation, with specialist coaches, fitness experts, even statisticians. So unless a new coach has brought most of these people in themselves, the degree to which the coaching organisation’s quality may be effected by changing one person may be muted. It’s just that the head coach is usually pretty much the only individual who is visible to outsiders.

  9. One version of “argument 1” here is that the coaches that get fired are worse than the ones that get retained; a simple model suggests that the newly hired coaches should fall somewhere in between. If X is the expected performance that a team will get if it goes through the process of hiring a new coach, then a rational team will fire a coach whose expected performance is below X and retain a coach whose expected performance is above X.

    This does not account for things like the cost of switching coaches, which would make the predictions less clear.

  10. Has anyone tried to apply VAM to coaching? It would work best, I think, for college or high school basketball or baseball, where on-base percentages and FG% maybe could function as your outcome variables and you could estimate an average change from freshman to sophomore players, and then estimate the group-level variance in that change across a whole bunch of coaches.

    Of course, I think this illustrates why VAM in general is problematic.

  11. I think one of the things we must do is to put statistics into context. Just because our models are giving us some conclusions, doesn’t mean we can claim that people are doing the wrong things because they differ from our models. Statistically insignificant and practically significant are two different things.

    The real question is how important is a coach to a team, and how much can a coach do.

    There are many reaons that make teams successful, the coach, the players, the owners and etc. Sometimes even the where the team is makes a different (in NHL, for example, a team would play teams in its division more than other teams, so if you are in a strong division, that’s disadvantageous). How important these factors are vs. the quality of the coach is unclear. Would you rather have a good player (perhaps two or even three) or a good coach? Would you rather have another 10 million dollars in transfer budget than a good coach?

    Look at the Premier League, for instace. For the past 20 years or so, the top five places are more or less occupied by the same teams each year (and Chelsea, for example, fired more than 15 coachs in the past 20 years, but still was able to achieve top five almost every year). From that perspective, coach replacements do make little difference. But such comparisons are not at all valid without considering other factors like the quality of the players. Chelsea is a big name in soccer, and they can always attract top players. This is probably why they can performance decently regardless of the quality of the coach. Just creating a model to compare performances and conclude that coachs have little effect would be a very simplified (and unrealistic) approach. Many other factors need to be considered before we can say something.

Comments are closed.