Steve Heston reminded me of the claim from business school professors Devin Pope and Jonah Berger, published several years ago, that basketball teams do better when they’re behind by one point at halftime. I discussed this a couple times (see here and here).
Here’s the graph from the original paper:
You can see the usual problem with regression discontinuity analysis here: the big jump at the discontinuity can be seen as an artifact of the noisily-estimated and implausible baseline curve.
The article was published in Management Science in 2011, and, according to Google, it has 233 citations—that’s a lot! From the abstract:
Analysis of more than 18,000 professional basketball games illustrates that being slightly behind at halftime leads to a discontinuous increase in winning percentage. Teams behind by a point at halftime, for example, actually win more often than teams ahead by one, or approximately six percentage points more often than expected. This psychological effect is roughly half the size of the proverbial home-team advantage. Analysis of more than 45,000 collegiate basketball games finds consistent, though smaller, results. . . .
Here are their results:
These look much better than the original graph above, and full credit to Pope and Berger for improving their analysis before publication. The coefficient estimates are 0.058 (with standard error 0.015) for the pros and 0.025 (with standard error 0.010) for the college games. They did their analysis using all NBA games between the 1993/1994 season and March 1, 2009, and all NCAA games between the 1999/2000 season and March 22, 2009, restricting to games that were within 10 points of being tied at halftime.
The statement in the abstract of the paper, “Teams behind by a point at halftime, for example, actually win more often than teams ahead by one . . .” is misleading. If you look at the graph, you can see the probabilities are essentially equal, so they’re leading with a chance pattern in the data. Also, they don’t mention it in the abstract, but in the NCAA games, teams behind by a point at halftime actually win less often than teams ahead by one. The effect being smaller in college than the pros seems surprising, as I’d think that less experienced players would be more subject to psychological factors. But, who knows, and, in any case, the difference between those two estimates is explainable by noise.
They found an effect. Do I “believe it”? I don’t know.
On one hand, yeah, you see something there in the data, and the second and third fitted curves above look reasonable—nothing like those regression discontinuity disasters that arise from time to time. And the idea that being behind at halftime could be a benefit—that’s not ridiculous to me. Players and coaches do have to decide how hard to play during the second half, and it doesn’t seem implausible to think that halftime strategy decisions could be slightly discontinuous with respect to being in the lead or being bheind.
On the other hand, there’s the usual story of forking paths, also in this case a potential bias arising from the functional form of the fitted model. Also there could be a bias because they fit a logistic curve but the true underlying curve won’t be logistic. I’m not completely sure, but it’s worth raising as a concern that this model isn’t quite fitting the effect of being in the lead at halftime vs. being behind at halftime; it’s fitting this with respect to a particular parametric curve, and I’m thinking that effects are small enough that misspecification of the curve could induce systematic error in the estimate. I guess this particular question could be addressed by fake-data simulation.
This is a great example for us to consider: not obviously wrong, not obviously right either.
Bringing the Bayes, and thinking about a potential replication study
I wonder if anyone’s followed up? We now have another 13 years of data.
What if I had to guess (or “bet”) what would happen if these new data were analyzed in the same way? What do I think we’d see?
We can consider a series of analyses:
1. Start by taking the 2011 result at face value and being Bayesian. For the NBA, the data estimate is 0.058 with standard error 0.015. What’s our prior? I’d center it at zero—really no reason to think ahead of time that there’d be a jump in probability at zero. What about the prior sd? I’m not sure, but we can start with their statement that 0.06 is approximately half the size of the home-count advantage. If the home-court advantage is 0.12, then I’d think any halftime effect would be much less—let’s say a prior sd of 0.01, so it’s highly unlikely to see an effect of more than 2 percentage points in either direction. My posterior estimate of effect size is then (1/0.015^2)*0.058/((1/0.015^2) + (1/0.01^2)) = 0.018, with standard deviation 1/sqrt((1/0.015^2) + (1/0.01^2)) = 0.08.
2. But the above analysis doesn’t seem quite right, given the strong disagreement between prior and likelihood. I guess the point is that we shouldn’t quite believe the prior or the likelihood here: we shouldn’t believe the prior because maybe I’m suffering from a failure of imagination and the effect could actually be larger, and we shouldn’t believe the likelihood because it doesn’t account for model errors or selection in which model was used to summarize the data. We could, say, double the sd of the prior and the likelihood, in which case we’d still get a posterior estimate of 0.018, but now with a standard deviation of 0.016.
3. All that’s for the NBA. We should also do the NCAA. Indeed, had the NCAA result been larger, I assume the published article would’ve focused on that part of the analysis. For the college games the estimate is 0.025 with standard error 0.01; combining that with the normal(0, 0.01) prior gives a posterior mean of 0.0125 with standard deviation 0.007; again, following step 2 above let’s double the likelihood and prior uncertainties, so now we have a posterior mean of 0.015 with standard deviation 0.014.
4. If I take those analyses seriously, I’d have to say I’m something like 85% sure that the true effect is positive, where “true effect” is defined as some increase in the average probability of winning, if you’re behind at halftime, compared to what would be expected under a smooth model. Do I really think 85%? Maybe. If you asked me what would I expect is the true effect going forward, if it were possible to get data on zillions of games and estimate this very precisely, I guess I’d give less than an 85% probability of a positive effect. Maybe a 60% probability? And my best estimate of the effect size would be less than 0.01. The point is that the effect size itself varies: past data give some insight into the future, but this particular effect seems so fragile (not in the statistical estimation sense, but fragile in the sense that any effect is some unstable combination of strategy and psychology that could well change in different leagues or different eras of the game) that I wouldn’t want to think of this as some near-constant effect going forward. As they say in psychology, it’s domain-specific, and the “domain” isn’t just sports, or basketball, or even the NBA and NCAA, but rather these leagues at these particular times.
5. What about actual new data? That’s tough, because even 13 years of new data is not a lot; any estimates will be noisy compared to actual effect sizes. Still, I’d be interested in seeing what comes up.
6. Suppose someone did a preregistered analysis on the new data, and further suppose, just for simplicity, that the new sample size is the same as in the Devin and Pope (2011) article. In that case, what’s my probability that the new results are “successful” (as defined using the conventional way, as an estimate that is positive and more than two standard errors from zero)? Even setting aside potential changes in the effect over time, the probability of a “successful replication” is surprisingly low! For the NBA study, the standard error of the data estimate was 0.015, so in a replication we’d need an estimate of at least 0.03. My posterior (see item 2 above) probability of this happening is 1 – pnorm(0.03, 0.018, 0.016) = 23%. For the NCAA, we’d need an estimate of at least 0.02, which has a posterior (see item 3 above) probability of 1 – pnorm(0.02, 0.015, 0.014) = 36%. Considering these two studies as independent events, the probability of both these happening is 8% and the probability of at least one of them happening is 51%. I think that if a new study were performed and just one of the two comparisons reached the statistical significance threshold, it would be considered a success. So there you have it.
The big picture: Basketball
As noted above, I think that any effect of being ahead or behind at halftime will be context-specific, and I’d guess the best way of studying this would be to look at something like how many minutes are played by bench players in the third quarter. Just looking at won-lost isn’t so great because there’s not a lot of information in binary data.
At this point, you might say: But isn’t everything context-specific? The home-court advantage: that’s context-specific too! But we have no problem talking about that, without needing a million qualifiers. The difference is that the home-court advantage is large and persistent, two things we can’t really say with any confidence regarding the behind-at-halftime effect. Yes, the larger of the two estimates reported in the published paper was a 6 percentage point increase in win probability, and that ain’t nothing—but, as discussed above, we have lots of reasons to think the true effect was much smaller.
The big picture: Bayesian analysis of empirical studies
It was interesting to go through all the steps. The experience was similar to when we tried to think hard about probabilities in election forecasts:
Do we really believe the Democrats have an 88% chance of winning the presidential election?
Or my favorite simple example in Section 3 of our article, Holes in Bayesian Statistics.
It can be challenging to think Bayesianly with the goal of coming up with a believable and coherent set of inferences, but I think the effort is worth it.
P.S. In comments, Mike points to a new article, Does Losing Lead to Winning? An Empirical Analysis for Four Sports, by Bouke Klein Teeselink, Martijn J. van den Assem, and Dennie van Dolder, that appears to have performed the replication that I was looking for! They find:
When we revisit the phenomenon for basketball, we only find supportive evidence for NBA matches from the period analyzed in Berger and Pope. There is no significant effect for NBA matches from outside this sample period, for NCAA matches, or for matches from the Women’s NBA.
This is as expected given our analysis above. I’m glad I’d not head of this new paper before writing the above post, as I think it was instructive to think through my priors before seeing any new data.
P.P.S. Just to clarify one issue: Whether or not there’s ultimately a persistent and measurable behind-at-halftime effect, it’s still better to be up by 1 than down by 1. Any reasonable estimate of the being-behind-at-halftime effect will be smaller than the natural difference in probability corresponding to moving up by 2 in the point differential. The discussion of the behind-at-halftime effect is entirely about whether the curve of Pr (win | halftime point differential) is flatter right around zero than in the negative or positive zones.
Curious if you’ve seen this already?
https://www.dennievandolder.com/publication/does-losing-lead-to-winning/
Mike:
No, but this seems very relevant to the topic!
Just seeing this paper for the first time. I have to ask: is there any reasonable reason (!) that the data for this paper should not be made available? I’m sure the data is “owned” by someone, but for journals to publish things like this without requiring disclosure of the data just seems more and more irresponsible to me. Also, it feeds the continuing mis-attribution of credit: the emphasis is on the analysis rather than the data curation, and the analysis may be the weakest and least valuable part of their effort. I would be happy to give them credit for assembling the data and making it amenable to analysis (and there are plenty of forking paths involved with that part). But I am reluctant to give them much credit for the analysis, which I find unconvincing. Instead, they have a “major” publication, based on the analysis with the data treated as if it didn’t involve much meaningful work.
Dale:
All I can say is that standards have changed. In Red State Blue State we have an appendix stating where all our data came from, but we didn’t even think of posting our data and code. It just never came up. Yes, it would’ve taken some work to put all our data and code together in one place, but in retrospect I think it would’ve been worth it. It might even be worth it now, but I just don’t have the energy to go through and find everything.
Anyway, the point is that the norm used to be to not share data or code. I’m glad the norm has changed.
Isn’t this data easily retrievable from public sources?
Last time I looked for NBA data it was a scraping ordeal. I think you can buy it from places, but it’s never clear if that’s legal or not (are you just buying data that someone else scraped? Can you further share this data or do you just need to point people to the commercial supplier?).
Regarding legality, I think I saw someone say “you can’t copyright facts,” but I don’t know legal stuff and that sounds too general to be true.
Anyway, if this is not true and sports data is simple to come by, let me know what sport it is and how to get it. I’d like to have some in my back pocket.
Andrew – Yes, I guess I’ll admit that things are improving. According to Management Science (https://www.informs.org/Blogs/ManSci-Blogs/From-the-Editor/Updates-and-Insights-on-Data-and-Code-Disclosure-Policy) they do have a policy on data/code availability, and according to their editorial statement around half of the papers provide all of the data and code required for replication. I’ll take their word for it since I don’t have access to that journal. Indeed that is an improvement. They still say 39% use proprietary data (which I view as similar to redacted materials in most regulatory proceedings and almost any national defense related materials).
le – no, I don’t believe this data (or almost any data) is “easily” retrievable from public sources. In fact, there is nothing “easy” about it, even if it is readily available. The truth is that there are man forking paths involved with getting publicly available data ready for analysis. I have requested data from authors in the past where they responded “it is readily available from public courses,” only to find that I could not match there results – mostly because there were too many choices that needed to be made about what to include/exclude, any recoding of data that might have been made, how to treat missing data or not-applicable entries, etc. etc.
In that case, is it desirable for publications to include their retrieved and pre final analysis data? I’m genuinely asking as someone that is not in any of these fields.
In this particular case what is needed for the analysis? Half time scores and final scores. I don’t have access to the paper but I would hope that the author’s described what to do in the case of overtime games and any other exclusion criteria.
All that said, I see the benefit of having their data published with the paper. I’m just not sure quite what that means. Is it right before calculating percentages and half time differentials? Is it all the games they downloaded before excluding some of them for REASONS? What is the expectation now a days? Maybe to publish the raw retrieved data and the full analysis code (including our analysis steps)?
Oh. Your response to Andrew actually answers my question if what you would like to see. Should have read that first.
> The coefficient estimates are 0.058
Maybe it’s in the paper, but the value here is the difference between the two endpoints of the lines extrapolated to zero (tied games)? If so, I don’t know how, even if we take this at face value, how to take advantage of this information. Should teams angle to be down, but tie the game as close as possible to halftime (so like approach the limit from the left so they’re on the left function)? It seems like there’s a resolution problem in the x-axis here.
It’s weird to think about a discontinuity here because there would certainly have been lots of games tied at halftime — there should be a data point there (like in the original plot) and then this extrapolation thing is just kindof a strange thing to look at.
Then it’s back to just looking at the data points I suppose. It is pretty surprising to me that in the raw NBA data down by one doesn’t seem like a noticeable disadvantage compared to up by one. The NCAA data is much closer to what I’d expect.
I agree. I still can’t get past the choice of a discontinuity here. why? is there any plausible mechanism for expecting an actual discontinuity?
far more interesting for me is that college games have lower come-from-behind wins than NBA games. i can propose two plausible hypothesis
1. NBA games are longer, have 4 quarters. so more time for the team that is down to make a run and get back in the game]
2. NBA shoots more 3s (although college game is fast catching up in % of 3s shot). 3 point shots make coming back from deficits more likely. This hypothesis is testable (college ball has been shooting more and more 3s with better averages over time, are we seeing more teams back?)
“is there any plausible mechanism for expecting an actual discontinuity?”
This suggests an interesting test for any discontinuity analysis: move the “discontinuity” around randomly. How does the outcome change? In this case moving the discontinuity by one-point increments looks like it wouldn’t change anything, that whichever way the advantage goes, the scale of the “advantage” would be about the same.
This is really interesting – I tried to (casually) figure out something related a while back about close games and how it relates to team quality. Basically, my question was, if you know a game is tied at halftime, would you rather have played poorly in the first half or played well (ie, which effect is stronger in the second half of a game, mean reversion or the signal you get from how a team is playing in the first half).
It’s not an easy thing to measure, but – not that surprisingly – mean reversion wins out.
https://benn.substack.com/p/how-to-feel-about-a-tie
I’m not sure if that explains any of the results here, but it at least highlights that close games often come from teams playing abnormally well or poorly in the first half, which adds yet another context specific variable to the mix.
Benn: your comments are interesting but how would you see a game like the recent Chiefs/Bengals playoff game in terms of “regression to the mean”. KC took a 21-10 lead to half time but was outscored 14-0 in first 29 minutes and fifty-seven seconds of the second half, barely tying the game on a last second field goal. One analysis I saw showed that Cincinatti dramatically changed defensive strategy in the second half after getting picked apart by Mahomes in the first half, switching to a 3-man rush with eight in coverage, and locking down Mahomes.
It begs the question: did Cincinatti get outplayed or outstrategized in the first half? Was the second half for KC a “regression to the mean” or just an excellent strategy by Cincinatti defensive coordinator? Taking this thought back to the tie game at half time, if the score was tied at half time, would Cincinatti have made any adjustments?
My first rection was that a one-point lead in hoops is effectively a tie, so it’s hard to see how any emotional or psychological advantage would emerge from that. But it does mean that the teams were evenly matched in the first half. If one team has a notably worse record than the other, then the tie could suggest an effective strategy by the “lesser” team that could be countered by the “better” team in the second half. Since major shifts often occur at half time, apparently it’s hard to counter an effective strategy on-the-run, mid half, so couldn’t this all be on the coaching staff too?
just thinking about this stuff makes me think there couldn’t possibly be any durable advantage to a one-point deficit at half time, except through a total fluke.
I have all play-by-play data since the 1997-1998 season scraped and parsed for my site (darko.app). I tested it for that time period, and the result does not appear to hold up in recent years. Breaking it down by scenario:
Home team leads by 1 and has the ball to start the 2nd half:
539 games, 310 home wins (57.5%)
Home team trails by 1 and has the ball to start the 2nd half:
541 games, 282 home wins (52.1%)
Road team leads by 1 and has the ball to start the 2nd half:
549 games, 212 road wins (38.6%)
Road team trails by 1 and has the ball to start the 2nd half:
551 games, 222 road wins (40.3%)
I’ve provided these breakdowns since it’s not totally obvious to me how to think of the various home court advantage biases at play here (there is a home court advantage to winning the opening tipoff – https://twitter.com/kmedved/status/1478129488319889413), and perhaps there’s something to be teased out of the breakdowns. Aggregating all scenarios, I get a total of 1110 wins when leading by 1, and 1070 wins when trailing by 1 (so it’s better to be leading than trailing).
As a sanity check, I rechecked using data from 1997-98 through 2008-2009 only (mirroring the study period as best I could), and I got 535 wins when leading by 1, and 558 wins when trailing by 1 (so better to trailing than leading). So I’m getting the same core result as the authors of the study – it just doesn’t hold up in a larger sample.
> my site (darko.app)
Very cool site! Also I see the lead to https://github.com/dblackrun/pbpstats which I will keep in my back pocket.
> Aggregating all scenarios, I get a total of 1110 wins when leading by 1, and 1070 wins when trailing by 1
I think in my head I was kindof Zoolander-ing the number of NBA games there would be if all data was available. Something to the effect of, “well if we look at all games from all time that has to be like infinity or something!”
But 2000 binary outcomes over 20-30 years (whatever this window is) isn’t a whole lot of information considering there are like 300 players active in the NBA each year lol.
May also be relevant. Challenge to hot hands and small numbers.
“Aren’t we smart, fellow behavioural scientists
12 June 2020
“Below is the text of my presentation at Nudgsestock on 12 June 2020.
…: And even of those that replicate in the lab, many become an ineffective, or even dangerous tool when we try to apply them in the complex outside world.
“Let me tell you a story to illustrate.
The hot hand fallacy
“This story comes from great work by Joshua Miller and Adam Sanjurjo. It stands as one of the starkest examples of where I have been forced to change my beliefs.
…
“So, this isn’t just about the misperception of the hot hand, but also about the failure of people to see their error when presented with evidence about it.
…
Let’s delve into how Gilovich, Vallone and Tversky showed the absence of a hot hand.
…
“We need to stop making glib assumptions about what other people want and how they can best achieve their objectives.”..
https://www.jasoncollins.blog/arent-we-smart-fellow-behavioural-scientists/
Yes, see here, here, and here.
Is there sort of a regression to the mean going on here? Like if the naive result is a tie, then you’d expect the down team to do better. Of course, maybe this is the “just came missed bet on red, so bet it again” fallacy of Vegas (where future results should not be affected by past).
I guess there could be things like having used trick plays or pinch hitters up that would like earlier and later results. Maybe “using up top pitching” (actually probably that really is an effect, the using up pitching).