Quantifying Officiating Impact in the NBA: A Referee Impact Metric Analysis Using ESPN Win-Probability Data

Leo Benaharon; Nirek Duma

arxiv: 2605.17845 · v1 · pith:37NVHBGKnew · submitted 2026-05-18 · 📊 stat.AP

Quantifying Officiating Impact in the NBA: A Referee Impact Metric Analysis Using ESPN Win-Probability Data

Nirek Duma , Leo Benaharon This is my paper

Pith reviewed 2026-05-20 00:52 UTC · model grok-4.3

classification 📊 stat.AP

keywords NBA officiatingreferee impactwin probabilityfoul callssports analyticsobservational metricsgame leverage

0 comments

The pith

The Referee Impact Metric aggregates absolute win-probability shifts from foul calls to measure referee influence separately from simple foul counts.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces the Referee Impact Metric (RIM) to quantify officiating by summing how much each foul call moves a team's win probability in a given game. This addresses the limitation of prior work that treats all fouls equally regardless of game leverage or situation. A reader would care because it shifts focus from raw foul rates to context-sensitive impact, allowing comparisons across referees, teams, and home/away settings. The authors apply the metric to four recent NBA seasons and run linear regressions that control for home status, team identity, opponent, season, and postseason context. They report that RIM remains distinct from foul volume and disparity, with some team and referee-team patterns still visible after those adjustments, though they frame the results as screening signals open to further testing.

Core claim

The central claim is that RIM, defined as the sum of absolute win-probability movements attached to each foul event in a game, is empirically distinct from both foul volume and foul disparity. When the metric is examined across 2021-2022 to 2024-2025 NBA seasons, regular-season and postseason referee distributions show variation, and linear controls for home status, team, opponent, season, and series state leave several team-side and referee-team associations intact. The authors present these associations as observational patterns that warrant additional scrutiny with different win-probability models rather than as evidence of intent or responsibility by individual officials.

What carries the argument

The Referee Impact Metric (RIM), a game-level sum of the absolute changes in ESPN win probability produced by each foul call.

If this is right

RIM distributions can be compared between regular season and postseason for individual referees.
Team-side and home/away heterogeneity in RIM values can be measured after basic contextual controls.
Referee-team interaction patterns remain detectable after conditioning on home status, team, opponent, season, and series state.
The metric supplies an observational screening signal rather than a direct attribution of misconduct to any official.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same win-probability aggregation method could be tested in other sports that publish play-by-play outcome models.
Adding explicit fatigue or substitution covariates to the underlying win-probability model would provide a direct robustness check on the current distinctions.
Extending the analysis to individual foul types or to late-game versus early-game calls would clarify which situations drive the observed team patterns.

Load-bearing premise

The ESPN win-probability model accurately isolates the marginal effect of each foul call on game outcome without systematic bias from unmodeled factors such as player fatigue, substitution timing, or referee-specific tendencies that correlate with call timing.

What would settle it

Recalculating RIM and the reported patterns with an alternative win-probability model that explicitly includes player fatigue and substitution timing, then checking whether the distinctions from foul volume and the post-control associations disappear.

Figures

Figures reproduced from arXiv: 2605.17845 by Leo Benaharon, Nirek Duma.

**Figure 2.** Figure 2: Regular-season average RIM plotted against average foul disparity for referees with at [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Figure 3: Regular-season RIM distribution with a one-standard-deviation band and a side table of [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

**Figure 4.** Figure 4: Bottom seven, mean, and top seven regular-season referees by average RIM, using a [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

**Figure 5.** Figure 5: Volume and leverage jointly explain why some referees have high or low average RIM. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

**Figure 6.** Figure 6: Quarter-specific average RIM by referee. Eric Lewis remains elevated across all four [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗

**Figure 7.** Figure 7: Postseason average absolute foul disparity and average absolute game RIM by normalized [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗

**Figure 8.** Figure 8: Postseason RIM distribution with standard-deviation bands and the bottom ten and top [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗

**Figure 9.** Figure 9: Combined regular-season and postseason home/away summary using signed foul disparity [PITH_FULL_IMAGE:figures/full_fig_p010_9.png] view at source ↗

**Figure 10.** Figure 10: Regular-season team-specific home/away splits. The league average is close to neutral, [PITH_FULL_IMAGE:figures/full_fig_p011_10.png] view at source ↗

**Figure 11.** Figure 11: Largest regular-season referee-team outliers by excess signed team RIM. Positive values [PITH_FULL_IMAGE:figures/full_fig_p012_11.png] view at source ↗

**Figure 12.** Figure 12: Largest regular-season referee-team outliers by excess signed foul disparity. Positive [PITH_FULL_IMAGE:figures/full_fig_p013_12.png] view at source ↗

**Figure 13.** Figure 13: Pregame series-score effects with 95% confidence intervals in the postseason. Mirrored [PITH_FULL_IMAGE:figures/full_fig_p014_13.png] view at source ↗

**Figure 14.** Figure 14: Regular-season team-side effects with omitted-variable robustness. Positive values favor [PITH_FULL_IMAGE:figures/full_fig_p015_14.png] view at source ↗

**Figure 15.** Figure 15: Residual regular-season referee-team effects with 95% confidence intervals. Positive [PITH_FULL_IMAGE:figures/full_fig_p016_15.png] view at source ↗

read the original abstract

Over the past century, basketball analytics has moved from simple box-score rates toward complex context-aware measures that evaluate events by their expected effect on game outcomes. Officiating analysis has not made the same transition: existing work and public discussion still rely heavily on foul rates, foul differentials, reviewed late-game correctness labels, or team/player benefit from calls. This leaves an empirical gap because a low-leverage foul in a decided game should not be treated as equivalent to a whistle that materially shifts win probability in a close game. To address this gap, we introduce the Ref Impact Metric (RIM), a game-level statistic that aggregates the absolute win-probability movement attached to foul events, measuring the impact of each referee for each game. Using ESPN game-summary and win-probability data for NBA seasons 2021-2022 through 2024-2025, we show that RIM is empirically distinct from both foul volume and foul disparity, identify regular-season and postseason referee distributions, and examine home/away, team-side, and referee-team heterogeneity. We then use linear controls intentionally as stress tests: conditioning on home status, team, opponent, season, and postseason series state asks which descriptive outliers persist after basic contextual adjustment. The results show that several team-side and referee-team patterns remain visible after conditioning, but omitted-variable robustness diagnostics indicate that these patterns should be interpreted as observational screening signals rather than evidence of intent, misconduct, or whistle-level responsibility by any single official. Our contribution to the literature is foundational, and we emphasize that this framework should be tested with different win probability models and further causal inference.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

This paper defines a new RIM from absolute win-probability shifts tied to fouls and shows it differs from foul volume after basic controls, but the whole thing rests on the ESPN model cleanly isolating officiating effects.

read the letter

This paper's core idea is worth knowing: they build a Referee Impact Metric by adding up the absolute win-probability changes that come with each foul call, using ESPN's data. They find that this RIM stands apart from raw foul numbers and some disparities, even after conditioning on home status, team, opponent, season, and series state. The patterns they see in team and referee effects are presented as screening signals only. What works here is the straightforward definition of the metric and the use of actual game data from 2021 through 2025. Treating the regressions as stress tests rather than final answers shows some restraint. They also note that different win-probability models should be tried, which is the right attitude for this kind of work. The main weakness is that everything rests on the ESPN win-probability model doing a good job of capturing the marginal impact of a foul without bias from unmeasured factors. Things like fatigue, substitutions, or when calls happen could be driving both the call and the probability shift. Without error bars, model checks, or comparisons to other WP sources, it's hard to know how robust the distinctness result is. The abstract does not describe any of those diagnostics. This is useful for people who track NBA officiating or build on sports analytics. It gives a new descriptive tool that could be refined. A reader who wants to see how context-aware measures change the conversation about refs would get something out of it. I would send this to peer review. The new aggregation is clear enough that referees can evaluate whether the measurement holds up and what additional checks are needed.

Referee Report

3 major / 3 minor

Summary. The manuscript introduces the Referee Impact Metric (RIM), which aggregates absolute win-probability shifts attached to foul events using ESPN game-summary and win-probability data across NBA seasons 2021-2022 through 2024-2025. It claims RIM is empirically distinct from foul volume and foul disparity, presents regular-season and postseason referee distributions along with home/away, team-side, and referee-team heterogeneity, and applies linear controls for home status, team, opponent, season, and postseason series state as stress tests, finding that several patterns remain visible after conditioning while emphasizing that results are observational screening signals rather than causal evidence of intent or misconduct.

Significance. If the central distinction and persistence claims hold after robustness checks against alternative win-probability models and with added uncertainty quantification, RIM would represent a meaningful advance over raw foul-rate metrics by incorporating game leverage and context. The deliberate use of linear controls as stress tests rather than causal identification is a methodological strength that aligns with the paper's cautious framing.

major comments (3)

[Abstract and §4] Abstract and §4 (results on distinctness): the claim that RIM is empirically distinct from foul volume and foul disparity is load-bearing for the contribution but rests on unspecified empirical comparisons; no correlation coefficients, regression R² values, or formal tests are reported to quantify how much unique variation RIM captures after accounting for volume and disparity.
[§3 and §5] §3 (metric construction) and §5 (conditioning results): the persistence of team-side and referee-team patterns after linear controls on home status, team, opponent, season, and series state is presented without error bars, standard errors, or model diagnostics; this undermines evaluation of whether the stress-test residuals are statistically distinguishable from zero or driven by the ESPN WP specification.
[§3] §3 (data and WP aggregation): the central distinction and heterogeneity claims require that the ESPN win-probability model isolates marginal foul effects without systematic bias from unmodeled factors such as fatigue or substitution timing; no robustness checks to alternative WP models or sensitivity analyses are described, which is a load-bearing limitation given the skeptic concern.

minor comments (3)

[Abstract] Abstract: the phrase 'omitted-variable robustness diagnostics' is used but not defined; specify the exact checks performed and report them in a dedicated subsection.
[§2] §2 (literature review): add citations to existing sports-analytics work on context-aware metrics and win-probability models to better situate the contribution.
[Figures/Tables] Figures and tables: ensure all panels include axis labels, legends, and sample sizes; clarify how RIM is normalized across games of different lengths.

Simulated Author's Rebuttal

3 responses · 1 unresolved

We thank the referee for the constructive and detailed report, which highlights important areas for strengthening the empirical claims. We agree that additional quantification, uncertainty measures, and explicit discussion of limitations will improve the manuscript. Below we respond point-by-point to the major comments and indicate the revisions we will make.

read point-by-point responses

Referee: [Abstract and §4] Abstract and §4 (results on distinctness): the claim that RIM is empirically distinct from foul volume and foul disparity is load-bearing for the contribution but rests on unspecified empirical comparisons; no correlation coefficients, regression R² values, or formal tests are reported to quantify how much unique variation RIM captures after accounting for volume and disparity.

Authors: We agree that the distinctness claim requires explicit quantification to be convincing. In the revised manuscript we will add Pearson and Spearman correlations between RIM and both foul volume and foul disparity. We will also report R² values from regressions of RIM on these two variables (and their interaction) to show the share of variation that remains unexplained. These results will be inserted into §4 and referenced in the abstract. revision: yes
Referee: [§3 and §5] §3 (metric construction) and §5 (conditioning results): the persistence of team-side and referee-team patterns after linear controls on home status, team, opponent, season, and series state is presented without error bars, standard errors, or model diagnostics; this undermines evaluation of whether the stress-test residuals are statistically distinguishable from zero or driven by the ESPN WP specification.

Authors: We accept that the absence of uncertainty quantification weakens the presentation of the conditioning results. The revised §5 will include standard errors and 95% confidence intervals for all reported coefficients from the linear models. We will also add basic model diagnostics (adjusted R², residual standard deviation, and a brief note on variance inflation factors) so readers can assess whether the remaining patterns are distinguishable from noise under the chosen specification. revision: yes
Referee: [§3] §3 (data and WP aggregation): the central distinction and heterogeneity claims require that the ESPN win-probability model isolates marginal foul effects without systematic bias from unmodeled factors such as fatigue or substitution timing; no robustness checks to alternative WP models or sensitivity analyses are described, which is a load-bearing limitation given the skeptic concern.

Authors: This is a substantive limitation we acknowledge. Because the analysis relies exclusively on the ESPN win-probability model, we cannot conduct direct robustness checks against alternative specifications with the current data. In the revision we will expand the discussion in §3 and the concluding section to explicitly address potential biases arising from unmodeled factors such as fatigue and substitution timing. We will also outline concrete sensitivity analyses that future work could perform with other win-probability models and will strengthen the language emphasizing that results are observational screening signals. revision: partial

standing simulated objections not resolved

Direct robustness checks against alternative win-probability models cannot be performed because the analysis is restricted to the ESPN data source; we will only be able to discuss this limitation and propose future checks rather than execute them.

Circularity Check

0 steps flagged

No circularity: RIM defined from external ESPN data; patterns are descriptive summaries

full rationale

The paper defines RIM directly as the aggregation of absolute win-probability deltas attached to foul events drawn from the independent ESPN win-probability series. Subsequent steps consist of empirical comparisons to foul volume/disparity, distributional summaries, and linear regressions that condition on home status, team, opponent, season, and series state. These are observational screening exercises whose outputs are not forced by construction to equal any fitted input or self-referential definition. No equations, ansatzes, or uniqueness claims reduce the reported distinctness or residual patterns back to the authors' own choices; the chain remains open to external data and model assumptions.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the assumption that the external win-probability series can be treated as an unbiased measure of foul impact and that linear conditioning on a small set of observables is sufficient to screen for obvious confounders. No free parameters are introduced; the metric itself is a direct sum. No new physical or social entities are postulated.

axioms (1)

domain assumption ESPN win-probability estimates isolate the marginal contribution of each foul call without material omitted-variable bias from timing, player state, or referee style.
Invoked when the paper treats absolute win-probability movement as the impact measure and when it interprets surviving patterns after linear controls.

pith-pipeline@v0.9.0 · 5829 in / 1521 out tokens · 26686 ms · 2026-05-20T00:52:40.318986+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

18 extracted references · 18 canonical work pages · 1 internal anchor

[1]

Michael Lewis.Moneyball: The Art of Winning an Unfair Game. W. W. Norton, New York, 2003

work page 2003
[2]

Thomas Dunne Books, New York, 2004

Alan Schwarz.The Numbers Game: Baseball’s Lifelong Fascination with Statistics. Thomas Dunne Books, New York, 2004

work page 2004
[3]

1946–47 BAA season summary

Basketball Reference. 1946–47 BAA season summary. https://www.basketball-reference. com/leagues/BAA_1947.html. Accessed April 27, 2026

work page 1946
[4]

Potomac Books, Washington, DC, 2004

Dean Oliver.Basketball on Paper: Rules and Tools for Performance Analysis. Potomac Books, Washington, DC, 2004

work page 2004
[5]

Oliver, Kevin Pelton, and Dan T

Justin Kubatko, Dean S. Oliver, Kevin Pelton, and Dan T. Rosenbaum. A starting point for analyzing basketball statistics.Journal of Quantitative Analysis in Sports, 3(3):1–24, 2007

work page 2007
[6]

POINTWISE: Predicting points and valuing decisions in real time with NBA optical tracking data

Dan Cervone, Alexander D’Amour, Luke Bornn, and Kirk Goldsberry. POINTWISE: Predicting points and valuing decisions in real time with NBA optical tracking data. InMIT Sloan Sports Analytics Conference, 2014. https://www.lukebornn.com/papers/cervone_ssac_2014.pdf

work page 2014
[7]

A Multiresolution Stochastic Process Model for Predicting Basketball Possession Outcomes

Daniel Cervone, Alexander D’Amour, Luke Bornn, and Kirk Goldsberry. A multiresolution stochastic process model for predicting basketball possession outcomes.Journal of the American Statistical Association, 111(514):585–599, 2016. arXiv:1408.0777

work page internal anchor Pith review Pith/arXiv arXiv 2016
[8]

Deshpande and Shane T

Sameer K. Deshpande and Shane T. Jensen. Estimating an NBA player’s impact on his team’s chances of winning.Journal of Quantitative Analysis in Sports, 12(2):51–72, 2016. arXiv:1604.03186

work page arXiv 2016
[9]

NBA officiating last two minute reports archive

National Basketball Association. NBA officiating last two minute reports archive. https: //official.nba.com/nba-officiating-last-two-minute-reports-archive/ . Accessed April 27, 2026

work page 2026
[10]

Racial discrimination among NBA referees.The Quarterly Journal of Economics, 125(4):1859–1887, 2010

Joseph Price and Justin Wolfers. Racial discrimination among NBA referees.The Quarterly Journal of Economics, 125(4):1859–1887, 2010

work page 2010
[11]

Pope, Joseph Price, and Justin Wolfers

Devin G. Pope, Joseph Price, and Justin Wolfers. Awareness reduces racial bias.Management Science, 64(11):4988–4995, 2018

work page 2018
[12]

Joseph Price, Marc Remer, and Daniel F. Stone. Subperfect game: Profitable biases of NBA referees.Journal of Economics & Management Strategy, 21(1):271–300, 2012

work page 2012
[13]

No referee bias in the NBA: New evidence with leagues’ assessment data

Christian Deutscher. No referee bias in the NBA: New evidence with leagues’ assessment data. Journal of Sports Analytics, 1(2):91–96, 2015

work page 2015
[14]

The effect of the crowd on home bias: Evidence from NBA games during the COVID-19 pandemic.Journal of Sports Economics, 23(7):950–975, 2022

Hua Gong. The effect of the crowd on home bias: Evidence from NBA games during the COVID-19 pandemic.Journal of Sports Economics, 23(7):950–975, 2022

work page 2022
[15]

Quantifying implicit biases in refereeing using NBA referees as a testbed.Scientific Reports, 13:4664, 2023

Konstantinos Pelechrinis. Quantifying implicit biases in refereeing using NBA referees as a testbed.Scientific Reports, 13:4664, 2023. arXiv:2210.13687

work page arXiv 2023
[16]

Mocan and Eric Osborne-Christenson

Naci H. Mocan and Eric Osborne-Christenson. In-group favoritism and peer effects in wrongful acquittals: National Basketball Association referees as judges.Journal of Law and Economics, 67(4):731–766, 2024. 22

work page 2024
[17]

NBA scoreboard and game summary data

ESPN. NBA scoreboard and game summary data. https://www.espn.com/nba/scoreboard. Accessed April 27, 2026

work page 2026
[18]

Favoritism under social pressure.Review of Economics and Statistics, 87(2):208–216, 2005

Luis Garicano, Ignacio Palacios-Huerta, and Canice Prendergast. Favoritism under social pressure.Review of Economics and Statistics, 87(2):208–216, 2005. 23

work page 2005

[1] [1]

Michael Lewis.Moneyball: The Art of Winning an Unfair Game. W. W. Norton, New York, 2003

work page 2003

[2] [2]

Thomas Dunne Books, New York, 2004

Alan Schwarz.The Numbers Game: Baseball’s Lifelong Fascination with Statistics. Thomas Dunne Books, New York, 2004

work page 2004

[3] [3]

1946–47 BAA season summary

Basketball Reference. 1946–47 BAA season summary. https://www.basketball-reference. com/leagues/BAA_1947.html. Accessed April 27, 2026

work page 1946

[4] [4]

Potomac Books, Washington, DC, 2004

Dean Oliver.Basketball on Paper: Rules and Tools for Performance Analysis. Potomac Books, Washington, DC, 2004

work page 2004

[5] [5]

Oliver, Kevin Pelton, and Dan T

Justin Kubatko, Dean S. Oliver, Kevin Pelton, and Dan T. Rosenbaum. A starting point for analyzing basketball statistics.Journal of Quantitative Analysis in Sports, 3(3):1–24, 2007

work page 2007

[6] [6]

POINTWISE: Predicting points and valuing decisions in real time with NBA optical tracking data

Dan Cervone, Alexander D’Amour, Luke Bornn, and Kirk Goldsberry. POINTWISE: Predicting points and valuing decisions in real time with NBA optical tracking data. InMIT Sloan Sports Analytics Conference, 2014. https://www.lukebornn.com/papers/cervone_ssac_2014.pdf

work page 2014

[7] [7]

A Multiresolution Stochastic Process Model for Predicting Basketball Possession Outcomes

Daniel Cervone, Alexander D’Amour, Luke Bornn, and Kirk Goldsberry. A multiresolution stochastic process model for predicting basketball possession outcomes.Journal of the American Statistical Association, 111(514):585–599, 2016. arXiv:1408.0777

work page internal anchor Pith review Pith/arXiv arXiv 2016

[8] [8]

Deshpande and Shane T

Sameer K. Deshpande and Shane T. Jensen. Estimating an NBA player’s impact on his team’s chances of winning.Journal of Quantitative Analysis in Sports, 12(2):51–72, 2016. arXiv:1604.03186

work page arXiv 2016

[9] [9]

NBA officiating last two minute reports archive

National Basketball Association. NBA officiating last two minute reports archive. https: //official.nba.com/nba-officiating-last-two-minute-reports-archive/ . Accessed April 27, 2026

work page 2026

[10] [10]

Racial discrimination among NBA referees.The Quarterly Journal of Economics, 125(4):1859–1887, 2010

Joseph Price and Justin Wolfers. Racial discrimination among NBA referees.The Quarterly Journal of Economics, 125(4):1859–1887, 2010

work page 2010

[11] [11]

Pope, Joseph Price, and Justin Wolfers

Devin G. Pope, Joseph Price, and Justin Wolfers. Awareness reduces racial bias.Management Science, 64(11):4988–4995, 2018

work page 2018

[12] [12]

Joseph Price, Marc Remer, and Daniel F. Stone. Subperfect game: Profitable biases of NBA referees.Journal of Economics & Management Strategy, 21(1):271–300, 2012

work page 2012

[13] [13]

No referee bias in the NBA: New evidence with leagues’ assessment data

Christian Deutscher. No referee bias in the NBA: New evidence with leagues’ assessment data. Journal of Sports Analytics, 1(2):91–96, 2015

work page 2015

[14] [14]

The effect of the crowd on home bias: Evidence from NBA games during the COVID-19 pandemic.Journal of Sports Economics, 23(7):950–975, 2022

Hua Gong. The effect of the crowd on home bias: Evidence from NBA games during the COVID-19 pandemic.Journal of Sports Economics, 23(7):950–975, 2022

work page 2022

[15] [15]

Quantifying implicit biases in refereeing using NBA referees as a testbed.Scientific Reports, 13:4664, 2023

Konstantinos Pelechrinis. Quantifying implicit biases in refereeing using NBA referees as a testbed.Scientific Reports, 13:4664, 2023. arXiv:2210.13687

work page arXiv 2023

[16] [16]

Mocan and Eric Osborne-Christenson

Naci H. Mocan and Eric Osborne-Christenson. In-group favoritism and peer effects in wrongful acquittals: National Basketball Association referees as judges.Journal of Law and Economics, 67(4):731–766, 2024. 22

work page 2024

[17] [17]

NBA scoreboard and game summary data

ESPN. NBA scoreboard and game summary data. https://www.espn.com/nba/scoreboard. Accessed April 27, 2026

work page 2026

[18] [18]

Favoritism under social pressure.Review of Economics and Statistics, 87(2):208–216, 2005

Luis Garicano, Ignacio Palacios-Huerta, and Canice Prendergast. Favoritism under social pressure.Review of Economics and Statistics, 87(2):208–216, 2005. 23

work page 2005