The reviewed record of science sign in
Pith

arxiv: 2607.00420 · v1 · pith:NCHDWXW2 · submitted 2026-07-01 · cs.HC

A Simple Solution to Improving Human Supervision of Algorithms: Evidence from Smart Vending

Minda Zhao , Brian Rongqing Han , Xin Chen , Tao Zhu This is my paper

Reviewed by Pith2026-07-02 06:54 UTCgrok-4.3pith:NCHDWXW2open to challenge →

classification cs.HC
keywords human supervision of AIoverride policy designfield experimentinventory managementsmart vending machinesdiscretion limitsselective filtering
0
0 comments X

The pith

Limiting overrides to two per machine lets workers reduce inventory by 1.28% without losing sales.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests whether limiting how often workers can override an AI's inventory decisions improves outcomes. It finds that capping downward overrides at two per vending machine reduces stock levels by 1.28 percent while keeping sales steady, because workers focus overrides on the most valuable items. Free overrides, by contrast, cut both inventory and sales. This matters for any setting where humans supervise autonomous systems, as it shows a low-cost way to capture useful private information without letting bias degrade performance.

Core claim

In a randomized experiment with 553 workers managing smart vending machines, a policy limiting downward overrides to two per machine produced a 1.28 percent inventory reduction with no sales decline. Workers under this constraint selected higher-value SKUs to override compared to those with unlimited overrides, which reduced inventory by 1.95 percent but also cut sales by 1.19 percent. The effect was confirmed through local average treatment effects and was strongest among experienced workers, high-incentive products, and growth-stage SKUs.

What carries the argument

The constrained override policy that limits the number of overrides per decision episode to enable selective filtering of high-value cases.

If this is right

  • Gains are largest for experienced workers.
  • High-incentive SKUs and growth-stage SKUs benefit most.
  • A simulated personalized policy further increases sales probability by 9.1 percent.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same limit-based approach might improve human oversight in other operational AI systems such as logistics scheduling.
  • Managers could experiment with adjusting the override cap based on worker tenure or product type to amplify the benefits.

Load-bearing premise

That the lack of sales loss under the two-override limit results specifically from workers choosing better SKUs to override, rather than from unrelated differences between the worker groups or machines.

What would settle it

Finding that the SKUs overridden under the constrained policy are no higher value than those overridden without the limit, yet sales still hold steady, would challenge the claim that selective filtering drives the result.

read the original abstract

Organizations increasingly deploy autonomous artificial intelligence (AI) systems for operational decisions, such as inventory replenishment. Yet fully granting override rights can degrade performance due to human bias and noise, while prohibiting them may overlook valuable private information. This raises a key question: How should override rights be structured to improve human supervision of autonomous AI? Methodology/results: We propose a constrained override policy that limits overrides per decision episode to enable selective filtering that prioritizes high-value overrides. We tested it through a randomized field experiment with 553 workers at a major Chinese smart vending machine retailer that manages more than 59,000 machines and 4,000 SKUs. Workers were assigned to no overrides, free overrides, or a two-per-machine limit on downward overrides. Free overrides reduce inventory by 1.95% but also cut sales by 1.19%. Constrained overrides reduce inventory by 1.28% without harming sales, as workers select better SKUs to override, confirmed via local average treatment effects. Gains are largest for experienced workers, high-incentive SKUs, and growth-stage SKUs. A simulated personalized policy further increases sales probability by 9.1%. Managerial implications: Academics gain novel insights from the causal effects of discretion design in human-supervised AI, emphasizing selective filtering to enhance decision quality. Managers can benefit from a scalable, low-cost policy for operations such as retail, logistics, and resource planning, reducing excess inventory without sales loss while harnessing private human information, with no need for algorithmic redesign, information customization, or additional training.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 0 minor

Summary. The manuscript reports results from a randomized field experiment with 553 workers at a Chinese smart vending retailer managing over 59,000 machines. It compares no-override, free-override, and constrained (two downward overrides per machine) policies. Free overrides reduce inventory by 1.95% and sales by 1.19%; constrained overrides reduce inventory by 1.28% with no sales loss. The authors interpret the sales preservation as evidence of selective filtering of higher-value SKUs by workers, supported by LATE analysis. Effects are larger for experienced workers, high-incentive SKUs, and growth-stage SKUs; a simulated personalized policy raises sales probability by 9.1%.

Significance. If the causal identification is robust, the work supplies field-experiment evidence on structuring limited human discretion to improve AI-supervised operational decisions, with direct implications for retail inventory, logistics, and resource allocation. The scale of the experiment (thousands of machines, thousands of SKUs) and the low-cost nature of the intervention are strengths that could inform practice without requiring model retraining or information redesign.

major comments (1)
  1. [Abstract] Abstract: the LATE step is presented as confirming that sales preservation occurs because workers select better SKUs under the two-override limit. However, the exclusion restriction—that the constraint affects outcomes solely through override count and selection—is not shown to hold against plausible alternatives (e.g., changes in worker attention, effort reallocation across machines, or compliance patterns). This assumption is load-bearing for the selective-filtering mechanism claim.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive comment on the identification strategy. We address the concern regarding the LATE exclusion restriction below and propose a targeted revision to strengthen the presentation without altering the core claims.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the LATE step is presented as confirming that sales preservation occurs because workers select better SKUs under the two-override limit. However, the exclusion restriction—that the constraint affects outcomes solely through override count and selection—is not shown to hold against plausible alternatives (e.g., changes in worker attention, effort reallocation across machines, or compliance patterns). This assumption is load-bearing for the selective-filtering mechanism claim.

    Authors: We appreciate the referee raising this identification issue. The LATE in the manuscript instruments the realized number of downward overrides with the randomized two-override constraint to isolate the effect of selective filtering on sales. While direct tests of the exclusion restriction are not feasible, the field-experiment design randomizes the policy at the worker-machine level with no changes to information, incentives, or task structure, making it unlikely that the constraint operates through attention shifts or effort reallocation (workers manage the same machines under all arms). Compliance is high by design, as the limit is mechanically enforced. Nevertheless, to address the concern transparently, we will revise the abstract to state the identifying assumption more explicitly and add a short robustness subsection discussing why alternative channels are implausible given the low-cost, rule-based intervention. This constitutes a partial revision focused on clarity rather than new empirical work. revision: partial

Circularity Check

0 steps flagged

No circularity: results from randomized field experiment with no self-referential derivations

full rationale

The paper reports causal effects from a randomized field experiment assigning workers to override policies, with outcomes measured on inventory and sales. The LATE estimates are standard instrumental-variables applications of the experimental assignment as instrument; they do not reduce any fitted parameter or equation to the input data by construction. No self-citations, ansatzes, or uniqueness theorems are invoked as load-bearing steps. The central claims are therefore externally falsifiable via the experimental design rather than tautological.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the internal validity of the randomized experiment and the interpretation that LATE differences reflect selective filtering; no new theoretical entities or fitted parameters are introduced.

axioms (1)
  • domain assumption Randomized assignment of workers to treatment groups identifies causal effects of the override policies.
    Standard identification assumption required for interpreting the experiment as causal.

pith-pipeline@v0.9.1-grok · 5816 in / 1214 out tokens · 41315 ms · 2026-07-02T06:54:24.167716+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

233 extracted references · 233 canonical work pages

  1. [1]

    Management Science , volume=

    Profit implications of judgmental adjustments to forecast inputs: Evidence from a large-scale field experiment , author=. Management Science , volume=. 2025 , publisher=

    work page 2025

  2. [2]

    Abdel-Karim, Benjamin M and Pfeuffer, Nicolas and Carl, K Valerie and Hinz, Oliver , journal=. How

    work page

  3. [3]

    The Review of Economic Studies , volume=

    Optimal delegation , author=. The Review of Economic Studies , volume=. 2008 , publisher=

    work page 2008

  4. [4]

    Acta Psychologica , volume=

    A timely account of the role of duration in decision making , author=. Acta Psychologica , volume=. 2001 , publisher=

    work page 2001

  5. [5]

    Journal of Public Economics , volume=

    Higher education as a filter , author=. Journal of Public Economics , volume=. 1973 , publisher=

    work page 1973

  6. [6]

    AEA Papers and Proceedings , volume =

    The Allocation of Decision Authority to Human and Artificial Intelligence , author =. AEA Papers and Proceedings , volume =. 2020 , month =

    work page 2020

  7. [7]

    Management Science , volume=

    Human-Algorithm Collaboration with Private Information: Naïve Advice-Weighting Behavior and Mitigation , author=. Management Science , volume=

    work page

  8. [8]

    Management Science , volume=

    Helping small businesses become more data-driven: A field experiment on eBay , author=. Management Science , volume=. 2024 , publisher=

    work page 2024

  9. [9]

    Management Science , volume=

    Improving Human Sequential Decision Making with Reinforcement Learning , author=. Management Science , volume=

    work page

  10. [10]

    Journal of Operations Management , volume=

    Demand planning for the digital supply chain: How to integrate human judgment and predictive analytics , author=. Journal of Operations Management , volume=. 2023 , publisher=

    work page 2023

  11. [11]

    American Economic Review , volume=

    The rapid adoption of data-driven decision-making , author=. American Economic Review , volume=. 2016 , publisher=

    work page 2016

  12. [12]

    What can machine learning do?

    Brynjolfsson, Erik and Mitchell, Tom , journal=. What can machine learning do?. 2017 , publisher=

    work page 2017

  13. [13]

    The economics of artificial intelligence: An agenda , pages=

    Artificial Intelligence and Behavioral Economics , author=. The economics of artificial intelligence: An agenda , pages=. 2018 , publisher=

    work page 2018

  14. [14]

    Available at SSRN 3640862 , year=

    How Forced Intervention Facilitates Long-term Algorithm Adoption , author=. Available at SSRN 3640862 , year=

    work page

  15. [15]

    Believing in analytics: Managers’ adherence to price recommendations from a

    Caro, Felipe and de Tejada Cuenca, Anna S. Believing in analytics: Managers’ adherence to price recommendations from a. Manufacturing & Service Operations Management , volume=. 2023 , publisher=

    work page 2023

  16. [16]

    Proceedings of ICIS 2023 , year=

    Algorithm-Human-Algorithm: A New Classification Approach to Integrating Judgemental Adjustments , author=. Proceedings of ICIS 2023 , year=

    work page 2023

  17. [17]

    Available at SSRN 4365121 , year=

    The impact of a more diversified recommender system on digital platforms: Evidence from a large-scale field experiment , author=. Available at SSRN 4365121 , year=

    work page

  18. [18]

    arXiv preprint arXiv:2211.11028 , year=

    Algorithmic decision-making safeguarded by human knowledge , author=. arXiv preprint arXiv:2211.11028 , year=

    work page arXiv

  19. [19]

    Management Science , volume=

    Information sharing in supply chains: An empirical and theoretical valuation , author=. Management Science , volume=. 2015 , publisher=

    work page 2015

  20. [20]

    2004 , publisher=

    Order Statistics , author=. 2004 , publisher=

    work page 2004

  21. [21]

    Is your machine better than you?

    de V. Is your machine better than you?. Management Science , volume=. 2023 , publisher=

    work page 2023

  22. [22]

    The Review of Economic Studies , volume=

    Authority and communication in organizations , author=. The Review of Economic Studies , volume=. 2002 , publisher=

    work page 2002

  23. [23]

    Psychological Science , volume=

    People reject algorithms in uncertain decision domains because they have diminishing sensitivity to forecasting error , author=. Psychological Science , volume=. 2020 , publisher=

    work page 2020

  24. [24]

    , author=

    Algorithm aversion: People erroneously avoid algorithms after seeing them err. , author=. Journal of Experimental Psychology: General , volume=. 2015 , publisher=

    work page 2015

  25. [25]

    Management Science , volume=

    Overcoming algorithm aversion: People will use imperfect algorithms if they can (even slightly) modify them , author=. Management Science , volume=. 2018 , publisher=

    work page 2018

  26. [26]

    Psychonomic Bulletin & Review , volume=

    Optimal or not; depends on the task , author=. Psychonomic Bulletin & Review , volume=. 2019 , publisher=

    work page 2019

  27. [27]

    Production and Operations Management , volume=

    Modeling strategic behavior in the competitive newsvendor problem: An experimental investigation , author=. Production and Operations Management , volume=. 2017 , publisher=

    work page 2017

  28. [28]

    International Journal of Forecasting , volume=

    Effective forecasting and judgmental adjustments: An empirical evaluation and strategies for improvement in supply-chain planning , author=. International Journal of Forecasting , volume=. 2009 , publisher=

    work page 2009

  29. [29]

    Management Science , volume=

    Competition-based dynamic pricing in online retailing: A methodology validated with field experiments , author=. Management Science , volume=. 2018 , publisher=

    work page 2018

  30. [30]

    Will humans-in-the-loop become borgs?

    F. Will humans-in-the-loop become borgs?. MIS Quarterly , volume=

    work page

  31. [31]

    Information Systems Research , volume=

    Cognitive challenges in human--artificial intelligence collaboration: Investigating the path toward productive delegation , author=. Information Systems Research , volume=. 2022 , publisher=

    work page 2022

  32. [32]

    Management Science , volume=

    The best decisions are not the best advice: Making adherence-aware recommendations , author=. Management Science , volume=. 2024 , publisher=

    work page 2024

  33. [33]

    Proceedings of the Conference on Fairness, Accountability, and Transparency , pages=

    Disparate interactions: An algorithm-in-the-loop analysis of fairness in risk assessments , author=. Proceedings of the Conference on Fairness, Accountability, and Transparency , pages=

    work page

  34. [34]

    1963 , publisher=

    Analysis of Inventory Systems , author=. 1963 , publisher=

    work page 1963

  35. [35]

    Connecting Customers and Merchants Offline: Experimental Evidence from the Commercialization of Last-mile Stations at

    Brian Rongqing Han and Tianshu Sun and Leon Yang Chu and Lixia Wu and Yinghui Xu , journal =. Connecting Customers and Merchants Offline: Experimental Evidence from the Commercialization of Last-mile Stations at

    work page

  36. [36]

    Available at SSRN 4195014 , year=

    Human-algorithmic bias: Source, evolution, and impact , author=. Available at SSRN 4195014 , year=

    work page

  37. [37]

    Management Science , volume=

    Discretionary task ordering: Queue management in radiological services , author=. Management Science , volume=. 2018 , publisher=

    work page 2018

  38. [38]

    Management Science , volume=

    Eliciting human judgment for prediction algorithms , author=. Management Science , volume=. 2021 , publisher=

    work page 2021

  39. [39]

    Econometrica , volume=

    Identification and estimation of local average treatment effects , author=. Econometrica , volume=. 1994 , publisher=

    work page 1994

  40. [40]

    Journal of Personality and Social Psychology , volume=

    When choice is demotivating: Can one desire too much of a good thing? , author=. Journal of Personality and Social Psychology , volume=. 2000 , publisher=

    work page 2000

  41. [41]

    Augmenting medical diagnosis decisions?

    Jussupow, Ekaterina and Spohrer, Kai and Heinzl, Armin and Gawlitza, Joshua , journal=. Augmenting medical diagnosis decisions?. 2021 , publisher=

    work page 2021

  42. [42]

    When will workers follow an algorithm?

    Kawaguchi, Kohei , journal=. When will workers follow an algorithm?. 2021 , publisher=

    work page 2021

  43. [43]

    Management Science , volume=

    Field experiment on the profit implications of merchants’ discretionary power to override data-driven decision-making tools , author=. Management Science , volume=. 2020 , publisher=

    work page 2020

  44. [44]

    Evaluating human behaviour in response to

    Khosrowabadi, Naghmeh and Hoberg, Kai and Imdahl, Christina , journal=. Evaluating human behaviour in response to. 2022 , publisher=

    work page 2022

  45. [45]

    Available at SSRN 4773773 , year=

    Human-computer interactions in demand forecasting and labor scheduling decisions , author=. Available at SSRN 4773773 , year=

    work page

  46. [46]

    To engage or not to engage with

    Lebovitz, Sarah and Lifshitz-Assaf, Hila and Levina, Natalia , journal=. To engage or not to engage with. 2022 , publisher=

    work page 2022

  47. [47]

    Production and Operations Management , volume=

    The risk of algorithm transparency: How algorithm complexity drives the effects on the use of advice , author=. Production and Operations Management , volume=. 2022 , publisher=

    work page 2022

  48. [48]

    Production and Operations Management , volume=

    Transshipment between overconfident newsvendors , author=. Production and Operations Management , volume=. 2021 , publisher=

    work page 2021

  49. [49]

    Production and Operations Management , volume=

    Overconfident distribution channels , author=. Production and Operations Management , volume=. 2019 , publisher=

    work page 2019

  50. [50]

    Production and Operations Management , volume=

    Examining Behavioral Biases in Discretionary Pricing: Evidence from Field and Lab Experiments , author=. Production and Operations Management , volume=. 2024 , publisher=

    work page 2024

  51. [51]

    Management Science , volume=

    Algorithm aversion: Evidence from ridesharing drivers , author=. Management Science , volume=. 2023 , publisher=

    work page 2023

  52. [52]

    Lou, Bowen and Wu, Lynn , journal=

    work page

  53. [53]

    1 + 1 > 2?

    Lu, Tian and Zhang, Yingjie , journal=. 1 + 1 > 2?. 2025 , publisher=

    work page 2025

  54. [54]

    humans: The impact of artificial intelligence chatbot disclosure on customer purchases , author=

    Frontiers: Machines vs. humans: The impact of artificial intelligence chatbot disclosure on customer purchases , author=. Marketing Science , volume=. 2019 , publisher=

    work page 2019

  55. [55]

    Production and Operations Management , volume=

    How to compete against a behavioral newsvendor , author=. Production and Operations Management , volume=. 2015 , publisher=

    work page 2015

  56. [56]

    Management Science , volume=

    The editor and the algorithm: Recommendation technology in online news , author=. Management Science , volume=. 2024 , publisher=

    work page 2024

  57. [57]

    Management Science , volume=

    The effectiveness of field price discretion: Empirical evidence from auto lending , author=. Management Science , volume=. 2015 , publisher=

    work page 2015

  58. [58]

    Production and Operations Management , volume=

    Behavioral ordering, competition and profits: An experimental investigation , author=. Production and Operations Management , volume=. 2019 , publisher=

    work page 2019

  59. [59]

    Journal of the Operational Research Society , volume=

    The overconfident newsvendor , author=. Journal of the Operational Research Society , volume=. 2017 , publisher=

    work page 2017

  60. [60]

    Management Science , volume=

    Overconfidence in newsvendor orders: An experimental study , author=. Management Science , volume=. 2013 , publisher=

    work page 2013

  61. [61]

    Journal of Educational Psychology , volume=

    Estimating causal effects of treatments in randomized and nonrandomized studies , author=. Journal of Educational Psychology , volume=

    work page

  62. [62]

    Bounds for expectations of

    Rychlik, Tomasz , journal=. Bounds for expectations of. 1998 , publisher=

    work page 1998

  63. [63]

    Sementic Scholar , pages=

    Decision-making under cognitive constraints: Evidence from the emergency department , author=. Sementic Scholar , pages=

    work page

  64. [64]

    The Quarterly Journal of Economics , volume=

    A behavioral model of rational choice , author=. The Quarterly Journal of Economics , volume=. 1955 , publisher=

    work page 1955

  65. [65]

    Journal of Monetary Economics , volume=

    Implications of rational inattention , author=. Journal of Monetary Economics , volume=. 2003 , publisher=

    work page 2003

  66. [66]

    Management Science , volume=

    Algorithm Reliance: Fast and Slow , author=. Management Science , volume=

    work page

  67. [67]

    The Quarterly Journal of Economics , volume=

    Job market signaling , author=. The Quarterly Journal of Economics , volume=. 1973 , publisher=

    work page 1973

  68. [68]

    Available at SSRN 4851815 , year=

    Reluctant to decrease: Human-algorithm collaboration in tactical inventory decisions , author=. Available at SSRN 4851815 , year=

    work page

  69. [69]

    Manufacturing & Service Operations Management , volume=

    Bounded rationality in newsvendor models , author=. Manufacturing & Service Operations Management , volume=. 2008 , publisher=

    work page 2008

  70. [70]

    Management Science , volume=

    Predicting human discretion to adjust algorithmic prescription: A large-scale field experiment in warehouse operations , author=. Management Science , volume=. 2022 , publisher=

    work page 2022

  71. [71]

    Information Systems Research , volume=

    Motivating effective mobile app adoptions: Evidence from a large-scale randomized field experiment , author=. Information Systems Research , volume=. 2019 , publisher=

    work page 2019

  72. [72]

    Production and Operations Management , volume=

    Behavioral drivers of routing decisions: Evidence from restaurant table assignment , author=. Production and Operations Management , volume=. 2020 , publisher=

    work page 2020

  73. [73]

    Management Science , volume=

    Ordering behavior in retail stores and implications for automated replenishment , author=. Management Science , volume=. 2010 , publisher=

    work page 2010

  74. [74]

    Management Science , volume=

    The Power of Disagreement: A Field Experiment to Investigate Human-Algorithm Collaboration in Loan Evaluations , author=. Management Science , volume=

    work page

  75. [75]

    The Value of

    Wolczynski, Nicholas and Saar-Tsechansky, Maytal and Wang, Tong , journal=. The Value of

    work page

  76. [76]

    , author=

    Decision field theory: a dynamic-cognitive approach to decision making in an uncertain environment. , author=. Psychological review , volume=. 1993 , publisher=

    work page 1993

  77. [77]

    , author=

    Judgment under Uncertainty: Heuristics and Biases: Biases in judgments reveal some heuristics of thinking under uncertainty. , author=. science , volume=. 1974 , publisher=

    work page 1974

  78. [78]

    Journal of Public Economics , volume=

    Choice proliferation, simplicity seeking, and asset allocation , author=. Journal of Public Economics , volume=. 2010 , publisher=

    work page 2010

  79. [79]

    Proceedings of the ACM on Human-computer Interaction , volume=

    Understanding the role of human intuition on reliance in human-AI decision-making with explanations , author=. Proceedings of the ACM on Human-computer Interaction , volume=. 2023 , publisher=

    work page 2023

  80. [80]

    University of Miami Business School Research Paper , number=

    The role of AI assistants in livestream selling: Evidence from a randomized field experiment , author=. University of Miami Business School Research Paper , number=

    work page

Showing first 80 references.