pith. sign in

arxiv: 2511.09612 · v2 · pith:BDZ4LROTnew · submitted 2025-11-12 · 💻 cs.HC

When Thinking Pays Off: Incentive Alignment for Human-AI Collaboration

Joshua Holstein , Patrick Hemmer , Gerhard Satzger , Wei Sun This is my paper

Pith reviewed 2026-05-17 22:06 UTC · model grok-4.3

classification 💻 cs.HC
keywords incentive alignmenthuman-AI collaborationoverreliance on AIbehavioral experimentdecision-makingAI adviceincentive design
0
0 comments X

The pith

Redesigning incentives to reward independent judgment reduces human overreliance on AI advice.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Humans systematically follow AI recommendations even when their own judgment would produce better results, which prevents teams from using the strengths of both. The paper traces this overreliance to standard incentive structures that do not reward people for thinking when it matters. It introduces an alternative incentive mechanism intended to align rewards with the actual complementarities between human and AI capabilities. A behavioral experiment with 180 participants shows that the new mechanism cuts overreliance and raises decision quality. The work also demonstrates that poorly chosen incentives can distort behavior and lower overall performance instead.

Core claim

Prevailing incentive structures in human-AI decision-making act as a structural driver of overreliance. The authors propose an alternative incentive mechanism designed to counteract this misalignment. In a behavioral experiment with 180 participants, the mechanism significantly reduces overreliance. The results further show that appropriately designed incentives enhance collaboration and decision quality, while poorly designed incentives can distort behavior, introduce unintended consequences, and degrade performance. Effective collaboration therefore requires context-sensitive incentive design.

What carries the argument

An alternative incentive mechanism that realigns rewards with task context and human-AI complementarities to reduce overreliance.

If this is right

  • The proposed incentive mechanism significantly reduces overreliance on AI advice.
  • Appropriately designed incentives enhance collaboration and improve decision quality.
  • Poorly designed incentives distort behavior, create unintended consequences, and degrade performance.
  • Effective human-AI collaboration requires shifting to context-sensitive incentive design.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same incentive logic could apply to other human-AI settings such as medical diagnosis or creative work.
  • Organizations may need to audit existing performance metrics when introducing AI tools to avoid creating hidden dependence.
  • Longer-term studies could test whether reduced overreliance persists or whether humans lose skills from less independent practice.

Load-bearing premise

The incentive effects and overreliance patterns observed in the lab tasks accurately reflect real-world human-AI collaboration settings and stem primarily from incentive misalignment.

What would settle it

A field deployment of the proposed incentive mechanism in an actual organization that shows no significant drop in overreliance would challenge the central claim.

Figures

Figures reproduced from arXiv: 2511.09612 by Gerhard Satzger, Joshua Holstein, Patrick Hemmer, Wei Sun.

Figure 1
Figure 1. Figure 1: Overview of the 30-instance selection scheme. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of processed instances across experimental conditions. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Weighted average human-AI team performance across treatments with 95% confidence intervals. Reference lines show [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Weighted average participant reliance across treatments with 95% confidence intervals. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of instances with confidences to complementary subsets. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
read the original abstract

Collaboration with artificial intelligence (AI) has improved human decision-making across various domains by leveraging the complementary capabilities of humans and AI. Yet, humans systematically overrely on AI advice, even when their independent judgment would yield superior outcomes, fundamentally undermining the potential of human-AI complementarity. Building on prior work, we identify prevailing incentive structures in human-AI decision-making as a structural driver of this overreliance. To address this misalignment, we propose an alternative incentive mechanism designed to counteract systemic overreliance. We empirically evaluate this approach through a behavioral experiment with 180 participants, finding that the proposed mechanism significantly reduces overreliance. We also show that while appropriately designed incentives can enhance collaboration and decision quality, poorly designed incentives may distort behavior, introduce unintended consequences, and ultimately degrade performance. These findings underscore the importance of aligning incentives with task context and human-AI complementarities, and suggest that effective collaboration requires a shift toward context-sensitive incentive design.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper identifies prevailing incentive structures as a driver of human overreliance on AI advice in collaborative decision-making, proposes an alternative incentive mechanism to promote complementarity, and evaluates it via a behavioral experiment with 180 participants that reports a statistically significant reduction in overreliance. It further shows that well-designed incentives can improve outcomes while poorly designed ones can degrade performance, emphasizing context-sensitive incentive design.

Significance. If the experimental results are robust, the work provides actionable evidence that incentive alignment can reduce overreliance and enhance human-AI complementarity, with implications for designing decision-support systems. The inclusion of both positive and negative incentive effects strengthens the practical takeaway. The empirical approach with participant data offers a falsifiable test of the mechanism rather than purely theoretical claims.

major comments (2)
  1. [§4, §5] §4 (Experimental Design) and §5 (Results): The central claim that the mechanism reduces overreliance rests on the assumption that the chosen tasks contain a non-trivial fraction of trials where independent human judgment is verifiably superior to AI advice. The manuscript does not report baseline human-only vs. AI-only accuracies or the distribution of complementarity cases, making it difficult to rule out that the measured reduction reflects task-specific effects rather than the incentive mechanism itself.
  2. [§5] §5 (Results): While the abstract and results claim a statistically significant reduction, the manuscript provides insufficient detail on the specific statistical tests, effect sizes, confidence intervals, or controls for confounds such as experimenter demand characteristics or novelty effects from the new payment rule. These omissions weaken the ability to assess whether the reduction is attributable to improved incentive alignment rather than demand artifacts common in behavioral HC experiments.
minor comments (2)
  1. [§3] The notation for the proposed incentive mechanism (e.g., any payoff functions or alignment parameters) could be introduced earlier with a clear example to improve readability for readers unfamiliar with the specific formulation.
  2. [Figures in §5] Figure captions and axis labels in the results figures should explicitly state the dependent variable (e.g., overreliance rate) and include error bars or statistical annotations for clarity.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive and detailed comments, which highlight important aspects for strengthening the interpretation of our experimental results on incentive alignment in human-AI collaboration. We address each major comment below and outline the revisions we will make.

read point-by-point responses

  1. Referee: [§4, §5] §4 (Experimental Design) and §5 (Results): The central claim that the mechanism reduces overreliance rests on the assumption that the chosen tasks contain a non-trivial fraction of trials where independent human judgment is verifiably superior to AI advice. The manuscript does not report baseline human-only vs. AI-only accuracies or the distribution of complementarity cases, making it difficult to rule out that the measured reduction reflects task-specific effects rather than the incentive mechanism itself.

    Authors: We agree that explicit reporting of baseline accuracies and complementarity distributions is essential to substantiate the claim and rule out task-specific confounds. The tasks were chosen based on established domains in prior human-AI decision-making research where complementarity has been documented, and the experiment included a no-AI baseline condition to measure human-only performance. In the revised manuscript, we will add to §4 a description of how AI-only accuracy was computed on the identical task set and include in §5 a new analysis or table reporting the proportion of trials in which independent human judgment outperformed AI advice. This will allow direct assessment of whether the incentive mechanism enhances use of human strengths in complementarity cases. revision: yes

  2. Referee: [§5] §5 (Results): While the abstract and results claim a statistically significant reduction, the manuscript provides insufficient detail on the specific statistical tests, effect sizes, confidence intervals, or controls for confounds such as experimenter demand characteristics or novelty effects from the new payment rule. These omissions weaken the ability to assess whether the reduction is attributable to improved incentive alignment rather than demand artifacts common in behavioral HC experiments.

    Authors: We will substantially expand the statistical reporting in the revised §5. This includes specifying the exact tests (e.g., t-tests or linear mixed-effects models with appropriate random effects for participants), reporting effect sizes (Cohen's d or equivalent), and including 95% confidence intervals for the primary comparisons on overreliance rates and decision accuracy. To address potential confounds, we will add a dedicated paragraph discussing the between-subjects design, standardized instructions across conditions, and the inclusion of both well-designed and poorly-designed incentive arms (the latter degrading performance), which helps isolate the mechanism from general demand or novelty effects. We will also note any pre-registration details and offer to include additional robustness checks on the existing dataset. revision: yes

Circularity Check

0 steps flagged

No circularity in empirical evaluation of incentive mechanism

full rationale

The paper proposes an incentive mechanism to address overreliance in human-AI collaboration, building on prior work, and evaluates it via a behavioral experiment with 180 participants. The central claims rest on observed experimental outcomes rather than any derivation chain, equations, or self-referential definitions. No fitted inputs are renamed as predictions, no uniqueness theorems are imported from self-citations, and no ansatzes are smuggled in. The results are grounded in new participant data, making the study self-contained against external benchmarks with no load-bearing reductions to its own inputs.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on standard assumptions from behavioral economics and experimental psychology regarding how monetary incentives influence decision behavior and the generalizability of lab findings to applied settings.

axioms (1)
  • domain assumption Standard assumptions in experimental psychology about participant behavior and incentive effects.
    Used to interpret the results of the behavioral experiment.

pith-pipeline@v0.9.0 · 5463 in / 1087 out tokens · 44441 ms · 2026-05-17T22:06:18.313548+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

13 extracted references · 13 canonical work pages · 1 internal anchor

  1. [1]

    Bansal, B

    G. Bansal, B. Nushi, E. Kamar, E. Horvitz, and D. S. Weld. Is the most accurate ai the best teammate? optimizing ai for teamwork. InProceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11405–11414, 2021a. G. Bansal, T. Wu, J. Zhou, R. Fok, B. Nushi, E. Kamar, M. T. Ribeiro, and D. Weld. Does the whole exceed its parts? the effec...

    work page 2021

  2. [2]

    Buçinca, S

    Z. Buçinca, S. Swaroop, A. E. Paluch, F. Doshi-Velez, and K. Z. Gajos. Contrastive explanations that anticipate human misconceptions can improve human decision-making skills. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1–25,

    work page 2025

  3. [3]

    Hemmer, M

    P. Hemmer, M. Schemmer, N. Kühl, M. Vössing, and G. Satzger. Complementarity in human-ai collaboration: Concept, sources, and evidence.arXiv preprint arXiv:2404.00029,

    work page arXiv

  4. [4]

    Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task

    Association for Computing Machinery. ISBN 9798400705083. N. Kosmyna, E. Hauptmann, Y . T. Yuan, J. Situ, X.-H. Liao, A. V . Beresnitzky, I. Braunstein, and P. Maes. Your brain on chatgpt: Accumulation of cognitive debt when using an ai assistant for essay writing task.arXiv preprint arXiv:2506.08872,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    H.-P. Lee, A. Sarkar, L. Tankelevitch, I. Drosos, S. Rintel, R. Banks, and N. Wilson. The impact of generative ai on critical thinking: Self-reported reductions in cognitive effort and confidence effects from a survey of knowledge workers. In Proceedings of the 2025 CHI conference on human factors in computing systems, pages 1–22,

    work page 2025

  6. [6]

    are you really sure?

    S. Ma, X. Wang, Y . Lei, C. Shi, M. Yin, and X. Ma. “are you really sure?” understanding the effects of human self-confidence calibration in ai-assisted decision making. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pages 1–20,

    work page 2024

  7. [7]

    Pinski, M

    M. Pinski, M. Adam, and A. Benlian. Ai knowledge: Improving ai delegation through human enablement. InProceedings of the 2023 CHI conference on human factors in computing systems, pages 1–17,

    work page 2023

  8. [8]

    Schemmer, P

    M. Schemmer, P. Hemmer, M. Nitsche, N. Kühl, and M. Vössing. A meta-analysis of the utility of explainable artificial intelligence in human-ai decision-making. InProceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pages 617–626,

    work page 2022

  9. [9]

    Spitzer, J

    P. Spitzer, J. Holstein, K. Morrison, K. Holstein, G. Satzger, and N. Kühl. Don’t be fooled: The misinformation effect of explanations in human-ai collaboration.arXiv preprint arXiv:2409.12809,

    work page arXiv

  10. [10]

    X. Wang, Z. Lu, and M. Yin. Will you accept the ai recommendation? predicting human behavior in ai-assisted decision making. InProceedings of the ACM Web Conference 2022, page 1697–1708,

    work page 2022

  11. [11]

    Zhang, Q

    Y . Zhang, Q. V . Liao, and R. K. Bellamy. Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making. InProceedings of the 2020 conference on fairness, accountability, and transparency, pages 295–305,

    work page 2020

  12. [12]

    very low

    12 When Thinking Pays Off: Incentive Alignment for Human-AI CollaborationPREPRINT Appendix This appendix provides detailed methodological specifications and supplementary analyses supporting our main findings. We present comprehensive implementation details for our behavioral experiment, including AI system training, instance selection criteria, and incen...

    work page 2022

  13. [13]

    This procedure caps extreme weights while preserving the overall weighting structure, preventing any single participant from disproportionately influencing results

    at the 5th and 95th percentiles to the weight distributions within each treatment group before conducting comparative analyses. This procedure caps extreme weights while preserving the overall weighting structure, preventing any single participant from disproportionately influencing results. Statistical Testing.All comparative analyses for human-AI team p...

    work page 1992