Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

Who Deserves the Reward? SHARP: Shapley Credit-based Optimization for Multi-Agent System

T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SHARP claims that per-agent marginal-credit rewards, computed by counterfactually masking each agent, stabilize multi-agent LLM training and lift average match scores by 23.66% over single-agent and 14.05% over multi-agent baselines.

desk verdict A concrete reward-decomposition recipe with a real claim problem: the 'Shapley' credit term is leave-one-out on noisy rollouts, but the ablations and benchmark sweep make it worth a careful referee. read the letter →

arxiv 2602.08335 v2 pith:Y4BNYRXE submitted 2026-02-09 cs.AI

Yanming Li , Xuelin Zhang , WenJie Lu , Ziye Tang , Maodong Wu , Haotian Luo , Tongtong Wu , Zijie Peng
show 6 more authors
Hongze Mi Yibo Feng Naiqiang Tan Chao Huang Lian Peng Li Shen
This is my paper
classification cs.AI
keywords multi-agentreinforcementlearningcreditassignmentShapleyvaluelargelanguagemodelstool-augmentedreasoningcounterfactualmaskinghierarchicalplanner-workerrewarddecomposition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SHARP claims that the hard part of training a multi-agent LLM system is deciding which agent deserves credit for the final answer, and that a reward split by per-agent marginal contribution solves it. The framework rewards each agent with a combination of a global accuracy signal, a counterfactual masking term that measures how task success changes when that agent is removed, and a tool-validity term. These per-agent rewards are normalized within trajectory groups and used to update one shared policy that instantiates both planner and worker roles. Across four tool-use and multi-hop reasoning benchmarks, the paper reports average match gains of 23.66% over single-agent RL baselines and 14.05% over multi-agent baselines. If true, this makes multi-agent LLM training more stable and accurate than broadcast-reward training, and offers a way to see which subagents are actually helping.

What carries the argument

The load-bearing object is the counterfactual masking mechanism: for each trajectory τᵢ and each worker m, credit is R_acc(full trajectory) − R_acc(trajectory with m masked out). This leave-one-out accuracy delta is intended to isolate the marginal causal impact of each worker; the planner's credit is the mean of its workers' positive deltas, scaled by a factor λ. These per-agent credits are added to a broadcast accuracy reward and a tool-validity reward, and the sum is normalized across a group of sampled trajectories with a group-relative advantage before the clipped policy-gradient update. The group-relative normalization is what the paper credits with stabilizing the resulting policy upd

What would settle it

Take a fixed set of trajectories and compute each worker's credit two ways: keeping the planner's final answer fixed while masking the worker, versus rerunning the policy on the masked context. If the credits disagree in sign or ordering for more than a few workers, the reward signal is undefined. A second decisive check: replace the per-agent credit values with random noise of the same mean and variance; if accuracy does not drop, the specific causal values are not what carry the gain.

Watch

Extended reading notes

Core claim

The central claim is that per-agent marginal credit—defined as the accuracy difference between a full trajectory and the same trajectory with a given agent's contribution masked out—is a better optimization signal than a single team-wide reward for a shared-policy hierarchical multi-agent system. The paper calls this Shapley-based credit and assigns worker agents this counterfactual delta, while the planner receives a damped average of its workers' positive credits. These are combined with a global accuracy term and a tool-process reward, then converted into group-relative advantages that update the shared policy. The paper reports that this reward decomposition consistently outperforms sing

Load-bearing premise

The load-bearing premise is that the counterfactual masking step—constructing the masked trajectory by removing agent m and measuring R_acc(full) minus R_acc(masked)—produces a well-defined, causally meaningful delta; the paper does not state whether the policy is rerun on the masked context, whether the planner's final answer is kept fixed, or how sampling noise is controlled, so this delta carries the entire learning signal.

Editorial extensions

If this is right

  • Multi-agent LLM training can be made more stable: the paper reports monotonic accuracy gains over 180 training steps on a benchmark while broadcast-reward baselines fluctuate.
  • Per-agent credit changes coordination structure: the reported proportion of harmful subagent calls falls from 5.48% to 4.40%, and the planner's average credit score rises.
  • The benefit scales with model size: on a multi-hop QA benchmark the gap over a single-agent baseline widens from small at 0.6B parameters to 14.41 points at 8B.
  • Credit density trades training cost for deployment efficiency: computing credit for a larger fraction of subagents raises training latency but lowers per-sample inference tokens while improving accuracy.
  • Planner and worker credit are synergistic: ablations show joint credit reaches 50.76 while planner-only and worker-only variants reach lower scores, indicating more than additive gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If counterfactual masking truly isolates causal contribution, the same reward decomposition should transfer to other shared-policy multi-agent settings—long-horizon research agents, code-generation pipelines, or tool-use hierarchies—without changing the optimizer, since the mechanism is role-agnostic.
  • The paper's coordination analysis suggests a testable extension: use per-agent credit as a live inference-time filter, pruning subagents with persistently negative credit instead of only penalizing them during training.
  • Because planner credit is the mean of positive worker credits, a planner could be rewarded for delegating to many mildly useful workers; an extension that subtracts a redundancy penalty might address the paper's own observation that useful subagents remain a minority of calls.
  • The credit term as written is leave-one-out, not a full Shapley coalition average; a direct comparison of the two would show whether the Shapley machinery itself matters or whether any sensible per-agent delta would produce the same training dynamics.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SHARP (Shapley-based Hierarchical Attribution for Reinforcement Policy), a joint optimization method for multi-agent LLM systems with a shared policy instantiating planner and worker roles. The method augments GRPO-style group-relative advantages with a decomposed reward: a broadcast binary accuracy reward, a per-agent 'marginal credit' reward computed by counterfactually ablating each agent, and a tool-process reward. Experiments on MuSiQue, GAIA-text, WebWalkerQA, and FRAMES claim consistent gains over single-agent and multi-agent RL baselines, with an average improvement of 23.66% and 14.05%, respectively, plus additional analyses of scaling, stability, forgetting, and coordination. The central technical claim is that the counterfactual masking procedure in Eq. (11) provides a Shapley-based marginal credit signal that improves credit assignment.

Significance. If the counterfactual credit mechanism were well-defined and the reported gains reproducible, SHARP would be a useful contribution: the paper targets a genuine bottleneck in multi-agent LLM RL, uses a shared-policy self-play formalism that is practical, and provides broad benchmark coverage plus scaling and stability analyses. The paper also ships a detailed algorithmic workflow (Algorithm 1) and identifies a concrete failure mode for harmful subagents in Appendix F, which is informative. However, the significance is currently conditional: the core credit signal is underspecified and, as written, does not match the Shapley formalism advertised in the title and abstract. Because the empirical gains are attributed to this signal (Figure 3), the correctness of the central claim cannot be assessed from the submitted manuscript.

major comments (4)
  1. [§4.3, Eq. (11) and Algorithm 1, line 13] The counterfactual trajectory τ_i^{\m} is never formally defined. Appendix C only says 'systematically masking the contributions of agent m' and provides no algorithm for constructing τ_i^{\m}. Appendix F confirms that the counterfactual is a separately generated rollout: in F.1 the planner states 'the tool's response was masked' and produces a different answer (85% vs. the full-trajectory 92%). This has two consequences. If the final answer is retained from the original trajectory, R_acc(τ_i^{\m}) = R_acc(τ_i) for any unchanged answer, making all worker credits zero and the method degenerate to broadcast reward. If the policy is rerun with the agent masked, then R_acc(τ_i) and R_acc(τ_i^{\m}) come from independent stochastic rollouts, and the difference is confounded by sampling noise; no temperature, seed, or repeated-sampling controls are reported. Since this signal enters the total r
  2. [§4.3, Eqs. (10)–(13)] Eq. (11) is a leave-one-out marginal contribution R_acc(τ_i) − R_acc(τ_i^{\m}), not the Shapley value shown immediately above it. The Shapley value requires averaging over all coalitions S with weights ω(S) = |S|!(|N|−|S|−1)!/|N|!. Eq. (13), which defines the planner credit as an average of positive worker credits, is further from Shapley. The paper calls this an 'approximation' but provides no approximation bound, no argument that leave-one-out is a reasonable Shapley surrogate in this setting, and no comparison with a direct Shapley estimate. Given that 'Shapley-based' is the paper's headline claim and the basis of the title and abstract, either implement Shapley averaging (which is feasible for the small number of workers in the case studies) or rename the method and temper the claims that the credit signal is 'mathematically grounded' in Shapley values.
  3. [§5.2, Table 1] The main empirical claim is that SHARP 'significantly outperforms' MATPO and other baselines, but Table 1 reports single numbers without error bars, number of seeds, or significance tests. The margin over the strongest multi-agent baseline MATPO is 1.75 average points (32.56 vs. 30.81), and per-benchmark margins vary widely (e.g., 50.76 vs. 47.00 on MuSiQue but 37.29 vs. 37.10 on FRAMES). The abstract's 23.66% and 14.05% gains are averages over heterogeneous baselines, and without variance information the 'significant' language is not supported. Please add multiple-seed results with confidence intervals or significance tests, or soften the claims accordingly.
  4. [§4.2, Eq. (9) and Appendix C] The tool-process reward R_tool uses a scalar function φ(·) described only as 'evaluates the validity and executability of each tool usage.' No definition of φ is provided anywhere, so the third component of the tripartite reward is not reproducible. Even though γ=0.1 in the reported configuration, a well-specified reward decomposition should define all of its terms. Please provide the exact mapping from (a_m,i,j, s_m,i,j) to φ.
minor comments (5)
  1. [§6] The section heading 'Conlucions' is a typo for 'Conclusions.'
  2. [§4.2] The three 'axiomatic principles' are not formal axioms but design desiderata. Consider renaming them 'design principles' to avoid overclaiming.
  3. [Appendix C] The sentence 'each agent is constrained toa single tool call per step' has a formatting typo; also clarify whether this applies to both planner and workers and whether the planner can call a worker more than once per subtask.
  4. [References] The reference 'CARD (Anonymous, 2026)' with a URL to an OpenReview forum page is unusual for a main-text citation and should be updated with author names if available.
  5. [§2 / Appendix C] The phrase 'systematically masking the contributions' in Appendix C is vague; if the counterfactual is a rerun, please state that explicitly there, not only implicitly via Appendix F.

Circularity Check

2 steps flagged · score 4.0 of 10

Marginal credit is built as a delta of the terminal accuracy, and RQ4 validates the method with that same training signal; the headline benchmark comparisons remain independent.

  1. self definitional [Section 4.3, Eq. (10)-(12); Algorithm 1 line 13]
    "we define the value function of a trajectory as v(τ) = R_acc(τ) ... Given a realized trajectory τ_i and a specific worker agent m ∈ M_i, we quantify its credit by credit_i,m = R_acc(τ_i) − R_acc(τ_i^{m}) ... Consequently, the marginal credit reward for worker agents is assigned as R_mc_i,m = credit_i,m."

    The claimed per-agent 'Shapley-based marginal credit' is, by Eq. (10)-(12), literally the difference of the binary terminal accuracy R_acc, the same signal already broadcast to all agents in Eq. (7) and used as the benchmark metric. Thus the credit is not an independently grounded attribution; it is a definitional rearrangement of the aggregate outcome. Whatever τ_i^{m} is, R_mc is a function of R_acc by construction, so the claim that this 'mathematically isolates each agent's causal impact' restates the definition rather than following from it.

  2. fitted input called prediction [Section 5.5 (RQ4), Figure 6]
    "We analyze coordination using Shapley-based credit signals, defining the planner score as the planner’s average Shapley value and labeling subagents with positive (negative) credit as useful (harmful)... SHARP reduced the proportion of harmful subagent calls ... from 5.48% to 4.40%."

    Algorithm 1 trains the policy with R_mc_i,m = credit_i,m inside R̄_i,m = αR_b + βR_mc + γR_tool, with β=0.9 (Appendix C). RQ4 then labels subagents 'useful' or 'harmful' by the sign of the same credit signal and reports that training lowered the 'harmful' fraction. This is not independent evidence: the evaluation metric is the training objective by definition, so the reported coordination improvement is the optimized quantity moving in the direction it was optimized to move.

full rationale

SHARP's headline accuracy gains (Table 1) are measured on held-out or zero-shot test sets (MuSiQue held-out; GAIA-text, WebWalkerQA, FRAMES, DocMath-Eval without training data), so the main performance claim is not circular. There is no load-bearing self-citation: the anonymous CARD citation is only a baseline, not a justification of SHARP's mechanism. The circularity is partial and localized: (1) Eq. (10)-(12) construct the marginal credit as a delta of the same binary R_acc used for broadcast reward and evaluation, so the credit signal has no independent grounding; and (2) RQ4 uses that same training-time credit to define 'useful/harmful' agents and then reports improvement in it, which is self-referential. In addition, Appendix F shows the counterfactual τ_i^{m} is a separately sampled rollout ('the tool’s response was masked' and the planner outputs a different number), so the causal-isolation claim is confounded by sampling noise; that is a validity problem rather than a circularity step. The Shapley label is also loose, since Eq. (11) is a leave-one-out delta rather than an average over coalitions. Weighing these, the central benchmark results remain empirically independent, so the score is 4 rather than 6+.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central method rests on five assumptions listed above, most importantly that ablation credit can stand in for Shapley and that the counterfactual rollout is well-defined. Free parameters α,β,γ,λ,δ,ϵ,G are hand-set; α,β,γ are explicitly tuned on the evaluated benchmarks. No new physical or conceptual entities are introduced; the 'counterfactual trajectory' is a construction within the method, not an independent entity.

free parameters (4)
  • reward weights (α, β, γ) = 0.9, 0.9, 0.1
    Appendix C: 'reward coefficients specifically tuned to α=0.9, β=0.9, γ=0.1.' These weight the three reward terms and directly set the optimization signal; no validation split is reported.
  • planner credit scale λ = not reported
    Eq. 13 defines planner credit as λ times the mean positive worker credit; λ is an unexplained free scale and its value is never given.
  • GRPO constants ϵ and δ = 0.2, 1e-6
    Hyperparameters for clipping and normalization are specified but hand-set; they affect the stability and magnitude of the per-agent advantage.
  • group size G (rollouts per query) = 8
    8 rollouts per input are used for group-relative normalization; this choice affects variance of the advantage estimates but is not justified.
assumptions (5)
  • ad hoc to paper Leave-one-out ablation credit in Eq. 11 is an acceptable approximation of the Shapley value
    Eq. 10 sets v(τ)=Racc(τ), and Eq. 11 uses only the grand-coalition marginal v(full)−v(without agent m). Shapley requires a weighted average over all coalitions; the paper asserts the approximation without proof.
  • domain assumption Counterfactual trajectories τ_i^{\m} can be generated by masking agent m while preserving other agents and interactions
    Appendix C and Algorithm 1 state masking but do not specify the generative procedure; the validity of the credit signal depends entirely on this.
  • domain assumption Binary accuracy Racc is a sufficient terminal value function for agent-level credit assignment
    Eq. 6 and Eq. 10 use only final-answer correctness, ignoring partial progress, tool quality, and answer length; all agent credits are derived from this binary signal.
  • domain assumption Shared policy with role-specific prompts creates distinguishable agent identities whose credits can be assigned separately
    §3.2 uses parameter sharing; credit is assigned per role, but all updates modify the same πθ, so the separation of 'agent-specific' gradients is conceptual rather than structural.
  • domain assumption Tool validity function ϕ(a^m, s^m) correctly scores execution quality
    Eq. 9 defines the tool-process reward through an unspecified scalar ϕ; no details or examples are given for how validity and executability are scored.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Who Deserves the Reward? SHARP: Shapley Credit-based Optimization for Multi-Agent System." pith.science (2026). https://pith.science/paper/Y4BNYRXE

@misc{pith2026260208335,
  author       = {Pith},
  title        = {Pith review of: Who Deserves the Reward? SHARP: Shapley Credit-based Optimization for Multi-Agent System},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y4BNYRXE}},
  note         = {Machine review of arXiv:2602.08335}
}
read the original abstract

Integrating Large Language Models (LLMs) with external tools via multi-agent systems offers a promising new paradigm for decomposing and solving complex problems. However, training these systems remains notoriously difficult due to the credit assignment challenge, as it is often unclear which specific functional agent is responsible for the success or failure of decision trajectories. Existing methods typically rely on sparse or globally broadcast rewards, failing to capture individual contributions and leading to inefficient reinforcement learning. To address these limitations, we introduce the Shapley-based Hierarchical Attribution for Reinforcement Policy (SHARP), a novel framework for optimizing multi-agent reinforcement learning via precise credit attribution. SHARP effectively stabilizes training by normalizing agent-specific advantages across trajectory groups, primarily through a decomposed reward mechanism comprising a global broadcast-accuracy reward, a Shapley-based marginal-credit reward for each agent, and a tool-process reward to improve execution efficiency. Extensive experiments across various real-world benchmarks demonstrate that SHARP significantly outperforms recent state-of-the-art baselines, achieving average match improvements of 23.66% and 14.05% over single-agent and multi-agent approaches, respectively.

Figures

Figures reproduced from arXiv: 2602.08335 by the authors.

Figure 1
Figure 1. Existing credit assignment policy for all agents (left) and the precise strategy of SHARP for each individual agent (right). 1. Introduction The evolution of Large Language Models (LLMs) has en￾abled a fundamental shift from static knowledge retrieval to dynamic and tool-augmented interactions in complex real￾world scenarios (Lewis et al., 2020; Wang et al., 2024). In￾tegrating LLMs with external tools via multi-age… view at source ↗
Figure 2
Figure 2. Overview of SHARP framework. The pipeline involves (a) hierarchical interaction between planner and worker agents via a shared policy; (b) tripartite reward system integrating global accuracy, marginal credit, and tool process rewards; (c) marginal credit mechanism isolating agents’ contribution via Shapley values; (d) SHARP workflow using group-relative policy for stable alignment. Multi-Agent Reward Assignment Rul… view at source ↗
Figure 3
Figure 3. Left: Ablation studies on MuSiQue and GAIA-text comparing full SHARP with variants that remove planner-level or worker￾level Shapley credit. Middle: The corresponding accuracy differences (∆ Accuracy) measured relative to the no-Shapley baseline on each benchmark. Right: Evaluation on DocMath-Eval across four document-level reasoning settings, including Simple-Short (SS), Simple-Long (SL), Complex-Short (CS), and Co… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Parameter scalability on MuSiQue from 0.6B to 8B. SHARP shows consistent improvement as the model size increases and achieves a larger advantage over the baselines at larger scales. Observation 2: Planning strategy and execution quality. As illustrated in [PITH_FULL_I…
Figure 5
Figure 5. Figure 5: Training-step scalability on GAIA-text from 0 to 180 steps. SHARP improves steadily as training progresses and avoids the instability observed in the baseline; shaded areas denote 95% confidence intervals. DocMath-Eval with four document-level reasoning tasks, in￾cludi…
Figure 7
Figure 7. Figure 7: Analysis of the cost v.s. performance trade-off across varying credit sparsification levels. We adjust the credit sparsification probability p, where a p-fraction of subagent invocations undergo Shapley-based marginal credit assignment, while the remaining (1 − p) are …

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Element-Aware Group Learning for E-Commerce Image Generation

    cs.CV 2026-08 conditional novelty 6.0 of 10

    EAGLE-GRPO uses kernel ridge regression to split GRPO image rewards across structured prompt elements, and reports better e-commerce image generation results than standard GRPO.

  2. Coalition Free Energy and Adaptive Precision in Multi-Agent Cooperation

    cs.GT 2026-05 unverdicted novelty 6.0 of 10

    The paper introduces the GT-FEP variational framework deriving a non-monotonic Shapley-value vs. precision relationship and proposes the APC adaptive algorithm that performs comparably to tuned fixed precision on real...

  3. From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models

    cs.CL 2026-04 unverdicted novelty 5.0 of 10

    A survey of credit assignment techniques in LLM reinforcement learning that distinguishes maturing methods for reasoning from new approaches needed for agentic settings and provides supporting resources.

  4. Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Traces

    cs.CL 2026-05 unverdicted novelty 4.0 of 10

    This survey organizes RL for LLM multi-agent systems into reward families, credit units, and five orchestration sub-decisions, notes the absence of explicit stopping-decision training in its paper pool, and releases a...

Reference graph

Works this paper leans on

20 extracted references · 1 linked inside Pith · cited by 4 Pith papers

  1. [8]

    •Identify what is missing

    Before each tool call: •Summarize what is known. •Identify what is missing. •Choose the most relevant tool. •Verify all required parameters

  2. [18]

    param1":

    If the task does not require tool use, answer directly. Agent Specific Objective You are a task-solving agent that uses tools step-by-step to answer the user’s question. Your goal is to provide complete, accurate, and well-reasoned answers using additional tools. 22 Who Deserves the Reward? SHARP: Shapley Credit-based Optimization for Multi-Agent System S...

    2025

  3. [19]

    Analyze the user’s request and set clear, achievable sub-goals

  4. [20]

    Start with a concise, numbered, step-by-step plan before taking any action

  5. [21]

    Work through these sub-goals sequentially and adjust the plan as needed

  6. [22]

    Use tools strategically to accomplish each sub-goal

  7. [23]

    Tool-Use Guidelines

    Revise earlier steps if new information emerges. Tool-Use Guidelines

  8. [24]

    Each step must involve a single tool call, unless the task is already solved

Show all 20 references
  1. [25]

    •Identify what is missing

    Before each tool call: •Summarize what is known. •Identify what is missing. •Choose the most relevant tool. 24 Who Deserves the Reward? SHARP: Shapley Credit-based Optimization for Multi-Agent System •Verify all required parameters

  2. [26]

    All tool queries must include full context

  3. [27]

    Each call should retrieve actionable information

    Avoid vague queries. Each call should retrieve actionable information

  4. [28]

    Tool-Use Communication Rules

    Extract and summarize partial information if a tool result is incomplete. Tool-Use Communication Rules

  5. [29]

    Do not include tool results in your response

  6. [30]

    Do not present the final answer until the entire task is complete

  7. [31]

    Do not mention tool names

  8. [32]

    Do not engage in unnecessary back-and-forth

  9. [33]

    Do not use non-existent tools

  10. [34]

    Respond in the same language as the user’s message

  11. [35]

    Agent Specific Objective You are an agent that performs the task of searching and browsing the web for specific information and generating the desired answer

    If the task does not require tool use, answer directly. Agent Specific Objective You are an agent that performs the task of searching and browsing the web for specific information and generating the desired answer. Your task is to retrieve reliable, factual, and verifiable inf...

  12. [2026]

    What percentage of the world’s Jews in 1931 were Ashkenazi Jews, representing the bulk of modern Jewry?

    URL https://openreview.net/forum? id=JgvJdICc6P. Chan, A., Wei, K., Huang, S., Rajkumar, N., Perrier, E., Lazar, S., Hadfield, G. K., and Anderljung, M. Infras- tructure for ai agents.arXiv preprint arXiv:2501.10114, 2025. Chen, H., Covert, I. C., Lundberg, S. M., and Lee, S.-...

    2025 arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.