REVIEW 4 major objections 4 minor 10 references
Who Gets the Reward & Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agents
T0 review · 4 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read A single system-level score for a multi-LLM team can be decomposed into signed, credit-conserving message-level rewards and repair-aware preferences, giving each agent local supervision without losing cooperation.
desk verdict A clear, honest conceptual framework that unifies Shapley credit, PRM refinement, and repair-aware failure preferences; the algebra is sound but the load-bearing counterfactual simulation and two unproven claims need real work before the training signals can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Shapley value over a coalition game where v(S) is the evaluator score of a simulated trajectory with only agents in S active and all others replaced by a fixed baseline policy. Coalition counterfactuals are simulated by replaying the original trace until the first turn of a removed agent, then regenerating with frozen seeds. On the success route, message-level allocation uses weights ω_i,t = 1/k_i + λ(s_i,t − s̄_i) that sum to one, so credit is conserved exactly; on the failure route, a prefix judge performs O(log T) binary search to locate the first error, and a failure-alignment judge labels later messages as repair attempts or failure-aligned.
What would settle it
Run the same multi-agent task twice: once letting the full team complete it, and once actually removing one agent (or replacing it with a no-op) and observing the real outcome. Compare the real ablation score to the simulated coalition score v(S) used in the Shapley computation; if the two disagree systematically across tasks, the credit assignment is unfaithful. A precise test would fix one agent, evaluate several tasks, and check whether the simulated-vs-actual score difference is within a small tolerance.
Extended reading notes
Core claim
The paper's central claim is that global evaluation can be bridged to local supervision in multi-LLM systems through two complementary routes. On the success route, each agent's credit is its Shapley value in a cooperative game whose value function is the evaluator score of a simulated run with only a subset of agents active; this credit is then refined into per-message rewards using signed alignment labels, and the weights are constructed so that per-message rewards sum exactly to the agent's credit and the agent credits sum exactly to the system score. On the failure route, rather than dividing a zero reward, the framework localizes the first harmful message and constructs repair-aware pre
Load-bearing premise
The load-bearing assumption is that replaying a trace with some agents replaced by baseline outputs faithfully simulates what the system would have produced if those agents had been absent—if those simulated coalitions diverge from reality, every Shapley credit and therefore every message reward is an artifact of the simulation.
Editorial extensions
If this is right
- If the framework is correct, a single system-level evaluator score can provide dense per-message training signals for all agents, removing the need for hand-labeled step-level rewards in multi-agent LLM systems.
- The success route naturally discourages redundancy and sabotage: duplicated work yields near-zero Shapley marginal credit, and harmful behavior lowers the total pool instead of increasing one's own share.
- The failure route avoids the common pitfall of monotone first-error PRMs by rewarding later repair attempts, so agents can learn corrective behavior from failed episodes.
- All signals are bounded and sum exactly to the system score, meaning no credit inflation, free-riding, or runaway incentive can arise from the decomposition itself.
- Because the outputs are formatted as dense rewards and preference pairs, standard post-training methods such as RLHF, DPO, and GRPO can consume them without architectural changes.
Reading between the lines
- The most testable extension is to compare actual leave-one-out runs against the simulated coalition scores: if the simulation is faithful, the per-message rewards inherit that fidelity; if not, the entire success-route credit is an artifact of the simulator.
- The framework's reliance on a judge for per-message alignment labels suggests a natural sensitivity study: measure how much training outcome degrades as judge label noise increases, since credit conservation holds regardless but the direction of credit shifts with mislabels.
- Because the evaluator is treated as a black box bounded to [0,1], the same machinery could be applied to any measurable system outcome—execution success, rubric scores, or user satisfaction—making the pathway potentially generic across task domains.
- A concrete empirical validation would be to train a small multi-agent system on a benchmark task using only these signals and compare improvement against single-agent-style outcome-only RL; the paper leaves this as future work.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes a theoretical framework for converting a single system-level evaluation score of a multi-LLM agent system into per-message, signed, credit-conserving training signals. In the success route, a cooperative game is defined over subsets of agents via a coalition simulator, Shapley values yield agent-level credits, and a PRM-style judge produces signed message-level alignment labels that redistribute each agent's credit across its messages. In the failure route, a binary search localizes the first harmful message, and repair-aware preference pairs are constructed for DPO/GRPO-style training. The paper states formal-looking guarantees of credit conservation, boundedness, anti-competition, and repair-awareness, and positions the work as a conceptual contribution with empirical validation left to future work.
Significance. If the framework's assumptions hold, the paper would provide a useful unification of cooperative game-theoretic credit assignment with process reward modeling for multi-LLM systems. The algebraic construction is transparent and reproducible, the author's distinction between success and failure regimes is sensible, and the explicit acknowledgment that this is a conceptual contribution is honest. However, the practical value hinges critically on the fidelity of the coalition simulator and on the monotonicity of the prefix judge, neither of which is established. The 'proofs' advertised in the introduction are partly definitional identities and partly informal assertions, so the theoretical contribution needs to be sharpened before the central claims can be accepted.
major comments (4)
- [§4.1.1, Eqs. (2)–(3)] The coalition value v(S) is defined through simulated outputs y_S, obtained by replaying the original trace until the first turn of a removed agent and then regenerating with frozen seeds. The entire success route—every Shapley value φ_i and every message reward r_{i,t}—is computed from these simulated counterfactuals. The paper never validates that this replay approximates the true absence counterfactual (running the system without the agent from the start), nor gives conditions under which it does. If the simulated y_S diverges from the actual counterfactual, φ_i measures the simulator, not the MAS, and the central claim that system evaluation is transformed into agent credit collapses. This is load-bearing and must be addressed: either prove a robustness/invariance property of the Shapley ranking under this replay approximation, or include an empirical validation protocol in the plann
- [§5, 'Anti-competition' guarantee] The 'Anti-competition' bullet is asserted without proof. Shapley efficiency, symmetry, and dummy-player axioms do not, by themselves, guarantee that duplicating another agent's work yields near-zero credit or that sabotage cannot increase an agent's own share. In games with redundancy, duplication can increase the probability of at least one successful outcome and hence can carry positive marginal value; in some games a player's marginal contribution can rise when the total pool shrinks. A formal statement with explicit conditions on the coalition value function (e.g., submodularity, concavity, or specific structural assumptions) is needed, or the claim should be presented as an informal intuition rather than a theorem. Since 'preserving cooperation' is a headline contribution, this is not a cosmetic issue.
- [§4.2, First-error localization] Binary search for t* = min{t : J_pref(H_t)=ERR} requires J_pref to be prefix-monotone: if a prefix is ERR, all later prefixes are also ERR. The text explicitly says 'we do not assume monotone traces' and argues that later repair messages should be judged in context, which is at odds with binary search. If J_pref can return OK after an ERR prefix, binary search is not guaranteed to find the first error. The manuscript must either define a monotone prefix predicate and justify it, or replace the binary search with a linear scan or a predicate-aware search that can handle non-monotone judgments. This is load-bearing for the failure route's preference construction.
- [§4.1.2, Eqs. (4) and (5)] Equations (4) and (5) define α_i and r_i, and Eq. (11) follows algebraically from the definitions of the weights and the efficiency of Shapley values. Credit conservation is therefore a built-in identity of the construction, not an empirical or game-theoretic discovery. The paper should stop presenting 'we prove credit conservation' as a separate substantive guarantee; the substantive question is whether φ_i correctly reflects the agent's true marginal contribution, which returns to the simulator-fidelity issue in §4.1.1. This is a framing concern, but it matters because the paper's contribution list explicitly promises 'proofs' of these properties.
minor comments (4)
- [§4.1.2, Complexity] In the complexity paragraph, 'PRM labeling: a pass over messages to obtain s_i,t and ω_i,t is O(T) time and o(T) space' appears to be a typo; it should be O(T) space, not little-o.
- [§4.2, First-error localization] The statement 'credits must sum to zero, which blurs responsibility' in the failure-route motivation is slightly misleading: when R_sys=0, Shapley credits are all zero, which gives no signal, not merely a blurred one. The intended point is clear but could be phrased more precisely.
- [§4.1.1, 'Simulating coalitions'] The term 'frozen seeds' is undefined. If the seeds are shared across all simulated coalitions, the Monte Carlo variance may be underestimated; if they are resampled, the comparison across coalitions may be noisy. This should be clarified.
- [§4.1.2, 'Clipping and normalization'] The optional clipping of φ_i or r_i,t to [−1,1] changes the total credit pool, so it no longer satisfies the exact conservation identity (11). The text says it does not change 'relative proportions'; please state explicitly that clipping is applied before the conservation guarantee is invoked, or that conservation is approximate after clipping.
Circularity Check
Self-contained conceptual framework; the credit-conservation guarantee is a definitional identity, but no prediction reduces to its inputs.
-
self definitional
[Sec. 4.1.1, Eqs. (4)-(5); Sec. 5 'Theoretical guarantees']
"For interpretability, we also define the credit ratio α_i ≜ φ_i / Σ_j φ_j = φ_i / R_sys ... Finally, the reward for agent i is r_i ≜ α_i · R_sys = φ_i, so that Σ_i r_i = R_sys."
The credit-conservation guarantee is not derived from evidence; it is installed by definition. Equation (4) defines α_i as φ_i/R_sys, and Eq. (5) defines r_i as α_i R_sys, so r_i is identically φ_i. Summing then uses only Shapley efficiency and the definition v(A) = score(E_A(x, y_A)) = R_sys, so Σ_i r_i = R_sys is a tautology of the construction. This does not invalidate the framework's attribution content, but presenting it as a proven property overstates a definitional identity.
full rationale
The paper is a conceptual framework with no fitted parameters and no empirical predictions; its central objects are defined, not estimated. The claimed guarantees—credit conservation, boundedness, anti-competition, repair-awareness—follow algebraically from the definitions: φ_i is the Shapley value of the coalition game v(S) = score(E_S(x, y_S)), r_i is defined to equal φ_i via α_i = φ_i/R_sys, and ω_i,t is constructed to sum to one. These are identities of the construction, not empirical discoveries. The only substantive tautology is the credit-conservation guarantee in Eqs. (4)-(5), which is a minor definitional overstatement rather than a load-bearing circularity: the framework's actual content lies in choosing the coalition game and the PRM-style signal refinement, neither of which is derived from the conclusion it claims. There are no load-bearing self-citations: the 'earlier version' note is a priority claim, not a cited result, and the Shapley axioms are standard external mathematics. The replay-based coalition simulation in Sec. 4.1.1 is unvalidated—if the simulated y_S diverges from true counterfactual absence, the Shapley credits may measure the simulator rather than the system—but that is an empirical validity concern, not circularity, and the paper explicitly defers experiments to future work (Abstract, Sec. 6).
Assumptions & free parameters
free parameters (2)
- lambda (alignment strength) =
range [0, 1/2), no fitted value
- M (Monte Carlo permutations) =
unspecified; 'a few hundred samples suffice' (Appendix A.3)
assumptions (5)
- domain assumption System evaluator E returns bounded score R_sys in [0,1] with score(fail)=0 and score(success(r))=r
- domain assumption Counterfactual coalition outputs y_S can be faithfully simulated by replaying the trace until the removed agent's first turn and regenerating with frozen seeds plus baseline outputs
- domain assumption A domain-tuned judge J can label each message s_i,t in {-1,0,+1} as aligned/counter-aligned/neutral with the agent's contribution direction
- ad hoc to paper The prefix judge J_pref is monotone enough for binary search to locate the first error via t* = min{t: J_pref(H_t)=ERR}
- standard math Shapley value axioms (efficiency, symmetry, dummy) and uniqueness
Cite this review
Pith. "Pith review of Who Gets the Reward & Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agents." pith.science (2026). https://pith.science/paper/G3I74FDU
@misc{pith2026251110687,
author = {Pith},
title = {Pith review of: Who Gets the Reward & Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agents},
year = {2026},
howpublished = {\url{https://pith.science/paper/G3I74FDU}},
note = {Machine review of arXiv:2511.10687}
}
read the original abstract
Large Language Models (LLMs) in multi-agent systems (MAS) have shown promise for complex tasks, yet current training methods lack principled ways to connect system-level evaluation with agent- and message-level learning. We propose a theoretical framework that unifies cooperative game-theoretic attribution with process reward modeling to transform system evaluation to agent credit to response-level signals. Unlike prior approaches that rely only on attribution (Shapley) or step-level labels (PRM), our method produces local, signed, and credit-conserving signals. In success cases, Shapley-based credit assignment fairly allocates outcomes across agents and is refined into per-message rewards that promote cooperation while discouraging redundancy or sabotage; in failure cases, first-error localization yields repair-aware preferences that penalize harmful steps while rewarding corrective attempts. The resulting signals are bounded, cooperative, and directly compatible with reinforcement- or preference-based post-training, providing a unified and auditable pathway from global evaluation to local supervision in LLM multi-agent training. Our contribution is conceptual: we present a theoretical foundation and training signals, leaving empirical validation for future work.
Reference graph
Works this paper leans on
-
[1]
Signed, credit-conserving rewards.In success episodes, each message receives a signed reward ri,t with X t∈Ti ri,t =ϕ i, X i,t ri,t =R sys ∈[0,1], so supervision isbudgetedby the realized outcome and already shaped like a reward func- tion—making RL-style optimization natural
-
[2]
Multi-agent grounding via Shapley.All message signals are scaled by the agent’s Shapley credit ϕi, aligning step-level learning with each agent’smarginal contributionto system performance
-
[3]
A.2.1 PLUGGING THE SIGNALS INTO POST-TRAINING Success route (RL-style).Use {ri,t} as per-message rewards for each agent policy πi
Failure-aware preferences.When Rsys = 0, we localize the first harmful message and con- struct contrastive pairs Ht⋆−1, y+, y− whilestill rewardingsubsequent repair attempts—unlike monotone-invalidating schemes that mark all post-error steps invalid. A.2.1 PLUGGING THE SIGNALS INTO POST-TRAINING Success route (RL-style).Use {ri,t} as per-message rewards f...
2009
-
[6]
Hanhan Zhou, Tian Lan, and Vaneet Aggarwal
URLhttps://proceedings.mlr.press/v258/zhao25c.html. Hanhan Zhou, Tian Lan, and Vaneet Aggarwal. Pac: Assisted value factorization with counterfactual predictions in multi-agent reinforcement learning. InAdvances in Neural Information Processing Systems (NeurIPS), volume 35, pp. 15757–15769,
-
[7]
URL https://proceedings.neurips.cc/paper_files/paper/2022/ hash/65338cfb603d4871a2c38e53a3e039c9-Abstract-Conference.html. A APPENDIX A.1 RELATEDSHAPLEYLITERATURE For completeness, we summarize additional strands of work where Shapley values have been applied across economics, political science, and machine learning. Foundations in cooperative game theory...
2022
-
[2013]
Jacob Menick, Victoria Krakovna, Lawrence Chan, Michael Laskin, et al
URLhttps://arxiv.org/abs/1306.4265. Jacob Menick, Victoria Krakovna, Lawrence Chan, Michael Laskin, et al. Teaching language models to support answers with verified quotes.arXiv preprint arXiv:2203.11147, 2022. Sumeet Ramesh Motwani, Chandler Smith, Rocktim Jyoti Das, Rafael Rafailov, Ivan Laptev, Philip H. S. Torr, Fabio Pizzati, Ronald Clark, and Christ...
arXiv 2022
-
[2020]
Eric Zelikman, Yuhuai Wu, Noah D
URLhttps://proceedings.mlr.press/v108/yue20a/yue20a.pdf. Eric Zelikman, Yuhuai Wu, Noah D. Goodman, and Maxwell Nye. Star: Bootstrapping reasoning with reasoning. InAdvances in Neural Information Processing Systems (NeurIPS), 2022. URL https://arxiv.org/abs/2203.14465. Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. ReST-MCTS*...
arXiv 2022
-
[2022]
URLhttps://arxiv.org/abs/2203.02155. Joshua Owotogbe. Assessing and enhancing the robustness of llm-based multi-agent systems through chaos engineering.arXiv preprint arXiv:2505.03096, 2025. Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavi...
arXiv 2025
Show all 10 references
-
[2023]
Tabish Rashid, Mikayel Samvelyan, Christian De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson
URLhttps://arxiv.org/abs/2305.18290. Tabish Rashid, Mikayel Samvelyan, Christian De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. InInternational Conference on Machine Lear...
2018 arXiv
-
[2025]
Javier Castro, Daniel G´omez, and Juan Tejada
URLhttps://arxiv.org/abs/2505.20417. Javier Castro, Daniel G´omez, and Juan Tejada. Polynomial calculation of the shapley value based on sampling.Computers & Operations Research, 36(5):1726–1730, 2009. doi: 10.1016/j.cor.2008. 04.004. Mert Cemri, Melissa Z Pan, Shuyi Yang, Lak...
2009 arXiv
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.