REVIEW 4 major objections 5 minor 12 references
The paper claims that an agent's contribution in an LLM-based multi-agent system can be read off a single realized dialogue as semantic support logic, without counterfactual reruns.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 14:01 UTC pith:M5BVSKL3
load-bearing objection The semantic-support idea is genuinely interesting and the Shapley algebra is correct, but the paper's central reduction is a definitional identity, not a verified single-trajectory guarantee. the 4 major comments →
Semantic Cooperative Games for Contribution Attribution in LLM-Based Multi-Agent Systems
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
SCG/SLIC treats a workflow's language flow as the observable trace in which agents generate, preserve, or transform task-relevant semantic nodes. Starting from the value-bearing output nodes read by a rubric, SLIC traces backward along the dialogue, builds a semantic generation hypergraph with links owned by agents, extracts minimal semantic supports, applies Boolean absorption to strip redundant paths, and expands the remaining support logic into multilinear coefficients from which the Semantic Shapley Value is allocated analytically — no coalition is rerun. Theorem 1 states that under set-based, fully observable, order-independent conditions this induced value equals the classical coalitio
What carries the argument
The central object is the semantic generation hypergraph G=(S,L,N;α,ω,O,I) built from one realized trajectory: semantic nodes S connected by directed hyperedges L that record how each non-initial semantic node is produced from its predecessors, with an ownership map α assigning each link to an agent. Around this graph the paper wraps two operations that do the real work: Boolean absorption, which reduces the recovered support logic to inclusion-minimal agent subsets so that redundant rewrite paths receive no credit, and Möbius/multilinear expansion of the resulting support function, whose coefficients are allocated to agents by Shapley-style equal splitting. Because the value function is ind
Load-bearing premise
The load-bearing premise is the paper's Assumption 1: every value-relevant semantic node in the language trace must be classifiable as newly created, carried over, or derived, with its originating agent readable from one run — a premise the paper itself flags as its first limitation (Appendix B).
What would settle it
Take the same agent outputs expressed in two lexically different but semantically equivalent trajectories that give identical coalition values; SLIC's prediction is that the recovered semantic support logic — and hence the SSV profile — is the same for both. If the two runs produce different rankings, the source-identification premise is false.
If this is right
- Attribution stops requiring reruns: with SCG, one realized trajectory supplies the full value function, so contribution scores become cheap enough for routine auditing and pruning.
- Under set-based, fully observable, no-order-dependence workflows, SLIC's Semantic Shapley Value exactly equals the classical Shapley value (Theorem 1).
- On a controlled medical workflow, SLIC lowers attribution L1 error to 6.61 versus 15.08 for leave-one-out and raises Kendall rank agreement with the exact Monte Carlo reference to 0.814.
- In multi-role workflows, SSV profiles align with medium and strong perturbation-induced score drops (mean Spearman ρ = 0.868), and they flag privilege–capability mismatches where a low-contribution end-stage agent can still collapse the system.
- The 4-agent extension cuts corrected computation cost by 93.3% while keeping low attribution error, indicating the advantage is not limited to the 3-agent setting.
Where Pith is reading between the lines
- If the source-identification step can be hardened, SLIC makes per-run contribution auditing nearly free: any logged workflow transcript can be re-attributed without extra model calls, enabling dashboards that flag which agent's semantics supported each rubric hit.
- The paper's documented privilege–capability divergence suggests real systems may need two complementary scores — semantic contribution for credit, structural failure-sensitivity for blast radius — rather than one.
- The failure modes the authors list (OR-miss, polarity inversion, asymmetric penalty) are a ready-made stress test: run SLIC on paraphrased, negated, and reordered variants of the same semantic content; attribution stability under such perturbations is the property that would eventually decide how far the method generalizes.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Semantic Cooperative Games (SCG), a representation of an LLM multi-agent workflow trace as a semantic generation hypergraph, and defines the Semantic Shapley Value (SSV) on the Boolean support logic recovered from a single realized trajectory. The companion algorithm SLIC constructs this hypergraph via backward tracing from rubric output nodes, extracts minimal agent-level supports, applies Boolean absorption, and computes SSV without counterfactual reruns. The authors claim (Theorem 1, Eq. 6) that under set-based, fully observable, no-order-dependence conditions, SSV coincides exactly with the classical Shapley value of the underlying coalition game. Experiments on a HealthBench-based medical workflow report high consistency with an exact Monte Carlo Shapley baseline at 85.7% cost reduction, and experiments on four diagnostic multi-role workflows report Spearman alignment between SSV and perturbation-induced score-drop profiles. The paper includes a lengthy appendix with the derivation, worked examples, error analyses, and a NeurIPS-style checklist.
Significance. If the reduction claim and empirical validation held, the paper would offer a meaningful practical advance: single-trajectory, counterfactual-free Shapley-style attribution for LLM multi-agent systems, with a clear interpretable object (semantic support logic) rather than black-box reruns. The framework is cleanly defined, the Möbius-inversion allocation is standard, and the paper is honest about several limitations, including parser reliability and judge instability. It also ships code and reports concrete cost measurements, and the perturbation experiment is a genuinely external behavioral probe rather than a fit-then-predict cycle. However, the theoretical reduction is weaker than stated, and the empirical support for the central 'exact Shapley' claim is undermined by post-hoc exclusion of ambiguous rubric items, small samples, and an unvalidated semantic parser. The paper is therefore more convincing as a proposal for a new structural attribution signal than as a proof of exact equivalence to classical Shapley.
major comments (4)
- [Theorem 1 / Appendix F, Eq. (5)] Equation (5), V_sem_G(C)=v(C) for all C, is asserted but not established. Appendix F defines v_ok(C):=f_ok(1_C) from the recovered support logic and then defines v(C) as the weighted sum of these v_ok(C). With these definitions, Eq. (5) holds by construction and Theorem 1 reduces to the well-known fact that the Shapley value of any Boolean game can be computed from its Möbius coefficients. What must be shown is that the support function f_ok recovered from one realized trajectory equals the actual rubric-hit function under every coalition C, including coalitions whose successful paths are not visible in the single trace. The theorem's informal conditions ('set-based value reachability, full observability, no masking/order dependence') are never formally defined, and Assumption 1 merely asserts source identifiability. As written, the headline 'counterfactual-free exact Shapley' claim is u
- [§4.1.2, Appendix J.3.3, Checklist item 7] The main consistency result rests on a post-hoc filtered dataset: the paper removes rubric items with 2:1 judge disagreement from the main analysis, and Appendix J.3 reports that 22.33% of rubrics have such disagreement for at least one subset. This exclusion is acknowledged and motivated, but it directly affects the measured agreement with MC-GT, since the oracle v(S) itself is unstable on the removed items. Combined with the small sample (50 cases) and the checklist's own admission that error bars are 'not particularly meaningful,' the reported L1=6.61 and Kendall tau_b=0.814 cannot support the strong claim that SLIC 'matches' classical Shapley. The paper should either report results on the full rubric set, quantify the sensitivity of the headline metrics to the de-ambiguation rule, or explicitly restrict the claim to judge-stable rubric items throughout the abstract and conclusion.
- [Appendix I, Appendix J.2, Micro Hit Rate] The semantic extraction step is the bridge between the realized trajectory and the support logic, yet it is not validated as a component. The parser is an 'independent LLM side-channel' with a text-to-graph transform Phi described only informally (Appendix I). In the controlled HealthBench experiment, SLIC's Micro Hit Rate is 0.857 (150/175), with documented OR-miss and polarity-inversion errors (Appendix J.2). Each such error changes Möbius coefficients and hence SSV. Because the central claim is that SSV is exact and counterfactual-free, the paper needs either a gold-standard evaluation of the parser's recall/precision on support-structure recovery, or a formal statement of conditions under which the recovered logic is guaranteed to equal the true support logic. The current Appendix B limitation paragraph concedes this is the first limitation, but the main text's claims do not carry th
- [§4.2, Table 3] Experiment 2's alignment evidence is suggestive but quantitatively thin. Each Spearman rho is computed on only five agents (A–E), the four settings share the same workflow skeleton, and no confidence intervals or significance tests are reported. The weak-perturbation mean rho=0.612 is modest, and the paper's own structural analysis explains why LOO/C3-style interventions can diverge. This experiment supports 'SSV tracks strong perturbation profiles in healthy workflows' as an exploratory finding, but it should not be presented as strong evidence for the general diagnostic value of SSV without either more agents per workflow, more diverse workflow topologies, or interval estimates.
minor comments (5)
- [Throughout] The manuscript contains several unfinished or inconsistent editorial artifacts: 'Code is avaiable' (Abstract), a stray 'zAcross' (Section 4.2.3), and multiple references to 'the thesis' in appendices (e.g., J.1, K.1, K.2) that appear to be leftovers from a dissertation. These should be cleaned.
- [Appendix F] The proof itself is standard and correct for the game it defines, but it should be relabeled as a derivation of the Möbius/S安宁 Shapley formula for a Boolean game, rather than a proof of equivalence between two independently defined value functions.
- [Checklist] The NeurIPS checklist answers are incomplete or internally inconsistent: 'Claims' says yes without justification, 'Theory assumptions and proofs' says yes despite the gap above, and 'Experiment statistical significance' says no. The paper should align the checklist with the actual status of the theoretical and empirical claims.
- [§4.1.3, Table 1] The cost-reduction numbers for SLIC and the Holistic LLM Judge are identical (85.7%), which is expected since both use one pass, but the table does not state that SLIC's semantic analysis adds a separate LLM parser call beyond the rubric judge; the API-call accounting should be explicit about which calls are included.
- [Appendix K.3.3] The 'privilege–capability misalignment' discussion is interesting, but it is presented after the fact and not formalized. A precise definition (e.g., a measure of structural sensitivity vs. semantic contribution) would make the claim testable in future work.
Circularity Check
Theorem 1's 'reduction to classical Shapley' is definitional: Appendix F constructs the classical game from the recovered support logic, so Eq. (5) is an identity, not an independent derivation.
specific steps
-
self definitional
[Theorem 1 (Eqs. 5–6); Appendix F, proof of Theorem 1]
"Under the set-based, fully observable, and no-masking / no-order-dependence conditions stated in the theorem, whether the node ok is hit depends only on the subset of participating agents. Hence fok can be naturally viewed as a Boolean characteristic function on the coalition space 2N : for any C⊆N, let 1C ∈ {0,1}|N| be its indicator vector and define the coalition hit function vok(C) := fok(1C). ... Finally, aggregate over all activated output nodes. Let v(C) := X ok∈O:x ok (G)=1 ω(ok)v ok (C), C⊆N."
Theorem 1 claims Eq. (5), V_sem_G(C)=v(C), i.e. that the semantic induced game equals an externally given classical coalition value function, and then concludes SSV_i(G)=φ_i(v). But in the proof, v_ok is defined as f_ok(1_C), where f_ok is the support logic recovered from the single trajectory, and v is then defined as the weighted sum of these v_ok. Thus Eq. (5) is true by construction: the 'classical' game is the semantic value function induced by the recovered Boolean support logic, not an independently evaluated coalition game. No step demonstrates that f_ok(1_C) equals the actual rubric hit under coalition C; that bridge is merely delegated to the informal 'full observability / no-masking' conditions. The algebraic part of the proof (Shapley of a Boolean game equals the Möbius allocat
full rationale
The paper is not a fit-then-predict cycle: in Experiment 1, MC-GT scores actual agent subsets and SLIC's SSV is computed from the realized trajectory with no parameters fitted to the MC values, and Experiment 2 uses perturbation-induced score drops as an external behavioral probe. There is also no load-bearing self-citation chain. However, the central theoretical claim in Theorem 1—that SSV reduces exactly to the classical Shapley value—is partially circular as written. Appendix F defines the coalition hit function v_ok from the recovered support logic f_ok and then defines v as the weighted sum of those v_ok; therefore Eq. (5), V_sem_G(C)=v(C), is an identity rather than a proof that the recovered support logic matches the true rubric value function. The equivalence between the single-trajectory support logic and the actual coalition hit function is exactly what the unformalized 'full observability' and 'no masking' assumptions would need to supply, but the proof never establishes it. The paper's own Appendix B concedes this dependency as the first limitation, and Appendix J.2 documents concrete recovery errors (OR-miss, polarity inversion) that shift SSV. Thus the method has substantial independent empirical content, but the exact-Shapley reduction claim is largely definitional with respect to the recovered Boolean game, warranting partial circularity rather than a clean bill of health.
Axiom & Free-Parameter Ledger
axioms (5)
- domain assumption Assumption 1: Every value-relevant semantic node is native, derived, or persistent, and if derived/persistent its prior source in the input state can be identified.
- domain assumption Value decomposes into weighted Boolean rubric items: v(z)=Σ_k w_k r_k(z) with r_k ∈ {0,1}.
- ad hoc to paper An LLM side-channel parser can reliably extract the text-to-graph transform Φ (nodes, links, ownership, absorption-relevant redundancy).
- domain assumption Theorem 1 conditions: set-based value reachability, full observability, no masking or order dependence along the realized language flow.
- standard math Shapley axioms and Möbius inversion are accepted as the allocation standard.
invented entities (3)
-
Semantic generation hypergraph G=(S,L,N;α,ω,O,I)
independent evidence
-
Semantic Shapley Value (SSV) / semantic support logic
independent evidence
-
Privilege–capability misalignment (structural privilege vs semantic contribution)
no independent evidence
read the original abstract
Contribution attribution has become a central problem in LLM-based multi-agent systems, where final outputs are produced through multiple agents, message exchanges, and ordered workflow dependencies. Existing attribution methods often rely on counterfactual valuation, such as removing agents or comparing score changes across altered agent subsets. In language-mediated workflows, these methods require repeated model calls, introduce high variance, and do not explicitly capture the intermediate semantic states through which agents produce, preserve, and transform task-relevant information. We propose Semantic Cooperative Games (SCG), a framework that represents a realized language flow as a semantic generation hypergraph and induces an agent-level semantic value function on this structure. We define the Semantic Shapley Value (SSV) to allocate contribution over semantic support logic, and introduce SLIC, a single-trajectory algorithm that constructs the semantic hypergraph, recovers minimal semantic supports, applies Boolean absorption, and computes SSV without rerunning agent subsets. We prove that SSV reduces to the classical Shapley value under standard set-based, fully observable, and no-order-dependence conditions. On a medical benchmark satisfying these conditions, SLIC reduces computation cost by 93.3% while remaining highly consistent with a Monte Carlo Shapley baseline. In more general multi-role workflows, SSV aligns with perturbation-induced score-drop profiles and exposes cases where semantic contribution and failure impact diverge. Overall, SLIC provides a fast, counterfactual-free, and interpretable attribution method for complex LLM-based multi-agent systems.
Figures
Reference graph
Works this paper leans on
-
[1]
similar disease
Rubric boundaries are not hard enough.Criteria such as “similar disease” or “overly technical” lack a fully executable boundary and easily drift into semantic gray zones
-
[2]
based on the current symptoms,
Identify link ℓ2:from the trigger phrase “based on the current symptoms,” take the existing symptom node s2 together with the related sign node s1 as input and point to output s3, thus generating the directed linkℓ 2 = ({s1, s2},{s 3})
-
[3]
weight is only 50kg,
Identify link ℓ4:from the factual dependency “weight is only 50kg,” take the sign node s1 as input and point to outputs 4, thus generating the linkℓ 4 = ({s1},{s 4}). In other words, generating semantic links is itself part of an agent’s behavior. After traversing the global log, the semantic transformation events that actually occurred form the following...
-
[4]
The mitigation strategy adopted in the paper is therefore to apply 3-run majority voting to each rubric and remove all items with2:1disagreement from the main analysis
There is an upper bound on judging consistency.Even with temperature=0, slight dif- ferences in generation path and semantic alignment across API calls can still flip a binary verdict. The mitigation strategy adopted in the paper is therefore to apply 3-run majority voting to each rubric and remove all items with2:1disagreement from the main analysis. The...
-
[6]
For implicit mentions, partial hints, or conditional phrasing, the judge threshold becomes extremely unstable
Negative rubrics and conditional triggers are highly sensitive.Penalty items often depend on whether a violation is triggered. For implicit mentions, partial hints, or conditional phrasing, the judge threshold becomes extremely unstable
-
[7]
Subset concatenation changes the context.The same rubric evaluated on A|B versus A|B|C is presented under different context lengths and information coverage, which shifts the LLM’s attention distribution and introduces combinatorial sensitivity noise
-
[9]
We first examine the overall correlation between SSV and system loss under ordinary pertur- bations, to verify whether this semantic contribution signal can capture node importance at all
-
[10]
We then analyze performance differences across nodes at different positions, with special attention to whether late-stage nodes are systematically underestimated
-
[11]
work” has already been completed by upstream nodes, “how the final output is presented
Finally, we introduce amplified hallucination perturbations, that is, explicit disruptive factors injected into the target agent’s prompt, in order to distinguish whether the observed bias comes from the attribution definition itself or from structural imbalance in the workflow. This mirrors the logic of the thesis: establish aggregate validity first, the...
-
[15]
Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects
Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or ...
-
[1989]
doi: 10.2307/1911054. 11 Sirui Hong et al. MetaGPT: Meta programming for a multi-agent collaborative framework. In International Conference on Learning Representations, 2024. Ehud Kalai and Dov Samet. On weighted shapley values.International Journal of Game Theory, 16 (3):205–222, 1987. doi: 10.1007/BF01756292. Guohao Li, Hasan Abed Al Kader Hammoud, Hani...
Pith/arXiv arXiv 2024
-
[2024]
doi: 10.18653/v1/2024.acl-long.810. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?": Explaining the predictions of any classifier. InProceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135–1144, 2016. Lloyd S. Shapley. A value for n-person games. In H. W. Kuhn and A. ...
Pith/arXiv arXiv 2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.