{"id":"fe4b968c-fcaf-4857-878a-e83dfc3062b4","arxiv_id":"2506.11285","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Shapley Machine reshapes rewards and TD targets so per-agent value functions approximately satisfy the Shapley axioms, and it beats the prior POAM baseline in several NAHT test environments.","lead":"The paper derives a multi-agent reinforcement learning algorithm that assigns credit to individual agents by interpreting each agent's learned value as a Shapley value from cooperative game theory. It links this credit to the classical TD(lambda) method and tests it in open teams with unknown and varying teammates.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Assumption 4.2, the hinge of Prop 4.6, is numerically inconsistent: each basis-game value at the grand coalition equals V(Ñ,s), so the assumed RHS is q·V(Ñ,s), not an n-step return; the derivation of Eq. (6) needs a concrete check before the Shapley claim can stand.","rationale":"The paper has two separable claims: (a) Shapley Machine is a practical NAHT algorithm that often matches or beats POAM; (b) its learned V_i are Shapley values because the algorithm enforces Efficiency, Additivity, and Symmetry. Claim (a) is supported by the experiments, the ablations, and the released code, and I do not contest it. Claim (b) is the paper's headline theoretical contribution and depends on constructing a cooperative game whose Shapley value is computed by TTD(λ). The bridge is Assumption 4.2, which maps basis games ordered by coalition size to n-step returns. The reader's weakest_assumption identifies exactly this, and I agree. My stress-test sharpens the point: the assumption is not merely unverified; its statement is internally inconsistent under the paper's own Definition 2.1. Every basis game v^z_{C,s} evaluates to z on any superset D, and in particular on the grand coalition Ñ; because the paper rewrites z=V(Ñ,s), each term on the RHS of Assumption 4.2 is V(Ñ,s). Summing same-size coalitions gives q·V(Ñ,s), whereas the LHS is an expected return of scale V(Ñ,s). The 1/q(i) rescaling in Eq. (6) cannot repair this, because the same V(Ñ,s_t) is then asserted to equal different n-step returns for different i. One could imagine a repaired assumption (e.g., defining basis games through marginal contributions rather than carrier games), but that repair is not in the paper and would change the derivation. Therefore Proposition 4.6, and the central claim that enforcing the three axioms on a TD target yields Shapley values, is currently unsupported. The empirical law m≈number of basis games is interesting and the ablations are a good-faith check, but the result is consistent with TTD(λ) being a good heuristic and does not by itself validate the equality. The verdict should remain CONDITIONAL, with the explicit condition being a valid derivation or a direct empirical test of Assumption 4.2.","tokens_in":25102,"tokens_out":7278,"duration_ms":84990,"concrete_test":"Instantiate Assumption 4.2 on a minimal NAHT environment (e.g., two controlled agents plus one uncontrolled agent in a small finite MDP with known transition and reward). Compute V(Ñ,s_t) by dynamic programming and E[G_{t:t+1}|s_t] exactly. Compare with the RHS of Assumption 4.2 for the size-1 basis games: Σ_{i:p..q} v^{V}_{C_i,s_t}(Ñ)=q·V(Ñ,s_t). If the two sides differ, as expected, the derivation of Eq. (6) and Prop 4.6 is unsupported; then re-derive Eq. (6) without Assumption 4.2, or replace it with a stated identity linking the explicit Shapley weights 1/|C| of basis games to n-step returns. This single test discriminates between a harmless exposition gap and a missing theorem.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is Assumption 4.2 (Sec. 4.1), which converts the cooperative-game basis into the n-step-return sequence used to build the TTD(λ) target (Eqs. 6-11). The assumption is not just unproved; as written it cannot hold generically. For the basis games in Definition 2.1, v^{v_s(Ñ)}_{C,s}(Ñ)=v_s(Ñ)=V(Ñ,s) for every nonempty C⊆Ñ. Hence the RHS of Assumption 4.2, Σ_{i=p}^q v(...)(Ñ), equals q·V(Ñ,s) with q the number of same-size coalitions, whereas the LHS E[G_{t:t+k}|s_t] is on the order of V(Ñ,s). For k=1 the equality would require R_t+γE[V(s_{t+1})]=qV(Ñ,s), which is false for generic MDPs (and if V>0, the RHS generally exceeds the LHS). The next sentence then divides by q(i) and assigns different n(i) to different coalition sizes, yielding Eq. (6), in which V(Ñ,s_t) is simultaneously equal to different n-step returns; these equalities are mutually incompatible unless all those returns coincide. Since Eqs. (9)-(11) and hence Proposition 4.6 inherit this step, the axiomatic characterization of V_i as Shapley values is not currently derived. Appendix B supplies only a variance heuristic ('more agents require more samples') and does not establish the equality. This is a correctness risk, not a disagreement with consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a game-theoretic framework for n-agent ad hoc teamwork (NAHT). It represents each NAHT process as an element of a direct-sum vector space of cooperative games over states, generated by constant-valued basis games indexed by coalitions. Under Assumption 4.2, which maps these basis games to a sequence of expected n-step returns, the paper derives a truncated TD(λ)-like target, defines per-agent shaped rewards from an Efficiency condition, and proposes Shapley Machine, an algorithm claimed to learn Shapley-value credit allocations by satisfying Efficiency, Additivity, and Symmetry. Experiments on MPE and SMAC benchmarks compare Shapley Machine with POAM and include ablations and horizon-m studies.","tokens_in":25538,"tokens_out":5283,"duration_ms":54552,"significance":"If the central derivation were correct, the paper would offer a novel, principled link between cooperative game theory (Shapley value) and TD(λ)-style credit assignment in open multi-agent systems, and would provide a useful axiomatic design template. The paper is commendable for releasing code, for attempting to derive the algorithm from axioms rather than from an explicit Shapley formula, and for including ablation studies. However, the core theoretical claim rests on an assumption that, as stated, is numerically inconsistent with the definition of basis games, and the implementation weakens the Efficiency axiom; these issues need to be resolved before the significance claim can be accepted.","major_comments":[{"comment":"The assumption cannot hold for the basis games defined in Definition 2.1. For every nonempty C⊆Ñ, v^{vs(Ñ)}_{C,s}(Ñ)=vs(Ñ)=V(Ñ,s). Therefore the right-hand side of the assumed equality, Σ_{i=p}^q v^{vs(Ñ)}_{C_i,s}(Ñ), equals q·V(Ñ,s), where q is the number of coalitions of that size, while the left-hand side Eπ[G_{t:t+k}|s_t] is a discounted return whose expected magnitude is V(Ñ,s). For k=1 the equality would require R_t+γE[V(s_{t+1})]=qV(Ñ,s), which is generically false. Dividing by q(i) and assigning different horizons n(i) to different coalition sizes produces Eq. (6), which states that the same V(Ñ,s_t) equals different n-step returns; these equalities are mutually incompatible unless all those returns coincide. Appendix B offers only a variance heuristic, not a proof. Because Eqs. (7)–(11) and Proposition 4.6 inherit this step, the axiomatic characterization of V_i as Shapley values is not currently derived.","section":"Section 4.1, Assumption 4.2 and Eq. (6)"},{"comment":"The proof assumes the per-agent Bellman allocation ϕ_i(V(Ñ,s_t)) = ϕ_i(R_t) + γϕ_i(V(Ñ,s_{t+1})) without deriving it from the dynamics. This identity is precisely the n-step-return allocation whose validity is at issue; R_{t,i} is then defined by solving for ϕ_i(R_t), and Eq. (13)/(23) is imposed as a regularization to enforce the resulting decomposition. The Shapley conclusion is therefore circular with respect to Assumption 4.2: the allocated Bellman equation is assumed, not proved.","section":"Proposition C.2 and Eq. (12)"},{"comment":"The implementation introduces a coefficient α∈(0,1) in R_{t,i}=R_t−αΣ_{j≠i}(V_j(s_t)−γV_j(s_{t+1})). This means Σ_i R_{t,i}≠R_t in general, so the Efficiency axiom is not exactly satisfied by the shaped rewards. Similarly, Eq. (23) is enforced only as a soft regularization term (the efficiency loss), not as a hard constraint. Proposition 4.6's statement that Shapley Machine 'fulfills Efficiency' is therefore not supported by the implemented algorithm; at most one can say the algorithm is biased toward Efficiency.","section":"Appendix D.1.1, Eq. (22)"},{"comment":"The experiments show that m=20 and m=31 give similar performance in 5v6, and m=7 is best in MPE, but exact equality with the number of nonempty coalitions is not established; m=20 versus m=31 is a counterexample to the claim that the optimal horizon is exactly the number of basis games. In 8v9 and 10v11, m is set to the episode length rather than to the theoretical number of basis games, so Figure 15 demonstrates only a monotone empirical relationship, not the theory's prediction. The empirical results are suggestive but should be framed as approximate or additional evidence, not as verification of Assumption 4.2.","section":"Section 5.2 and Figure 5a"}],"minor_comments":[{"comment":"The phrase 'multinational distribution' should be 'multinomial distribution'.","section":"Appendix D.4"},{"comment":"The theorem cites reference [18], but the statement as given ('given an additively factored immediate reward function...') is not located in that reference; please provide a precise citation or proof.","section":"Theorem 4.5"},{"comment":"The proof of Proposition 4.6 is a single sentence and does not engage with the concrete implementation details in Appendix D; it should be expanded to show each axiom is satisfied by the actual loss functions.","section":"Appendix C.4"},{"comment":"Eq. (13) is introduced as a condition in the main text and later used as a regularization loss; the relation between the two should be stated explicitly.","section":"Eq. (13) and Appendix D.1.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's claim to be the first to directly relate cooperative game theory to RL concepts is stronger than warranted given reference [42] and the long line of Shapley-value MARL work; the novelty lies in the axiomatic/TD(λ) connection. The central assumption needs a concrete numerical test before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the empirical algorithm—a POAM variant with a shaped per-agent reward and truncated TD(lambda)—is a solid, well-ablated contribution to n-agent ad hoc teamwork. Second, the mathematical derivation that the learned values are Shapley values has a load-bearing flaw: Assumption 4.2 is numerically inconsistent as written.\n\nWhat's new: the state-space cooperative game model, the idea of setting the TTD(lambda) horizon to the number of basis games, and shaping rewards through other agents' value differences rather than an explicit Shapley formula. Experiments on MPE and SMAC (5v6, 3sv5z, 8v9, 10v11) show Shapley Machine matches or beats POAM, and the ablations show each component contributes. Code is public.\n\nThe soft spot: Assumption 4.2 states that basis-game values correspond to a sequence of n-step returns. But for the grand coalition, v^{v_s(N)}_{C,s}(N)=v_s(N) for every nonempty C, so the RHS of the assumption is q·V(N,s), not an n-step return, which is of order V(N,s). That cannot hold generically; Eq. (6) and the TTD(lambda) target inherit the issue. Appendix B offers only a variance heuristic, not a proof. Proposition 4.6 is one sentence. The implementation further weakens Efficiency with alpha<1, and Eq. (13) is a soft regularizer. So the strong claim that V_i are Shapley values is not currently derived.\n\nThat said, this is an addressable flaw, not a dead end. The empirical gains do not depend on the strong claim, and the axiomatic framing is a promising lens. The paper is honest about its limits, including manual tuning of m and weak performance on large-scale tasks.\n\nFor readers in MARL credit assignment or NAHT, this is worth discussing and citing if the authors revise. I would send it to review with a request to fix or weaken the central claim. If they cannot fix Assumption 4.2, they should cut the Shapley claim and present it as a practical credit-shaping scheme.","headline":"A useful empirical POAM variant, but the Shapley-value derivation has a load-bearing numerical error; worth peer review if the authors fix the math or soften the claim.","tokens_in":26046,"tokens_out":2860,"would_cite":false,"duration_ms":30649,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["91A12","91A06"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that per-agent value functions trained with a TD(λ)-like target that satisfies Shapley's Efficiency, Additivity, and Symmetry axioms are Shapley-value credit allocations in dynamic n-agent ad hoc teamwork, and derives the…","keywords":["Shapley value","ad hoc teamwork","multi-agent reinforcement learning","credit assignment","temporal difference learning","cooperative game theory","open multi-agent systems"],"falsifier":"In a small NAHT task (e.g., three controlled agents) after training, fix the joint policy, enumerate all non-empty coalitions, compute each agent's explicit Shapley value from the corresponding state values, and compare with the $V_i$ learned by Shapley Machine; a systematic mismatch would falsify Proposition 4.6, while a direct test of Assumption 4.2 would compare $\\mathbb{E}_\\pi[G_{t:t+k}\\,|\\,s_t]$ with the summed basis-game values for coalitions of the matching size.","tokens_in":24879,"feed_emoji":"🤝","tokens_out":12271,"duration_ms":114496,"temperature":0.7,"pith_summary":"The paper sets out to give n-agent ad hoc teamwork (NAHT), where only some agents in an open team are controlled, a principled credit-assignment theory. It models an NAHT process as a state-dependent vector space of cooperative games generated by basis games, then shows that requiring per-agent value functions to satisfy the three Shapley axioms turns a $\\mathrm{TD}(\\lambda)$-like return into a game-theoretic credit target. The resulting Shapley Machine learns each controlled agent's value as its Shapley value, so the reward signal each agent optimizes reflects its contribution to the ad hoc team. If the theory holds, heuristic reward shaping is replaced by an axiomatically grounded scheme, and the truncation horizon in $\\mathrm{TD}(\\lambda)$ should track the number of non-empty coalitions of the team.","feed_headline":"Shapley Machine: fair credit for teammates you cannot control","feed_subtitle":"A new RL algorithm derives Shapley values from TD(lambda)-style returns, splitting team rewards among controlled agents.","key_machinery":"The carrying object is the state-space cooperative game space $\\mathcal{G}_{\\mathrm{NAHT}} = \\bigoplus_{s\\in S}\\mathcal{G}(s)$, where each fixed state $s$ has a vector space generated by basis games $v^1_{C,s}$: cooperative games that pay one unit to every coalition containing $C$ and zero otherwise, so any subgame decomposes as $v_s = \\sum_{C\\subseteq \\tilde{N}} k_C\\, v^1_{C,s}$. Assumption 4.2 is the bridge: it identifies basis games ordered by coalition size with expected $n$-step returns $\\mathbb{E}_\\pi[G_{t:t+k}\\,|\\,s_t]$, and the geometric weights of truncated $\\mathrm{TD}(\\lambda)$ supply the coefficients $k_C$. Efficiency fixes each agent's reward share $R_{t,i}$; Additivity converts the target into a weighted $\\lambda$-return; Symmetry is realized by giving identical agent types identical inferred teammate representations, so they receive equal value estimates.","core_discovery":"The paper's central claim is Proposition 4.6: Shapley Machine fulfills Efficiency, Additivity, and Symmetry, so it learns $V_i$ as Shapley values for dynamic scenarios. The derivation builds a state-space cooperative game model $\\mathcal{G}_{\\mathrm{NAHT}} = \\bigoplus_{s\\in S}\\mathcal{G}(s)$, decomposes each state's subgame into basis games $v^1_{C,s}$ indexed by coalitions $C$, and imposes Assumption 4.2, which identifies the basis-game values ordered by coalition size with expected $n$-step returns of matching horizons. This identification turns the Shapley axioms into recursions on returns, yielding a truncated $\\mathrm{TD}(\\lambda)$ update whose horizon $m$ equals the number of basis games and whose per-agent reward is $R_{t,i} = R_t - \\sum_{j\\neq i}(V_j(s_t) - \\gamma V_j(s_{t+1}))$. The paper further argues that the earlier POAM algorithm satisfies only Additivity and Symmetry, making it a Banzhaf Machine rather than a Shapley Machine, and reports MPE and SMAC experiments where Shapley Machine matches or outperforms POAM.","pith_inferences":["If Assumption 4.2 is the right bridge between game values and n-step returns, then any MARL algorithm training critics with truncated λ-returns is implicitly choosing a cooperative-game decomposition, and its solution concept can be identified by which Shapley axioms its target satisfies.","A direct empirical check is available: in a small NAHT task, fix the learned joint policy, enumerate all coalitions, compute explicit Shapley values from the resulting state values, and compare them with the $V_i$ learned by Shapley Machine; agreement would confirm Proposition 4.6, while systematic mismatch would localize the failure to Assumption 4.2 or the relaxed efficiency loss.","The framework suggests an adaptive variant that learns the basis-game weights $k'_C$ from data instead of fixing them by a geometric distribution, a direction the paper itself flags as future work and which would extend the method beyond superadditive games.","The paper's vector-space picture implies a geometric measure of openness bias: the angle between the restricted game vector (controlled agents only) and the full game vector quantifies how much credit assignment is distorted by ignoring uncontrolled agents."],"forward_implications":["Per-agent critics trained by Shapley Machine are Shapley values, so policy gradients act on a credit signal that reflects each controlled agent's marginal contribution to the team's state value.","The TD(λ) target acquires a game-theoretic reading: its truncation horizon should equal the number of non-empty coalitions of the ad hoc team (7 for a 3-agent team, 31 for a 5-agent team), matching the paper's experiments where m=7 beats longer horizons on MPE.","POAM, the existing NAHT baseline, is classified as a Banzhaf Machine: it satisfies Additivity and Symmetry but not Efficiency, so it learns Banzhaf indices rather than Shapley values.","In the tested MPE and SMAC NAHT tasks, Shapley Machine converges faster and reaches lower critic loss than POAM, with test performance better in three scenarios and matching in the remaining two.","Under superadditivity the basis coefficients satisfy $k_C \\geq 0$, which is consistent with the geometric distribution used to set the TTD(λ) weights."],"supporting_citations":[{"why":"Defines the NAHT problem and supplies the POAM baseline and benchmark tasks used in the experiments.","marker":"[6]"},{"why":"Proves that Shapley value is the unique payoff allocation satisfying Efficiency, Additivity, and Symmetry on the cooperative game space, the theorem Proposition 4.6 relies on.","marker":"[16]"},{"why":"Introduces TD(λ) and the λ-return, the RL target whose truncated form carries the game-theoretic values.","marker":"[10]"},{"why":"Defines truncated TD(λ), used by Shapley Machine with horizon equal to the number of basis games.","marker":"[21]"},{"why":"Connects superadditive cooperative games to cooperative MARL and supports the $k_C \\geq 0$ condition for the game class.","marker":"[11]"},{"why":"Provides the Dec-POMDP framework and the decomposability result used to justify realizing the Efficiency axiom.","marker":"[18]"},{"why":"Gives the original definition of the Shapley value for n-person games, the solution concept the algorithm aims to learn.","marker":"[43]"}],"fun_headline_variants":["Shapley Machine: fair credit in open multi-agent teams","Shapley Machine: game theory meets RL for ad hoc teamwork","Shapley Machine: Shapley values from TD(lambda) returns","Shapley Machine: cooperative game theory for agent credit"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the values of basis games ordered by coalition size line up exactly with expected $n$-step returns of matching horizons (Assumption 4.2); if that mapping fails, the learned values are a weighted TD target but not Shapley values.","fun_headline_variants_meta":{"raw":{"variants":["Shapley Machine: fair credit in open multi-agent teams","Shapley Machine: game theory meets RL for ad hoc teamwork","Shapley Machine: Shapley values from TD(lambda) returns","Shapley Machine: cooperative game theory for agent credit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000311,"raw_usage":{"total_tokens":1866,"prompt_tokens":1132,"completion_tokens":734,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":748,"completion_tokens_details":{"reasoning_tokens":661}},"tokens_in":748,"tokens_out":734,"duration_ms":7458,"temperature":1.0,"reasoning_tokens":661,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:12:02.566737+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In a small NAHT task (e.g., three controlled agents) after training, fix the joint policy, enumerate all non-empty coalitions, compute each agent's explicit Shapley value from the corresponding state values, and compare with the $V_i$ learned by Shapley Machine; a systematic mismatch would falsify Proposition 4.6, while a direct test of Assumption 4.2 would compare $\\mathbb{E}_\\pi[G_{t:t+k}\\,|\\,s_t]$ with the summed basis-game values for coalitions of the matching size.","supporting_citations":[{"cited_title":"N-agent ad hoc teamwork.Advances in Neural Information Processing Systems, 37:111832– 111862, 2025","cited_arxiv_id":null,"evidence_quote":"Defines the NAHT problem and supplies the POAM baseline and benchmark tasks used in the experiments."},{"cited_title":"On the uniqueness of the shapley value.International Journal of Game Theory, 4(3):131–139, 1975","cited_arxiv_id":null,"evidence_quote":"Proves that Shapley value is the unique payoff allocation satisfying Efficiency, Additivity, and Symmetry on the cooperative game space, the theorem Proposition 4.6 relies on."},{"cited_title":"Truncating temporal differences: On the efficient implementation of td (lambda) for reinforcement learning.Journal of Artificial Intelligence Research, 2:287–318, 1994","cited_arxiv_id":null,"evidence_quote":"Defines truncated TD(λ), used by Shapley Machine with horizon equal to the number of basis games."},{"cited_title":"Shapley q-value: A local reward approach to solve global reward games","cited_arxiv_id":null,"evidence_quote":"Connects superadditive cooperative games to cooperative MARL and supports the $k_C \\geq 0$ condition for the game class."},{"cited_title":"A value for n-person games.Annals of Mathematics Studies, 28:307–318, 1953","cited_arxiv_id":null,"evidence_quote":"Gives the original definition of the Shapley value for n-person games, the solution concept the algorithm aims to learn."}],"review_version":1}