{"id":"4babb564-44a8-4616-bb88-a82bd61fc37f","arxiv_id":"2608.08604","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"MAGPIE converts agent-specific preference feedback into per-agent reward models and a summed global reward, with a claimed epsilon-Nash equilibrium guarantee.","lead":"Researchers introduce MAGPIE, a multi-agent reinforcement learning method where each agent is trained from preference feedback given by its own dedicated teacher, instead of a single hand-designed global reward. The paper claims this decentralized preference scheme can converge to a Nash equilibrium while matching reward-engineered baselines on benchmark tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Affine identifiability per agent breaks the reward-aggregation claim: Theorems 2 and 4 require exact or common reward scales that Theorem 3 rules out, so the Nash-equivalence result is unproven.","rationale":"The paper's central claim is that optimizing decentralized agent-specific preferences converges to a Nash equilibrium and that optimizing the sum of learned local reward models is equivalent to training that equilibrium. The reader identifies Assumption 4 as the weakest assumption; I agree it is strong and untested, since preference data in the experiments come from deterministic rules built from the same quantities as the ground-truth reward. However, the more decisive problem is internal to the theorem chain. Theorem 3 explicitly states that preferences identify each local reward only up to a per-agent affine transformation, while Theorem 2 silently requires exact equality between learned and true local reward sums. Theorem 4 then assumes a shared affine scale and offset across all agents without deriving it from Theorem 3. This is not a matter of external realism; it is a mathematical inconsistency between the paper's own identifiability result and its aggregation theorems. Because the scale ambiguity is per teacher and the losses in Eq. 17 are trained independently for each agent, nothing forces the α_i and β_i to match. With mismatched scales, the aggregate learned reward can order joint policies differently than the true sum of local rewards, so maximizing it need not improve the true global objective. The ε-Nash conclusion is further aimed at an artificial rescaled objective rather than at the true reward r, so it cannot support the advertised equivalence. A minimal finite-MDP counterexample with per-agent reward scalings would settle whether the aggregation step is salvageable, but as written the theorems do not deliver the central claim. The experimental section cannot compensate because the preference teachers are hand-built from reward components, which assumes away the very alignment problem Assumption 4 is supposed to guarantee. Verdict remains REJECT.","tokens_in":26143,"tokens_out":8404,"duration_ms":99581,"concrete_test":"Run an exhaustive tabular counterexample search in a finite two-agent MDP: choose true local rewards r_1, r_2 and transitions satisfying Assumptions 4, 5, and 7, with global reward r = r_1 + r_2. Generate Bradley-Terry preferences from r_1 and r_2, then solve Eq. 17 to obtain affine-equivalent reward models with deliberately different scales, e.g., r_hat_1 = 2r_1 and r_hat_2 = r_2. Check whether any policy maximizing J under r_hat = r_hat_1 + r_hat_2 is not a Nash equilibrium of r. A single such instance refutes Theorem 2; if no instance exists over a broad grid, the aggregation step may still be true, but the provided proof would still need repair.","verdict_should_be":"REJECT","load_bearing_attack":"The load-bearing gap is in the reward-modeling half of the central claim. Theorem 3 (Eq. 20) shows that each local reward model r_hat_i is identifiable from agent i's preferences only up to an agent-dependent affine transformation: r_hat_i ≈ α_i r_i + β_i with per-agent α_i, β_i. But Theorem 2's proof needs the much stronger statement, used after Eq. 14, that 'the discounted sum of r_hat_j over a segment equals R_j(σ)' — exact equality of learned and true local rewards. Preference equivalence cannot certify equality; it only fixes each reward up to its own scale and offset. Consequently the step from Σ_j R_j(σ') > Σ_j R_j(σ*) to Σ_j R_hat_j(σ') > Σ_j R_hat_j(σ*) is invalid: with α_i ≠ α_j, the sum of differently scaled rewards can rank two joint policies in the opposite order of the unscaled sum even when every individual r_hat_i ranks segments like r_i. Theorem 4 attempts to absorb the ambiguity by assuming a single shared α and β in Eq. 31, but Theorem 3 provides no coupling among agents' affine constants: each teacher is independent and each reward model is trained with its own loss. With per-agent constants, the error bound in Eq. 33 does not follow, and the 'true preference objective' R_true(s,A) = α Σ_i r_i(s,A) + nβ is not shown to coincide with the global reward r of Assumptions 4 and 6. Thus an equilibrium under Σ_i r_hat_i, or an ε-Nash under R_true, does not transfer to r. Even granting Assumption 4 in full, the aggregation theorem and the finite-sample ε-Nash theorem are unsupported as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MAGPIE, a multi-agent reinforcement learning framework that replaces a hand-engineered global reward with agent-specific preference signals. Each agent has a dedicated teacher who compares trajectory segments; the paper learns per-agent reward models from these preferences via a Bradley-Terry objective, aggregates them by summation, and then trains a QMIX-style policy with the aggregated reward. The theoretical claims are that iterative local-preference updates converge to a Nash equilibrium (Theorem 1), that maximizing the summed learned rewards yields a Nash equilibrium (Theorem 2), that each learned reward model is identifiable up to an affine transformation (Theorem 3), and that with finite preference data the resulting policy is an ε-Nash equilibrium (Theorem 4). An appendix gives a stochastic-approximation convergence proof (Theorem 5). Experiments on three MPE tasks and a sequential production-line task compare MAGPIE against QMIX with the true task reward, with additional ablations and scaling results.","tokens_in":26583,"tokens_out":4509,"duration_ms":50600,"significance":"If the theoretical claims were sound, the paper would offer a practical alternative to reward engineering in heterogeneous cooperative MARL, and the release of source code plus the breadth of experiments would make it a useful contribution. The paper also correctly identifies limitations of existing multi-agent preference-based methods, such as global trajectory evaluation and cross-agent comparability. However, the central theoretical guarantees are not established: Assumption 4 makes local preferences equivalent to global-return comparisons, turning the convergence theorems into properties of global-reward optimization, and the per-agent affine ambiguity in Theorem 3 breaks the reward-aggregation arguments in Theorems 2 and 4. The empirical evaluation uses rule-based teachers constructed from the reward components, so the experiments do not test the framework under the noisy or genuinely local preference models that motivate the work.","major_comments":[{"comment":"Assumption 4 states that σ' ≻_i σ holds if and only if the expected true global return under (π'_i, π_−i) exceeds that under (π_i, π_−i). This makes each teacher's local preference a perfect indicator of global-return improvement. Theorem 1 then proves that monotone improvement of the global return terminates at a Nash equilibrium, which is a standard property of global-reward optimization and does not demonstrate that decentralized preferences can replace global reward information. If Assumption 4 is relaxed so that teachers evaluate only an agent's local contribution or are noisy, the proofs of Theorems 1, 2, and 4 lose their foundation, yet the paper never tests such relaxation; the experimental teachers are deterministic rules built from reward components and thus satisfy Eq. 3 by construction.","section":"Section IV-A, Assumption 4 (Eq. 3)"},{"comment":"The proof of Theorem 2 needs the exact equality 'the discounted sum of \\hat r_j over a segment equals R_j(σ)', but Theorem 3 guarantees only \\hat r_i = α_i r_i + β_i with per-agent constants. With α_i ≠ α_j, the sum Σ_j \\hat R_j(σ) can rank two joint policies in the opposite order of Σ_j R_j(σ) even if each individual \\hat r_i ranks segments like r_i. The step from the inequality for R_j to the inequality for \\hat r_j is therefore invalid, and the conclusion that a maximizer of \\hat r is a Nash equilibrium for the true reward r is unproven. Theorem 4 (Eq. 31) assumes a single shared α and β across agents, but Theorem 3 provides no coupling among the per-agent affine constants, so the ε-Nash guarantee relative to R_true(s,A) = α Σ_i r_i(s,A) + nβ does not transfer to the true global reward r.","section":"Section IV-A, Theorem 2 (Eqs. 14-16) and Theorem 3 (Eq. 20)"},{"comment":"The finite-sample proof has several gaps. Strict properness of the cross-entropy loss gives equality of preference distributions at the population minimizer, but the passage from L_µ(\\hat r_{ψ*}) → L_µ(r*_i) to uniform convergence of \\hat r_{ψ*} to r*_i is asserted rather than proved. The later L2 bound in Eq. 30 is said to follow from 'strong convexity of L_µ in a neighborhood of r*_i', but the strong convexity is not established and the logistic loss is not strongly convex on an unbounded reward class without additional constraints. The Rademacher complexity R_N(F) is invoked without specifying the function class F used by the neural reward model in Eq. 17, so the sample-complexity bound in Eq. 21 cannot be evaluated as stated.","section":"Section IV-B, Theorem 3 proof (Eqs. 25-30)"},{"comment":"Assumption 7 is a condition on the true local rewards r_i, but the only mechanism proposed to enforce it is the soft regularization loss L_i^reg in Eq. 37. No theorem shows that minimizing L^reg guarantees Eq. 11, and the regularization uses the learned models \\hat r_i, which are only affine-identified with respect to the true r_i. Consequently, the aggregated objective is not shown to satisfy the premise of Theorem 2, and the claim that the regularization 'promotes' Assumption 7 is not backed by a quantitative statement.","section":"Section IV-C, Eq. 37 and Assumption 7 (Eq. 11)"},{"comment":"The convergence result rests on assumptions that the optimal global Q-function lies in the decomposable subspace Q_d and that the projection Π is non-expansive; these are stated without verification and are not properties of the implemented QMIX algorithm, which uses a nonlinear monotonic mixing network. The proof also asserts that the projected noise ΠM_{k+1} remains a martingale difference sequence without checking the required measurability and integrability conditions. These gaps weaken the appendix's claim that the practical MAGPIE procedure converges to an ε-Nash policy.","section":"Appendix A, Theorem 5 (Eqs. 40-49)"}],"minor_comments":[{"comment":"The sentence 'proving that the n-to-nexpert-agent preference mechanism' contains a typo; it should read 'n-to-n expert-agent' or similar.","section":"Contributions, page 2"},{"comment":"The notation R_N(F) is used without defining the function class F or the norm used in the L2(μ) error; please specify F (e.g., the neural network family with bounded weights) and the measure μ explicitly.","section":"Theorem 3, Eq. 21"},{"comment":"Assumption 5 is stated for a true local reward r_i, but Theorem 2 says the learned \\hat r_i satisfies Assumption 5; this conflation of true and learned rewards should be clarified, as it is central to the affine-ambiguity issue.","section":"Assumption 5 vs. Theorem 2"},{"comment":"The dependency graph references 'C1', but the text labels the result as Corollary 1; the labels should be made consistent.","section":"Figure 2"},{"comment":"The hyperparameter table lists segment length H as 3/5/10/20, but the MPE experiments use H = 3, 5, 10 and the production task uses H = 5, 10, 20; please make the per-task choices explicit.","section":"Section V-A and Table I"},{"comment":"Algorithm 1 does not specify what happens when the teacher is indifferent (p = 0.5), which Assumption 1 allows; a tie-breaking or no-update rule should be stated.","section":"Algorithm 1, line 9"}],"recommendation":"reject","confidential_remarks":"The manuscript's main theoretical result is undermined by a circular premise (Assumption 4 equates local preference with global-return comparison) and by the per-agent affine ambiguity in Theorem 3 that invalidates the aggregation arguments in Theorems 2 and 4. The experiments use teacher rules derived from the same reward components as the oracle, so they do not validate the method under the noisy or local-only preference models that motivate the paper. These issues appear to be fundamental to the framework as presented, rather than local fixable gaps, and I therefore recommend rejection. The work may be salvageable if the authors reformulate the assumptions (e.g., provable aggregation under a coupling condition on affine scales) and test with noisy or human-like teachers, but that is a substantial revision beyond the current scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on 2608.08604. The main thing you should know is that this is basically an extended version of the authors' own 2025 CAC paper (their ref [13]) — same name, same method, same core claims. The arXiv version adds a finite-sample analysis and an appendix convergence proof, but the new theory has a load-bearing gap. Theorem 3 only identifies each local reward up to a per-agent affine transformation; Theorem 2 then treats the learned rewards as exactly equal to the true local rewards, and Theorem 4 quietly assumes a single shared α and β for all agents. With per-agent scales, the sum of learned rewards can rank joint policies differently from the sum of true rewards, so the Nash-equivalence and ε-Nash claims are not proven as written. The stress-test note is right: this isn't a minor technical fix, it's the hinge of the paper.\n\nThe paper does some things well. The problem framing is sensible: having a dedicated teacher per agent avoids the cognitive overload of global trajectory evaluation and sidesteps the invalid cross-agent comparisons that plague current methods. The experiments are fairly thorough — ablations, comparisons to preference-based baselines, a production-line task, and the code is released. Within the narrow setting where teachers are deterministic rules built from the same quantities as the ground-truth reward (so the assumptions hold by construction), MAGPIE matches QMIX. That's a useful proof-of-concept, but it doesn't validate the reward-free promise, since the teachers are effectively reward components in disguise.\n\nOther soft spots: the assumptions do a lot of heavy lifting. Assumption 4 equates local preference with global-return comparison — that's essentially the conclusion and Assumption 7 assumes that local improvements don't hurt the team. The finite-sample part of Theorem 3 is also sloppy: strong convexity of the loss is asserted, the function class is unspecified, and the L2 bound doesn't follow from the displayed uniform-convergence argument.\n\nWho is this for? Someone working on preference-based MARL will find the agent-specific decomposition and the experimental setup worth reading. But as a submission, it's not ready. The authors need to fix the aggregation theory (e.g., per-agent normalization or a scale-invariant aggregation), test with noisy or human teachers, and clearly state the relationship to their CAC paper. I'd desk reject the current version, but I'd tell the authors exactly what to fix; there's a decent journal paper underneath.","headline":"The Nash-equivalence claim is unproven because per-agent reward scalings aren't coupled, and the core method already appeared in the authors' 2025 CAC paper; still, the problem framing and experiments are useful.","tokens_in":27074,"tokens_out":4684,"would_cite":false,"duration_ms":51242,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that per-agent preference feedback, aggregated by summing learned local reward models, provably trains cooperative multi-agent teams to a Nash equilibrium without any global reward function.","keywords":["multi-agent reinforcement learning","preference-based reinforcement learning","agent-specific preferences","Nash equilibrium","reward learning","monotonic aggregation","cooperative MARL"],"falsifier":"Run the paper's production-line task with a teacher whose preference rule depends only on the agent's own stage, for example the first-stage agent prefers segments that empty its own queue fastest, then sample 10,000 segment pairs and count how often the teacher prefers a segment whose true global return is lower. If the violation rate is positive and the regularizer does not remove it, the premise of Theorems 1 and 2 is violated; the paper does not report such a test with human or noisy teachers.","tokens_in":25896,"feed_emoji":"🤖","tokens_out":8282,"duration_ms":76035,"temperature":0.7,"pith_summary":"The paper introduces a method for training cooperative multi-agent systems without a global reward function. Instead of one reward for the whole team, each agent has a dedicated teacher that compares trajectory segments and states which behavior is better for that agent. The paper proves that if each teacher's preference is exactly aligned with the true global return, repeatedly updating agents along teacher-preferred directions reaches a Nash equilibrium. It then learns one reward model per agent from those comparisons, sums them into a global reward, and proves that maximizing this summed reward is equivalent to training a Nash equilibrium policy, with an epsilon-Nash guarantee under finite preference data. Experiments on benchmark coordination tasks and a three-stage production-line task show performance comparable to a baseline trained with hand-designed ground-truth rewards.","feed_headline":"Per-agent preferences reach Nash equilibrium, no rewards needed","feed_subtitle":"Summing learned per-agent preference rewards provably yields equilibrium policies and matches reward-engineered baselines.","key_machinery":"The load-bearing object is the per-agent logistic preference-reward model. Each agent $i$ has a reward model $\\hat r_\\psi^i$; the probability that segment $\\sigma^{(1)}$ is preferred over $\\sigma^{(0)}$ is the softmax of the discounted sums of $\\hat r_\\psi^i$ over the two segments, and the model is trained by cross-entropy on preference pairs. These local rewards are aggregated additively, $\\hat r(s,A)=\\sum_i \\hat r_\\psi^i(s,A)$, a monotonic combination whose choice is justified by Lemma 1 and Assumption 7. The identity that carries the theory is Theorem 2: any joint policy maximizing expected return under the summed learned rewards is a Nash equilibrium for the true global reward, because a teacher-preferred local improvement is, by assumption, a global-return improvement. A regularizer penalizes locally preferred segments that reduce the summed reward, guarding Assumption 7.","core_discovery":"The paper's central claim is that a cooperative multi-agent team can be trained entirely from agent-specific preference feedback, with no global reward function, and still provably reach a Nash equilibrium. Under Assumption 4, where each teacher's pairwise preference between trajectory segments is true exactly when the expected global return improves, the paper proves that iteratively adopting teacher-preferred unilateral policy changes terminates at a Nash equilibrium. It then shows that fitting one reward model per agent to those preferences and summing them, $\\hat r(s,A)=\\sum_i \\hat r_i(s,A)$, yields a global reward whose optimal policies are Nash equilibria of the true reward (Theorem 2); with finite preference data the guarantee degrades to an $\\epsilon$-Nash equilibrium with $\\epsilon=2n\\epsilon_r(N)/(1-\\gamma)$ (Theorem 4). The paper's experiments assert that this preference-only pipeline reaches performance comparable to a baseline trained on hand-crafted ground-truth rewards across three benchmark coordination tasks and a three-stage production-line scheduling task.","pith_inferences":["Extension: a natural follow-up not pursued by the paper is to replace Assumption 4's strict 'if and only if' with a margin or probabilistic alignment, which would yield preference-based equilibrium bounds for noisy or systematically biased teachers.","Extension: additive aggregation is sufficient but not necessary; learning state- or role-dependent aggregation weights could relax Assumption 7's condition that one agent's gain always outweighs any collective loss.","Extension: because the convergence arguments assume finite policy spaces and perfect preference consistency, extending the method to continuous policies would require an additional exploration or consistency mechanism.","Extension: the theory suggests a practical teacher-auditing test before deployment: check whether the teacher's local preferences correlate with global-return ordering on held-out segment pairs; the regularizer is the stated safeguard for violations, but its strength is a single scalar hyperparameter."],"forward_implications":["Preference feedback can fully replace global reward engineering in cooperative MARL: the learned aggregate reward comes with a Nash-equilibrium guarantee, so no task-specific reward formula is needed.","Any centralized-training MARL algorithm that consumes a scalar reward signal can be switched to preference-driven learning by replacing the environment reward with the summed local reward models.","Heterogeneous teams are naturally supported, because each teacher evaluates one agent's contribution rather than comparing agents to each other.","Finite preference data yields a graceful guarantee: the trained policy is an $\\epsilon$-Nash equilibrium with $\\epsilon$ growing linearly in the number of agents and shrinking as more comparisons are collected.","Lightweight automated teacher rules suffice in practice, so human annotators are not required; this makes the method feasible in industrial settings such as production-line scheduling."],"supporting_citations":[{"why":"Supplies the preference-based reward-learning model and cross-entropy objective that the method generalizes from single-agent to per-agent feedback.","marker":"[36]"},{"why":"Provides the paired-comparison probability model and strict-properness property used to identify the reward up to affine scaling.","marker":"[45]"},{"why":"Supplies the value-decomposition MARL algorithm used for policy optimization and as the reward-engineered oracle baseline.","marker":"[6]"},{"why":"Supplies the additive decomposition template and the decomposable Q-function assumption used in the appendix convergence analysis.","marker":"[5]"},{"why":"Guarantees existence of a Nash equilibrium in finite games, used to show Algorithm 1 terminates.","marker":"[43]"},{"why":"Provides the ODE and stochastic-approximation method used to prove almost-sure convergence of the learned Q-function.","marker":"[53]"},{"why":"Supplies the Rademacher-complexity uniform-convergence bounds behind the finite-sample reward estimation error.","marker":"[46]"},{"why":"Provides the interactive preference collection and experience relabeling techniques adopted for sample efficiency.","marker":"[9]"}],"fun_headline_variants":["Per-agent prefs provably reach Nash in MARL","Agent-specific preference training hits Nash, no rewards","MARL from preferences alone: provable Nash equilibrium","No global reward? Learn Nash via per-agent preferences","Decentralized preference MARL converges to Nash"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each agent's teacher prefers a new behavior exactly when that change increases the team's true global return; if teachers judge only local contribution, are noisy, or are biased, this 'if and only if' fails and the Nash-equilibrium guarantees no longer follow.","fun_headline_variants_meta":{"raw":{"variants":["Per-agent prefs provably reach Nash in MARL","Agent-specific preference training hits Nash, no rewards","MARL from preferences alone: provable Nash equilibrium","No global reward? Learn Nash via per-agent preferences","Decentralized preference MARL converges to Nash"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000148,"raw_usage":{"total_tokens":1179,"prompt_tokens":922,"completion_tokens":257,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":182}},"tokens_in":538,"tokens_out":257,"duration_ms":3144,"temperature":1.0,"reasoning_tokens":182,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:30:08.758005+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's production-line task with a teacher whose preference rule depends only on the agent's own stage, for example the first-stage agent prefers segments that empty its own queue fastest, then sample 10,000 segment pairs and count how often the teacher prefers a segment whose true global return is lower. If the violation rate is positive and the regularizer does not remove it, the premise of Theorems 1 and 2 is violated; the paper does not report such a test with human or noisy teachers.","supporting_citations":[{"cited_title":"Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the value-decomposition MARL algorithm used for policy optimization and as the reward-engineered oracle baseline."},{"cited_title":"Value-decomposition networks for cooperative multi-agent learning based on team reward,","cited_arxiv_id":null,"evidence_quote":"Supplies the additive decomposition template and the decomposable Q-function assumption used in the appendix convergence analysis."},{"cited_title":"The o.d.e. method for convergence of stochastic approximation and reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Provides the ODE and stochastic-approximation method used to prove almost-sure convergence of the learned Q-function."},{"cited_title":"Rademacher and gaussian complex- ities: Risk bounds and structural results,","cited_arxiv_id":null,"evidence_quote":"Supplies the Rademacher-complexity uniform-convergence bounds behind the finite-sample reward estimation error."}],"review_version":1}