{"id":"974e5719-2081-4226-8971-73ee2ff282e8","arxiv_id":"2501.18944","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"O-MAPL trains cooperative MARL agents end-to-end from trajectory preferences by maximizing a Bradley-Terry likelihood in soft Q-space with linear value factorization and local weighted behavior cloning.","lead":"O-MAPL learns cooperative multi-agent policies directly from pairwise trajectory preferences, skipping the separate reward-modeling stage used by prior methods. The algorithm outperforms existing baselines on StarCraft and MuJoCo benchmarks, but many reported gains are within one standard deviation and no code or data has been released.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The stability/convergence theory is proven only for fixed scalar mixing weights, but the deployed hypernetwork mixing (Eqs. 6–7) makes weights state/action-dependent and nonlinear in θ, so the guarantees do not cover the implemented algorithm.","rationale":"The paper has genuine independent components: the Q-space preference likelihood is a reasonable multi-agent extension of IPL; the WBC extraction theorem is clever; the empirical study is broad (SMACv1/v2, MAMuJoCo, rule-based and LLM labels), and the reported numbers mostly favor O-MAPL. My stress-test focuses on the load-bearing link between theory and implementation. The reader's weakest assumption is exactly the one I would flag: the proofs in Prop. 4.1, Thm. 4.4 and Prop. 4.5 use fixed scalar mixing weights, while Section 5 uses hypernetwork-generated, observation/action-dependent weights. The proof of Thm. 4.4 factorizes e^{(Qtot−Vtot)/β} using state-independent bq and bv and per-agent scalar weights; with Wθ(o,a) this factorization does not survive, so the closed-form local policy is not guaranteed. Prop. 4.1's concavity in w is also not valid if the hypernetwork is nonlinear in θ; and concavity in function space does not imply concavity in neural-network parameters. This does not falsify the empirical results, but it means the paper's advertised 'stable, consistent training' is theoretically unsubstantiated for the deployed method. I would keep the CONDITIONAL verdict and ask for either a revised proof covering Eq. (6)–(7), or a precise statement of the narrowed class of mixing networks, plus the fixed-weight ablation. Secondary issues: the Prop. 4.2 counterexample uses a non-ReLU exponential-branch function, so it does not demonstrate non-convexity for two-layer ReLU mixing; the claim 'consistently outperforms' is not supported in every table row (e.g., SMACv1 corridor rule-based IPL-VDN 93.9 vs O-MAPL 93.2; several differences are within one standard deviation); and no code or significance tests are provided. None of these changes the central qualitative finding.","tokens_in":34853,"tokens_out":5988,"duration_ms":57390,"concrete_test":"Implement O-MAPL with the same local networks but replace the hypernetwork with fixed non-negative scalar weights w_i^q, w_i^v and constants b^q, b^v (the exact mixing family in Thm. 4.4). Train on the same rule-based SMACv2 datasets with 4 seeds. If this theory-conforming variant matches or improves the reported hypernetwork performance and training variance, the mismatch is benign; if the hypernetwork version is substantially better, the reported gains are not explained by the provided theory. Additionally, numerically check Prop. 4.1: for fixed trained q and v, evaluate L along interpolations θ(λ) = λθ1 + (1−λ)θ2 in hypernetwork parameter space; if L(θ(λ)) < λL(θ1) + (1−λ)L(θ2) anywhere, concavity in θ fails and the stability claim needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that O-MAPL trains stable, consistent policies because the preference objective is concave and the extreme-V loss is convex (Prop. 4.1), and because local WBC preserves global-local consistency (Thm. 4.3, Thm. 4.4). These results are proved for mixing networks with fixed scalar per-agent weights: Theorem 4.4 explicitly assumes Mw[q] = Σ_i w^q_i q_i + b^q and Mw[v] = Σ_i w^v_i v_i + b^v, with w^q_i, w^v_i, b^q, b^v constants (Appendix A.4). The implemented architecture in Section 5 instead uses hypernetworks: Vtot(o) = v^T W^o_θ + b^o_θ and Qtot(o,a) = q^T W^{o,a}_θ + b^{o,a}_θ, where W^o_θ and W^{o,a}_θ are generated from global observations and joint actions (Eqs. 6–7). The proof of Theorem 4.4 relies crucially on factoring e^{(Qtot−Vtot)/β} into a product over agents with agent-independent constants e^{(bq−bv)/β}; with hypernetwork weights that depend on (o,a), this factorization fails, so the closed-form local policy and the correction terms η(si)/∆(si) are not derived for the trained model. Likewise, Prop. 4.1's claim of concavity in 'the parameters of the mixing networks' is not established when the weights are outputs of a nonlinear hypernetwork. Concavity of L in the vector-valued function q does not transfer to concavity in the neural-network parameters ψ_q either. Hence the stated theoretical basis for stable and consistent end-to-end training does not cover the architecture whose results are reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces O-MAPL, an offline multi-agent preference-based RL algorithm that learns soft Q-functions directly from pairwise trajectory preferences using a linear value-factorization mixing network within the CTDE paradigm, and then extracts local policies via weighted behavior cloning. The authors claim theoretical guarantees of concavity/convexity of the learning objectives and global-local consistency of the extracted policies, and report experiments on SMACv1/SMACv2 and MAMuJoCo with both rule-based and LLM-generated preference labels, concluding that O-MAPL consistently outperforms existing baselines.","tokens_in":35251,"tokens_out":6442,"duration_ms":57531,"significance":"If the claims hold, O-MAPL would be a useful contribution: it is a one-phase preference-based MARL method that avoids explicit reward modeling, and it comes with a substantial empirical study including LLM-generated preferences and documented dataset construction. The paper also provides a principled local policy extraction procedure that preserves global-local consistency by construction. However, the theoretical results are proved only for constant scalar mixing weights, while the implemented architecture uses hypernetwork-generated state/action-dependent weights, so the stated guarantees do not cover the actual algorithm. In addition, the empirical advantage over baselines is often within one standard deviation, and no significance testing is reported.","major_comments":[{"comment":"The theoretical results that the paper uses to justify stable and consistent training are proved only for a linear mixing network with constant per-agent weights and biases (Prop. 4.1, Thm. 4.4, Prop. 4.5; see the assumption in Appendix A.4). The deployed architecture in Section 5 instead generates the mixing weights from hypernetworks that take the global observation and joint action as inputs, so W_theta^o and W_theta^{o,a} depend on (o,a) and on the parameters theta. With state/action-dependent weights, the product factorization of e^{(Qtot-Vtot)/beta} over agents used in the proof of Thm. 4.4 fails, and the closed-form local policy in Eq. (5) is not derived for the implemented model. Likewise, concavity of the preference loss L in q and w does not transfer to concavity in the network parameters psi_q and theta when the mixing weights are outputs of a nonlinear hypernetwork. The paper should either restrict the theoretical claims to the constant-weight setting and clearly state that the hypernetwork is an approximation, or extend the analysis to cover the deployed architecture.","section":"Appendix A.2 (Prop. 4.2)"},{"comment":"The counterexample used to prove non-convexity under two-layer mixing networks is not a two-layer ReLU network: it is a single-layer network with a piecewise exponential activation. The claim that the preference loss is no longer concave when Mw[q] and Mw[v] are two-layer feed-forward ReLU networks is therefore not established. Since this proposition is cited in the main text as a reason for preferring a single-layer linear mixer over two-layer alternatives, the proof needs to be corrected or replaced with a valid counterexample for the stated architecture class.","section":"Appendix A.2 (Prop. 4.2)"},{"comment":"In the proof of Theorem 4.4, the quantity eta(s_i) is defined as a sum over all joint next states and actions with s'_i = s_i, which includes the local action a'_i. However, in the expression for g_{s_i}, the local action is already separated out in the factor mu_i(a'_i|s_i) and in the log term. As written, eta(s_i) therefore double-counts the local action and is not independent of the local action as claimed. The definition should marginalize only over the other agents' actions and states, i.e., over s' and a'_{-i} with s'_i = s_i. The closed-form solution for pi_i^* may still be valid after this correction, but the current derivation is not correct as stated.","section":"Appendix A.4 (Theorem 4.4)"},{"comment":"The claim that O-MAPL 'consistently and significantly outperforms' baselines is not supported by statistical evidence. Several reported differences are within one standard deviation of the baseline (e.g., Hopper-v2: O-MAPL 1114.4 +/- 154.1 vs SL-MARL 890.0 +/- 88.7; SMACv2 protoss 5 vs 5, rule-based: O-MAPL 54.3 +/- 24.2 vs IPL-VDN 54.5 +/- 25.9). Since the results are averaged over only four seeds, the authors should report paired significance tests (or at least confidence intervals and effect sizes) for the main comparisons, or soften the 'significantly' claim to 'reports higher mean scores'.","section":"Section 6, Tables 1 and 2"}],"minor_comments":[{"comment":"In the definition of the local WBC loss, the expression 'log pi_i(o_i|s_i; omega_i)' appears to contain a typo; it should read 'log pi_i(a_i|o_i; omega_i)' since the policy should condition on the local observation and evaluate the probability of the local action.","section":"Section 5, Algorithm 1 and WBC loss"},{"comment":"The proof header in Appendix A.2 states 'Proposition 4.1' but the result being proved is Proposition 4.2; this should be corrected.","section":"Appendix A.2 heading"},{"comment":"The dataset descriptions refer to 'Meng et al.' and 'Xiangsen et al.' without corresponding entries in the reference list; the authors should add the full citations for the offline dataset sources.","section":"Appendix B.1.1 and B.1.2"},{"comment":"The text states that the mixing networks use 'non-negative weights' but does not specify how non-negativity is enforced in the hypernetworks (e.g., via softplus or absolute value); this implementation detail should be stated for reproducibility.","section":"Section 5, Eqs. (6)-(7)"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the core idea is promising, but the main risk is the gap between the theoretical analysis (constant scalar mixing weights) and the implemented architecture (hypernetwork-generated, state/action-dependent weights). The stability and consistency claims are load-bearing for the paper's contribution, so this gap should be addressed either by restricting the theory to the constant-weight setting or by extending it to the deployed model. I also recommend asking for significance testing or effect sizes given the high variance in the benchmark results. The invalid counterexample in Prop. 4.2 and the double-counting issue in the proof of Thm. 4.4 should be corrected; these are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"My take: this is a straightforward, honest extension of single-agent inverse preference learning to cooperative offline MARL. The local WBC extraction with correction terms is a nice reusable idea, and the experimental sweep across SMAC and MAMuJoCo is wide. But the theoretical scaffolding as written does not cover the deployed algorithm. The proofs of Prop 4.1 and Thm 4.4 assume fixed scalar mixing weights and constants, while Section 5 uses hypernetworks that generate mixing weights from the global observation and joint action. The factorization of e^{(bq-bv)/beta} that makes the closed-form local policy work fails when bq and bv depend on (o,a), so the stated convexity and global-local consistency guarantees do not apply to the trained model. This is load-bearing in the paper's framing, not a cosmetic gap. Also, Prop 4.2's counterexample is not a two-layer ReLU mixing network; it is a hand-built piecewise function, so the non-convexity claim for standard two-layer mixing is not actually proved. Empirically, O-MAPL often comes out ahead, but the tables show large standard deviations and no significance testing; many differences are within noise. The text repeatedly says \"significant\" or \"consistently outperforms\" when the data only supports \"higher on average.\" No code or data is released, which matters here because the datasets come partly from the authors' own OMIGA and ComaDICE papers. On the positive side, the baselines are sensible (IIPL, IPL-VDN, SL-MARL), the GLC theorem for local WBC is correct under its stated assumptions, and the appendix is unusually thorough about dataset construction and LLM annotation costs. The limitations section honestly notes the focus on cooperative settings and sample efficiency. So the core idea deserves a serious referee, but the current version overclaims. My recommendation: send it to peer review with a request for major revision: align the theory with the actual mixing architecture or explicitly restrict the claims, add significance tests, and release the code and data.","headline":"A useful incremental adaptation of inverse preference learning to cooperative MARL, but the convexity/consistency theory is proved for fixed mixing weights while the implementation uses hypernetworks, and the benchmark evidence is noisy.","tokens_in":35765,"tokens_out":2533,"would_cite":false,"duration_ms":24802,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"O-MAPL learns cooperative multi-agent policies end-to-end from pairwise trajectory preferences in soft Q-space, without an explicit reward model.","keywords":["offline multi-agent reinforcement learning","preference-based reinforcement learning","inverse soft Q-learning","value factorization","centralized training decentralized execution","weighted behavior cloning","Bradley-Terry model","LLM preference annotation"],"falsifier":"Take a trained O-MAPL model, freeze the data, and compute the preference likelihood $L$ along convex combinations of two Q-parameter vectors while letting the hypernetwork recompute the mixing weights; a violation of concavity along some segment would directly contradict the applicability of Proposition 4.1 to the deployed architecture.","tokens_in":34645,"feed_emoji":"🎮","tokens_out":7980,"duration_ms":72072,"temperature":0.7,"pith_summary":"This paper introduces O-MAPL, an offline algorithm that learns cooperative multi-agent policies directly from pairwise trajectory preferences. It avoids the two-phase pipeline of reward modeling followed by MARL, which the authors argue is unstable and data-hungry. Instead, it exploits the one-to-one correspondence between reward functions and soft Q-functions under maximum-entropy RL, turning preference learning into a Q-learning problem. A linear value-factorization mixer keeps the preference likelihood concave, and local policies are extracted by weighted behavior cloning. The authors report that O-MAPL consistently outperforms BC, independent IPL, VDN-based IPL, and a supervised two-phase baseline on SMAC and MAMuJoCo tasks.","feed_headline":"No reward model needed: preference data trains cooperative agents","feed_subtitle":"It converts trajectory comparisons into soft-Q targets and local policies, beating baselines on SMAC and MAMuJoCo","key_machinery":"The load-bearing identity is the inverse soft Bellman operator $(T^* Q_{\\text{tot}})(s,a)=Q_{\\text{tot}}(s,a)-\\gamma \\mathbb{E}_{s'}[V_{\\text{tot}}(s')]$, which gives a one-to-one map from soft Q-functions to rewards, so the Bradley-Terry preference likelihood can be written entirely in Q-space. A linear mixing network $M_w$ combines local $q_i$ and $v_i$ into global $Q_{\\text{tot}}$ and $V_{\\text{tot}}$; the paper's theory, including concavity of the preference likelihood, convexity of the extreme-V loss, and global-local consistency, relies on this linearity. The extreme-V (XQL) loss is used to fit $V_{\\text{tot}}$ to the log-sum-exp of $Q_{\\text{tot}}$ without enumerating actions. Finally, local weighted behavior cloning replaces direct value-based policy extraction, so local policies are proper probability distributions and remain globally consistent with the joint optimal policy.","core_discovery":"The central claim is that multi-agent preference-based RL can be solved in one phase: maximize a Bradley-Terry preference likelihood over trajectories with rewards replaced by the inverse soft Bellman operator applied to a global soft Q-function, factorize that global Q and V through a linear mixing network into local values, and extract decentralized policies by weighted behavior cloning. The paper proves that with a single-layer linear mixer the preference loss is concave in the local Q functions and mixing weights, the extreme-V loss is convex in the local value functions, and the local weighted-BC solution is globally consistent with the optimal decomposed global policy. It further gives a closed form for the recovered local policy, with a normalization correction that depends on other agents' values, supporting credit assignment. On SMACv1, SMACv2, and MAMuJoCo, the paper reports that O-MAPL achieves the highest win rates or returns on nearly all tasks, with especially clear gains on LLM-labeled preference data.","pith_inferences":["The formal guarantees are proved for constant mixing weights, while the deployed hypernetwork makes the weights functions of observations and actions; if that gap is real, the practical stability may come from the architecture's inductive bias rather than from the stated theorems.","Because the global-local consistency theorem holds for any mixing structure, weighted behavior cloning may extend the approach beyond linear mixers or toward mixed cooperative-competitive settings, though the paper evaluates only cooperative tasks.","The reported preference data volume is small, roughly one to two thousand trajectory pairs per task, so sample-efficient preference collection, including LLM annotation in state-rich environments, seems the main scaling bottleneck."],"forward_implications":["Offline cooperative MARL can be trained from preference comparisons alone, removing the separate reward-modeling stage and the misalignment it introduces.","The linear mixing architecture gives a concave preference objective, so in theory the Q-space optimization has a unique global maximum rather than many local plateaus.","Weighted behavior cloning makes extracted local policies valid probability distributions while preserving global-local consistency, even if the mixing network is nonlinear.","The method generalizes across discrete and continuous action spaces under centralized training with decentralized execution, as demonstrated on SMACv1, SMACv2, and MAMuJoCo.","LLM-labeled preferences can substitute for hand-designed or rule-based labels, and in these experiments they often yield higher final win rates on SMAC tasks."],"supporting_citations":[{"why":"Supplies the pairwise comparison model used as the preference likelihood over trajectory pairs.","marker":"Bradley & Terry, 1952"},{"why":"Establishes the inverse soft-Q relation that lets the paper rewrite reward learning as Q-learning in preference space.","marker":"Garg et al., 2021"},{"why":"Provides the extreme-V / XQL loss used to fit the global value function to the log-sum-exp of Q without action enumeration.","marker":"Garg et al., 2023"},{"why":"The single-agent inverse preference learning algorithm that O-MAPL extends, and the source of the rule-based preference labeling protocol.","marker":"Hejna & Sadigh, 2024"},{"why":"OMIGA supplies an offline MARL baseline and offline datasets of varying quality used for preference sampling.","marker":"Wang et al., 2022"},{"why":"VDN defines the value-decomposition baseline IPL-VDN, which aggregates local Q and V by simple summation.","marker":"Sunehag et al., 2017"},{"why":"ComaDICE supplies offline MARL datasets and supports the choice of a single-layer mixing structure in offline settings.","marker":"Bui et al., 2025"},{"why":"DPM provides the two-phase multi-agent PbRL baseline and the LLM-based preference annotation method the paper follows.","marker":"Kang et al., 2024"}],"fun_headline_variants":["Preference data alone trains cooperative agents","End-to-end preference learning for multi-agent systems","Soft Q mixing enables offline preference-based MARL","One-phase multi-agent preference optimization","O-MAPL: learning cooperative policies from preferences"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the formal guarantees, proven for constant per-agent mixing weights, still describe the deployed hypernetwork-based mixer whose weights depend on the global observation and joint action.","fun_headline_variants_meta":{"raw":{"variants":["Preference data alone trains cooperative agents","End-to-end preference learning for multi-agent systems","Soft Q mixing enables offline preference-based MARL","One-phase multi-agent preference optimization","O-MAPL: learning cooperative policies from preferences"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000689,"raw_usage":{"total_tokens":3083,"prompt_tokens":868,"completion_tokens":2215,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":2148}},"tokens_in":484,"tokens_out":2215,"duration_ms":15850,"temperature":1.0,"reasoning_tokens":2148,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T21:51:48.854227+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained O-MAPL model, freeze the data, and compute the preference likelihood $L$ along convex combinations of two Q-parameter vectors while letting the hypernetwork recompute the mixing weights; a violation of concavity along some segment would directly contradict the applicability of Proposition 4.1 to the deployed architecture.","supporting_citations":[],"review_version":1}