{"id":"4a0f2e99-55be-4fd6-ba97-ee964c8cc6db","arxiv_id":"2505.18595","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MisoDICE labels unlabeled mixed-quality multi-agent demonstrations with LLM and preference learning, then trains policies with a value-decomposed DICE objective and reports gains on SMAC tasks.","lead":"The paper proposes MisoDICE, a two-stage system for learning cooperative multi-agent policies from unlabeled demonstrations that mix expert and poor-quality behaviors. Stage one uses LLM preferences plus preference-based RL to label expert trajectories; stage two uses a value-decomposed DICE imitation objective. On StarCraft benchmarks the method beats several baselines, but the theory has derivation errors.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (5) is not the Lagrangian of Eqs. (2)–(4): expanding it gives the opposite sign on the (1+α)log w term and on the ν terms inside A, so the convex Eq. (6) and Prop. 5.1 do not follow from the stated KL objective.","rationale":"The reader identified both the affine-factorization assumption and the sign error in Eq. (5). I focus on the sign error because it attacks the derivation of the objective itself, whereas the factorization issue is an approximation caveat that would remain even if the algebra were correct. The sign error is internal and decisive: the paper's own Lagrangian, expanded under its stated multiplier convention, does not produce the printed L(ν,w), the printed w*, or the convex objective in Eq. (6). Consequently, the abstract's central claim that MisoDICE 'yields a convex policy optimization objective' is not established for the stated KL objective, and the global-local consistency result is attached to a different optimization problem. The empirical results are extensive, internally consistent, and suggestive, so this is not grounds for rejecting the paper outright; the correct response is to require a corrected derivation, a precise statement of which objective is actually optimized, and a weakening of the convexity claim if the sign issue cannot be fixed. This preserves the reader's conditional verdict, hence UNCHANGED. A machine-checked proof is absent and no code is accessible from the manuscript, so an independent algebraic re-derivation is the cheapest decisive test of the central claim.","tokens_in":48350,"tokens_out":15534,"duration_ms":134707,"concrete_test":"Independently expand Eq. (5) with w=ρ/ρU and compare to the printed L(ν,w). Then, on a toy MDP (one state, two actions, γ=0, known ρE and ρU), solve the correct inf_w Lagrangian and compare the resulting policy with (i) the paper's Eq. (8) weight and (ii) the direct solution of Eq. (2). If the paper's Eq. (8) does not recover the direct optimum, or if Eq. (6)'s stationary point differs from the correct dual, the derivation is invalid. A code-level check is also worthwhile: instrument the loss in Eq. (6) and verify whether Algorithm 1 minimizes or maximizes it, since the printed derivation requires a sign convention not stated in the paper.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Let x=ρ_π, u=ρ_U, w=x/u, and keep ν as the multiplier in Eq. (5). Expanding the printed Lagrangian gives L=(1−γ)E_P0[ν]+E_U[ w((1+α)log w − A'_ν) ], with A'_ν=log(ρ_E/ρ_U)+ν−γE_{s'|s,a}[ν(s')]. The paper reports w(A_ν−(1+α)log w) with A_ν=log(ρ_E/ρ_U)+γEν−ν. These differ in the sign of the logarithmic term and in the sign of the ν terms inside A; no re-labelling of ν removes both discrepancies. Since the original problem is min over ρ, the inner reduction should also be inf over w, not the max_w used in §5.1. With the correct algebra, the stationary w* is exp((log(ρ_E/ρ_U)+ν−γEν)/(1+α)−1), and the substituted value is not the '+ exponential' in Eq. (6); the resulting dual is concave, not convex. Therefore Prop. 5.1's convexity and the claim that Eq. (6) is equivalent to Eq. (2) are unsupported by the derivation. The empirical section may still stand, but the central theoretical claim is exactly the one that fails.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MisoDICE, a two-stage framework for offline multi-agent imitation learning from unlabeled, mixed-quality demonstrations. Stage 1 labels trajectories by first using an LLM to produce pairwise preferences, then training the O-MAPL preference-based MARL method on these preferences to recover a reward signal and rank the trajectories, selecting a top-k expert set. Stage 2 introduces a DICE-based multi-agent IL algorithm that matches occupancy measures of the learned policy to expert and union data, factorizes the global value function via a linear mixing network, and trains local policies by weighted behavior cloning. The paper claims a convex policy optimization objective, global-local policy consistency, and superior empirical performance on SMACv1, SMACv2, and MAMuJoCo benchmarks.","tokens_in":48713,"tokens_out":15594,"duration_ms":117023,"significance":"If correct, the paper would address a practical and underexplored problem: learning from unlabeled demonstrations of heterogeneous quality in cooperative multi-agent systems. The combination of LLM-based labeling with a DICE-style multi-agent IL objective is novel, and the empirical evaluation is broad, covering multiple benchmarks and ablations. The paper also provides machine-checkable proof sketches for consistency and convexity. However, the central theoretical claims are load-bearing for the paper's contribution, and they are compromised by algebraic errors in the derivation of the core objective, as detailed below. The empirical results may still be of interest, but the paper's main selling point—a convex, theoretically grounded multi-agent IL objective—is not currently supported.","major_comments":[{"comment":"The printed Lagrangian L(ν,ρ) in Eq. (5) is not the Lagrangian of the constrained problem in Eq. (2)-(4). Expanding L(ν,ρ) with w = ρ/ρU yields (1-γ)E[ν] + E_{(s,a)~ρU}[ w( -log(ρE/ρU) + (1+α)log w + γE[ν(s')] - ν(s) ) ], but the paper reports w( log(ρE/ρU) + γE[ν(s')] - ν(s) - (1+α)log w ). The two expressions differ in the sign of the log-ratio term and in the sign of the (1+α)log w term; no re-labelling of ν or w removes both discrepancies. Consequently, the following claims that L is linear in ν and convex in w, and that the inner problem over w admits the closed-form solution w* = exp(A/(1+α)-1), are unsupported. This is load-bearing because Eq. (6) and Proposition 5.1 inherit the error, and the implemented algorithm is based on this reduction.","section":"Section 5.1, Eq. (5)"},{"comment":"The derivation from Eq. (5) to Eq. (6) is not shown; the main text refers to the appendix, but Appendix A.1 simply writes the simplified objective as (1-γ)E[ν] + (1+α)E[exp(A/(1+α)-1)] without deriving it from the printed Lagrangian. Since Eq. (5) is not the correct Lagrangian for the problem in Eq. (2)-(4), the simplified objective does not correspond to the stated KL minimization. Thus Proposition 5.1's conclusion that the training objective is convex in (φ,ν) does not apply to the actual problem, and the proof in A.1 is a proof of a different statement.","section":"Section 5.1 and Appendix A.1"},{"comment":"The theoretical results (Propositions 5.1 and 5.2) assume the mixing network Mφ is linear in its inputs and parameters to preserve convexity. However, the implementation described in Appendix B.3.5 uses a hypernetwork-based mixer that generates state-dependent mixing weights, which is generally nonlinear. The paper does not reconcile this mismatch, so the convexity guarantee does not hold for the algorithm actually evaluated. This disconnect between the theory and the implemented system needs to be addressed.","section":"Section 5.2 and Appendix B.3.5"},{"comment":"The closed-form local policy expression relies on the linear decomposition r(s,a) = Σ_i φ*_i r_i(s_i,a_i) + φ*_0, which is introduced as an assumption that is \"often feasible or can be approximated in practice.\" No error bound, validation, or discussion of when this decomposition holds is provided. Without such support, the claim that optimizing local policies via the decentralized weighted BC recovers the globally optimal joint policy is only an approximate statement, and the theoretical guarantee of global-local consistency is not quantitatively justified.","section":"Section 5.4, Prop 5.5"}],"minor_comments":[{"comment":"The column headers mix baseline names and β values in a confusing way; it should be stated explicitly that BC is evaluated with three variants (β=0.0, 0.5, 1.0).","section":"Table 1"},{"comment":"The notation νtot is used for the global Lagrange multiplier and later ννν denotes the set of local value functions; the transition should be defined more carefully to avoid confusion.","section":"Section 5.1"},{"comment":"The learning curves do not show error bars or confidence intervals, which makes it difficult to assess the variability of the reported improvements; please include error bars or explain their omission.","section":"Section 7, Figure 1"},{"comment":"The paper claims to be the first to address unlabeled mixed-quality demonstrations in multi-agent IL; consider softening this claim in light of prior work on imperfect demonstrations and multi-agent IL with noisy data.","section":"Introduction"},{"comment":"The proof of Proposition 5.4 is correct but terse; a brief explanation of why each local optimizer π*_i is optimal for the local weighted BC objective would improve readability.","section":"Appendix A.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a preprint with a NeurIPS checklist attached, which is unusual for a journal submission and may need reformatting. The authors rely heavily on their own prior work (O-MAPL, ComaDICE) for datasets and components, which is acceptable but should be clearly positioned. The main concern is the correctness of the theoretical derivation; the empirical results may still be salvageable after a major revision that either fixes the derivation or substantially softens the theoretical claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Mike — quick read of MisoDICE (arXiv:2505.18595). The empirical package is solid and the problem is a real one, but the theory at the core does not hold together. Eq. (5) is not the Lagrangian of Eqs. (2)–(4): expanding it gives E_U[ w((1+α)log w − (log(ρE/ρU)+ν−γEν)) ] plus the initial-state term, not the w(log(ρE/ρU)+γEν−ν−(1+α)log w) the paper reports. Both the log-term and the ν terms have flipped signs, so no re-labelling of ν fixes it. That makes the 'convex in w' sentence wrong for the expression as written, and Prop. 5.1's convexity does not follow from the stated KL objective. The appendix uses a different, positive-exponential form that is internally consistent but is not derived from Eq. (5); the main text and appendix are inconsistent. This is load-bearing: the paper's headline theoretical claim is that the factorized objective minimizes D_KL(ρπ||ρE)+αD_KL(ρπ||ρU) and is convex, and that claim is unsupported.\n\nWhat is genuinely new: the problem setting — offline multi-agent IL from unlabeled mixed-quality demonstrations — is a real gap, and the two-stage pipeline (LLM/O-MAPL labeling, then DICE with value decomposition) is a reasonable combination. Prop. 5.4 and the closed-form local policy in Prop. 5.5 are nice, though Prop. 5.5 leans on a reward-decomposition assumption that is strong and only hand-waved. The experiments are extensive — SMACv1/v2 and MAMuJoCo, multiple ablations — and MisoDICE is usually ahead of the baselines, though several gaps are within error bars. The α and top-k ablations are tuned on the same benchmarks, which is mild tuning rather than circularity. Also, the checklist claims code was uploaded, but I see no link in the manuscript.\n\nNet: the empirical core may survive, but the theory needs a rewrite, not a patch. I'd send this to peer review with instructions to focus on the Lagrangian derivation and the convexity claims, and to require code release. As it stands it's a reject-and-resubmit rather than an accept.","headline":"Real problem and extensive experiments, but the central DICE derivation has a sign error that breaks the claimed convexity/equivalence; the paper needs a major theory revision before it can be trusted.","tokens_in":49196,"tokens_out":7088,"would_cite":false,"duration_ms":57127,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MisoDICE learns cooperative multi-agent policies from unlabeled, mixed-quality demonstrations by combining an LLM-based labeling stage with a DICE-style imitation objective whose linear value factorization makes the optimization convex…","keywords":["multi-agent imitation learning","offline imitation learning","unlabeled mixed-quality demonstrations","DICE","value decomposition","centralized training decentralized execution","preference-based reinforcement learning","large language model labeling"],"falsifier":"Take a two-agent cooperative MDP whose optimal value function is known to be non-additive — for instance, a task where neither agent gains anything alone and success requires strict interleaving — collect unlabeled mixed-quality rollouts, and run MisoDICE; the gap between the recovered policy's return and the known optimum isolates the cost of the linear factorization. A second, cheaper check: on a small tabular problem, verify numerically whether the closed-form objective in Eq. (5) reproduces the optimum of the constrained program in Eqs. (3)-(4), since the sign of the $\\log w$ term determines whether the objective being optimized is the one whose convexity is proved.","tokens_in":48146,"feed_emoji":"🤖","tokens_out":11713,"duration_ms":91657,"temperature":0.7,"pith_summary":"The paper studies offline imitation learning in cooperative multi-agent settings where the only data are unlabeled demonstrations of mixed quality — some trajectories near-expert, most not — and no reward function is available. Its proposal, MisoDICE, is a two-stage pipeline: a labeling stage generates pairwise trajectory preferences with a large language model, refines them into a learned Q-function using offline preference-based multi-agent RL, and ranks trajectories to recover an expert subset; an imitation stage then minimizes a DICE-style divergence objective $D_{\\mathrm{KL}}(\\rho^{\\pi} \\| \\rho^{E}) + \\alpha D_{\\mathrm{KL}}(\\rho^{\\pi} \\| \\rho^{U})$ over joint occupancy measures. The central theoretical claim is that with a linear mixing network aggregating per-agent values, this objective is convex in both the mixer and the value parameters, and that independently optimizing each agent's weighted behavior-cloning objective yields the globally optimal factorized joint policy. The empirical claim, on SMACv1 and SMACv2, is that this beats behavior-cloning variants, independent per-agent DemoDICE, and supervised-reward MARL, most clearly when expert trajectories are scarce. If right, the framework turns an awkward practical situation — imperfect, unlabeled team data — into a convex problem with a principled decomposition.","feed_headline":"MisoDICE imitates experts from unlabeled, mixed-quality demos","feed_subtitle":"A convex DICE-style objective with per-agent value decomposition beats prior baselines on SMACv1 and SMACv2.","key_machinery":"The load-bearing object is the linear mixing network that assembles per-agent values into a global value function, $M_{\\phi}[\\boldsymbol{\\nu}(s)] = \\sum_{i\\in N} \\phi_i \\nu_i(s_i) + \\phi_0$, paired with its twin for occupancy-ratio discriminators, $M_{\\eta}[\\mathbf{c}(s,a)] = \\sum_{i\\in N} \\eta_i c_i(s_i,a_i) + \\eta_0$. Because of this linearity, the optimal weighing function $w^*_{\\boldsymbol{\\nu}}(s,a) = \\exp(A^{\\mathrm{tot}}_{\\boldsymbol{\\nu}}(s,a)/(1+\\alpha) - 1)$ is an exponential of a linear function, so the reduced training objective is convex in the value and mixing parameters, and the discriminator objective is concave in its parameters. The same decomposition splits the global advantage into per-agent terms, which is what makes weighted behavior cloning factorize into independent local problems whose solutions provably multiply back into the optimal joint policy; Proposition 5.5 then closes the loop with the explicit local-policy form $\\pi^*_i(a_i|s_i) \\propto \\mu^U_i(a_i|s_i)\\exp(\\phi^*_i q^*_i(s_i,a_i)/(1+\\alpha))$.","core_discovery":"MisoDICE claims that offline multi-agent imitation from unlabeled mixed-quality demonstrations reduces to a convex policy-optimization problem. The derivation starts from the single-agent DemoDICE objective — a KL divergence to the expert occupancy measure plus a regularizing KL divergence to the union of expert and suboptimal data — and lifts it to joint state-action space, where optimizing over the global occupancy measure directly is intractable. Tractability comes from a linear value factorization: the global Lagrange multiplier is written $M_{\\phi}[\\boldsymbol{\\nu}(s)] = \\sum_i \\phi_i \\nu_i(s_i) + \\phi_0$, which makes the reduced objective $L(\\phi, \\boldsymbol{\\nu})$ convex in both the mixing parameters and the local values (Prop 5.1), whereas a nonlinear mixer makes it non-convex (Prop 5.2). The same factorization is applied to the occupancy-ratio discriminator, $M_{\\eta}[\\mathbf{c}(s,a)] = \\sum_i \\eta_i c_i(s_i,a_i) + \\eta_0$, keeping the ratio-estimation objective concave (Prop 5.3). With these pieces, the paper proves that local weighted behavior cloning, run per agent with globally shared weights, composes exactly into the optimal joint policy (Prop 5.4), and that each local policy has the closed form $\\pi^*_i(a_i|s_i) \\propto \\mu^U_i(a_i|s_i) \\exp(\\phi^*_i q^*_i(s_i,a_i)/(1+\\alpha))$ (Prop 5.5). The experimental claim is that the full pipeline consistently achieves the highest mean return among the compared methods on SMACv2, with the advantage attributed to occupancy matching, the linear mixing architecture, and the LLM-initialized labeling.","pith_inferences":["The faithfulness of the linear mixing family is directly testable: on any benchmark where the optimal joint value function is available, one can fit $M_{\\phi}$ to it and measure the approximation gap; the paper does not quantify how far SMACv2 value functions are from the affine span of per-agent values.","The two claimed error sources — labeling noise in Phase 1 and factorization error in Phase 2 — are never separated; an experiment that feeds ground-truth labels into Phase 2 and compares with the LLM-labeled run would isolate the labeling pipeline's contribution.","If the convexity guarantee matters more than the particular mixer, the framework suggests a broader recipe: any linear-in-parameters decomposition of value, ratio, and reward makes multi-agent DICE convex; a natural extension would be sparse or state-dependent linear mixers that enlarge the representable family while keeping the exp-of-linear structure.","The LLM-as-labeler step is likely the transfer bottleneck: the paper reports LLMs fail to produce meaningful preferences in MaMuJoCo and substitutes a rule-based oracle, implying the labeling stage, not the imitation stage, determines how far the method generalizes outside structured game domains."],"forward_implications":["A single non-adversarial objective replaces minimax imitation training, so multi-agent imitation from mixed-quality data inherits the stability benefits DICE brought to single-agent offline IL.","Once the value factorization is trained, agent policies are extracted independently and in parallel from the closed-form weighted behavior cloning, making decentralized execution policies a by-product of centralized training.","Suboptimal data becomes a feature rather than noise: the $\\alpha D_{\\mathrm{KL}}(\\rho^{\\pi} \\| \\rho^{U})$ term is claimed to stabilize learning and keep coverage, which predicts graceful degradation as expert trajectories become rarer.","The pipeline claims expert annotation is unnecessary: LLM-initialized preferences refined through preference-based RL suffice to separate expert from poor trajectories, which if true removes a major data bottleneck for multi-agent IL."],"supporting_citations":[{"why":"Supplies the single-agent DemoDICE objective — KL to expert plus regularizing KL to imperfect data — whose dual reduction the paper adapts to the multi-agent setting.","marker":"Kim et al. [2021]"},{"why":"Provides O-MAPL, the offline preference-based MARL algorithm whose learned soft Q-function is re-purposed in Phase 1 to recover rewards and rank trajectories; also the source of the quality-categorized datasets.","marker":"Bui et al. [2025]"},{"why":"Source (with Wang et al.) of the offline SMACv1/SMACv2 trajectories with quality labels from which the unlabeled mixed dataset is sampled.","marker":"Bui et al. [2024]"},{"why":"Co-source of the offline trajectories that make up the unlabeled mixed-quality dataset.","marker":"Wang et al. [2022]"},{"why":"Defines SMACv1, one of the two benchmark suites where MisoDICE is evaluated.","marker":"Samvelyan et al. [2019]"},{"why":"Defines SMACv2, the main benchmark of the paper's empirical comparison.","marker":"Ellis et al. [2023]"},{"why":"MIFQ, the CTDE multi-agent imitation method extending IQ-Learn, which is the closest prior approach and the main lineage MisoDICE builds on.","marker":"Mai et al. [2024b]"},{"why":"QMIX, the canonical mixing-network value decomposition under CTDE that motivates the global-from-local aggregation architecture.","marker":"Rashid et al. [2020]"}],"fun_headline_variants":["Convex multi-agent imitation from unlabeled mixed-quality demos","LLM labels pick experts for multi-agent imitation","DICE principle scales to multi-agent offline IL","MisoDICE: convex multi-agent IL without expert labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guarantees hold only if the optimal joint value function, the data-quality ratio, and the reward can all be exactly represented as weighted sums of per-agent components; if the best team behavior requires interactions that no linear combination of individual values can capture, the convex objective optimizes the wrong target and local-global consistency no longer implies good behavior.","fun_headline_variants_meta":{"raw":{"variants":["Convex multi-agent imitation from unlabeled mixed-quality demos","LLM labels pick experts for multi-agent imitation","DICE principle scales to multi-agent offline IL","MisoDICE: convex multi-agent IL without expert labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000834,"raw_usage":{"total_tokens":3713,"prompt_tokens":1094,"completion_tokens":2619,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":710,"completion_tokens_details":{"reasoning_tokens":2554}},"tokens_in":710,"tokens_out":2619,"duration_ms":16609,"temperature":1.0,"reasoning_tokens":2554,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:29:28.790496+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a two-agent cooperative MDP whose optimal value function is known to be non-additive — for instance, a task where neither agent gains anything alone and success requires strict interleaving — collect unlabeled mixed-quality rollouts, and run MisoDICE; the gap between the recovered policy's return and the known optimum isolates the cost of the linear factorization. A second, cheaper check: on a small tabular problem, verify numerically whether the closed-form objective in Eq. (5) reproduces the optimum of the constrained program in Eqs. (3)-(4), since the sign of the $\\log w$ term determines whether the objective being optimized is the one whose convexity is proved.","supporting_citations":[],"review_version":1}