{"id":"23ef89f7-5c41-4f93-8a1f-59e449794793","arxiv_id":"2506.12497","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A cooperative MARL algorithm that regularizes each agent's policy toward the Sinkhorn barycenter of the team's visitation distributions, with a claimed but insufficiently proven geometric convergence guarantee.","lead":"This paper proposes a multi-agent reinforcement learning method in which every agent's policy is softly pulled toward a statistical average of all agents' policies, defined using optimal transport. The intended payoff is faster and more coherent team coordination than independent learning, but the central convergence proof rests on an unproved contraction step and the experiments are limited to a toy grid world.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.1's geometric contraction is asserted, not derived: the proof drops the reward-gradient term, leaves the constant C unconstructed, and silently changes the contraction factor from 1−αλC to 2(1−αλC) without stating the required αλC > 1/2.","rationale":"The reader's weakest_assumption pinpoints exactly the step on which the central claim rests. I read the proof of Theorem 3.1 and found the same gap: the contraction inequality is a one-sentence assertion that does not follow from strong convexity and Lipschitz continuity as stated, because the policy update is not a pure gradient step on the OT penalty. The reward-gradient term α∇E[R] is present in the update equation but absent from the contraction argument, and no regularity assumption connects parameter movement to visitation-measure movement in W_{p,ε}. The proof also changes the contraction factor mid-proof: the theorem states κ = 1 − αλC < 1, while the proof requires κ = 2(1 − αλC) < 1, i.e. αλC > 1/2, a condition never mentioned. These are internal inconsistencies, not merely disagreements with consensus. The central claim of the paper is the geometric convergence of all policies to a common distribution; without this step, the algorithm is an interesting heuristic but the advertised guarantee is unproven. The experimental section is a single no-error-bar comparison and cannot substitute for the missing proof. I therefore see no reason to overturn the reader's rejection; the verdict stays as the reader stated, so I mark it UNCHANGED rather than issuing a new one.","tokens_in":8407,"tokens_out":5308,"duration_ms":63262,"concrete_test":"Re-derive Theorem 3.1 keeping every term: substitute θ_i^{(t+1)} = θ_i^{(t)} + α(g_R − λg_OT) into a first-order expansion of W_{p,ε}(μ_i(θ^{(t+1)}), μ*^{(t)}) and identify the contribution α⟨∇_θ W, g_R⟩; then instantiate a minimal two-agent, two-action gridworld in which the team reward strongly favors opposite actions for the two agents, set αλ ∈ {0.6, 0.9}, run Algorithm 1 for 200 iterations over 20 seeds, and record max_{i,j} W_{p,ε}(μ_i^{(t)}, μ_j^{(t)}). If the empirical contraction ratio remains above 1 for a sustained interval, or cannot be bounded by 2(1−αλC) for any admissible C>0, the omitted reward-gradient term demonstrably breaks the claimed contraction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is the geometric contraction D(t+1) ≤ κD(t) in Theorem 3.1, and that claim rests entirely on a single unproved inequality: W_{p,ε}(μ_i^{(t+1)}, μ*^{(t)}) ≤ (1−αλC) W_{p,ε}(μ_i^{(t)}, μ*^{(t)}). This is the load-bearing step, and it is not established. The actual update is θ_i^{(t+1)} = θ_i^{(t)} + α(∇_θ E[R] − λ∇_θ W_{p,ε}); the asserted contraction ignores the +α∇_θ E[R] term completely. No stated assumption controls the effect of the reward-gradient term on the W_{p,ε} distance to the barycenter, and no regularity condition connects a parameter step to the visitation-measure change μ_i^{(t+1)} in the metric. The theorem's hypothesis κ = 1 − αλC < 1 is also not the factor used in the proof, which sets κ = 2(1 − αλC) < 1 and thereby requires αλC > 1/2; this stronger condition is never stated, and C is never constructed or bounded. Because the triangle inequality is the only bridge from per-agent contraction to pairwise D(t+1) ≤ κD(t), the geometric-rate conclusion collapses unless the missing per-agent contraction, including the reward term, is actually proven. The attached limitation section concedes that strong-convexity and density-boundedness assumptions seldom hold, but that concession concerns Theorem 3.2; the more fundamental gap is the unproved contraction in Theorem 3.1 itself.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a cooperative MARL algorithm in which the team's consensus policy is the entropic-regularized p-Wasserstein barycenter of the agents' empirical visitation measures. Each agent maximizes the expected team reward minus a Sinkhorn-divergence penalty to this barycenter, and the paper claims in Theorem 3.1 that the maximal pairwise policy discrepancy contracts geometrically under Lipschitz and compactness assumptions. Theorem 3.2 claims a fast statistical rate for the Sinkhorn barycenter. A small cooperative-navigation experiment compares the method with independent PPO, KL-regularized, and parameter-sharing baselines. The reviewer finds that the proof of Theorem 3.1 does not establish the central contraction claim: the key inequality is asserted without proof, the reward-gradient term is omitted, the contraction constant is never constructed, and the proof uses a different contraction factor from the theorem statement. The appendix proof of Theorem 3.2 also contains unresolved inconsistencies. The empirical results are transparent but preliminary.","tokens_in":8761,"tokens_out":8449,"duration_ms":95138,"significance":"The high-level idea of using an OT barycenter as a soft consensus target rather than hard parameter sharing is appealing and worth exploring. The paper is clearly structured, the algorithm is specified in detail, and the inclusion of a limitations section is commendable. However, the main advertised contribution is the geometric convergence theorem, and that contribution is not supported by the provided argument. The experimental evaluation is a proof-of-concept on a single small task without error bars. If a rigorous proof could be supplied under explicit, verifiable assumptions, this framework would be a useful contribution; in its current form, the paper does not meet the standard for publication.","major_comments":[{"comment":"The load-bearing inequality W_{p,ε}(μ_i^{(t+1)}, μ^{*(t)}) ≤ (1 − αλC) W_{p,ε}(μ_i^{(t)}, μ^{*(t)}) is asserted without proof. The update is a parameter-space gradient step, and μ_i^{(t+1)} is the empirical visitation distribution after executing the updated policy; neither the strong convexity of the entropic OT problem nor L-Lipschitz continuity of the policy-gradient mapping, as stated, implies a contraction of this quantity. The reward-gradient term +α∇_{θ_i} E[R] is absent from the inequality, and no assumption in the theorem controls its effect on the Sinkhorn distance to the barycenter. Without a derivation of this inequality, the conclusion D^{(t+1)} ≤ κ D^{(t)} does not follow.","section":"§3, Theorem 3.1 proof"},{"comment":"The theorem statement imposes κ = 1 − αλC < 1, but the proof sets κ = 2(1 − αλC) < 1, which is a strictly stronger condition requiring αλC > 1/2. This stronger condition is not stated, and the constant C is never constructed or bounded in terms of the entropic regularizer and Lipschitz constants. As a result, the hypothesis of the theorem is not verifiable from the stated assumptions.","section":"§3, Theorem 3.1"},{"comment":"The proof invokes the triangle inequality for W_{p,ε}. The entropic-regularized Sinkhorn divergence is not a metric, and the triangle inequality is not a general property of this divergence for ε > 0. If the authors intend to use a distance that satisfies the triangle inequality, they must state and prove that property for their W_{p,ε}; otherwise the step bounding W_{p,ε}(μ_i^{(t+1)}, μ_j^{(t+1)}) by the sum of two distances to μ^{*(t)} is unjustified.","section":"§3, Theorem 3.1 proof"},{"comment":"The displayed formula ∇_{θ_i} W^p_{p,ε}(μ_i^{(t)}, μ^{*(t)}) = ∫ d(x,y)^p ∇_{θ_i} log π_i(a|s;θ_i) dγ_i^{*(t)}(x,y) is not justified as written, because μ_i^{(t)} is the empirical visitation distribution induced by the current policy and therefore depends on θ_i. Differentiating only the action probabilities while holding the state-action measure fixed omits the state-distribution term, so the update implemented in Algorithm 1 is not demonstrably the update analyzed in Theorem 3.1.","section":"§3, gradient of the consensus penalty"},{"comment":"The proof sketch and the appendix give inconsistent bounded-difference bounds: the sketch states D^p/(N m ε_m) = O(m^{-2}), while (A.1) uses D^p/(N m), a factor of ε_m = D^p/m apart. Step 2's metric-entropy bound log M_γ ≤ C_d γ^{-d} for P(X) is asserted without reference to the dimension or structure of X, and Step 3's inequality F(μ) ≥ F(μ*) + λ_min W_{p,ε}(μ, μ*) does not follow from λ_min-strong convexity in any stated norm. The claimed rate in Eq. (1) is therefore not established; the limitation section's concession that the strong-convexity and density-boundedness assumptions seldom hold does not repair this gap.","section":"Appendix A.1, proof of Theorem 3.2"}],"minor_comments":[{"comment":"The paper alternates between W^p_{p,ε} in the objectives and W_{p,ε} in Theorem 3.1 and in the definition of D^{(t)}; since contraction in W_{p,ε} does not automatically imply contraction in W^p_{p,ε}, the notation and the logical relation between the two should be clarified.","section":"§2, §3, notation"},{"comment":"The assumption that the policy-gradient operators are L-Lipschitz 'in the induced distributions' is not defined precisely; the objects, norms, and the space on which the Lipschitz condition is imposed should be specified.","section":"§3, Theorem 3.1 assumptions"},{"comment":"Figure 3 reports point estimates without error bars, number of seeds, or training details, so the claim of approximately 2× improvement over IPPO is not substantiated beyond a single run.","section":"§4, Figure 3"},{"comment":"The iterative update for v^{(ℓ+1)} appears to be a simplified fixed-point update, whereas standard Sinkhorn-barycenter algorithms maintain per-measure dual potentials; this step should be clarified or referenced.","section":"§3, barycenter computation"}],"recommendation":"reject","confidential_remarks":"The central advertised result is a convergence theorem that is not proved; the manuscript is not suitable for publication in its current form. The author would be better served either by providing a rigorous proof under explicit, verifiable assumptions or by reframing the paper as an empirical methods study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: the paper has a genuinely new idea – using an entropic-regularized Wasserstein barycenter of agent visitation measures as a soft consensus penalty in cooperative MARL – but the theory is not there yet. The main proof (Theorem 3.1) is a sequence of assertions, not a derivation, and the experiment is a single grid-world run with no error bars or seeds. As a research proposal it is interesting; as a paper it is not ready.\n\nWhat is new: the specific combination of Sinkhorn barycenters with policy-gradient updates for multi-agent consensus does not appear in the cited literature, which surveys KL regularizers, parameter sharing, and CTDE. That is a legitimate gap, and the framework is clearly described. The algorithm – alternating barycenter computation and gradient ascent – is straightforward to implement, and the intuitive appeal of aligning policies while preserving heterogeneity is real.\n\nWhere it falls short: the central claim is geometric contraction of pairwise policy discrepancies. The proof's load-bearing step is an assertion that a gradient step contracts each agent's distance to the barycenter, with an unspecified constant C. The reward-gradient term in the update is dropped without any assumption controlling its effect on the OT distance, and the parameter-to-measure map is never analyzed. The proof also silently changes the contraction factor from (1−αλC) to 2(1−αλC), requiring αλC>1/2, a condition never stated. This is not a minor gap; the theorem is effectively assumed. Theorem 3.2's proof has its own issue: the metric-entropy term after the net argument grows with m for d>1 (and is O(1) for d=1), so the claimed O(1/m) rate does not follow. The limitations section honestly admits that the strong-convexity and density assumptions seldom hold, but that concerns Theorem 3.2; Theorem 3.1's problem is more fundamental.\n\nThe experiment: a 3-agent navigation task, one learning curve, no variance, and a comparison that mixes the OT penalty with centralized training. The qualitative action maps are nice, but they do not substitute for statistics.\n\nBottom line: the idea might be worth pursuing, but as written the core contribution – a convergence guarantee – is not established. I would not cite it for the theory, and I would not use it as a basis for empirical claims. If it crossed my desk as an editor, I would likely send it to reviewers because the concept is interesting enough to warrant feedback, but I would expect heavy revision or a major rewrite of the proofs.\n\nRecommendation: treat it as a pointer to a possible research direction, not as a citable result.","headline":"Promising barycenter-consensus idea for cooperative MARL, but the central convergence proof is an assertion and the experiment is too thin to carry the claims.","tokens_in":9304,"tokens_out":4054,"would_cite":false,"duration_ms":45385,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cooperative MARL agents can be aligned by penalizing each policy's Sinkhorn distance to a team Wasserstein barycenter, with geometric convergence to a common policy.","keywords":["multi-agent reinforcement learning","cooperative MARL","optimal transport","Wasserstein barycenter","Sinkhorn divergence","policy consensus","geometric convergence","policy gradient"],"falsifier":"Run the WBC algorithm on a two-agent grid task and record $\\log D(t)$ over iterations; a lack of linear decay (i.e., $D(t+1)/D(t)$ not consistently below 1) would contradict the geometric-rate claim. Alternatively, compute the constant $C$ explicitly for a Gaussians-with-known-barycenter instance and check whether $1-\\alpha\\lambda C < 1$ under the paper's stated assumptions.","tokens_in":8159,"feed_emoji":"🎯","tokens_out":13047,"duration_ms":125030,"temperature":0.7,"pith_summary":"This paper introduces a consensus mechanism for cooperative multi-agent reinforcement learning based on optimal transport. The team's collective strategy is defined as the entropic-regularized $p$-Wasserstein barycenter of the agents' joint state–action visitation measures, and each agent's policy objective is augmented with a soft penalty proportional to its Sinkhorn divergence from this barycenter. The paper proves that, under compactness and Lipschitz assumptions, the maximal pairwise Sinkhorn divergence between agents contracts at a geometric rate, so all policies converge to a common distribution. A cooperative navigation experiment shows the method reaches higher final reward and converges faster than independent PPO, KL-regularized, and parameter-sharing baselines.","feed_headline":"Policy gaps close geometrically with a barycenter penalty","feed_subtitle":"Pulling each agent's policy toward the team's optimal-transport midpoint yields shared behavior and faster coordination.","key_machinery":"The central object is the entropic-regularized $p$-Wasserstein barycenter, the measure $\\mu^{*(t)}$ minimizing $\\frac{1}{N}\\sum_i W_{p,\\varepsilon}^p(\\mu, \\mu_i^{(t)}) + \\varepsilon\\, \\mathrm{KL}(\\mu \\| \\eta)$, computed by alternating Sinkhorn (iterative Bregman projection) updates on the discretized joint state–action space. The mechanism that couples consensus to learning is the gradient identity $\\nabla_{\\theta_i} W_{p,\\varepsilon}^p(\\mu_i^{(t)}, \\mu^{*(t)}) = \\int d(x,y)^p\\, \\nabla_{\\theta_i}\\log\\pi_i(a|s;\\theta_i)\\, d\\gamma_i^{*(t)}(x,y)$, where $\\gamma_i^{*(t)}$ is the optimal coupling between $\\mu_i^{(t)}$ and the barycenter; this makes the entire loop differentiable by sampling from the visitation measure. The convergence proof uses the contraction of a single gradient step toward the barycenter (attributed to strong convexity of the regularized OT problem and Lipschitz continuity of the policy-gradient mapping), combined with the triangle inequality for $W_{p,\\varepsilon}$ and an induction over iterations.","core_discovery":"The central claim is that consensus in cooperative MARL can be achieved geometrically: rather than sharing parameters or a centralized critic, agents are drawn together by a soft penalty equal to their entropic-regularized $p$-Wasserstein distance to a common barycenter. The paper's main theoretical result (Theorem 3.1) states that if the state–action space is compact, rewards are bounded and Lipschitz, policy-gradient operators are $L$-Lipschitz in the induced distributions, and the step size and consensus weight satisfy $\\kappa = 1 - \\alpha\\lambda C < 1$ for a constant $C > 0$, then the maximal pairwise discrepancy $D(t) = \\max_{i,j} W_{p,\\varepsilon}(\\mu_i^{(t)}, \\mu_j^{(t)})$ obeys $D(t+1) \\leq \\kappa D(t)$ and hence converges to zero at a geometric rate. The paper also proves a fast-rate bound (Theorem 3.2) on the accuracy of the estimated barycenter as a function of sample size. Together these results position the Wasserstein barycenter as a principled, differentiable consensus anchor that preserves specialized behavior while aligning the team.","pith_inferences":["A constructive proof that makes the contraction constant $C$ explicit in terms of $\\varepsilon$, the Lipschitz constants, and the strong-convexity parameter would turn Theorem 3.1 into a practical recipe for choosing $\\alpha$ and $\\lambda$, a step the paper leaves open.","The same barycenter-consensus penalty could plausibly be applied to critics or value distributions rather than visitation measures; whether the geometric-rate guarantee carries over to off-policy settings is a direct testable extension.","The predicted geometric decay of pairwise Sinkhorn divergence is measurable in training logs: if the log-ratio $D(t+1)/D(t)$ is not eventually bounded below 1 on a standard benchmark, the claimed contraction rate would fail, even if the method still performs well."],"forward_implications":["Coordination no longer requires parameter sharing, centralized critics, or hand-designed reward shaping; a soft OT penalty provides a differentiable, geometry-aware alignment signal.","The theorem yields a finite-time bound on policy disagreement: after $t$ iterations, the maximal pairwise Sinkhorn divergence is at most $\\kappa^t D(0)$, so the time to reach a desired consensus tolerance is $\\mathcal{O}(\\log(1/\\delta))$.","The fast-rate barycenter estimation result implies that, under strong convexity and bounded density, only $m = O(\\eta^{-1} \\log(N/\\delta))$ samples per agent are needed to achieve barycenter error $\\eta$, making the consensus step statistically efficient.","The empirical comparison suggests the same mechanism is competitive with or better than existing consensus regularizers: about 2× the final reward of independent PPO and roughly 50% higher than a KL-regularized baseline on the cooperative navigation task."],"supporting_citations":[{"why":"Supplies the Sinkhorn divergence and the efficient algorithm for computing entropic optimal transport distances, the core of the barycenter penalty.","marker":"(Cuturi, 2013)"},{"why":"Provides the computational OT background, including barycenters and iterative Bregman projections, used to compute the consensus measure.","marker":"(Peyré et al., 2019)"},{"why":"Supplies the theory of Wasserstein distances and metric-entropy estimates for probability measures used in the fast-rate proof.","marker":"(Villani et al., 2009)"},{"why":"Provides the Lipschitz and bounded-difference lemmas for Sinkhorn divergences that the statistical rate theorem relies on.","marker":"(Genevay et al., 2018)"}],"fun_headline_variants":["MARL policies align geometrically via barycenter penalty","Barycenter penalty yields geometric policy alignment","Wasserstein barycenter drives geometric MARL consensus","Geometric rate convergence from OT barycenter penalty","Geometric consensus in MARL via Wasserstein barycenter"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The geometric convergence proof depends on an unproved contraction step: a single gradient step with weight $\\alpha\\lambda$ is asserted to shrink each agent's Sinkhorn distance to the current barycenter by a factor $(1-\\alpha\\lambda C)$, where $C$ is never constructed and the reward-gradient term is ignored.","fun_headline_variants_meta":{"raw":{"variants":["MARL policies align geometrically via barycenter penalty","Barycenter penalty yields geometric policy alignment","Wasserstein barycenter drives geometric MARL consensus","Geometric rate convergence from OT barycenter penalty","Geometric consensus in MARL via Wasserstein barycenter"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000445,"raw_usage":{"total_tokens":2243,"prompt_tokens":932,"completion_tokens":1311,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":1237}},"tokens_in":548,"tokens_out":1311,"duration_ms":11152,"temperature":1.0,"reasoning_tokens":1237,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:48:22.597048+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the WBC algorithm on a two-agent grid task and record $\\log D(t)$ over iterations; a lack of linear decay (i.e., $D(t+1)/D(t)$ not consistently below 1) would contradict the geometric-rate claim. Alternatively, compute the constant $C$ explicitly for a Gaussians-with-known-barycenter instance and check whether $1-\\alpha\\lambda C < 1$ under the paper's stated assumptions.","supporting_citations":[{"cited_title":"Learning generative models with sinkhorn divergences","cited_arxiv_id":null,"evidence_quote":"Provides the Lipschitz and bounded-difference lemmas for Sinkhorn divergences that the statistical rate theorem relies on."}],"review_version":1}