{"id":"96b26526-102f-41cf-ab48-bddeeb218a9f","arxiv_id":"2501.10605","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"WAVE adds an adaptively weighted Sinkhorn approximation of the Wasserstein distance between successive Q-value distributions to the critic loss in actor-critic reinforcement learning.","lead":"This paper adds a penalty to the critic's error term in actor-critic reinforcement learning, penalizing large changes in the estimated value distribution from one training step to the next. The goal is smoother, more stable learning in continuous-control tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's O(1/k) rate for the WAVE critic rests on strong convexity of Lreg, which is not satisfied by the implemented MLP critic; even if it were, the proof omits the condition 2am > 1 required for the stated rate.","rationale":"Choosing the most load-bearing concern: the paper's headline theoretical result is the O(1/k) critic MSE convergence rate. Theorem 1 is the only proof of that rate, and it depends critically on strong convexity of Lreg. This assumption is not satisfied by the neural-network critic used in the experiments, and the proof contains an additional omitted condition (2am > 1) for the claimed rate to follow from the displayed recurrence. This is a formal gap, not a disagreement with consensus. The reader's weakest_assumption points to the same strong-convexity issue, so I agree. I also reviewed Theorems 2 and 3; Theorem 3's derivative of the Sinkhorn distance with respect to Q(s,a) is not a standard functional derivative and the lower-bound inequality is asserted without proof, but the collapse of Theorem 1 already undermines the abstract's convergence-rate claim. The paper does offer a plausible empirical observation (Sinkhorn penalty may smooth critic updates), but the absence of seeds, error bars, and code prevents verification; the verdict should remain REJECT.","tokens_in":9141,"tokens_out":9844,"duration_ms":99248,"concrete_test":"On the exact WAVE critic architecture (twin Q-network, 3×256 hidden units, LayerNorm), estimate the minimum eigenvalue of the Hessian of Lreg over held-out batches at several training checkpoints using autograd. If the minimum eigenvalue is non-positive (or no uniform m > 0 exists), the strong-convexity assumption in Theorem 1 fails for the implemented algorithm. Separately, check whether the scheduled learning rate a/k satisfies 2am > 1; if not, O(1/k) would not follow even under strong convexity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is an O(1/k) rate for the critic's MSE. In Section 3, Theorem 1 derives this from a standard strongly convex SGD argument, with the key assumption that the regularized loss Lreg(θQ) = LTD(θQ) + λkWε(hat Qk, hat Qk−1) is strongly convex with parameter m > 0. For the implemented twin Q-network with 3×256 hidden units and LayerNorm, LTD is nonconvex in θQ; the Sinkhorn penalty is convex in the Q-value vector but not in the network parameters, so the sum is not strongly convex. Thus the rate does not apply to the algorithm evaluated in Section 4. The paper itself concedes in Section 5 that the theoretical guarantees 'may not be universally applicable across all neural network architectures or RL tasks.' Furthermore, even granting strong convexity, the recursion φ_{k+1} ≤ (1 − 2am/k)φ_k + a^2G^2/k^2 yields the quoted O(1/k) only when 2am > 1; the paper never states this constraint on a and m. As written, the proof of the main convergence guarantee is incomplete and conditional on an assumption that the implemented method does not satisfy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes WAVE, a modification of actor-critic deep reinforcement learning in which the critic's loss is augmented by a Sinkhorn-approximated Wasserstein penalty between consecutive Q-value distributions, with an adaptively tuned regularization coefficient. The paper claims three contributions: an O(1/k) convergence-rate guarantee for the critic's mean squared error, a stability guarantee from the Wasserstein regularization, and empirical improvements over a TD3 baseline on Inverted Pendulum, Acrobot, and a custom 2D Robot Navigation task. The theoretical section presents three theorems, and the experiments report cumulative-average reward curves.","tokens_in":9404,"tokens_out":5421,"duration_ms":56831,"significance":"The motivating idea, penalizing large shifts in the Q-value distribution to stabilize actor-critic training, is reasonable and relevant, and the paper is clearly written. The authors are also candid in Section 5 about the limitations of their assumptions. However, the theoretical statements are the central contribution, and each of the three theorems has a load-bearing gap: Theorem 1 relies on strong convexity that the implemented neural-network critic does not satisfy, Theorem 2 assumes the very negative covariance it claims to establish, and Theorem 3 contains an invalid algebraic step in the contraction argument. The experimental section is too thin to substitute for the theory: it reports single curves without error bars or seeds, compares against only one baseline, and provides no code or reproducible artifacts. If the regularization idea were analyzed in a setting where it provably holds, or if the empirical study were substantially expanded, the work could be of interest; as it stands, the central claims are not supported.","major_comments":[{"comment":"The central O(1/k) convergence claim is not established for the implemented algorithm. The proof assumes that the regularized loss Lreg(θQ)=LTD(θQ)+λk Wε(Q̂k, Q̂k−1) is strongly convex with parameter m>0. The implemented critic is a twin Q-network with three hidden layers of 256 units and LayerNorm; LTD is nonconvex in θQ, and the Sinkhorn penalty is convex in the Q-value vector, not in the network parameters, so there is no reason for the sum to be strongly convex. Section 5 concedes that the theoretical guarantees 'may not be universally applicable across all neural network architectures or RL tasks.' In addition, the theorem bounds E[||θk−θ*||²], whereas the abstract promises a rate for the critic's mean squared error; no Lipschitz conversion from parameter error to Q-value MSE is supplied, so even the stated conclusion does not match the abstract's claim.","section":"Section 3, Theorem 1"},{"comment":"Even if strong convexity were granted, the proof is incomplete. The recursion φ_{k+1} ≤ (1 − 2am/k)φ_k + a²G²/k² yields the quoted O(1/k) rate only when 2am > 1; the paper never states this constraint on the learning-rate constant a and the strong-convexity constant m. For 2am ≤ 1, the standard analysis gives a slower rate, and no such case is treated. The proof also invokes 'the Lipschitz continuity of ∇θQ Lreg' in order to bound the cross term, but that assumption is not part of the theorem statement; the stated assumptions only include Lipschitz continuity of Q and boundedness of the gradient of Lreg.","section":"Section 3, Theorem 1 proof"},{"comment":"The stability guarantee is circular. The proof computes Var(gTD + λk gW) = Var(gTD) + λk² Var(gW) + 2λk Cov(gTD, gW) and then assumes Cov(gTD, gW) ≤ 0 'because gW generally opposes changes suggested by gTD.' That negative correlation is precisely the effect the theorem claims to establish, and no independent argument or empirical evidence is provided. The additional bound ||∇θQ Lreg|| ≤ GTD + λk GW only controls the magnitude of the gradient, not the variance of the stochastic parameter updates, so it does not support the variance-reduction conclusion.","section":"Section 3, Theorem 2"},{"comment":"The contraction argument contains an invalid step. From |TλQ1(s,a) − TλQ2(s,a)| ≤ γ E_{s′,a′}[|Q1(s′,a′) − Q2(s′,a′)|] − λc1|Q1(s,a) − Q2(s,a)|, the proof immediately concludes |TλQ1(s,a) − TλQ2(s,a)| ≤ (γ − λc1)|Q1(s,a) − Q2(s,a)|. This is not a valid inequality: the first term is an expectation over next states and actions, which cannot be identified with the pointwise value |Q1(s,a) − Q2(s,a)|. The preceding derivative-difference bound is also merely asserted ('By properties of the Sinkhorn distance and Lipschitz continuity') and is not proved. Consequently, the claimed contraction factor γλ = γ(1 − cλ) is not derived, and the acceleration claim is unsupported.","section":"Section 3, Theorem 3"},{"comment":"The empirical claim of superior performance is not supported by the reported evidence. The paper shows one cumulative-average-reward curve per environment, without standard deviations, number of seeds, or confidence intervals, and it compares only against a single baseline (TD3). The text gives a numeric improvement claim for the Robot Navigation environment, but the figure does not provide the aggregate data needed to verify it. Given that the theoretical guarantees do not apply to the implemented network, these experiments are the main evidence for WAVE, and as reported they are insufficient to support the abstract's statement that WAVE 'achieves superior performance compared to standard actor-critic methods.'","section":"Section 4"}],"minor_comments":[{"comment":"Lemma 1 is stated without proof or citation. The Sinkhorn distance is a biased approximation of the Wasserstein distance for ε>0, and differentiability holds only under conditions on the measures and cost; the statement should be made precise and referenced.","section":"Section 2, Lemma 1"},{"comment":"The statement that λk decreases as the agent's performance improves is not guaranteed by the update rule: λk is a function of a moving average R̄k, which can fluctuate, and when R̄k ≤ Rth the parameter is fixed at λmax rather than decreasing.","section":"Section 2, Lemma 2"},{"comment":"The learning-rate notation is inconsistent: Section 2 uses ηk for the critic update, while Theorem 1 uses αk = a/k. The relationship between the analyzed schedule and the update actually used in Algorithm 1 is never specified.","section":"Section 2 and Section 3"},{"comment":"The proof says that a 'standard result' for the recursion yields φ_k ≤ C/k, but no reference or derivation is given. Please provide a proof or a citation, and also state the condition on 2am needed for the rate.","section":"Section 3, Theorem 1 proof"},{"comment":"The caption of Figure 2 shows the adaptation formula but the axis label uses λ(t) while the text uses λk; the notation should be unified, and the figure should indicate what is plotted (the value of the regularization parameter over episodes).","section":"Section 4, Figure 2"}],"recommendation":"reject","confidential_remarks":"The paper overclaims relative to its proofs. The Section 5 limitations paragraph is honest but does not repair the gap between the theory and the implemented algorithm. I would not encourage resubmission in the current form; a substantial reframing of the theoretical claims and a much more rigorous experimental study would be needed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful part of this paper is the algorithm: an adaptively weighted Sinkhorn penalty between successive Q-value distributions added to a TD3-style critic loss. That specific combination is new, and the three small experiments do show a modest, visible improvement over TD3. The paper is clearly written and the authors are honest enough to state in Section 5 that their theoretical assumptions may not hold across architectures. That honesty stops at the abstract, which promises O(1/k) convergence and stability guarantees the body does not deliver.\n\nTheorems 1–3 are the problem. Theorem 1 is a standard strongly-convex SGD proof that does not use the Wasserstein term at all; it assumes L_reg is strongly convex, which is false for an MLP critic with a TD error and a Sinkhorn penalty. The stress-test note is right that even granting strong convexity, the proof omits the condition 2am > 1 needed for the stated O(1/k) rate. Theorem 2 assumes Cov(g_TD, g_W) ≤ 0, which is exactly the variance-reduction effect the theorem claims to prove. Theorem 3, the result most specific to the method, differentiates the Sinkhorn distance with respect to Q(s,a) in a way that is not defined for empirical distributions, and then turns an expectation bound into a pointwise one without justification. These are not minor gaps.\n\nThe experiments are illustrative at best: no error bars, no seed count, no code, and comparison only to TD3, not to the Wasserstein-based methods cited in Section 1.1. That said, the absence of negatives is also real: the method does appear to train stably in these environments, and the adaptive schedule on λ is sensible even if it needs tuning.\n\nFor a reader, the value here is the algorithmic idea and the empirical hint, not the theory. I would not cite it in its current form, and I would not send it to a serious referee expecting a publishable result. The authors should either remove the theoretical claims entirely or ground them in a setting where they actually hold, and they need to add basic experimental rigor. As is, it is a useful pointer for someone thinking about Wasserstein-based regularizers, but not a paper that supports its stated guarantees.","headline":"A genuinely new algorithmic idea undermined by overstated and flawed theoretical claims; the empirical observation is plausible but under-reported.","tokens_in":9958,"tokens_out":2010,"would_cite":false,"duration_ms":22333,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"WAVE adds an adaptively weighted Wasserstein penalty between successive Q-value distributions to the critic loss, and the paper claims this makes the regularized critic's mean squared error fall at O(1/k) while the regularized Bellman…","keywords":["Wasserstein distance","optimal transport","actor-critic methods","reinforcement learning","Sinkhorn approximation","adaptive regularization","convergence analysis","continuous control"],"falsifier":"Compute the smallest eigenvalue of the Hessian of the regularized critic loss on a three-layer critic at the states a run of WAVE actually visits; if that eigenvalue is zero or negative for any visited point, strong convexity is violated and the O(1/k) proof does not apply. A more direct check is to run WAVE with the Wasserstein term removed and compare the empirical squared parameter error against the predicted C/k curve: if the regularized and unregularized errors have the same rate and nearly the same constant, the claimed stabilizing mechanism is not carrying the result.","tokens_in":8811,"feed_emoji":"🎯","tokens_out":7809,"duration_ms":73438,"temperature":0.7,"pith_summary":"The paper proposes that the instability of actor-critic reinforcement learning can be reduced by adding a Wasserstein penalty between successive Q-value distributions to the critic's loss, with the penalty's weight adapted to the agent's recent rewards. It claims that this Wasserstein Adaptive Value Estimation (WAVE) makes the critic's mean squared error decrease at an O(1/k) rate, the critic parameters converge at O(1/sqrt(k)), and the regularized Bellman operator contract faster than the unregularized one. The point of the claim is practical: deep actor-critic agents often fail because value estimates fluctuate, and a cheap, differentiable Sinkhorn approximation of the Wasserstein distance can be inserted into the critic update to enforce temporal consistency without changing the actor.","feed_headline":"Wasserstein penalty steadies actor-critic at provable O(1/k) rate","feed_subtitle":"Temporal consistency in Q-value estimates cuts update variance; WAVE adapts the penalty weight as rewards improve.","key_machinery":"The load-bearing object is the Sinkhorn distance $W_\\varepsilon(\\hat Q_k, \\hat Q_{k-1})$, the entropically regularized optimal-transport plan between consecutive empirical distributions of critic outputs. It is differentiable in the critic parameters, so its gradient can be added directly to the TD gradient, and it tracks a geometry-aware notion of how much the Q-value distribution moved between updates. The other carrying element is the adaptive coefficient $\\lambda_k$ defined from the moving average of episode rewards, and, in the convergence proof, the strong convexity of $L_{\\mathrm{reg}}$ with parameter $m$ together with bounded gradients, which converts the gradient descent recursion into the $1/k$ bound.","core_discovery":"On its own terms, the paper's central discovery is that the regularized critic loss $L_{\\mathrm{reg}}(\\theta^Q) = L_{\\mathrm{TD}}(\\theta^Q) + \\lambda_k W_\\varepsilon(\\hat Q_k, \\hat Q_{k-1})$ -- the temporal-difference error plus an adaptive Sinkhorn-approximated Wasserstein distance between the empirical Q-value distributions of consecutive iterations -- yields provably stable critic updates. Theorem 1 shows that when $L_{\\mathrm{reg}}$ is strongly convex, the expected squared parameter error obeys $\\mathbb{E}[\\|\\theta_k - \\theta^*\\|^2] \\le C/k$. Theorem 2 argues that the penalty lowers gradient variance by opposing TD gradients, and Theorem 3 claims the regularized Bellman operator is a contraction with factor $\\gamma(1 - c\\lambda)$, strictly smaller than $\\gamma$. The adaptive schedule sets $\\lambda_k$ high when the moving average of episode rewards is below a threshold and decays it exponentially once performance improves, so the stabilizing bias fades exactly as learning succeeds.","pith_inferences":["The O(1/k) rate is the same rate that strongly convex stochastic gradient descent gives without any Wasserstein penalty, so the theoretical reason to prefer WAVE would be a better constant or lower gradient variance rather than a faster exponent; I would want a direct measurement of squared-error constants with and without the penalty to test that.","The regularizer links iterations $k$ and $k-1$ through the previous Q-distribution, so the recursion analyzed in Theorem 1 is not a standard first-order Markov gradient method; a rigorous extension would require a two-timescale or delayed-gradient analysis that the paper does not provide.","The adaptive $\\lambda$ scheme depends on episodic rewards, so WAVE may need modification for sparse or delayed reward tasks, a limitation the paper acknowledges; testing it under reward-shaping or with a learned performance estimator is a natural follow-up."],"forward_implications":["Any actor-critic implementation can incorporate WAVE by evaluating one extra Sinkhorn cost per critic update; the actor update and policy-gradient structure remain unchanged.","If Theorem 1 holds, the regularized critic's squared parameter error is guaranteed to shrink as $C/k$, so the added penalty does not slow the asymptotic rate while it bounds the size of each update.","If Theorem 3 is correct, value-function estimates contract at factor $\\gamma(1 - c\\lambda)$ instead of $\\gamma$, making WAVE strictly faster than unregularized Bellman iteration.","The reward-driven $\\lambda$ schedule protects early training with high regularization and relaxes it automatically as performance improves, reducing the bias introduced by the penalty over time.","On the three continuous-control benchmarks reported, WAVE yields cumulative average reward above the TD3 baseline, including a stated gain of roughly 25% in the 2D robot-navigation task."],"supporting_citations":[{"why":"Provides the finite-time analysis of a regularized actor-critic method that WAVE extends to Wasserstein-based regularization.","marker":"[5]"},{"why":"Supplies the TD3 baseline algorithm and target-network machinery used in the experiments.","marker":"[6]"},{"why":"Establishes the prior Wasserstein actor-critic approach using approximate Q-posteriors and barycenters that motivates the regularizer.","marker":"[12]"},{"why":"Gives a mean-field analysis showing Wasserstein-constrained actor-critic can reach global optimality at sublinear rates, supporting the value of Wasserstein regularization.","marker":"[18]"},{"why":"Shows how Wasserstein barycenters propagate uncertainty over the state-action space, an idea the paper adopts for temporal consistency.","marker":"[13]"}],"fun_headline_variants":["WAVE tames actor-critic with adaptive Wasserstein penalty","Adaptive Wasserstein penalty yields stable actor-critic","Provable O(1/k) convergence via adaptive Wasserstein loss","WAVE critic: Wasserstein penalty for stable RL updates","Adaptive Wasserstein regularization steadies actor-critic"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The convergence proof assumes the regularized critic loss is strongly convex, but a neural-network critic with a temporal-difference error and a Sinkhorn penalty is not strongly convex in general, so the O(1/k) rate is not guaranteed for the implemented algorithm.","fun_headline_variants_meta":{"raw":{"variants":["WAVE tames actor-critic with adaptive Wasserstein penalty","Adaptive Wasserstein penalty yields stable actor-critic","Provable O(1/k) convergence via adaptive Wasserstein loss","WAVE critic: Wasserstein penalty for stable RL updates","Adaptive Wasserstein regularization steadies actor-critic"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000154,"raw_usage":{"total_tokens":1179,"prompt_tokens":882,"completion_tokens":297,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":215}},"tokens_in":498,"tokens_out":297,"duration_ms":3550,"temperature":1.0,"reasoning_tokens":215,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:02:34.441303+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the smallest eigenvalue of the Hessian of the regularized critic loss on a three-layer critic at the states a run of WAVE actually visits; if that eigenvalue is zero or negative for any visited point, strong convexity is violated and the O(1/k) proof does not apply. A more direct check is to run WAVE with the Wasserstein term removed and compare the empirical squared parameter error against the predicted C/k curve: if the regularized and unregularized errors have the same rate and nearly the same constant, the claimed stabilizing mechanism is not carrying the result.","supporting_citations":[{"cited_title":"Finite-Time Analysis of Entropy-Regularized Neural Natural Actor-Critic Algorithm","cited_arxiv_id":"2206.00833","evidence_quote":"Provides the finite-time analysis of a regularized actor-critic method that WAVE extends to Wasserstein-based regularization."},{"cited_title":"In: International conference on machine learning","cited_arxiv_id":null,"evidence_quote":"Supplies the TD3 baseline algorithm and target-network machinery used in the experiments."},{"cited_title":"In: Proceedings of the AAAI Conference on Artificial Intelligence","cited_arxiv_id":null,"evidence_quote":"Establishes the prior Wasserstein actor-critic approach using approximate Q-posteriors and barycenters that motivates the regularizer."},{"cited_title":"Advances in Neural Information Processing Systems34, 15993–16006 (2021)","cited_arxiv_id":null,"evidence_quote":"Gives a mean-field analysis showing Wasserstein-constrained actor-critic can reach global optimality at sublinear rates, supporting the value of Wasserstein regularization."},{"cited_title":"Advances in Neural Information Processing Systems 32 (2019)","cited_arxiv_id":null,"evidence_quote":"Shows how Wasserstein barycenters propagate uncertainty over the state-action space, an idea the paper adopts for temporal consistency."}],"review_version":1}