{"id":"08f0c354-02d1-47df-aa5c-5ae3fa93aea2","arxiv_id":"2507.21397","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"MOCHA combines weighted-Chebyshev scalarization with an MGDA-style actor-critic and claims O(epsilon^-2 log) sample complexity for finding epsilon-Pareto-stationary policies, with offline KuaiRand experiments.","lead":"MOCHA is a new actor-critic method for multi-objective reinforcement learning that mixes weighted-Chebyshev scalarization with MGDA-style gradient updates, claiming to explore Pareto-stationary trade-off policies with a finite-time sample complexity. It matters for applications like recommender systems that must balance conflicting engagement metrics, because it offers a principled way to sweep the trade-off surface instead of returning one arbitrary solution.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (7)'s K_p constraint is algebraically inconsistent with the KKT stationarity of the WC problem (6), so the actor's λ-update and Theorem 3's guarantee do not follow even before nonconvexity is considered.","rationale":"The reader's weakest assumption identifies the KKT/Wolfe dual step as load-bearing. Our stress test confirms this and finds a sharper, algebraic flaw: the stationarity condition written in Eq. (7) does not match the actual KKT condition of Eq. (6), independent of the nonconvexity objection. The actor's λ-update is the mechanism that produces the common policy improvement direction, and Theorem 3's bound on E[||∇J(θ_hat) λ_hat||^2] is meaningless if λ is not computed from the correct stationarity condition. Since the theorem is stated without proof, the claimed O~(ε^-2) sample complexity is not established. The experiments are real and the empirical comparison is informative, but they cannot substitute for a valid derivation of the core algorithm. We therefore maintain the reader's rejection without adjusting the verdict.","tokens_in":13167,"tokens_out":14618,"duration_ms":179887,"concrete_test":"Perform an analytical check of the dual step: instantiate the WC problem (6) with d = 1, M = 2, f(θ) = (θ, -θ) (so G = (1, -1)), and p = (0.25, 0.75). Solve the simplex conditions G diag(p) λ = 0 and K_p λ = 0 separately; the former gives λ = (0.75, 0.25) and the latter gives λ ≈ (0.634, 0.366). If the paper's claim that (7) is the Wolfe dual of (6) is correct, these sets must coincide. This mismatch can be included in a rebuttal as a direct algebraic counterexample to Eq. (7); alternatively, if the authors intended a different K_p, they must provide a corrected definition and re-derive Theorem 3.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of finite-time Pareto-stationary convergence and O~(ε^-2) sample complexity rests on the derivation of the WC-MGDA actor update in Eqs. (6)-(8). The paper states that the Wolfe dual of (6) is (7) with K_p λ = 0, where K_p = diag(√p)√(G^T G)diag(√p). But the KKT stationarity of the Lagrangian of (6), L = ρ + Σ_i λ_i(p_i(J_i^ub - J_i(θ)) - ρ), is ∂L/∂θ = -Σ_i λ_i p_i ∇J_i(θ) = 0, i.e. ∇J(θ) diag(p) λ = 0 (equivalently, with G = -∇J, G diag(p) λ = 0). The constraint K_p λ = 0, however, is equivalent to G diag(√p) λ = 0 because null(√(G^T G)) = null(G). These two conditions are not equivalent: for d = 1, M = 2, G = (1, -1), p = (0.25, 0.75), the KKT condition gives λ = (0.75, 0.25) on the simplex, while K_p λ = 0 gives λ ≈ (0.634, 0.366). Thus Eq. (7) is not the KKT/Wolfe dual of Eq. (6), and the QP (8) and the update θ_{t+1} = θ_t - α G_t(p ⊙ λ_t) do not enforce WC-scalarization stationarity. Theorem 3 is stated without proof, and no alternative justification is given; the nonconvexity of the objectives makes the invalidity worse because strong duality for the Wolfe dual is unavailable. Under these conditions the central sample-complexity claim is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MOCHA, a multi-objective actor-critic algorithm that combines weighted-Chebyshev scalarization with an MGDA-style actor update, aiming to systematically explore Pareto-stationary solutions in MORL. The main theoretical claim is a finite-time convergence bound to an ε-Pareto-stationary solution with sample complexity O~(ε^{-2}) for any positive weight vector p. The paper also reports experiments on the KuaiRand dataset comparing MOCHA with several baselines. The central algorithmic step is a derivation, via KKT conditions and a Wolfe dual, of the actor's λ-update from the weighted-Chebyshev problem; this derivation is the key premise for the stated convergence guarantees.","tokens_in":13602,"tokens_out":4787,"duration_ms":56979,"significance":"If the theoretical claims were correct, MOCHA would be a useful contribution to MORL: it would combine the Pareto-front exploration property of Chebyshev scalarization with finite-time Pareto-stationarity guarantees, extending prior MOAC results to systematic front exploration. The paper also presents experiments on a real-world recommendation dataset and attempts to visualize Pareto-stationarity exploration. However, the central derivation and convergence theorem are not verified in the manuscript: Theorem 3 is stated without proof, and the KKT/Wolfe-dual step that generates the actor update appears algebraically incorrect. These are load-bearing issues, not presentational ones, so the claimed contribution is not currently established.","major_comments":[{"comment":"Theorem 3 is stated without a proof and no appendix or supplementary material is provided. The finite-time Pareto-stationary bound E[||∇J(θ_hat_T)λ_hat_T||^2] ≤ O(...) and the sample-complexity claim in Corollary 4 are the main contributions of the paper, yet the reader cannot verify any of the steps (the handling of the Markovian TD critic, the momentum-based λ update, and the conversion of the critic error bound into total sample complexity). This alone prevents acceptance; a complete proof, or a precise derivation from prior theorems with all parameter choices specified, is required.","section":"IV-C, Theorem 3 and Corollary 4"},{"comment":"The derivation of Eq. (7) from Eq. (6) is algebraically incorrect. The Lagrangian of Eq. (6) is L = ρ + Σ_i λ_i(p_i(J_i^ub - J_i(θ)) - ρ). Stationarity with respect to θ gives ∇J(θ) diag(p) λ = 0 (equivalently G diag(p) λ = 0 with G = -∇J). In contrast, the constraint K_p λ = 0 in Eq. (7), where K_p = diag(√p)√(G^T G)diag(√p), is equivalent to G diag(√p)λ = 0, since null(√(G^T G)) = null(G). These are not equivalent conditions unless p_i = 1 or p_i = 0. For example, with d = 1, M = 2, G = (1, -1), p = (0.25, 0.75), the KKT condition gives λ = (0.75, 0.25) on the simplex, whereas K_p λ = 0 gives λ ≈ (0.634, 0.366). Thus the QP in Eq. (8) and the subsequent update θ_{t+1} = θ_t - α G_t(p ⊙ λ_t) do not enforce the WC-scalarization stationarity condition. This invalidates the claimed link between MOCHA and the weighted-Chebyshev objective.","section":"IV-B, Step 2-a, Eqs. (6)-(8)"},{"comment":"Even if the KKT condition were corrected to G diag(p)λ = 0, the paper's use of the Wolfe dual is unjustified for the nonconvex objectives it explicitly targets. The Wolfe dual is only equivalent to the primal under convexity and constraint qualification; with nonconvex J(θ) (as in neural-network policies), strong duality can fail. The manuscript offers no alternative justification for replacing Eq. (6) by the dual formulation in Eq. (7), so the actor update lacks a principled foundation. This is a separate correctness risk from the algebraic mismatch above.","section":"IV-B, Eq. (7)"},{"comment":"The experiments use weight vectors that violate the p_min > 0 assumption required by Theorem 3. Table II uses p = (0.2, 0.2, 0.2, 0, 0.4), which has p_min = 0; the one-hot vectors in Fig. 1 contain four zero entries; Table III contains ablation vectors with zeros (e.g., abl1 has dislike = 0) or values of 0.0001. Since Theorem 3 and Corollary 4 require p_min > 0, these experiments do not provide evidence for the theoretical claims. The paper should either restrict the experimental setting to positive weights or extend the theory to handle zero entries.","section":"V, Table II, Table III, and Fig. 1"}],"minor_comments":[{"comment":"The spelling of the method name is inconsistent: the abstract and title use 'Weighted-Chebychev' while the body uses 'Weighted-Chebyshev'; please standardize.","section":"Throughout"},{"comment":"The phrase 'solving for λ under the current θ-value yields a λ-weighting...' is not precise: Problem (8) contains both λ and θ as optimization variables, but the algorithm fixes θ when solving for λ. This should be stated as a conditional update, not as a joint solution.","section":"IV-B, Step 2-a, Eq. (8)"},{"comment":"In Corollary 4, the critic error condition is written as E[||w_i^t - w_i^{*,t}||^2] = O(ε) for all i ∈ [M], t ∈ [M]; the range for t should presumably be t ∈ [T], not [M]. Please correct this typo.","section":"IV-C, Corollary 4"},{"comment":"Reference [15] and reference [32] appear to be the same work (Xiao, Ban, and Ji, 'Direction-oriented multi-objective learning'), with different venue strings; please deduplicate or cite consistently.","section":"References"},{"comment":"After the critic loop, the notation sets w_i^t = w_i^k but w_i^k is already defined inside the loop; clarify whether w_i^t refers to the final critic iterate and use consistent superscripts.","section":"IV-B, Algorithm 1"}],"recommendation":"reject","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading for the algorithmic idea, but the central theory does not hold up as written. The genuine novelty is combining weighted-Chebyshev scalarization with an MGDA-style actor-critic to systematically explore Pareto-stationary solutions in MORL. That synthesis isn't in the prior work I'm aware of, including the same group's MOAC. The paper is clearly written and uses a real large-scale recommendation dataset, which is a plus.\n\nThe problems are serious. First, the derivation of the WC-MGDA update in Eqs. (6)-(8) rests on a Wolfe dual step that is algebraically wrong. The KKT stationarity of the WC problem (6) with respect to theta gives G diag(p) lambda = 0, but Eq. (7) imposes K_p lambda = 0 where K_p = diag(sqrt(p)) sqrt(G^T G) diag(sqrt(p)), which is equivalent to G diag(sqrt(p)) lambda = 0. These differ by a factor of p versus sqrt(p); the stress-test example with d=1, M=2, G=(1,-1), p=(0.25,0.75) shows the simplex solutions are different. So the lambda-update and the actor direction do not actually enforce the WC stationarity condition. This is not a minor technicality; it's the load-bearing premise that connects the algorithm to Theorem 3.\n\nSecond, Theorem 3 is stated without proof and there is no appendix. Even if the derivation were correct, the main claim would be unverifiable from the manuscript. Third, the experiments use weight vectors with zero entries (e.g., p=(0.2,0.2,0.2,0,0.4)) even though Theorem 3 assumes p_min > 0, and no error bars are reported. That weakens the empirical claims considerably.\n\nWhat's good: the idea is fresh, the problem is important, and the paper is well-structured. With a correct derivation, a full proof, and an honest treatment of the zero-weight cases, this could become a solid contribution to the MORL literature. In the current form, however, I would not accept it, and I wouldn't send it to a serious referee because the central mathematical claim is demonstrably wrong. If the authors fix the derivation and provide the proof, it's worth a second look.","headline":"Interesting algorithmic idea, but the central derivation has a load-bearing algebraic error and the main theorem is unproven; reject in this form.","tokens_in":14154,"tokens_out":7334,"would_cite":false,"duration_ms":75200,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MOCHA claims to explore Pareto-stationary policies in multi-objective RL at single-objective sample-complexity rates.","keywords":["multi-objective reinforcement learning","Pareto stationarity","weighted-Chebyshev scalarization","actor-critic","sample complexity","multiple-gradient descent","Pareto front exploration","offline recommendation"],"falsifier":"Set up a two-objective MOMDP with a known nonconvex Pareto front, e.g. a small tabular task where the set of achievable return pairs can be computed exactly; run MOCHA with refined $p$ grids and large batch sizes so the $O(1/B)$ and critic-error terms vanish. If the reached policies do not accumulate on the true Pareto-stationary set, or if the empirical stationarity gap exceeds the $O(1/T+\\zeta_{\\mathrm{approx}})$ prediction, the central claim fails. More directly, one can test the load-bearing duality step by searching for a parameter $\\theta$ of a nonconvex objective where the optimal value of $\\min \\rho$ subject to $p\\odot(J^*_{\\mathrm{ub}}-J(\\theta))\\le \\rho\\mathbf{1}$ is strictly less than the Wolfe dual value in Eq. (7); such a gap would sever the derivation of the $\\lambda$ update.","tokens_in":12943,"feed_emoji":"🎯","tokens_out":9258,"duration_ms":100319,"temperature":0.7,"pith_summary":"The paper proposes MOCHA, an actor-critic algorithm for multi-objective reinforcement learning, and argues that it can systematically explore Pareto-stationary policies — policies that no other policy dominates in every reward objective — with finite-time guarantees. The key claim is that for any positive weight vector $p$, MOCHA converges to a neighborhood of a Pareto-stationary solution with a bound of order $1/T$ plus batch, critic, and approximation error terms, and that the total sample complexity per exploration direction is $\\tilde{\\mathcal{O}}(\\epsilon^{-2})$ once learning rates are chosen as specified. This matters because most practical MORL methods either find one arbitrary Pareto-stationary policy or lack rates; MOCHA ties the weight vector $p$ to a specific trade-off and lets a user sweep $p$ to trace the Pareto-stationarity front. The convergence rate matches the single-objective actor-critic rate, so the multi-objective exploration does not, order-wise, cost extra samples.","feed_headline":"MOCHA explores Pareto-stationary policies at single-objective speed","feed_subtitle":"A weighted-Chebyshev actor-critic sweeps reward trade-offs with sample complexity matching single-objective RL.","key_machinery":"The load-bearing object is the Wolfe dual reformulation of the weighted-Chebyshev scalarization, a scalarization that converts a vector objective into $\\min_x \\|p \\odot F(x)\\|_\\infty$. To explore the Pareto front, the paper converts reward maximization into a regret-minimization problem $\\min_\\theta \\|p\\odot(J^*_{\\mathrm{ub}}-J(\\theta))\\|_\\infty$, introduces an auxiliary variable $\\rho$, and writes the Lagrangian dual with matrix $K_p=\\operatorname{diag}(\\sqrt{p})\\sqrt{G^\\top G}\\operatorname{diag}(\\sqrt{p})$. The condition $K_p\\lambda=0$ together with $\\mathbf{1}^\\top\\lambda=1$, $\\lambda\\ge 0$ characterizes a WC-guided common-descent weighting; because this condition need not hold at every iterate, the algorithm instead minimizes $\\|K_p\\lambda\\|^2 - u\\lambda^\\top(p\\odot(J^*_{\\mathrm{ub}}-J(\\theta)))$ subject to the simplex constraint, a convex QP that mixes MGDA's descent objective with WC's exploration objective. The policy update then moves along $G_t(p\\odot\\lambda_t)$, and the momentum update $\\lambda_t=(1-\\eta_t)\\lambda_{t-1}+\\eta_t\\hat\\lambda_t^*$ suppresses cumulative bias from the $\\lambda$-weighting. The role of this machinery is to turn a user-specified $p$ into a concrete gradient-combination direction while retaining the $O(1/T)$ convergence structure of single-objective actor-critic.","core_discovery":"On its own terms, the paper's central discovery is that the weighted-Chebyshev scalarization and the multiple-gradient descent algorithm can be combined inside an actor-critic loop so that the $\\lambda$-weighting that selects a common policy-improvement direction is explicitly guided by a user-chosen exploration weight $p$. Specifically, Algorithm 1 runs parallel TD critics, computes per-objective policy gradients from TD errors, solves a convex quadratic program over $\\lambda$ derived from the Wolfe dual of the WC problem, mixes the solution with momentum, and updates $\\theta_{t+1} = \\theta_t - \\alpha G_t(p \\odot \\lambda_t)$. Theorem 3 states that $E\\|\\nabla_\\theta J(\\theta_{\\hat T})\\lambda_{\\hat T}\\|_2^2$ is at most $O(1/(T(1+2\\sum_t\\eta_t/p_{\\min}^2)))+O(1/B)+O(\\max_j E\\|w^j_t-w^{j,*}_t\\|_2^2)+O(\\zeta_{\\mathrm{approx}})$. With $\\eta_t=p_{\\min}^2/t^2$, the first term is $O(1/T)$, and Corollary 4 converts the bound into an $\\tilde{\\mathcal{O}}(\\epsilon^{-2}\\log(\\epsilon^{-1}))$ sample complexity for an $\\epsilon$-Pareto-stationary solution. The paper also claims that the all-ones $p$ case recovers the weight-free MOAC convergence result, making MOCHA a strict generalization.","pith_inferences":["The same WC-dual-plus-MGDA trick should transfer to non-RL multi-objective optimization with stochastic gradients, which would extend the paper's insight beyond actor-critic to any setting where a common descent direction is needed.","A practical implication the authors leave implicit: to avoid tiny $p_{\\min}$ slowdowns, one could adaptively resample or soften weight vectors near simplex boundaries, or choose the exploration set so that $p_{\\min}$ is bounded below.","If the Wolfe duality step fails for nonconvex objectives, the $\\lambda$ update may still work as a heuristic; a testable extension is to measure on small MOMDPs with known Pareto fronts whether the dual-derived $\\lambda$ actually produces common ascent directions.","The paper's $\\tilde{O}(\\epsilon^{-2})$ claim is per exploration vector; the total cost of tracing a front over $n$ weight vectors is $n\\cdot\\tilde{O}(\\epsilon^{-2})$, so the real question for practice is how to choose the coarsest $p$-grid that still covers the front."],"forward_implications":["For each fixed weight vector $p$ with $p_{\\min}>0$, MOCHA reaches an $\\epsilon$-Pareto-stationary policy with $\\tilde{\\mathcal{O}}(\\epsilon^{-2})$ samples, so the cost of exploring one trade-off direction is the same order as single-objective actor-critic.","Sweeping $p$ over a uniform cover of the simplex yields a discrete approximation of the Pareto-stationarity front; the per-direction complexity does not grow with the number of objectives $M$.","Choosing $\\eta_t=p_{\\min}^2/t^2$ restores the $O(1/T)$ leading term, so the momentum schedule can be set from the smallest entry of $p$; small $p_{\\min}$ entries slow the leading term and demand more iterations.","With $p=\\mathbf{1}$ (after normalization), the results recover the weight-free MOAC bound in [4], which positions MOCHA as its generalization rather than a separate algorithm.","On the KuaiRand offline recommendation data, MOCHA reports higher Click, Like, and WatchTime scores than SDMGrad, TSCAC, and MOAC for the tested weight vector, and a wider radar-chart footprint as $p$ varies, matching the exploration claim."],"supporting_citations":[{"why":"Supplies the MORL actor-critic design (parallel TD critics, MGDA actor, M-independence analysis) that MOCHA extends, and the weight-free baseline whose rates MOCHA recovers when $p$ is all ones.","marker":"[4]"},{"why":"Defines the MGDA common-descent problem $\\min_\\lambda\\|K\\lambda\\|^2$ under a simplex constraint, the descent mechanism MOCHA's actor relies on.","marker":"[10]"},{"why":"Provides the Wolfe dual / Pareto-stationarity-induced reformulation that yields the $K_p\\lambda=0$ condition and the objective in Problem (8).","marker":"[19]"},{"why":"Supplies the equivalence between weak Pareto optimality and WC-scalarization minimizers (Lemma 1), which justifies sweeping $p$ to explore the front.","marker":"[20]"},{"why":"Single-objective actor-critic finite-time analysis that MOCHA's proof pattern and step-size choices follow.","marker":"[24]"},{"why":"Provides the KuaiRand offline dataset, reward signals, and the TSCAC baseline used in the experiments.","marker":"[2]"},{"why":"Provides the SDMGrad baseline, a direction-oriented multi-objective method compared against MOCHA for Pareto exploration.","marker":"[32]"}],"fun_headline_variants":["MOCHA: Pareto-stationary RL at single-objective sample cost","Weighted-Chebyshev actor-critic sweeps Pareto-stationary policies efficiently","MOCHA explores all reward trade-offs as fast as one objective","Multi-objective actor-critic with epsilon-2 sample complexity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Wolfe dual reformulation in Eq. (7) is exact for the generally nonconvex regret-minimization objective; if strong duality fails there, the $\\lambda$ weighting and hence the actor gradient update lose the principled connection to weighted-Chebyshev scalarization.","fun_headline_variants_meta":{"raw":{"variants":["MOCHA: Pareto-stationary RL at single-objective sample cost","Weighted-Chebyshev actor-critic sweeps Pareto-stationary policies efficiently","MOCHA explores all reward trade-offs as fast as one objective","Multi-objective actor-critic with epsilon-2 sample complexity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000288,"raw_usage":{"total_tokens":1769,"prompt_tokens":1108,"completion_tokens":661,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":724,"completion_tokens_details":{"reasoning_tokens":583}},"tokens_in":724,"tokens_out":661,"duration_ms":7269,"temperature":1.0,"reasoning_tokens":583,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:48:01.245148+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Set up a two-objective MOMDP with a known nonconvex Pareto front, e.g. a small tabular task where the set of achievable return pairs can be computed exactly; run MOCHA with refined $p$ grids and large batch sizes so the $O(1/B)$ and critic-error terms vanish. If the reached policies do not accumulate on the true Pareto-stationary set, or if the empirical stationarity gap exceeds the $O(1/T+\\zeta_{\\mathrm{approx}})$ prediction, the central claim fails. More directly, one can test the load-bearing duality step by searching for a parameter $\\theta$ of a nonconvex objective where the optimal value of $\\min \\rho$ subject to $p\\odot(J^*_{\\mathrm{ub}}-J(\\theta))\\le \\rho\\mathbf{1}$ is strictly less than the Wolfe dual value in Eq. (7); such a gap would sever the derivation of the $\\lambda$ update.","supporting_citations":[{"cited_title":"Finite-Time Convergence and Sample Complexity of Actor-Critic Multi-Objective Reinforcement Learning","cited_arxiv_id":"2405.03082","evidence_quote":"Supplies the MORL actor-critic design (parallel TD critics, MGDA actor, M-independence analysis) that MOCHA extends, and the weight-free baseline whose rates MOCHA recovers when $p$ is all ones."},{"cited_title":"Multiple-gradient descent algorithm (mgda) for multi- objective optimization,","cited_arxiv_id":null,"evidence_quote":"Defines the MGDA common-descent problem $\\min_\\lambda\\|K\\lambda\\|^2$ under a simplex constraint, the descent mechanism MOCHA's actor relies on."},{"cited_title":"A multi-objective/multi-task learning framework induced by pareto stationarity,","cited_arxiv_id":null,"evidence_quote":"Provides the Wolfe dual / Pareto-stationarity-induced reformulation that yields the $K_p\\lambda=0$ condition and the objective in Problem (8)."},{"cited_title":"Improving Sample Complexity Bounds for (Natural) Actor-Critic Algorithms","cited_arxiv_id":"2004.12956","evidence_quote":"Single-objective actor-critic finite-time analysis that MOCHA's proof pattern and step-size choices follow."},{"cited_title":"Two-stage constrained actor-critic for short video recommendation,","cited_arxiv_id":null,"evidence_quote":"Provides the KuaiRand offline dataset, reward signals, and the TSCAC baseline used in the experiments."},{"cited_title":"Direction-oriented multi-objective learning: Simple and provable stochastic algorithms,","cited_arxiv_id":null,"evidence_quote":"Provides the SDMGrad baseline, a direction-oriented multi-objective method compared against MOCHA for Pareto exploration."}],"review_version":1}