{"id":"b7b086ca-be21-443d-99a8-028abbd3d168","arxiv_id":"2608.03562","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper introduces robust general-utility RL, a minimax formulation over utility uncertainty sets, and proves convergence rates for projected gradient descent-ascent and prox-extragradient algorithms.","lead":"This paper proposes a minimax framework for reinforcement learning where the policy is trained to be robust to misspecification of the utility function, not just to environment dynamics. It provides two stochastic algorithms with convergence guarantees, one for concave utilities and one for the harder nonconcave case.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3's nonconcave guarantee hinges on Assumption 8 (Minty VI), which the paper never verifies for its own exploration-maximization example; a numerical feasibility check could settle whether the rate applies to that motivating problem.","rationale":"The paper's central claim for the nonconcave regime is conditional on Assumption 8. I checked the proof structure: Theorem 3's argument in Appendix B.18 uses Assumption 8 exactly to telescope the proximal-center displacement; without it, Lemma 1's σ∥z_{k+1}−z_k∥ term is uncontrolled and the O(log K/K) bound fails. The paper does not verify Assumption 8 for its Section 5.1 example, and composing the convex function −σ_min with the quadratic map W M W^T generically destroys monotonicity, so the assumption is not automatic. This is not an internal inconsistency—the theorem is true under the assumption—but it is a load-bearing gap for the motivating application. A numerical LP feasibility test on the paper's own tabular exploration-maximization instance would settle whether the assumption holds there. This concern aligns with the reader's weakest_assumption and does not change the CONDITIONAL verdict: the framework and concave-regime analysis stand, but the nonconcave guarantee needs either verification of Assumption 8 for the example, a weakened assumption such as weak MVI, or an explicit caveat that the example may fall outside Theorem 3.","tokens_in":47797,"tokens_out":7333,"duration_ms":67543,"concrete_test":"Fix the Section 5.1 tabular instance (S=10, A=5, d′=20, d=10, random ψ, nominal W̃, Ξ={W:∥W−W̃∥F≤1}). Sample z_i=(θ_i,W_i) uniformly in Θ×Ξ, say 10^4 points, including iterates from Algorithm 2 over several seeds. For each z_i compute F(z_i)=(∇θ fξ_i(λ_{θ_i}), −∇ξ fξ_i(λ_{θ_i})) using Theorem 1 and the closed form of −σ_min(W_i M(λ_{θ_i}) W_i^T). Solve the LP feasibility problem for z⋆∈Z with constraints ⟨F(z_i), z_i−z⋆⟩≥0 for all sampled i. If the LP is infeasible, Assumption 8 is false for this instance, so Theorem 3 does not cover the motivating example; if feasible, verify with denser sampling and check whether Algorithm 2's expected gradient-mapping residual decays at the predicted O(log K/K) rate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The single load-bearing premise is Assumption 8 (Minty VI) for Theorem 3. The proof of Theorem 3 in Appendix B.18, especially Eq. (55), uses z⋆ to control the anchor-drift term σ∥z_{k+1}−z_k∥ that enters through Lemma 1; without Assumption 8, the telescoping displacement bound collapses and the O(log K/K) stationarity guarantee for Algorithm 2 does not follow. The paper's own nonconcave example (Section 5.1), fξ(λ)=−σ_min(Wξ M(λ) Wξ^T), is never shown to satisfy Assumption 8. Because −σ_min is convex in positive-semidefinite arguments but Wξ M(λ) Wξ^T is quadratic in ξ, the saddle operator F(z)=(∇θ fξ(λθ), −∇ξ fξ(λθ)) is not generally monotone; for a generic feature map ψ, the global alignment condition ⟨F(z), z−z⋆⟩≥0 for all z will typically fail. The paper's assertion that Assumption 8 'is satisfied by various structured nonconvex–nonconcave problem classes' does not cover its own motivating utility-misspecification problem, so the headline nonconcave convergence claim may not apply to that example.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes robust general-utility reinforcement learning, formulated as min_theta max_{xi in Xi} f_xi(lambda_theta), where lambda_theta is the occupancy measure of policy theta and Xi is an ambiguity set over utility parameters. It argues that this strictly generalizes general-utility RL and recovers reward-robust RL and constrained RL as special cases. For utilities that are convex in lambda and strongly concave in xi, the paper develops a projected stochastic gradient descent-ascent method (Algorithm 1) and proves convergence to a first-order stationary point of the robust envelope Gamma(theta) at rate O(1/sqrt(K)). For nonconcave utilities, it develops a stochastic prox-extragradient method (Algorithm 2) and proves convergence to a first-order stationary point of the original problem at rate O(log K / K) under a Minty variational inequality assumption. Experiments on an LLM safety-alignment task and a tabular exploration-maximization task are reported as corroborating evidence.","tokens_in":48069,"tokens_out":17758,"duration_ms":164818,"significance":"If the results hold as stated, the framework is a natural and useful extension of general-utility RL, and the algorithmic contributions for the nonconvex-concave and nonconvex-nonconcave regimes are valuable. The paper's strengths include a detailed, mostly self-contained appendix: the policy gradient theorem, finite-sample stochastic gradient estimates, envelope smoothness, and the prox-extragradient variational-inequality analysis are all developed with explicit constants. The theorems are conditional on clearly stated assumptions and do not appear to fit constants to data. The main weaknesses are (i) a false sign identity in the formal claim that reward-robust RL is a special case, (ii) an unverified Minty variational inequality for the paper's own motivating nonconcave exploration-maximization example, and (iii) a mismatch between the stated 'concave utility' regime (strong concavity in xi) and the linear-in-xi examples used to advertise the unified view. These issues are fixable, but they need to be addressed before publication.","major_comments":[{"comment":"Appendix B.1 contains the identity min_theta max_R -<R, lambda_theta> = max_theta min_R <R, lambda_theta>. This is false; the correct identity is min_theta max_R -<R, lambda_theta> = -max_theta min_R <R, lambda_theta>. With f_xi(lambda_theta) = -<xi, lambda_theta>, problem (2) is the negative of the reward-robust objective in (3), not the objective itself. The optimal policy sets coincide, so the unification claim can be repaired by stating equivalence up to negation, but Proposition 1 and the printed equality in Appendix B.1 are wrong as written.","section":"Section 3.2, Proposition 1; Appendix B.1"},{"comment":"Theorem 3's nonconcave convergence guarantee is conditional on Assumption 8 (the Minty variational inequality), but the exploration-maximization utility in Section 5.1, f_xi(lambda) = -sigma_min(W_xi M(lambda) W_xi^T), is never shown to satisfy that assumption. Because -sigma_min is convex in the positive semidefinite argument while W_xi M(lambda) W_xi^T is quadratic in W_xi, the saddle operator F is not generally monotone, and the global alignment condition <F(z), z - z*> >= 0 for all z is a stringent requirement that will typically fail for a generic feature map. Without either a proof that a structured subclass of this example satisfies Assumption 8 or a numerical verification of the condition on the actual instance used in Section 6, the claim that Algorithm 2 provides provable convergence for the paper's motivating nonconcave example is unsupported.","section":"Sections 5.1-5.2, Theorem 3"},{"comment":"Assumption 6 requires f_xi(lambda) to be strongly concave in xi, but the unifying examples in Section 3.2 (reward-robust RL and the Lagrangian view of constrained RL) are linear in xi and therefore are not covered by Theorem 2 as stated. The text refers to this as the 'concave utility' regime and says it subsumes the examples in Section 3.2, but strong concavity is strictly stronger than concavity. The paper should either explicitly state that the unifying examples are outside the analyzed algorithmic regime, or provide a regularization or limiting argument explaining how the linear case can be recovered.","section":"Section 4, Assumption 6; Section 3.2"}],"minor_comments":[{"comment":"There are typographical errors such as 'proposestochastic' and 'hyperparemters' that should be corrected in a copyedit pass.","section":"Abstract and Introduction"},{"comment":"In the remark after Theorem 2, the expression E||nabla_theta G_Gamma(theta_k)||^2 should be E||G_Gamma(theta_k)||^2; the projected gradient mapping is already a first-order stationarity measure.","section":"Theorem 2, Remark"},{"comment":"The final bound in Theorem 4 appears to contain a duplicated factor L_{theta,xi}^2; please check whether the intended coefficient is L_{theta,xi}^2 L_xi / (mu_xi^3), consistent with the bias bound from Lemma 5, rather than L_{theta,xi}^4 / mu_xi^3.","section":"Theorem 4, Appendix B.3"},{"comment":"The experiments for Algorithm 2 report the objective f_xi(lambda_theta) and the proximal subproblem gradient norms ||G^k_{Theta,sigma}|| and ||G^k_{Xi,sigma}||, but Theorem 3 bounds the original gradient-mapping residual R_GM at a uniformly sampled iterate. These proximal residuals can decrease even if the anchor drift cycles, so the reported metrics do not directly corroborate the theorem's conclusion.","section":"Section 6, Algorithm 2 experiment"},{"comment":"The text states that results are averaged over five random seeds, but Figure 1 appears to show only mean curves; per-seed curves or error bars should be included so the reader can assess variability.","section":"Appendix C, Figure 1"},{"comment":"Theorem 3 should explicitly list the choices of T_k and the Monte-Carlo budgets that appear in Proposition 6 and Corollary 1, rather than referring only to Appendix B.16, so that the stated O(log K / K) rate is self-contained.","section":"Theorem 3, statement and proof"}],"recommendation":"major_revision","confidential_remarks":"The reader's conditional verdict is broadly appropriate. The core convergence proofs are coherent and the framework is a reasonable contribution, but the false sign identity in Proposition 1, the unverified Minty VI for the motivating nonconcave example, and the strong-concavity versus linear-example mismatch all need to be corrected or explicitly scoped in a revision. I do not see a load-bearing error that would force rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. The core idea is legitimate: make the utility functional itself the adversarial player and train against a set of utilities. That is a natural gap in general-utility RL, and the paper is the first to formalize it. The concave-utility analysis (robust envelope, PGDA with O(1/sqrt K) stationarity) is careful and checks out as far as I followed. The prox-extragradient treatment for nonconcave utilities using a prox-stabilized saddle problem is also a real technical contribution; the Lemma 1/Proposition 6 chain is well structured. Credit where due: the convergence proofs are self-contained, assumptions are stated, and the sample-complexity story in Appendix B is coherent. Self-citation to Liu, Sun, Zheng 2025 is used appropriately.\n\nSoft spots. Proposition 1's proof contains a false identity: min_theta max_R -<R,lambda_theta> = max_theta min_R <R,lambda_theta>. The correct identity has a minus sign: min max -f = -max min f. This means the claimed exact equivalence to reward-robust RL is at best an argmin correspondence, not a value equality as written. Easy to fix but it should be fixed. Proposition 2's Lagrangian equivalence to constrained RL is also stated more strongly than proven: with a compact Xi that merely contains an optimal multiplier, infeasible policies with small violations and much better primary cost can beat the true constrained optimum. You need a quantitative condition on the size of Xi or an additional assumption. Again fixable.\n\nThe bigger issue is Theorem 3. The O(log K/K) rate for the nonconcave regime depends entirely on Assumption 8 (Minty VI), and the paper never verifies that the motivating exploration-maximization example in Section 5.1 satisfies it. For a generic feature map, the saddle operator is not monotone and global alignment is unlikely to hold. So the headline nonconcave convergence guarantee may not apply to the paper's own motivating example. The empirical curves in Figure 1 are suggestive but don't establish the condition.\n\nBottom line: this is a serious paper with a genuinely useful framework and mostly sound math. The canonical reductions need repairs, and the nonconcave claim needs to either verify MVI for the example or be presented with the condition made prominent rather than buried. I'd send it to review; a competent referee would catch the same things.\n\nRecommendation: engage with it, but don't take the unification claims at face value until revised.","headline":"A useful minimax reframing of robust utility RL with plausible convergence theory, but the sign error in Proposition 1 and an unverified Minty VI requirement for the nonconcave regime need fixing before I'd rely on the unification claims.","tokens_in":48603,"tokens_out":5558,"would_cite":true,"duration_ms":51486,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes robust general-utility RL, a minimax formulation that trains policies against misspecification of the utility functional, and proves convergence of two stochastic algorithms in concave and nonconcave regimes.","keywords":["robust reinforcement learning","general utility reinforcement learning","minimax optimization","utility misspecification","nonconvex-nonconcave minimax","proximal point method","stochastic extragradient","Minty variational inequality"],"falsifier":"Run Algorithm 2 on the paper's exploration-maximization example and compute the residual $R_{\\mathrm{GM}}(\\theta,\\xi)$ at the returned iterate across many seeds; if the residual does not converge to zero, or if the iterates cycle, the example would be a counterexample to the applicability of Theorem 3, indicating that Assumption 8 fails for that instance. More directly, numerically check Assumption 8 by testing whether $\\langle\\nabla_\\theta f_\\xi(\\lambda_\\theta),\\theta-\\theta_\\star\\rangle - \\langle\\nabla_\\xi f_\\xi(\\lambda_\\theta),\\xi-\\xi_\\star\\rangle \\ge 0$ holds at every feasible point of the single feature-drift instance; if it fails at even one point, the condition is violated.","tokens_in":47591,"feed_emoji":"🛡️","tokens_out":7420,"duration_ms":54469,"temperature":0.7,"pith_summary":"The paper introduces robust general-utility reinforcement learning, framed as the minimax problem $\\min_{\\theta\\in\\Theta}\\max_{\\xi\\in\\Xi} f_\\xi(\\lambda_\\theta)$, where $\\lambda_\\theta$ is the occupancy measure of policy $\\theta$ and $\\Xi$ is a prescribed uncertainty set of utility parameters. The motivation is that utility functions used at deployment often deviate from those used in training, a gap prior general-utility RL did not address. The paper shows this formulation strictly generalizes standard general-utility RL (recovered when $\\Xi$ is a singleton) and unifies reward-robust RL and constrained RL as linear special cases. It then provides two provably convergent stochastic algorithms: a projected stochastic gradient descent-ascent method for strongly concave utilities, and a stochastic prox-extragradient method for nonconcave utilities, with stationarity guarantees at rates $O(1/\\sqrt{K})$ and $O(\\log K/K)$ respectively. Experiments on LLM safety alignment and exploration maximization corroborate the convergence behavior.","feed_headline":"Minimax framework trains RL against utility misspecification","feed_subtitle":"Convergent algorithms for both concave and nonconcave utilities unify reward-robust and constrained RL.","key_machinery":"The load-bearing object in the concave regime is the robust envelope $\\Gamma(\\theta):=\\max_{\\xi\\in\\Xi} f_\\xi(\\lambda_\\theta)$, together with its differentiability and smoothness, obtained by Danskin's theorem when $f$ is strongly concave in $\\xi$. In the nonconcave regime the load-bearing object is the prox-regularized subproblem $\\Phi_k(\\theta,\\xi)=f_\\xi(\\lambda_\\theta)+\\frac{\\sigma}{2}\\|\\theta-\\theta_k\\|^2-\\frac{\\sigma}{2}\\|\\xi-\\xi_k\\|^2$, whose saddle point is characterized by a variational inequality with a $(\\sigma-L_F)$-strongly monotone operator $F_k(z)=F(z)+\\sigma(z-z_k)$. The gap function $\\mathrm{Gap}_k(z)=\\max_{z'}\\langle F_k(z),z-z'\\rangle$ measures subproblem accuracy, and the Minty variational inequality (Assumption 8) keeps successive anchors from cycling. The stochastic extragradient update (prediction at a lookahead point, then correction) damps the rotational behavior induced by nonconcavity.","core_discovery":"The central claim is that robustness to utility misspecification is naturally modeled by the minimax objective $\\min_{\\theta\\in\\Theta}\\max_{\\xi\\in\\Xi} f_\\xi(\\lambda_\\theta)$, and that this objective can be solved provably in two regimes. Under strong concavity in $\\xi$, the robust envelope $\\Gamma(\\theta)=\\max_{\\xi\\in\\Xi} f_\\xi(\\lambda_\\theta)$ is differentiable with Lipschitz gradient, so the projected stochastic gradient descent-ascent method (Algorithm 1) converges to a first-order stationary point of the envelope at rate $O(1/\\sqrt{K})$ (Theorem 2). In the nonconcave regime, where the envelope may lose differentiability and plain ascent can cycle, the paper solves a sequence of prox-regularized subproblems $f_\\xi(\\lambda_\\theta)+\\frac{\\sigma}{2}\\|\\theta-\\theta_k\\|^2-\\frac{\\sigma}{2}\\|\\xi-\\xi_k\\|^2$, using stochastic extragradient updates; under a Minty variational inequality condition (Assumption 8), Algorithm 2 converges to a first-order stationary point of the original problem at rate $O(\\log K/K)$ (Theorem 3). The framework also provides a unified view of reward-robust RL and constrained RL as linear special cases, and exhibits a nonconcave exploration-maximization example motivated by feature drift.","pith_inferences":["The paper does not analyze the combined case where both transition dynamics and utility parameters are uncertain; a natural extension is to define the uncertainty set over $(p,\\xi)$ jointly, though the current theory would need new Lipschitz and strong-concavity conditions.","Because Assumption 8 is not verified for the feature-drift example, a numerical check of the Minty inequality on that example would settle whether Theorem 3 actually applies to the paper's own motivating nonconcave instance; if it fails, the prox-extragradient guarantee would need a weaker condition.","The framework's view of uncertainty sets suggests a practical recipe for reward hacking: choose $\\Xi$ as a ball of plausible evaluation utilities around the training utility; then the worst-case training objective is a direct defense against correlated proxies, and benchmark tasks with known proxy shifts could test the benefit.","The gap-based proof structure for Algorithm 2 leaves room for accelerating the inner variational-inequality solver; the paper does not claim an optimal rate, so an improved inner iteration count could potentially lower the $O(\\log K/K)$ outer rate."],"forward_implications":["When $\\Xi$ is a singleton, the objective reduces to standard general-utility RL, so the framework is a strict generalization of that setting.","With $f_\\xi(\\lambda)=-\\langle\\xi,\\lambda\\rangle$ the reward-robust RL objective is recovered; with $\\xi$ as Lagrange multipliers and $\\Xi$ containing an optimal multiplier, constrained RL is recovered, so both prior paradigms become instances of the same minimax formulation.","In the concave regime, Theorem 2 implies that a single-loop stochastic descent-ascent update on $\\theta$ and $\\xi$ drives the projected gradient mapping of the robust envelope to zero at rate $O(1/\\sqrt{K})$.","In the nonconcave regime, Theorem 3 implies that a uniformly sampled iterate of Algorithm 2 has expected gradient-mapping residual going to zero at rate $O(\\log K/K)$, provided the Minty variational inequality holds.","The exploration-maximization example shows that robustness to feature drift (a quadratic map $W_\\xi M(\\lambda)W_\\xi^\\top$ composed with $-\\sigma_{\\min}$) is generally nonconcave in $\\xi$, so the nonconcave algorithm addresses a practically motivated case rather than a purely theoretical one."],"supporting_citations":[{"why":"Prior robust general-utility RL under transition-dynamics uncertainty; supplies gradient-estimation error bounds used in Proposition 8 and the tabular experimental setup.","marker":"(Chen et al. 2024)"},{"why":"Introduced general-utility RL and the exploration-maximization example (Example 2.2) whose feature drift motivates the nonconcave regime.","marker":"(Zhang et al. 2020)"},{"why":"Provides the policy gradient theorem for general utilities and the occupancy-measure estimation scheme used in Algorithm 4.","marker":"(Barakat, Fatkhullin, and He 2023)"},{"why":"Proximal-point stabilization of nonconvex-nonconcave minimax that underlies Algorithm 2's prox-regularized subproblems.","marker":"(Grimmer et al. 2023)"},{"why":"Supplies the Minty variational inequality as a standard convergence condition for structured nonconvex-nonconcave minimax, adopted as Assumption 8.","marker":"(Diakonikolas, Daskalakis, and Jordan 2021)"},{"why":"Envelope theorem used to establish differentiability and smoothness of the robust envelope $\\Gamma(\\theta)$ in Proposition 5.","marker":"(Danskin 1966)"},{"why":"Gap function for variational inequalities used to measure subproblem accuracy in Proposition 6.","marker":"(Juditsky, Nemirovski, and Tauvel 2011)"},{"why":"Constrained RL primal-dual method recovered as an instance of Algorithm 1 when $\\xi$ are Lagrange multipliers.","marker":"(Ding et al. 2020)"}],"fun_headline_variants":["Minimax RL robust to utility misspecification","Unified minimax view unifies robust and constrained RL","Train RL against worst-case utility shifts","Provably convergent robust general-utility RL","Robust RL for arbitrary utilities, even nonconcave"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"In the nonconcave regime, the proof assumes the existence of a global Minty variational-inequality solution — a single point $z_\\star$ toward which the entire joint gradient field aligns — and the paper does not verify this condition for its own exploration-maximization example.","fun_headline_variants_meta":{"raw":{"variants":["Minimax RL robust to utility misspecification","Unified minimax view unifies robust and constrained RL","Train RL against worst-case utility shifts","Provably convergent robust general-utility RL","Robust RL for arbitrary utilities, even nonconcave"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000484,"raw_usage":{"total_tokens":2436,"prompt_tokens":1042,"completion_tokens":1394,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":658,"completion_tokens_details":{"reasoning_tokens":1321}},"tokens_in":658,"tokens_out":1394,"duration_ms":9958,"temperature":1.0,"reasoning_tokens":1321,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:48:45.662100+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Algorithm 2 on the paper's exploration-maximization example and compute the residual $R_{\\mathrm{GM}}(\\theta,\\xi)$ at the returned iterate across many seeds; if the residual does not converge to zero, or if the iterates cycle, the example would be a counterexample to the applicability of Theorem 3, indicating that Assumption 8 fails for that instance. More directly, numerically check Assumption 8 by testing whether $\\langle\\nabla_\\theta f_\\xi(\\lambda_\\theta),\\theta-\\theta_\\star\\rangle - \\langle\\nabla_\\xi f_\\xi(\\lambda_\\theta),\\xi-\\xi_\\star\\rangle \\ge 0$ holds at every feasible point of the single feature-drift instance; if it fails at even one point, the condition is violated.","supporting_citations":[],"review_version":2}