{"id":"9039d6f6-5d0e-4eb5-9ed0-16888a257366","arxiv_id":"2506.16753","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"VALT computes the optimal observation adversary from the agent's own value function via soft-constrained optimization, enabling sample-efficient off-policy robust RL without a separately trained attacker.","lead":"Researchers show that a reinforcement learning agent can be hardened against corrupted sensor inputs by computing the optimal attacker from the agent's own value function instead of training a separate adversarial agent. The result is an off-policy robust RL method that needs no extra environment interactions, which matters for deploying RL in safety-critical settings like robotics.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The soft-regularized adversary in Definition 4.1 is never shown to approximate the hard worst-case adversary of Eq. (2); with the finite α_attk values used (2–8), the trained policy may be robust only against the softened surrogate, not against the true worst case.","rationale":"The paper's own framing is honest: Section 4 introduces a soft-constrained adversary, and Appendices E.3 and D.2 acknowledge distribution mismatch and task-specific behavior policies. My reading agrees with the reader: the load-bearing step is the unquantified relationship between the soft fixed point used in training and the hard worst-case objective used at evaluation. The symmetry theorem is a correct statement about the soft game; it does not say the soft adversary is close to the hard adversary. Since all robustness numbers in Table 1 are against hard-style attacks (PGD, SA-RL, PA-AD), the headline claim would be misleading if a stronger or exact attacker exploits the gap. The empirical evidence is supportive but mixed: HalfCheetah and Ant show gains, Hopper and Walker2d show little or no gain, and clean-reward degradation is large; standard errors are high. The data-coverage part of the concern is directly supported by the paper's own Adv0 ablations. I also note that the proof of Theorem 4.8 contains a monotonicity/additivity slip, but the underlying soft-min Lipschitz argument appears repairable, so I do not base the verdict on that. I recommend no change to the reader's CONDITIONAL verdict: the idea is promising and the implementation is released, but acceptance should remain conditional on a transfer check or a scoped claim.","tokens_in":48344,"tokens_out":16896,"duration_ms":195029,"concrete_test":"Implement a small tabular SA-MDP (e.g., 5–10 states, 3–5 perturbations per state) and compute: (i) the exact hard max-min policy value via dynamic programming over B_ε; (ii) the VALT policy by iterating the soft Bellman operator of Eq. (7) with α_attk = 2, 4, 8 and the policy improvement step of Eq. (13); (iii) the hard worst-case value of that VALT policy under the exact adversary, and the soft-predicted value. If V_hard(VALT)/V_hard(max-min) < 0.8 at the paper's α_attk values, or if the ratio does not approach 1 as α_attk→0, the soft surrogate does not transfer to the hard worst-case objective.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that VALT trains against a worst-case observation adversary at single-agent sample cost — rests on treating the soft-constrained adversary of Definition 4.1 as a faithful surrogate for the hard max-min adversary in Eq. (2). The paper proves symmetry of the soft fixed point (Theorem 4.6) and contraction (Theorems 4.8 and 4.10), but no theorem bounds the gap between V_soft(π) and V_hard(π) = min_{ν∈B_ε} J(ν,π), nor gives a rate at which the soft optimum converges to the hard optimum as α_attk→0. The values used in Table 6 (α_attk = 2–8) are not small, so the soft adversary assigns non-negligible mass to non-worst perturbations and the policy can exploit this averaging. The same issue affects the implemented approximations: VALT-EPS uses two-step PGD on Q(s, μ(˜s)) rather than the exact argmin in Eq. (12), and VALT-SOFT fits ν_model by a KL loss on replay states; neither is shown to realize the operator whose contraction is proved. The behavior-policy experiments in Appendix E.3 (Adv0 collapse on HalfCheetah and Ant) show that the off-policy Q-function is not reliable at perturbed states outside the replay distribution, and no importance weighting or coverage bound is supplied. Thus the theoretical support establishes a well-defined soft game but not transfer to the hard worst-case robustness claimed in the abstract and evaluated by hard-style attacks.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VALT, an off-policy actor-critic method for robustness to bounded observation perturbations. It replaces the explicit alternating adversary-RL loop of ATLA with a soft-constrained adversary (Definition 4.1) whose value function is argued, via a symmetry theorem (Theorem 4.6), to be obtainable from the agent's own action-value function. Two instantiations are given: VALT-SOFT, based on a closed-form KL-regularized adversary (Lemma 4.7), and VALT-EPS, based on a Dirac-mixture approximation of an alpha-divergence-regularized adversary (Eq. (12)). Contraction of the resulting Bellman operators is proved (Theorems 4.8 and 4.10). Experiments on four MuJoCo benchmarks compare the methods with PPO- and SAC-based baselines and report improved sample efficiency and robustness under heuristic and learned attacks.","tokens_in":48661,"tokens_out":10938,"duration_ms":113445,"significance":"The proposed framework is potentially valuable: if the soft-adversary surrogate faithfully represents the hard worst-case problem, VALT would provide a principled off-policy alternative to alternating training at roughly single-agent sample cost. The paper's strengths include concrete contraction and symmetry theorems, a public implementation, and an evaluation protocol that uses external RL-trained attackers rather than fitting the evaluation to the method. The behavior-policy ablations in Appendix E.3 are also informative about where off-policy robust methods break down. The significance is tempered, however, by the absence of a formal link between the soft objective and the hard max-min objective, and by the gap between the idealized operators analyzed and the implemented approximations.","major_comments":[{"comment":"No theorem or bound connects the soft-constrained adversary in Definition 4.1 to the hard worst-case adversary in Eq. (2), and the finite values of alpha_attk used in the experiments (2, 4, and 8 in Table 6) are not accompanied by any analysis showing that the soft optimum is close to the hard max-min solution. Without such a bound, the abstract's claim that the method handles 'worst-case' observation perturbations is not supported by the theory; the experiments can only demonstrate robustness against the softened surrogate and against external attackers that are themselves not certified to reach the true worst case.","section":"Section 4.1, Eq. (6), and Table 6"},{"comment":"There is an inconsistency in the definition of H(pi composed with nu). Definition 4.1 and Proposition 4.11 treat pi composed with nu as a composed conditional distribution, whose entropy is the entropy of the mixture distribution, whereas the derivation of Lemma 4.7 in Appendix B.2 explicitly uses the averaged conditional entropy E_nu[H(pi(.|s_tilde))]. These are different functionals of nu and pi. Because the closed-form solution in Eq. (9) and the contraction proof in Theorem 4.8 rely on the linearity of the entropy term in nu, the entropy term must be defined consistently for VALT-SOFT to be the optimizer of the stated objective.","section":"Section 4.1, Lemma 4.7, and Appendix B.2"},{"comment":"The contraction theorems apply to idealized operators, not to the implemented algorithms. VALT-EPS approximates Eq. (12) by two-step PGD on Q(s, mu(s_tilde)) with a randomized start and a scheduled kappa_worst (Table 5), and VALT-SOFT fits a variational model nu_psi by minimizing the KL loss in Eq. (57) with periodic resets (Algorithm 3). No approximation error bounds or stationarity guarantees are provided for either implementation, so the theory does not directly certify the algorithms that are actually evaluated.","section":"Section 4.3, Appendix C.1, and Appendix C.2"},{"comment":"The replay buffer is collected under a behavior policy that uses uniform perturbations or a 50:50 mix rather than the soft adversary nu_soft, so the distribution used for the off-policy Bellman update need not cover the support of the soft adversary. The behavior-policy experiments in Appendix E.3, especially the Adv0 collapse in HalfCheetah and Ant, show that the learned Q-function is unreliable at perturbed states outside the replay distribution, and no importance weighting or coverage bound is supplied. This gap is acknowledged in Section 4.4, but it is load-bearing for the claim that the off-policy algorithm implements the symmetric soft game.","section":"Section 4.4 and Appendix E.3"}],"minor_comments":[{"comment":"The heading contains a typo: 'Decesion' should be 'Decision'.","section":"Section 3.1 heading"},{"comment":"The y-axis label reads 'Avrage Episodic Return' in both subfigures; it should be 'Average Episodic Return'.","section":"Figure 2"},{"comment":"These numbered items define Bellman operators and value functions rather than stating propositions; relabeling them as definitions would make the logical structure clearer.","section":"Propositions 4.4 and 4.5"},{"comment":"The representational assumption Q_nu(s, s_tilde) = E_pi[-Q'_pi(s, a_tilde)] should be stated explicitly as a condition on the adversary's value-function class, and the proof should show directly that the fixed point constructed in Appendix B.1 lies in that class.","section":"Theorem 4.6"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a machine-learning venue and the empirical contribution is substantial. The main risk is that the central claim is stated more strongly than the theory supports; the authors should either add a quantitative soft-to-hard gap result, or substantially weaken the worst-case framing and clarify which claims apply to the soft surrogate only. I would not recommend rejection, because the core contraction and symmetry results and the empirical comparisons appear useful, but the theoretical and implemented algorithms must be aligned before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this paper is worth your time. The central idea—replace the learned adversary in ATLA-style training with a soft-optimal adversary computed from the agent's own Q-function via sign symmetry—is genuinely new to observation-robust RL, and it removes the sample-efficiency bottleneck of prior on-policy methods. The contraction theory (Theorems 4.8 and 4.10) is standard f-divergence soft-max/min duality, cleanly re-derived and correctly attributed; that part holds up. The code is released, which helps.\n\nWhat the paper does well: the empirical study is substantial and honest. Against ATLA, PA-ATLA, WocaR, and Robust-SAC, VALT-EPS and VALT-SOFT show strong robustness gains on HalfCheetah and Ant under learned attacks, with clean ablations (w/o PE, w/o PI) and a behavior-policy coverage analysis in Appendix E.3. They also state their limitations plainly in Appendix I. That is a serious paper.\n\nThe soft spot—and it is a real one—is that the soft-constrained adversary in Definition 4.1 is never shown to approximate the hard worst-case adversary of Eq. (2). There is no bound on V_soft − V_hard, no rate as α_attk→0, and the α_attk values used (2–8) are not small. The policy may be robust to the softened surrogate, not to the true worst case. The implemented VALT-EPS uses two-step PGD, not the exact argmin; VALT-SOFT fits a model with a KL loss; neither is proven to realize the operator whose contraction is proved. And the off-policy coverage issue is not solved: Appendix E.3 shows that without adversarial samples in the replay buffer (Adv0) the Q-function is unreliable at perturbed states, and no importance weighting or coverage bound is offered. These concerns do not sink the core method, but they do mean the abstract's claim of worst-case robustness is stronger than the theory.\n\nWho should read it: anyone working on robust RL, especially on making adversarial training sample-efficient. It is a useful benchmark and a good cautionary example of how soft relaxation changes what you actually optimize.\n\nRecommendation: send it to peer review. It deserves a serious referee. I would ask the authors to either bound the soft-hard gap or rephrase to 'soft-constrained robustness,' and to add more seeds and tighter statistics. The condition is real but not fatal.","headline":"VALT is a genuinely off-policy way to train against a soft observation adversary, with clean contraction theory, but the paper never closes the gap to the hard worst-case attacker it claims to beat.","tokens_in":49268,"tokens_out":4318,"would_cite":true,"duration_ms":43400,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that worst-case observation robustness can be trained with a single off-policy actor-critic, because the adversary's soft-optimal value function is exactly the negative of the agent's soft-worst value function.","keywords":["adversarial observation robustness","off-policy reinforcement learning","soft actor-critic","symmetric policy evaluation","state-adversarial MDP","f-divergence regularization","virtual alternating training","worst-case value function"],"falsifier":"Train VALT-SOFT or VALT-EPS on a continuous-control benchmark, then attack the resulting policy with a much stronger learned adversary, such as SA-RL or PA-AD with many restarts, or with highly fine-grained PGD using dozens of random starts. If the worst-case return found at evaluation falls well below the soft-worst value that the training Bellman update optimizes—especially in regions where the uniform prior had little mass—the symmetry identity has not transferred from the soft adversary to true worst-case robustness.","tokens_in":48058,"feed_emoji":"🛡️","tokens_out":6333,"duration_ms":63092,"temperature":0.7,"pith_summary":"Off-policy methods have been hard to use for adversarial observation robustness because the standard recipe alternates two learners—an agent and an attacker—each needing its own environment rollouts. This paper tries to remove the attacker from the training loop entirely. It defines a soft-constrained adversary whose objective penalizes divergence from a prior over allowed perturbations, then proves a symmetry: the adversary's soft-optimal value is the negative of the agent's soft-worst value. Therefore the attacker's optimal policy can be read off from the agent's own action-value function, and robust training reduces to one soft-worst Bellman update inside an off-policy actor-critic like SAC. If this is right, robust policies against observation perturbations would come at roughly the sample cost of ordinary SAC, with no alternating adversarial RL.","feed_headline":"One policy learns worst-case robustness—no second agent","feed_subtitle":"Symmetric policy evaluation turns the attacker's value into the agent's mirror, so robustness costs no extra environment rollouts.","key_machinery":"The load-bearing object is the soft-constrained adversary of Definition 4.1, whose objective $\\tilde{J}[\\nu,\\pi] = J[\\nu,\\pi] + \\alpha_{\\mathrm{ent}}H(\\pi\\circ\\nu) + \\alpha_{\\mathrm{attk}}D_f(\\nu\\|p)$ replaces the hard constraint $\\nu\\in B_\\epsilon$ by a divergence penalty toward a prior $p$ over the $\\epsilon$-ball. Around this object the paper constructs two Bellman operators, $T^\\pi_{\\nu_{\\mathrm{soft}}}$ for the agent and $T^{\\nu_{\\mathrm{soft}}}_\\pi$ for the adversary, and proves that they are $\\gamma$-contractions whose fixed points are negatives of each other. This symmetry is what does the work: it turns adversarial training, normally a nested max-min problem, into a single soft-min evaluation from the agent's side. The two algorithms are then distinguished by which $f$-divergence yields the soft adversary—an exact KL form in VALT-SOFT and an extreme-$\\alpha$ form collapsed to a worst-plus-uniform mixture in VALT-EPS.","core_discovery":"The central discovery is Theorem 4.6: for a fixed agent policy, the Bellman evaluation of the soft-optimal adversary is the mirror image of the agent's soft-worst-case Bellman evaluation, and the two operators share fixed points of opposite sign, $V^{\\nu^\\star\\mathrm{soft}}_\\pi = -V^\\pi_{\\nu^\\star\\mathrm{soft}}$. Concretely, the adversary's action-value function can be expressed through the agent's as $Q^\\nu_\\pi(s,\\tilde{s}) = \\mathbb{E}_\\pi[-Q^\\pi_\\nu(s,\\tilde{a})]$, so an explicit adversarial RL process is unnecessary. The paper builds two practical variants on this identity. VALT-SOFT uses a KL-divergence penalty, whose soft-optimal adversary has the closed form $\\nu^\\star_{\\mathrm{soft}}(\\tilde{s}|s) \\propto p(\\tilde{s}|s)\\exp(-V^\\pi(s,\\tilde{s})/\\alpha_{\\mathrm{attk}})$, and learns a variational model of it. VALT-EPS instead uses the $\\alpha$-divergence family and approximates the adversary as a mixture of the worst state (found by projected gradient descent on the agent's value) and uniform noise over the epsilon-ball. Both Bellman operators are proven $\\gamma$-contractions, and policy improvement with the fixed soft adversary is proven monotonic. On MuJoCo control tasks the methods meet or beat alternating-training baselines in worst-case attacked score while using 0.5 to 3 million environment steps, matching plain SAC's sample budget.","pith_inferences":["We infer that the same symmetry argument should transfer to other off-policy bases, such as TD3 or DQN, with the soft-worst Bellman update replacing the ordinary target; implementing VALT on those bases and checking whether the contraction and robustness gains persist would be a direct test of the mechanism.","We infer that the robustness-accuracy trade-off is governed by the attack temperature $\\alpha_{\\mathrm{attk}}$: very small values approach the hard worst case but may reintroduce training instability, so scheduling $\\alpha_{\\mathrm{attk}}$ from large to small offers a principled curriculum, a possibility the paper only explores in one Ant experiment.","We infer a concrete failure mode from the paper's acknowledged data-coverage gap: if the replay buffer never contains states where the true worst-case attacker would go, the soft adversary's support is empty there, so evaluation-time attacks could find vulnerabilities the training adversary never selected.","We infer that the quantal-response-equilibrium interpretation, which the paper offers as an explanation for stability, also predicts a smooth interpolation between a uniform random attacker and the fully worst-case attacker; measuring that interpolation on a single benchmark would make the role of the f-divergence regularizer explicit."],"forward_implications":["Robust training against observation perturbation becomes compatible with off-policy actor-critic in continuous control, at a sample budget comparable to plain SAC rather than double the budget of alternating training.","No second policy or value network needs to explore the environment, because the adversary's soft-optimal value is computed from the agent's own action-value function.","The soft-constrained formulation gives a stable fixed-point learning target, avoiding the sharp-saddle instability that alternating max-min training can produce.","The learned soft-worst value doubles as a white-box robustness metric: the paper's ablations show that removing the adversary from policy improvement collapses training in high-dimensional tasks.","The framework naturally extends to discrete-action domains and to other off-policy bases, as the underlying symmetry and contraction arguments are not specific to SAC."],"supporting_citations":[{"why":"Defines alternating training with a learned optimal adversary (ATLA), the baseline that VALT replaces by substituting soft-worst evaluation for explicit adversarial RL.","marker":"Zhang et al. (2021)"},{"why":"Introduces State-Adversarial MDPs and the policy-consistency robustness bound that motivates the problem and supplies the +reg. regularizer used in all variants.","marker":"Zhang et al. (2020b)"},{"why":"Provides the maximum-entropy off-policy actor-critic (SAC) whose action-value function and entropy term carry the soft-worst evaluation.","marker":"Haarnoja et al. (2018a;b)"},{"why":"Supplies the f-divergence convex-duality derivation used to obtain the dual form of the soft-constrained adversary in Lemma 4.9.","marker":"Belousov & Peters (2017; 2019)"},{"why":"The projected gradient descent attack that VALT-EPS uses to locate the worst state inside the epsilon-ball, and also a standard evaluation attack.","marker":"Madry et al., 2018"},{"why":"Provides the PA-AD technique and the PA-ATLA baseline for high-dimensional observations, and the PA variant used for the Ant experiments.","marker":"Sun et al. (2022)"},{"why":"WocaR-RL is the closest prior attempt to avoid extra environment steps by estimating a worst-case value; VALT is compared against it and borrows its contraction-style proof technique.","marker":"Liang et al. (2022)"}],"fun_headline_variants":["Robust RL without a second agent—mirror the adversary","Adversarial robustness from symmetric value mirrors","Off-policy robustness: learn the defender, skip the attacker","Mirror-symmetric Bellman makes adversarial training free","No extra rollouts: virtual adversary via symmetry"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the soft-constrained adversary of Definition 4.1—the agent's value plus an entropy penalty and an f-divergence penalty toward a uniform prior over the epsilon-ball—is a faithful surrogate for the hard worst-case adversary; if that relaxation misses the true hardest perturbations, the trained policy can look robust in training but remain vulnerable at evaluation.","fun_headline_variants_meta":{"raw":{"variants":["Robust RL without a second agent—mirror the adversary","Adversarial robustness from symmetric value mirrors","Off-policy robustness: learn the defender, skip the attacker","Mirror-symmetric Bellman makes adversarial training free","No extra rollouts: virtual adversary via symmetry"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000305,"raw_usage":{"total_tokens":1784,"prompt_tokens":1016,"completion_tokens":768,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":691}},"tokens_in":632,"tokens_out":768,"duration_ms":7122,"temperature":1.0,"reasoning_tokens":691,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:20:49.106633+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train VALT-SOFT or VALT-EPS on a continuous-control benchmark, then attack the resulting policy with a much stronger learned adversary, such as SA-RL or PA-AD with many restarts, or with highly fine-grained PGD using dozens of random starts. If the worst-case return found at evaluation falls well below the soft-worst value that the training Bellman update optimizes—especially in regions where the uniform prior had little mass—the symmetry identity has not transferred from the soft adversary to true worst-case robustness.","supporting_citations":[],"review_version":2}