Pith. sign in

REVIEW 4 major objections 4 minor 44 references

Off-Policy Actor-Critic for Adversarial Observation Robustness: Virtual Alternative Training via Symmetric Policy Evaluation

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that worst-case observation robustness can be trained with a single off-policy actor-critic, because the adversary's soft-optimal value function is exactly the negative of the agent's soft-worst value function.

desk verdict VALT is a genuinely off-policy way to train against a soft observation adversary, with clean contraction theory, but the paper never closes the gap to the hard worst-case attacker it claims to beat. read the letter →

arxiv 2506.16753 v1 pith:NBSNIDSP submitted 2025-06-20 cs.LG cs.AIcs.RO

Kosuke Nakanishi , Akihiro Kubo , Yuji Yasui , Shin Ishii This is my paper · ORCID
classification cs.LGcs.AIcs.RO
keywords adversarialobservationrobustnessoff-policyreinforcementlearningsoftactor-criticsymmetricpolicyevaluationstate-adversarialMDPf-divergenceregularizationvirtualalternatingtrainingworst-casevaluefunction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Off-policy methods have been hard to use for adversarial observation robustness because the standard recipe alternates two learners—an agent and an attacker—each needing its own environment rollouts. This paper tries to remove the attacker from the training loop entirely. It defines a soft-constrained adversary whose objective penalizes divergence from a prior over allowed perturbations, then proves a symmetry: the adversary's soft-optimal value is the negative of the agent's soft-worst value. Therefore the attacker's optimal policy can be read off from the agent's own action-value function, and robust training reduces to one soft-worst Bellman update inside an off-policy actor-critic like SAC. If this is right, robust policies against observation perturbations would come at roughly the sample cost of ordinary SAC, with no alternating adversarial RL.

What carries the argument

The load-bearing object is the soft-constrained adversary of Definition 4.1, whose objective $\tilde{J}[\nu,\pi] = J[\nu,\pi] + \alpha_{\mathrm{ent}}H(\pi\circ\nu) + \alpha_{\mathrm{attk}}D_f(\nu\|p)$ replaces the hard constraint $\nu\in B_\epsilon$ by a divergence penalty toward a prior $p$ over the $\epsilon$-ball. Around this object the paper constructs two Bellman operators, $T^\pi_{\nu_{\mathrm{soft}}}$ for the agent and $T^{\nu_{\mathrm{soft}}}_\pi$ for the adversary, and proves that they are $\gamma$-contractions whose fixed points are negatives of each other. This symmetry is what does the work: it turns adversarial training, normally a nested max-min problem, into a single soft-min evaluation from the agent's side. The two algorithms are then distinguished by which $f$-divergence yields the soft adversary—an exact KL form in VALT-SOFT and an extreme-$\alpha$ form collapsed to a worst-plus-uniform mixture in VALT-EPS.

What would settle it

Train VALT-SOFT or VALT-EPS on a continuous-control benchmark, then attack the resulting policy with a much stronger learned adversary, such as SA-RL or PA-AD with many restarts, or with highly fine-grained PGD using dozens of random starts. If the worst-case return found at evaluation falls well below the soft-worst value that the training Bellman update optimizes—especially in regions where the uniform prior had little mass—the symmetry identity has not transferred from the soft adversary to true worst-case robustness.

Watch

Extended reading notes

Core claim

The central discovery is Theorem 4.6: for a fixed agent policy, the Bellman evaluation of the soft-optimal adversary is the mirror image of the agent's soft-worst-case Bellman evaluation, and the two operators share fixed points of opposite sign, $V^{\nu^\star\mathrm{soft}}_\pi = -V^\pi_{\nu^\star\mathrm{soft}}$. Concretely, the adversary's action-value function can be expressed through the agent's as $Q^\nu_\pi(s,\tilde{s}) = \mathbb{E}_\pi[-Q^\pi_\nu(s,\tilde{a})]$, so an explicit adversarial RL process is unnecessary. The paper builds two practical variants on this identity. VALT-SOFT uses a KL-divergence penalty, whose soft-optimal adversary has the closed form $\nu^\star_{\mathrm{soft}}(\tilde{s}|s) \propto p(\tilde{s}|s)\exp(-V^\pi(s,\tilde{s})/\alpha_{\mathrm{attk}})$, and learns a variational model of it. VALT-EPS instead uses the $\alpha$-divergence family and approximates the adversary as a mixture of the worst state (found by projected gradient descent on the agent's value) and uniform noise over the epsilon-ball. Both Bellman operators are proven $\gamma$-contractions, and policy improvement with the fixed soft adversary is proven monotonic. On MuJoCo control tasks the methods meet or beat alternating-training baselines in worst-case attacked score while using 0.5 to 3 million environment steps, matching plain SAC's sample budget.

Load-bearing premise

The load-bearing premise is that the soft-constrained adversary of Definition 4.1—the agent's value plus an entropy penalty and an f-divergence penalty toward a uniform prior over the epsilon-ball—is a faithful surrogate for the hard worst-case adversary; if that relaxation misses the true hardest perturbations, the trained policy can look robust in training but remain vulnerable at evaluation.

Editorial extensions

If this is right

  • Robust training against observation perturbation becomes compatible with off-policy actor-critic in continuous control, at a sample budget comparable to plain SAC rather than double the budget of alternating training.
  • No second policy or value network needs to explore the environment, because the adversary's soft-optimal value is computed from the agent's own action-value function.
  • The soft-constrained formulation gives a stable fixed-point learning target, avoiding the sharp-saddle instability that alternating max-min training can produce.
  • The learned soft-worst value doubles as a white-box robustness metric: the paper's ablations show that removing the adversary from policy improvement collapses training in high-dimensional tasks.
  • The framework naturally extends to discrete-action domains and to other off-policy bases, as the underlying symmetry and contraction arguments are not specific to SAC.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that the same symmetry argument should transfer to other off-policy bases, such as TD3 or DQN, with the soft-worst Bellman update replacing the ordinary target; implementing VALT on those bases and checking whether the contraction and robustness gains persist would be a direct test of the mechanism.
  • We infer that the robustness-accuracy trade-off is governed by the attack temperature $\alpha_{\mathrm{attk}}$: very small values approach the hard worst case but may reintroduce training instability, so scheduling $\alpha_{\mathrm{attk}}$ from large to small offers a principled curriculum, a possibility the paper only explores in one Ant experiment.
  • We infer a concrete failure mode from the paper's acknowledged data-coverage gap: if the replay buffer never contains states where the true worst-case attacker would go, the soft adversary's support is empty there, so evaluation-time attacks could find vulnerabilities the training adversary never selected.
  • We infer that the quantal-response-equilibrium interpretation, which the paper offers as an explanation for stability, also predicts a smooth interpolation between a uniform random attacker and the fully worst-case attacker; measuring that interpolation on a single benchmark would make the role of the f-divergence regularizer explicit.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes VALT, an off-policy actor-critic method for robustness to bounded observation perturbations. It replaces the explicit alternating adversary-RL loop of ATLA with a soft-constrained adversary (Definition 4.1) whose value function is argued, via a symmetry theorem (Theorem 4.6), to be obtainable from the agent's own action-value function. Two instantiations are given: VALT-SOFT, based on a closed-form KL-regularized adversary (Lemma 4.7), and VALT-EPS, based on a Dirac-mixture approximation of an alpha-divergence-regularized adversary (Eq. (12)). Contraction of the resulting Bellman operators is proved (Theorems 4.8 and 4.10). Experiments on four MuJoCo benchmarks compare the methods with PPO- and SAC-based baselines and report improved sample efficiency and robustness under heuristic and learned attacks.

Significance. The proposed framework is potentially valuable: if the soft-adversary surrogate faithfully represents the hard worst-case problem, VALT would provide a principled off-policy alternative to alternating training at roughly single-agent sample cost. The paper's strengths include concrete contraction and symmetry theorems, a public implementation, and an evaluation protocol that uses external RL-trained attackers rather than fitting the evaluation to the method. The behavior-policy ablations in Appendix E.3 are also informative about where off-policy robust methods break down. The significance is tempered, however, by the absence of a formal link between the soft objective and the hard max-min objective, and by the gap between the idealized operators analyzed and the implemented approximations.

major comments (4)
  1. [Section 4.1, Eq. (6), and Table 6] No theorem or bound connects the soft-constrained adversary in Definition 4.1 to the hard worst-case adversary in Eq. (2), and the finite values of alpha_attk used in the experiments (2, 4, and 8 in Table 6) are not accompanied by any analysis showing that the soft optimum is close to the hard max-min solution. Without such a bound, the abstract's claim that the method handles 'worst-case' observation perturbations is not supported by the theory; the experiments can only demonstrate robustness against the softened surrogate and against external attackers that are themselves not certified to reach the true worst case.
  2. [Section 4.1, Lemma 4.7, and Appendix B.2] There is an inconsistency in the definition of H(pi composed with nu). Definition 4.1 and Proposition 4.11 treat pi composed with nu as a composed conditional distribution, whose entropy is the entropy of the mixture distribution, whereas the derivation of Lemma 4.7 in Appendix B.2 explicitly uses the averaged conditional entropy E_nu[H(pi(.|s_tilde))]. These are different functionals of nu and pi. Because the closed-form solution in Eq. (9) and the contraction proof in Theorem 4.8 rely on the linearity of the entropy term in nu, the entropy term must be defined consistently for VALT-SOFT to be the optimizer of the stated objective.
  3. [Section 4.3, Appendix C.1, and Appendix C.2] The contraction theorems apply to idealized operators, not to the implemented algorithms. VALT-EPS approximates Eq. (12) by two-step PGD on Q(s, mu(s_tilde)) with a randomized start and a scheduled kappa_worst (Table 5), and VALT-SOFT fits a variational model nu_psi by minimizing the KL loss in Eq. (57) with periodic resets (Algorithm 3). No approximation error bounds or stationarity guarantees are provided for either implementation, so the theory does not directly certify the algorithms that are actually evaluated.
  4. [Section 4.4 and Appendix E.3] The replay buffer is collected under a behavior policy that uses uniform perturbations or a 50:50 mix rather than the soft adversary nu_soft, so the distribution used for the off-policy Bellman update need not cover the support of the soft adversary. The behavior-policy experiments in Appendix E.3, especially the Adv0 collapse in HalfCheetah and Ant, show that the learned Q-function is unreliable at perturbed states outside the replay distribution, and no importance weighting or coverage bound is supplied. This gap is acknowledged in Section 4.4, but it is load-bearing for the claim that the off-policy algorithm implements the symmetric soft game.
minor comments (4)
  1. [Section 3.1 heading] The heading contains a typo: 'Decesion' should be 'Decision'.
  2. [Figure 2] The y-axis label reads 'Avrage Episodic Return' in both subfigures; it should be 'Average Episodic Return'.
  3. [Propositions 4.4 and 4.5] These numbered items define Bellman operators and value functions rather than stating propositions; relabeling them as definitions would make the logical structure clearer.
  4. [Theorem 4.6] The representational assumption Q_nu(s, s_tilde) = E_pi[-Q'_pi(s, a_tilde)] should be stated explicitly as a condition on the adversary's value-function class, and the proof should show directly that the fixed point constructed in Appendix B.1 lies in that class.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor definitional reduction in the symmetry theorem; empirical robustness is externally validated.

  1. self definitional [Theorem 4.6 and Proposition 4.5 (Section 4.2); Appendix B.1]
    "Theorem 4.6 (Symmetry of γ-Contraction Properties). For a fixed agent policy π, if there exists a bounded function Q′π_ν(s, ã) such that Qν_π(s, s̃) = Eπ[−Q′π_ν(s, ã)], and if Tπ_νsoft is a γ-contraction operator, then Tνsoft_π is also a γ-contraction operator. Moreover, Tπ_νsoft and Tνsoft_π share fixed points with opposite signs, Vν⋆soft_π = maxν Vνsoft_π = − minν Vπ_νsoft = −Vπ_ν⋆soft."

    The adversary's reward is defined in Proposition 4.5 as c(s, s̃) = Eπ[−r(s, ã)], and the theorem's key hypothesis defines the adversary action-value function as Qν_π(s, s̃) = Eπ[−Q′π_ν(s, ã)]. Substituting these two identities into the adversary Bellman operator in Eq. (8) and multiplying by −1 yields exactly the agent's soft-worst Bellman operator in Eq. (7). The fixed-point relation Vν⋆soft_π = −Vπ_ν⋆soft is therefore a built-in consequence of the chosen representation rather than an independent derivation.

full rationale

The paper's empirical robustness claims are not circular: the evaluation in Table 1 uses independently trained attacker models (SA-RL/PA-AD with PPO and SAC) and heuristic attacks, so the reported worst-case scores are not a re-statement of the soft adversary used in training. The contraction and improvement theorems (4.8, 4.10, 4.12) are proved from the Bellman equations without importing a load-bearing self-citation. The main identified issue is one mild definitional reduction: Theorem 4.6's symmetric fixed-point relation follows from defining the adversary reward as Eπ[−r] and assuming the adversary Q is Eπ[−Q′π]; substituting these into Eq. (8) makes it the negative of Eq. (7), so the symmetry is essentially a property of the chosen representation. The paper is transparent about this assumption, and the practical algorithms (VALT-SOFT/VALT-EPS) are additionally supported by external attacks rather than by the symmetry theorem alone. The absence of a bound between the soft-regularized adversary and the hard worst-case adversary of Eq. (2) is a correctness or transfer risk, not circularity, since the soft objective is defined independently and the final evaluation is external.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the standard SA-MDP and robust RL setup plus the paper-specific assumption that the f-divergence-regularized soft adversary is a faithful stand-in for the hard worst-case adversary. The off-policy replay coverage assumption is acknowledged but unproven. The contraction results hold for the idealized soft-min operator, not for the implemented PGD or variational approximations, which introduces additional unverified steps.

free parameters (6)
  • alpha_attk (soft-adversary temperature) = 8.0 (HalfCheetah), 2.0 (Hopper), 4.0 (Walker2d), 4.0 (Ant)
    Controls how sharply the soft adversary concentrates on worst-case states; tuned per task in VALT-SOFT (Table 6).
  • kappa_worst scheduling (VALT-EPS) = linear 0 to 1.0 over specified step ranges
    Mixture weight between worst-case point mass and uniform perturbation; schedules chosen per task (Table 5).
  • behavior-policy adversarial ratio = 0.5 or 1.0 depending on task and method
    Fraction of data collected under adversarial perturbations; 1.0 for HalfCheetah (VALT-SOFT) and Ant (VALT-EPS), 0.5 otherwise (Tables 5 and 6).
  • robust regularizer coefficient kappa_reg = VALT-EPS: 30/3/None/30; VALT-SOFT: 1/3/None/30 across HalfCheetah/Hopper/Walker2d/Ant
    Weight for the Zhang et al. (2020b) policy-smoothness regularizer, tuned per task (Tables 5 and 6).
  • target entropy for Hopper = 0.2 instead of -3
    Hand-tuned stability correction for SAC on Hopper (Table 2).
  • VALT-SOFT adversary reset interval = 2000 steps, 1000 for Ant
    Periodic reset of the variational adversary model to avoid local minima (Appendix C.2).
assumptions (6)
  • domain assumption Assumption 4.2: the prior p has sufficient support, set to uniform over B_epsilon
    Ensures the soft adversary's support lies inside the L-infinity ball; however, the variational model in VALT-SOFT is not explicitly projected, so the implemented adversary may violate the constraint.
  • standard math Assumption 4.3: the f-divergence function f is convex and continuously differentiable
    Required for the Lagrangian and KKT derivations in Lemma 4.7 and Lemma 4.9.
  • ad hoc to paper The soft-constrained adversary (Eq. 6) approximates the hard worst-case adversary (Eq. 2) closely enough to yield robust policies
    No quantitative bound links the f-divergence-regularized optimum to the hard max-min optimum; the paper motivates this with QRE but does not prove transfer, and this is the main load-bearing modeling assumption.
  • domain assumption Representational assumption of Theorem 4.6: Q_nu^pi(s, s_tilde) = E_pi[-Q'^pi_nu(s, a_tilde)]
    The theorem assumes the adversary's Q decomposes as a negative expectation over the agent's policy; the paper argues the fixed point satisfies this by construction, but function approximation may not preserve the representation.
  • domain assumption Off-policy coverage: the replay buffer rho_R sufficiently covers the soft worst-case state distribution
    The Bellman min over nu is computed on samples from rho_R, which is mostly uniform or a 50:50 adversarial mix, not the induced worst-case distribution; the paper acknowledges this mismatch in Section 4.4 and Appendix E.3.
  • standard math Bounded rewards and discounting for the contraction and improvement arguments
    Standard MDP assumptions used in Theorems 4.8, 4.10, and 4.12.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Off-Policy Actor-Critic for Adversarial Observation Robustness: Virtual Alternative Training via Symmetric Policy Evaluation." pith.science (2026). https://pith.science/paper/NBSNIDSP

@misc{pith2026250616753,
  author       = {Pith},
  title        = {Pith review of: Off-Policy Actor-Critic for Adversarial Observation Robustness: Virtual Alternative Training via Symmetric Policy Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NBSNIDSP}},
  note         = {Machine review of arXiv:2506.16753}
}
read the original abstract

Recently, robust reinforcement learning (RL) methods designed to handle adversarial input observations have received significant attention, motivated by RL's inherent vulnerabilities. While existing approaches have demonstrated reasonable success, addressing worst-case scenarios over long time horizons requires both minimizing the agent's cumulative rewards for adversaries and training agents to counteract them through alternating learning. However, this process introduces mutual dependencies between the agent and the adversary, making interactions with the environment inefficient and hindering the development of off-policy methods. In this work, we propose a novel off-policy method that eliminates the need for additional environmental interactions by reformulating adversarial learning as a soft-constrained optimization problem. Our approach is theoretically supported by the symmetric property of policy evaluation between the agent and the adversary. The implementation is available at https://github.com/nakanakakosuke/VALT_SAC.

Figures

Figures reproduced from arXiv: 2506.16753 by the authors.

Figure 1
Figure 1. Overview of two learning frameworks for creating a robust agent by countering a corresponding adversary. Figure 1a illustrates the conventional approach, which alternates between training the agent via RL and training the adversary via another RL. This method requires twice as many environmental interactions due to the mutual dependency between the agent and the adversary. In contrast, our proposed approach, shown i… view at source ↗
Figure 2
Figure 2. Learning curves for four MuJoCo control tasks evaluated under no-attack settings. Subfigure 2a shows results for all algorithms, while Subfigure 2b focuses on sample-efficient off-policy methods. Solid lines indicate average evaluation scores, and shaded areas represent standard deviations across different random seeds. For simplicity, we abbreviate +reg., +wreg., and ”PA-”, and omit ”-SAC” unless explicitly stated.… view at source ↗
Figure 3
Figure 3. Learning curves of WocaR-SAC for four MuJoCo control tasks under no-attack settings. The solid lines represent the average evaluation scores, and the shaded areas indicate standard deviations across different seeds. While it performs well during the initial training phase, as the attack scale increases, training deteriorates across all four tasks. During training, we utilize the WocaR action-value function for deter… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Learning curves from the ablation study on VALT-EPS and VALT-SOFT for the Ant task under no-attack settings. We ablate the policy regularization term (+reg.), the adversary during policy evaluation (w/o PE), and the adversary during policy improvement (w/o PI). For bot…
Figure 5
Figure 5. Figure 5: Impact of different adversary assumptions in the behavior policy. Training curves for various adversary influence rates in VALT-SOFT-SAC on the HalfCheetah task under no-attack settings. Adv1.0 represents a setting where the behavior policy is fully influenced by the a…
Figure 6
Figure 6. Figure 6: Training performance of VALT-SOFT-SAC on the Ant task under no-attack evaluation, comparing different behavior policy assumptions and αattk schedules. Adv1.0 assumes full adversarial influence (a˜t ∼ π ◦ ν soft), while Adv0.0 uses the agent policy alone (at ∼ π). α = 4…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 31 canonical work pages

  1. [1]

    Then, we approximate the peak of the probability by a constant multiple of Dirac’s delta function asκworstδ(˜s⋆) and distribute the remaining probability equally as 1−κworst

    Now, we assume the priorp(˜s|s) is the uniform distribution overL∞-norm constrained range. Then, we approximate the peak of the probability by a constant multiple of Dirac’s delta function asκworstδ(˜s⋆) and distribute the remaining probability equally as 1−κworst. We represent this approximation as: ν⋆soft(˜s|s)≃    κworst + 1−κworst | ˜Sϵ| , if ˜s = ...

    work page

  2. [3]

    (57) 5: end for 6: else 7: Do nothing (pass) 8: end if C.3

    Algorithm 3 V ALT-SOFT-SAC Adversary Learning Procedure Require: Environmental stept, number of learning steps per environment step:num steps 1: if (t modnum steps) == 0 then 2: Reset the adversary policyνmodel ψ 3: forj = 1 tonum steps do 4: Optimize the adversary policy by minimizing the loss in Eq. (57) 5: end for 6: else 7: Do nothing (pass) 8: end if...

    work page 2021

  3. [4]

    This schedule consistently resulted in stable learning across all seeds and yielded robust behavior even under worst-case evaluations. These findings imply that introducing adversarial signals gradually—i.e., allowing the agent to collect sufficient data for reasonable behavior early on and emphasizing stronger perturbations later—can improve both optimiz...

    work page 2019

  4. [5]

    The contents are divided into three subsections: training for off-policy methods, training for on-policy methods, and evaluation (attackers). D.1. Settings for On-Policy Methods We use the code and settings provided by the original authors. For a fair comparison of robustness with SAC variants, we adopt the LSTM setting if it is available in the implement...

    work page 2021

  5. [6]

    Soft actor- critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor- critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Con- ference on Machine Learning (ICML) , pp. 1861–1870. PMLR, 2018a. Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V ., Zhu, H., Gupta, A., Abbeel, P., et al...

    arXiv

  6. [7]

    Average episodic rewards (± standard deviation) for median- seed models of our proposed methods (V ALT-EPS, V ALT-SOFT) and other SAC baselines across four MuJoCo tasks

    Comparison of performance with and without regularization terms. Average episodic rewards (± standard deviation) for median- seed models of our proposed methods (V ALT-EPS, V ALT-SOFT) and other SAC baselines across four MuJoCo tasks. All evaluations were conducted over twenty episodes with different seeds. Yellow indicates a setting where regularization ...

    work page 2022

  7. [8]

    In the ablation setting where the adversary is not assumed during policy evaluation (w/o PE), there is little change in evaluation scores. However, in the ablation setting where the adversary is not assumed during policy improvement (w/o PI), training collapses in both methods, resulting in catastrophic performance even in noise-free evaluations. 32 Off-P...

    work page 2000

  8. [9]

    We denote the ablation setting as w/o PE, and regularization is applied in all settings, including the ablation

    Additional comparison of performance with and without the soft worst-case adversary during policy evaluation (PE). We denote the ablation setting as w/o PE, and regularization is applied in all settings, including the ablation. Values indicate average episodic rewards (± standard deviation) for models trained with the median performance seed. All evaluati...

    work page 1953

Show all 44 references
  1. [10]

    instead of a single worst-case adversary, and updates the distribution using divergence-constrained optimization or percentile-based criteria. 15 Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training Interestingly, these methods improve not only robu...

    2020

  2. [11]

    We denote these settings asAdv*, where * indicates the adversary rate

    Comparison of performance under different adversary rates in the behavior policy. We denote these settings asAdv*, where * indicates the adversary rate. Values represent average episodic rewards (± standard deviation) for models trained with the median seed. All evaluations we...

    1953

  3. [13]

    B., Andrychowicz, M., Zaremba, W., and Abbeel, P

    Peng, X. B., Andrychowicz, M., Zaremba, W., and Abbeel, P. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 3803–3810. IEEE,

    2018

  4. [14]

    Robust deep reinforcement learning through adversar- ial attacks and training: A survey

    Schott, L., Delas, J., Hajri, H., Gherbi, E., Yaich, R., Boulahia-Cuppens, N., Cuppens, F., and Lamprier, S. Robust deep reinforcement learning through adversar- ial attacks and training: A survey. arXiv preprint arXiv:2403.00420,

    arXiv

  5. [16]

    L., Esfandiari, Y ., Lee, X

    Tan, K. L., Esfandiari, Y ., Lee, X. Y ., Sarkar, S., et al. Robustifying reinforcement learning agents via action 13 Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training space adversarial training. In 2020 American control conference (ACC), pp. 395...

    2020

  6. [18]

    Robust reinforcement learning using adversar- ial populations

    Vinitsky, E., Du, Y ., Parvate, K., Jang, K., Abbeel, P., and Bayen, A. Robust reinforcement learning using adversar- ial populations. arXiv preprint arXiv:2008.01825,

    2008 arXiv

  7. [20]

    are closely related to our proposed soft-constrained adversary approach. A.2. Additional Details on Robust RL for Adversarial Dynamics We additionally discuss robustness in dynamics, particularly focusing on adversarial training due to its diversity and importance. Broadly, ro...

    2005

  8. [22]

    incorporates the effect of uncertain parameters into theQ-function, and applies gradient-based optimization to find perturbations that degrade the value function. Compared to the external-force-based methods described earlier, these parameter-based methods typically assume tha...

    2021

  9. [24]

    Sincef(Q,st) is a monotonically increasing function for Q, then we can say: f(Q1,st)≤f(Q2 +ϵ,st) =ϵ +f(Q2,st) =∥Q1−Q2∥st,˜at +f(Q2,st) ↔f(Q1,st)−f(Q2,st)≤∥Q1−Q2∥st,˜at

    and Soft Q-learning (Haarnoja et al., 2017), we define ϵ = ||Q1(st, ˜at)−Q2(st, ˜at)||st,at. Sincef(Q,st) is a monotonically increasing function for Q, then we can say: f(Q1,st)≤f(Q2 +ϵ,st) =ϵ +f(Q2,st) =∥Q1−Q2∥st,˜at +f(Q2,st) ↔f(Q1,st)−f(Q2,st)≤∥Q1−Q2∥st,˜at. (28) In the sam...

    2017

  10. [29]

    While its effect is only slightly better in some tasks, we do not observe any disadvantages to using PER

    to accelerate learning outcomes. While its effect is only slightly better in some tasks, we do not observe any disadvantages to using PER. Therefore, we decide to continue to use this method. • Normalizer We normalize state inputs by recording running statistics as the agent r...

    2021

  11. [30]

    However, we observe that the agent’s learning became critically slow in HalfCheetah due to delays in updating the running statistics

    did. However, we observe that the agent’s learning became critically slow in HalfCheetah due to delays in updating the running statistics. Therefore, we decided to abandon this normalizer. In the following paragraphs, we describe the specific settings required for each method....

    2022

  12. [31]

    EnvironmentEnv

    only in the Ant task. EnvironmentEnv. stepsAgent lrEnt. targetReg. coeff.SGLD iter.PPO t×stepsAdv. ratioAdv. lrAdv. Val. lrAdv. Ent. coeff. Other settings HalfCheetah1.0M 3e-4−dim|A| 30.0 2 2048×2441 0.5 3e-5 1e-5 1e-3 Linear schedule: attack scale for reg. term from 0.0 to 0....

  13. [32]

    The SAC (agent) component retains the same settings as the base SAC, while the PPO (adversary) component follows the adversary settings of ATLA-PPO (Zhang et al., 2021)

    implementations into our SAC framework. The SAC (agent) component retains the same settings as the base SAC, while the PPO (adversary) component follows the adversary settings of ATLA-PPO (Zhang et al., 2021). The SAC training steps (i.e., environmental interactions) are evenl...

    2021

  14. [34]

    The solid lines represent the average evaluation scores, and the shaded areas indicate standard deviations across different seeds

    Learning curves of WocaR-SAC for four MuJoCo control tasks under no-attack settings. The solid lines represent the average evaluation scores, and the shaded areas indicate standard deviations across different seeds. While it performs well during the initial training phase, as ...

    2018

  15. [35]

    Compared to V ALT-EPS-SAC, V ALT-SOFT-SAC introduces more hyperparameters due to adversary policy training

    Hyperparameters for V ALT-SOFT-SAC. Compared to V ALT-EPS-SAC, V ALT-SOFT-SAC introduces more hyperparameters due to adversary policy training. However, the adversary learning rate (Adv. lr) was fixed across tasks without tuning, and others (e.g.,αattk, reset interval) showed ...

    2018

  16. [36]

    We perform multiple training runs to tune robust critic parameters (perturbation scale and regression term) around the benchmark’s attack scale

    for both PPO and SAC variants. We perform multiple training runs to tune robust critic parameters (perturbation scale and regression term) around the benchmark’s attack scale. Subsequently, we conduct multiple evaluations using all trained models and adopt the worst evaluation...

    2021

  17. [39]

    We denote these settings as w/o PE and w/o PI

    Comparison of performance with and without the soft worst-case adversary during Policy Evaluation (PE) and Policy Improvement (PI). We denote these settings as w/o PE and w/o PI. Average episodic rewards (± standard deviation) for median-seed models. All evaluations were condu...

    1953

  18. [43]

    Adv1.0 assumes full adversarial influence (˜at∼π◦νsoft), while Adv0.0 uses the agent policy alone (at∼π).α = 4 const

    Training performance of V ALT-SOFT-SAC on the Ant task under no-attack evaluation, comparing different behavior policy assumptions andαattk schedules. Adv1.0 assumes full adversarial influence (˜at∼π◦νsoft), while Adv0.0 uses the agent policy alone (at∼π).α = 4 const. denotes ...

    2022

  19. [44]

    Computation time (seconds) per 10k steps on HalfCheetah. Model act Policy Evaluation Policy Improvement Other Components cur q next q q back q opt p fwd reg p back p opt α learn adv learn SAC 18 6 21 17 22 24 – 23 13 7 – Robust-SAC 19 6 22 17 22 24 125 32 13 7 – SAC-PPO 25 6 2...

    2000

  20. [1972]

    Adversarial attacks on neural network policies

    Huang, S., Papernot, N., Goodfellow, I., Duan, Y ., and Abbeel, P. Adversarial attacks on neural network policies. arXiv preprint arXiv:1702.02284,

    arXiv

  21. [1998]

    under a fixed adversaryν. While we focus on the case of the maximum entropy scheme (Haarnoja et al., 2018a;b) in this study, other cases, such as deterministic policy gradient (DDPG) (Lillicrap et al., 2016; Fujimoto et al., 2018), can be similarly applied asπnew← arg max π Eν...

    2016

  22. [1999]

    and Munir, A

    Behzadan, V . and Munir, A. Whatever does not kill deep reinforcement learning, makes it stronger. arXiv preprint arXiv:1712.09344,

    arXiv

  23. [2005]

    and Dai, B

    Nachum, O. and Dai, B. Reinforcement learning via fenchel- rockafellar duality. arXiv preprint arXiv:2001.01866 ,

    2001 arXiv

  24. [2011]

    Openai gym

    Brockman, G., Cheung, V ., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. Openai gym. arXiv preprint arXiv:1606.01540,

    arXiv

  25. [2012]

    Adversar- ial examples in the physical world

    11 Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training Kurakin, A., Goodfellow, I., and Bengio, S. Adversar- ial examples in the physical world. arXiv preprint arXiv:1607.02533,

    arXiv

  26. [2013]

    and Song, D

    Kos, J. and Song, D. Delving into adversarial attacks on deep policies. arXiv preprint arXiv:1705.06452,

    arXiv

  27. [2014]

    On the effectiveness of interval bound propagation for training verifiably robust models

    Gowal, S., Dvijotham, K., Stanforth, R., Bunel, R., Qin, C., Uesato, J., Arandjelovic, R., Mann, T., and Kohli, P. On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint arXiv:1810.12715,

    arXiv

  28. [2015]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    arXiv

  29. [2016]

    Practical black-box attacks against 12 Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training machine learning

    Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, B.-Z., and Swami, A. Practical black-box attacks against 12 Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training machine learning. In Proceedings of the 2017 ACM on Asia Conference on Compu...

    2017

  30. [2017]

    and Peters, J

    Belousov, B. and Peters, J. f-divergence constrained policy improvement. arXiv preprint arXiv:1801.00056,

    arXiv

  31. [2018]

    Soft actor-critic for discrete action settings

    Christodoulou, P. Soft actor-critic for discrete action settings. arXiv preprint arXiv:1910.07207,

    1910 arXiv

  32. [2019]

    Mujoco: A physics engine for model-based control

    Todorov, E., Erez, T., and Tassa, Y . Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS), pp. 5026–5033. IEEE,

    2012

  33. [2020]

    for the convex relaxation and perform five iterations of PGD to ensure efficient computation within realistic time constraints. During policy improvement, they enhance the gradient ascent objective of the policy by incorporating the worst-case action value, aiming to balance p...

    2000

  34. [2021]

    Furthermore, Liu et al

    to enforce cost constraints effectively. Furthermore, Liu et al. (2023) conducted additional analyses and introduced two novel types of adversarial attacks: one that maximizes the cost and another that maximizes the reward while violating the safety constraints. Notably, the l...

    2023

  35. [2022]

    The limitations of deep learning in adversarial settings

    Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, B.-Z., and Swami, A. The limitations of deep learning in adversarial settings. In 2016 IEEE European symposium on Security and Privacy (EuroS&P), pp. 372–387. IEEE,

    2016

  36. [2023]

    In the following, we temporarily set aside strict notation and represent expressions like ” R a∈A·,da ” as ”P a∈A·” when discussing continuous action space

    14 Off-Policy Actor-Critic for Observation Robustness: Virtual Alternative Training APPENDIX We describe additional related works, provide more detailed theoretical derivations, offer additional evaluation results, and deeper discussions on the future research direction to hel...

    2011

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.