Pith. sign in

REVIEW 4 major objections 6 minor 51 references

Salience-Invariant Consistent Policy Learning for Generalization in Visual Reinforcement Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Saliency-masked value consistency plus policy KL-regularization improves zero-shot visual RL generalization, giving 14%, 39%, and 69% average gains over prior methods on DMC-GB, robotic manipulation, and CARLA.

desk verdict Clean empirical recipe with a sensible policy-consistency idea, but the DMC-GB numbers are inflated by test-set-dependent augmentation choice and the theory is wrong; worth a serious referee if both are fixed. read the letter →

arxiv 2502.08336 v2 pith:RDVDLMDF submitted 2025-02-12 cs.AI

Jingbo Sun , Songjun Tu , Qichao Zhang , Ke Chen , Dongbin Zhao This is my paper · ORCID
classification cs.AI
keywords visualreinforcementlearningzero-shotgeneralizationsaliencydataaugmentationpolicyconsistencyvaluedynamicsmodelKLdivergence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SCPL is a soft actor-critic variant that trains the encoder and value function to assign equal Q-values to an observation and its saliency-masked version, for both original and augmented views, so the agent's attention stays on task-relevant pixels. A dynamics module predicts next-state latents and rewards from augmented data, and a KL-divergence constraint keeps the policy's action distribution unchanged across perturbations. A theorem bounds the gap in expected return between two policies by a constant times their maximum KL divergence, which motivates the policy-consistency term. The paper reports state-of-the-art zero-shot generalization: average gains of 14% on the DMC-GB video-hard setting, 39% on the robotic manipulation hard setting, and 69% on CARLA. If true, this matters because agents would transfer to unseen visual environments without any adaptation data.

What carries the argument

The load-bearing object is the saliency attribute masked map: the gradient of the Q-network with respect to the input image, binarized with a $\rho$-quantile threshold so that a pixel is set to 1 only if its gradient belongs to the top $1-\rho$ quantile, and then multiplied elementwise with the observation. Equations (3)-(4) force $Q_\zeta(f_\theta(\hat{s}),a)$ to match $Q_\zeta(f_\theta(s),a)$ for both the original and the augmented observation, which drives the encoder and value function to ignore pixels outside the saliency mask. The dynamics module (Eqs. (6)-(8)) predicts the next latent and the reward from each view, and the policy module adds $\beta\,D_{\mathrm{KL}}(\pi_\phi(\cdot|e_s)\,\|\,\pi_\phi(\cdot|e_{s_\alpha}))$ to the SAC policy loss. Theorem 1, proved with the performance-difference lemma and the $\alpha$-coupling advantage bound, converts total-variation distance between training and test policies into a return-gap bound, and the standard inequality $D_{\mathrm{TV}}^2 \le D_{\mathrm{KL}}$ converts that into the KL bound that justifies the policy-consistency term.

What would settle it

On the DMC-GB video-hard benchmark, replace SCPL's gradient saliency masks with random binary masks of the same density; if the test return stays near 853 instead of dropping, saliency guidance is not carrying the claimed load. Alternatively, evaluate SCPL with human-annotated or segmentation-based oracle masks: if oracle-masked SCPL does not beat gradient-masked SCPL, the gradient saliency map is not the active ingredient.

Watch

Extended reading notes

Core claim

The paper's central claim is that visual overfitting in RL comes from two coupled failures, attention to task-irrelevant pixels and policy drift under perturbation, and that both can be corrected by three coordinated objectives on top of SAC: value consistency guided by saliency attribute maps, a dynamics objective over augmented observations, and policy consistency via KL divergence. It proves Theorem 1, which states that with $\alpha = D^{\max}_{\mathrm{TV}}(\pi_o,\pi_p)$ and $\epsilon = \max_{s,a}|A^\pi(s,a)|$, the return gap obeys $\eta(\pi_o)-\eta(\pi_p) \le \frac{2\epsilon\gamma}{(1-\gamma)^2}\alpha^2$; using $D_{\mathrm{TV}}(p\|q)^2 \le D_{\mathrm{KL}}(p\|q)$, the paper concludes the return gap is bounded by $C\,D^{\max}_{\mathrm{KL}}(\pi_o,\pi_p)$ with $C=\frac{2\epsilon\gamma}{(1-\gamma)^2}$. The empirical claim is that SCPL reaches 853 average return on DMC-GB video hard (14% above the best baseline), 65.1 on robotic manipulation hard (+39%), and 352 on CARLA (+69%).

Load-bearing premise

The whole method rests on the assumption that the Q-network's gradient saliency maps point at exactly the task-relevant pixels in both original and augmented images, so that masking and enforcing value equality on those pixels preserves task information instead of deleting it.

Editorial extensions

If this is right

  • Agents trained with SCPL should transfer to unseen backgrounds, colors, and video distractors without any test-environment data, because all objectives are computed from training observations and augmentations.
  • The KL bound gives a deployment diagnostic: if an agent's action distribution on a new observation stays close to its training distribution, the expected return gap is guaranteed small up to constants.
  • The shared encoder also supports next-latent and reward prediction, so the same representation can serve both model-free control and model-based planning or world-model objectives.
  • Because SCPL is built on top of SAC, the three auxiliary losses can be added to existing value-based or actor-critic pipelines without changing the base algorithm's architecture.
  • On the three benchmarks, the biggest reported gains occur in the hardest settings (video hard, robotic hard, and rainy or dark CARLA weather), suggesting the method's advantage grows with perturbation strength.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the saliency-masked value-consistency loss could be applied to offline RL, where augmented or perturbed observations are cheap; the KL policy term would then act as a conservative regularizer against distribution shift.
  • Editorial extension: the paper fixes the $\rho$-quantile per task (0.9 or 0.95) and reports sensitivity to it; an adaptive quantile chosen by a saliency-uncertainty or return-based criterion could remove that per-task tuning.
  • Editorial extension: the theory bounds the return gap by the maximum KL over states, but the experiments estimate consistency only on sampled observations; a state-covering worst-case KL estimator would make the bound actionable during deployment.
  • Editorial extension: the method's reliance on gradient saliency could be tested by replacing the learned masks with oracle masks from segmentation or human annotation; if oracle masks do not outperform learned masks, the saliency map is not the active ingredient.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes SCPL, a SAC-based method for zero-shot visual generalization. It adds a value consistency loss that matches Q-values on saliency-masked and original observations (Eqs. 3-4), a dynamics module that predicts next latents and rewards from original and augmented observations (Eqs. 6-8), and a policy-consistency KL loss between policies on original and augmented observations (Eqs. 12-14). The authors claim average improvements of 14% on DMC-GB video hard, 39% on Robotic hard, and 69% on CARLA, and provide Theorem 1 to motivate policy consistency by bounding the return gap with the KL divergence of the two policies.

Significance. If the empirical results were obtained under a fair protocol, SCPL would be a useful contribution: it combines saliency-guided Q-consistency with policy regularization and is evaluated on three benchmarks with ablations. The paper also gives a theoretical motivation for policy consistency. However, two load-bearing issues—test-set-dependent augmentation selection and an incorrect theorem—currently prevent the results from being accepted at face value. The implementation details are relatively complete, and the ablation study gives useful per-module information.

major comments (4)
  1. [Section 5.2 and Appendix D] The DMC-GB evaluation selects a different data augmentation per test setting: Table 1 states that SCPL uses random convolution for Color hard and random overlay for Video easy and Video hard, and Appendix D explicitly justifies this by saying each augmentation performs better in its assigned setting. This is a test-set-dependent choice: the augmentation hyperparameter is selected after observing the test distribution, whereas baselines are evaluated with fixed standard augmentations. The reported +14% on video hard, and the DMC-GB margins in general, are therefore not a clean zero-shot comparison. Please rerun with a single augmentation chosen before seeing test results, or report both augmentations on all settings, and adjust the claims accordingly.
  2. [Appendix B, Theorem 1 and Eq. (19)] The asserted O(alpha^2) bound is not correct as stated. The standard performance-difference bound is linear in the total-variation distance alpha (Kakade-Langford), not quadratic. In the proof, the step leading to Eq. (18) is invalid: the expectation over a coupled pair does not equal P(disagreement) times the expected difference, and the subsequent use of P(n_t > 0) to multiply each advantage term by alpha double-counts the coupling failure probability. Consequently Eq. (19) and the derived KL bound in Eq. (11) are not justified. Please correct the theorem to the standard linear bound, or remove the quantitative bound and present policy consistency only as a heuristic motivation.
  3. [Section 4.1 and Table 2] The method relies on vanilla-gradient saliency maps to identify task-relevant pixels, but the paper never defines a ground truth for 'task-relevant' regions. Table 2 reports ACC, AUC, and F1 without explaining what labels are used for these metrics. Please specify the ground-truth generation procedure and validate the saliency maps against it (for example, using known distractor masks or human annotations), because Eqs. (3)-(4) will actively hurt if the saliency maps are noisy or misaligned with task-relevant pixels.
  4. [Section 5.4, Tables 4 and 5] The Robotic and CARLA comparisons include only SAC, SVEA, and SGQN, yet the text concludes that SCPL 'outperforms the best prior methods' and claims state-of-the-art performance. Recent visual-RL generalization methods discussed earlier in the paper, such as MaDi, CG2A, and CNSN, are absent from these benchmarks. Please add those baselines, or limit the claim to the methods actually compared.
minor comments (6)
  1. [Eq. (12) and Theorem 1] The symbol alpha is used both for the SAC temperature in Eq. (12) and for the maximal total-variation distance in Theorem 1; please rename one of them to avoid ambiguity.
  2. [Algorithm 1, line 6] In the pseudocode, L_Q2 is formatted without the subscript style used in the main text; please make the notation consistent.
  3. [Figure 6] The t-SNE plots are qualitative; a quantitative clustering metric, such as silhouette score computed on the embeddings and actions, would better support the claim of consistent task-relevant representations.
  4. [Section 3] The sentence 'let s_t is the state' contains a grammar error; please revise.
  5. [Section 5.2] The phrase '15 visual perturbed control tasks' could be clearer as five DMC tasks evaluated under three perturbation settings.
  6. [Appendix J, Table 8] The text says rho=0.95 'slightly outperforms' rho=0.9, but some entries are lower for 0.95 (e.g., Ball in cup Color hard); please hedge the claim accordingly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported generalization gains come from benchmark comparisons, not from a self-referential derivation.

full rationale

Walking the paper's derivation chain, no load-bearing step reduces to its own input. The value consistency losses (Eqs. 3-4) do use saliency masks computed from the Q-network's own gradients, which is self-referential as a training regularizer, but the paper's central claims are the measured benchmark returns, and those returns are not derived from the saliency masks by construction; the masks are training signals rather than fitted predictors of the reported test scores. The policy-consistency theorem (Theorem 1) is imported from standard TRPO / Kakade-Langford bounds, with Lemmas 1 and 2 cited to [15] and [29] and the TV-to-KL step cited to [26]; this is independent, externally available support rather than a self-citation chain. The authors' self-citations ([18,19,22,32,36,37]) appear in related-work and application examples only and are not load-bearing for the algorithm's claimed contribution. The most serious protocol concern is the per-setting augmentation choice in Section 5.2 and Appendix D, where random convolution is used for Color hard and random overlay for Video easy/hard because each 'performs better' in those settings; this is a test-set-informed hyperparameter choice that weakens the zero-shot generalization claim, but it is not a circular reduction of a predicted quantity to a fitted parameter. Similarly, the saliency 'accuracy' metrics in Table 2 lack a stated ground truth, which is an evaluation gap, but no equation defines the reported improvements in terms of those metrics. Because no specific equation or claimed prediction can be exhibited as equivalent to its input by construction, the appropriate circularity score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests on three tuning knobs (rho, lambda, beta), an evaluation-time augmentation choice, and the assumptions that saliency maps reflect task relevance and that augmentation captures the test distribution. No new physical entities are introduced. The theory reuses known lemmas rather than deriving the algorithm.

free parameters (4)
  • Saliency quantile rho = 0.9 (walker tasks), 0.95 (other DMC tasks), 0.95 (Robot), 0.9 (CARLA)
    Controls the saliency mask threshold; tuned per task via 'rapid visual search' as admitted in Appendix J.
  • Value consistency coefficient lambda = 0.5
    Balances value loss and saliency consistency loss in Eq. (5); set by hand.
  • Policy consistency coefficient beta = 1
    Balances policy loss and KL consistency loss in Eq. (14); set by hand.
  • Data augmentation choice for DMC-GB = random convolution for Color hard; random overlay for Video easy and Video hard
    Selected after observing the test setting, so it is a discrete parameter fitted to benchmark performance rather than a fixed zero-shot recipe.
assumptions (3)
  • standard math The POMDP formulation and SAC objective are standard; policy difference lemmas (Lemma 1 from [15], Lemma 2 from [29]) hold.
    Used in Sections 3-4 and Appendix B to state and prove Theorem 1.
  • domain assumption Vanilla-gradient saliency maps of the Q-network identify task-relevant pixels.
    Section 4.1 relies on this to define the masked observations used in Eqs. (3)-(4).
  • domain assumption Augmented observations (random convolution, random overlay) are representative of test distribution shift.
    The dynamics and policy consistency modules, Sections 4.2-4.3, train on augmented data and expect it to transfer to test perturbations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Salience-Invariant Consistent Policy Learning for Generalization in Visual Reinforcement Learning." pith.science (2026). https://pith.science/paper/RDVDLMDF

@misc{pith2026250208336,
  author       = {Pith},
  title        = {Pith review of: Salience-Invariant Consistent Policy Learning for Generalization in Visual Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RDVDLMDF}},
  note         = {Machine review of arXiv:2502.08336}
}
read the original abstract

Generalizing policies to unseen scenarios remains a critical challenge in visual reinforcement learning, where agents often overfit to the specific visual observations of the training environment. In unseen environments, distracting pixels may lead agents to extract representations containing task-irrelevant information. As a result, agents may deviate from the optimal behaviors learned during training, thereby hindering visual generalization.To address this issue, we propose the Salience-Invariant Consistent Policy Learning (SCPL) algorithm, an efficient framework for zero-shot generalization. Our approach introduces a novel value consistency module alongside a dynamics module to effectively capture task-relevant representations. The value consistency module, guided by saliency, ensures the agent focuses on task-relevant pixels in both original and perturbed observations, while the dynamics module uses augmented data to help the encoder capture dynamic- and reward-relevant representations. Additionally, our theoretical analysis highlights the importance of policy consistency for generalization. To strengthen this, we introduce a policy consistency module with a KL divergence constraint to maintain consistent policies across original and perturbed observations.Extensive experiments on the DMC-GB, Robotic Manipulation, and CARLA benchmarks demonstrate that SCPL significantly outperforms state-of-the-art methods in terms of generalization. Notably, SCPL achieves average performance improvements of 14\%, 39\%, and 69\% in the challenging DMC video hard setting, the Robotic hard setting, and the CARLA benchmark, respectively.Project Page: https://sites.google.com/view/scpl-rl.

Figures

Figures reproduced from arXiv: 2502.08336 by the authors.

Figure 1
Figure 1. (Left) Saliency masked map of SVEA, SGQN, and SCPL (ours), which shows the attention regions of value functions [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of SCPL. The value consistency module is trained using the original and augmented observations [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The generation of saliency attribute masked maps. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: The performance of SAC, SVEA, SGQN, and SCPL in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Saliency attribute maps for SAC, SVEA, SGQN, and SCPL in [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: t-SNE maps of embeddings and actions learned with SVEA, SGQN, and SCPL for 20 motion situations, generated by [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: The network architecture of SCPL agent [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Samples from two data augmentations [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Training and test environments in DMC-GB. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Saliency attribute maps and saliency attribute masked maps in the DMC-GB [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Training and testing curves of SAC, SVEA, SGQN and SCPL in DMC-GB environments. [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Performance of Ablation Study [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Vision-based Robotic Manipulation Environments. This figure shows examples of training and testing observation [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Saliency attribute maps and saliency attribute masked maps in the Robotic Environment, [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: CARLA environments for generalization. Agents are trained in the weather of [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: Saliency attribute maps and saliency attribute masked maps of SAC, SVEA, SGQN, and SCPL in CARLA. The weather [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]
Figure 17
Figure 17. Figure 17: Example of 𝑀𝜌 (𝑄𝜃 , 𝑠, 𝑎) with different values of 𝜌 on cartpole swing up (top), ball in cup (middle), and finger spin (bottom) [PITH_FULL_IMAGE:figures/full_fig_p022_17.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 47 canonical work pages

  1. [1]

    Marcin Andrychowicz, Anton Raichuk, Piotr Stańczyk, et al. 2021. What matters for on-policy deep actor-critic methods? a large-scale study. In International Conference on Learning Representations (ICLR)

    work page 2021

  2. [2]

    David Bertoin, Adil Zouitine, Mehdi Zouitine, and Emmanuel Rachelson. 2022. Look where you look! Saliency-guided Q-networks for generalization in visual Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 35. 30693–30706

    work page 2022

  3. [3]

    Karl Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. 2021. Phasic Policy Gradient. In International Conference on Machine Learning (ICML), Vol. 139. PMLR, 2020–2027

    work page 2021

  4. [4]

    López, and Vladlen Koltun

    Alexey Dosovitskiy, Germán Ros, Felipe Codevilla, Antonio M. López, and Vladlen Koltun. 2017. CARLA: An Open Urban Driving Simulator. In Conference on Robot Learning (CoRL), Vol. 78. PMLR, 1–16

    work page 2017

  5. [5]

    Hanna, and Stefano V

    Mhairi Dunion, Trevor McInroe, Kevin Sebastian Luck, Josiah P. Hanna, and Stefano V. Albrecht. 2023. Temporal Disentanglement of Representations for Improved Generalisation in Reinforcement Learning. In International Conference on Learning Representations (ICLR)

    work page 2023

  6. [6]

    Linxi Fan, Guanzhi Wang, De-An Huang, Zhiding Yu, Li Fei-Fei, Yuke Zhu, and Animashree Anandkumar. 2021. SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies. InInternational Conference on Machine Learning (ICML). PMLR, 3088–3099

    work page 2021

  7. [7]

    Taylor, A

    Bram Grooten, Tristan Tomilin, Gautham Vasan, Matthew E. Taylor, A. Rupam Mahmood, Meng Fang, Mykola Pechenizkiy, and Decebal Constantin Mocanu

    work page

  8. [8]

    Jeongsoo Ha, Kyungsoo Kim, and Yusung Kim. 2023. Dream to generalize: zero-shot model-based reinforcement learning for unseen visual distractions. In Association for the Advancement of Artificial Intelligence (AAAI) , Vol. 37. 7802– 7810

    work page 2023

Show all 51 references
  1. [9]

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In International Conference on Machine Learning (ICML) , Vol. 80. PMLR, 1856–1865

    2018

  2. [10]

    Efros, Lerrel Pinto, and Xiaolong Wang

    Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyà, Pieter Abbeel, Alexei A. Efros, Lerrel Pinto, and Xiaolong Wang. 2021. Self-Supervised Policy Adapta- tion during Deployment. In International Conference on Learning Representations (ICLR)

    2021

  3. [11]

    Nicklas Hansen, Hao Su, and Xiaolong Wang. 2021. Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentation. In Advances in Neural Information Processing Systems (NeurIPS) . 3680–3693

    2021

  4. [12]

    Nicklas Hansen and Xiaolong Wang. 2021. Generalization in Reinforcement Learning by Soft Data Augmentation. In International Conference on Robotics and Automation (ICRA). IEEE, 13611–13617

    2021

  5. [13]

    Yangru Huang, Peixi Peng, Yifan Zhao, Guangyao Chen, and Yonghong Tian. 2022. Spectrum Random Masking for Generalization in Image-based Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS) , Vol. 35. 20393–20406

    2022

  6. [14]

    Rishabh Jangir, Nicklas Hansen, Sambaran Ghosal, Mohit Jain, and Xiaolong Wang. 2022. Look closer: Bridging egocentric and third-person views with transformers for robotic manipulation. IEEE Robotics and Automation Letters 7, 2, 3046–3053

    2022

  7. [15]

    Kakade and John Langford

    Sham M. Kakade and John Langford. 2002. Approximately Optimal Approximate Reinforcement Learning. In International Conference on Machine Learning (ICML) . 267–274

    2002

  8. [16]

    Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. 2020. Reinforcement learning with augmented data. In Advances in Neural Information Processing Systems (NeurIPS) , Vol. 33. 19884–19895

    2020

  9. [17]

    Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. 2020. Network Random- ization: A Simple Technique for Generalization in Deep Reinforcement Learning. In International Conference on Learning Representations (ICLR)

    2020

  10. [18]

    Haoran Li, Zhennan Jiang, Yuhui Chen, and Dongbin Zhao. 2024. Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regular- ization. In Advances in Neural Information Processing Systems (NeurIPS)

    2024

  11. [19]

    Haoran Li, Qichao Zhang, and Dongbin Zhao. 2020. Deep reinforcement learning- based automatic exploration for navigation in unknown environment. IEEE Transactions on Neural Networks and Learning Systems (TNNLS) 31, 6, 2064–2076

    2020

  12. [20]

    Lu Li, Jiafei Lyu, Guozheng Ma, Zilin Wang, Zhenjie Yang, Xiu Li, and Zhiheng Li

  13. [21]

    Xiang Li, Jinghuan Shang, Srijan Das, and Michael Ryoo. 2022. Does self- supervised learning really improve reinforcement learning from pixels?. In Ad- vances in Neural Information Processing Systems (NeurIPS) , Vol. 35. 30865–30881

    2022

  14. [22]

    In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS)

    Normalization Enhances Generalization in Visual Reinforcement Learning. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS). 1137–1146

  15. [23]

    Siao Liu, Zhaoyu Chen, Yang Liu, Yuzheng Wang, Dingkang Yang, Zhile Zhao, Ziqing Zhou, Xie Yi, Wei Li, Wenqiang Zhang, et al. 2023. Improving general- ization in visual reinforcement learning via conflict-aware gradient agreement augmentation. In Proceedings of the IEEE/CVF In...

    2023

  16. [24]

    Minsong Liu, Luntong Li, Shuai Hao, Yuanheng Zhu, and Dongbin Zhao. 2023. Soft Contrastive Learning with Q-irrelevance Abstraction for Reinforcement Learning. IEEE Transactions on Cognitive and Developmental Systems (TCDS) 15, 3, 1463–1473

    2023

  17. [25]

    Minting Pan, Xiangming Zhu, Yunbo Wang, and Xiaokang Yang. 2022. Iso-dream: Isolating and leveraging noncontrollable visual dynamics in world models. In Advances in Neural Information Processing Systems (NeurIPS) , Vol. 35. 23178– 23191

    2022

  18. [26]

    Seungyong Moon, JunYeong Lee, and Hyun Oh Song. 2022. Rethinking Value Function Learning for Generalization in Reinforcement Learning. In Advances in Neural Information Processing Systems (NeurIPS) , Vol. 35. 34846–34858

    2022

  19. [27]

    Roberta Raileanu and Rob Fergus. 2021. Decoupling value and policy for gen- eralization in reinforcement learning. In International Conference on Machine Learning (ICML). PMLR, 8787–8798

    2021

  20. [28]

    David Pollard. 2000. Asymptopia: an exposition of statistical asymptotic theory. http://www.stat.yale.edu/Ëœpollard/Books/Asymptopia

    2000

  21. [29]

    Jordan, and Philipp Moritz

    John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. 2015. Trust Region Policy Optimization. In International Conference on Machine Learning (ICML), Vol. 37. 1889–1897

    2015

  22. [30]

    Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. 2020. Mastering atari, go, chess and shogi by planning with a learned model. Nature 588, 7839, 604–609

    2020

  23. [31]

    Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Deep inside convolutional networks: Visualising image classification models and saliency maps. In International Conference on Learning Representations (ICLR)

    2014

  24. [32]

    Devon Hjelm, Aaron C

    Max Schwarzer, Ankesh Anand, Rishab Goel, R. Devon Hjelm, Aaron C. Courville, and Philip Bachman. 2021. Data-Efficient Reinforcement Learning with Self- Predictive Representations. In International Conference on Learning Representa- tions (ICLR)

    2021

  25. [33]

    Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An intro- duction. MIT press

    2018

  26. [34]

    Jingbo Sun, Xing Fang, and Qichao Zhang. 2023. Reinforcement learning driving strategy based on auxiliary task for multi-scenarios autonomous driving. In 2023 IEEE 12th Data Driven Control and Learning Systems Conference (DDCLS) . IEEE, 1337–1342

    2023

  27. [35]

    Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al

  28. [36]

    Zhentao Tang, Yuanheng Zhu, Dongbin Zhao, and Simon M Lucas. 2023. En- hanced rolling horizon evolution algorithm with opponent model learning: results for the fighting game AI competition. IEEE Transactions on Games (TOG) 15, 1, 5–15

    2023

  29. [37]

    Junjie Wang, Qichao Zhang, and Dongbin Zhao. 2022. Highway lane change decision-making via attention-based deep reinforcement learning. IEEE/CAA Journal of Automatica Sinica 9, 3, 567–569

    2022

  30. [38]

    Jingda Wu, Zhiyu Huang, and Chen Lv. 2022. Uncertainty-aware model-based reinforcement learning: Methodology and application in autonomous driving. IEEE Transactions on Intelligent Vehicles (TIV) 8, 1, 194–203

    2022

  31. [39]

    Songjun Tu, Jingbo Sun, Qichao Zhang, Yaocheng Zhang, Jia Liu, Ke Chen, and Dongbin Zhao. 2025. In-Dataset Trajectory Return Regularization for Offline Preference-based Reinforcement Learning. In Association for the Advancement of Artificial Intelligence (AAAI)

    2025

  32. [40]

    Denis Yarats, Ilya Kostrikov, and Rob Fergus. 2020. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations (ICLR)

    2020

  33. [41]

    Denis Yarats, Ilya Kostrikov, and Rob Fergus. 2021. Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels. In International Conference on Learning Representations (ICLR)

    2021

  34. [42]

    Keyu Wu, Min Wu, Zhenghua Chen, Yuecong Xu, and Xiaoli Li. 2022. Gener- alizing Reinforcement Learning through Fusing Self-Supervised Learning into Intrinsic Motivation. In Association for the Advancement of Artificial Intelligence (AAAI). 8683–8690

    2022

  35. [43]

    Zhecheng Yuan, Guozheng Ma, Yao Mu, Bo Xia, Bo Yuan, Xueqian Wang, Ping Luo, and Huazhe Xu. 2022. Don’t Touch What Matters: Task-Aware Lipschitz Data Augmentation for Visual Reinforcement Learning. In International Joint Conferences on Artificial Intelligence (IJCAI) . 3702–3708

    2022

  36. [44]

    Zhecheng Yuan, Zhengrong Xue, Bo Yuan, Xueqian Wang, Yi Wu, Yang Gao, and Huazhe Xu. 2022. Pre-trained image encoder for generalizable visual reinforce- ment learning. In Advances in Neural Information Processing Systems (NeurIPS) , Vol. 35. 13022–13037

    2022

  37. [45]

    Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. 2021. Improving sample efficiency in model-free reinforcement learning from images. In Association for the Advancement of Artificial Intelligence (AAAI) , Vol. 35. 10674–10681

    2021

  38. [46]

    Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba

  39. [48]

    Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. 2021. Learning Invariant Representations for Reinforcement Learning without Reconstruction. In International Conference on Learning Representations (ICLR)

    2021

  40. [51]

    These experimental results highlight the advantages of the components incorporated in SCPL

    SCPL consistently outperforms SODA in terms of generalization across both environments. These experimental results highlight the advantages of the components incorporated in SCPL. Setting Method Walker Stand Walker Walk Cartpole Ball Finger Average Video Easy SODA 965 771 742 ...

  41. [2017]

    ∞∑︁ 𝑡 =0 𝛾𝑡𝑟(𝑠𝑡) # − E𝑠0∼𝜋𝑝 h 𝑉𝜋𝑝(𝑠0) i = E𝜏∼𝜋𝑜

    Places: A 10 million image database for scene recognition.IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 40, 6, 1452–1464. APPENDIX A ARCHITECTURE OVERVIEW We implement the SCPL within the SAC framework. The network architecture of SCPL is depicted in F...

  42. [2018]

    arXiv preprint arXiv:1801.00690

    Deepmind control suite. arXiv preprint arXiv:1801.00690

    arXiv

  43. [2024]

    In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS)

    MaDi: Learning to Mask Distractions for Generalization in Visual Deep Reinforcement Learning.. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS) . 733–742

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.