Pith. sign in

REVIEW 2 major objections 71 references

A latent Q-Barrier shield filters or reweights actions in safe in-context RL using predicted future cost and remaining budget, improving reward-safety tradeoffs under OOD shifts without test-time updates.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-30 11:40 UTC pith:EVLMNV2S

load-bearing objection The paper adds an explicit Q-Barrier filter on top of pre-trained latent dynamics for safe ICRL without test-time updates, with a conditional proof and benchmark gains, but the OOD reliability of the safety margin is not directly checked. the 2 major comments →

arxiv 2605.25267 v1 pith:EVLMNV2S submitted 2026-05-24 cs.LG cs.AI

Latent Q-Barrier Shielding for Safe In-Context Reinforcement Learning

Minjae Kwon , Amir Moeini , Shangtong Zhang , Lu Feng This is my paper
classification cs.LG cs.AI
keywords safe in-context reinforcement learningQ-barrier shieldinglatent dynamicsensemble cost criticout-of-distribution robustnessbarrier marginreinforcement learning safety
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that pre-training a context representation, latent dynamics model, and ensemble cost critic allows construction of a shield that performs an explicit action-level safety check at deployment. Standard safe ICRL only conditions the policy on history, so the remaining budget influences behavior only indirectly; the shield makes the check direct by comparing predicted cumulative cost against the budget. A reader would care because this addresses poor tradeoffs that arise when pretraining-only methods encounter distribution shifts. The method includes a proof that any action satisfying the Q-Barrier condition leaves an approximately budget-safe continuation, up to Bellman and latent-prediction errors. On five benchmarks the shielded policy reaches higher return in four cases while matching or lowering average episode cost in all five.

Core claim

The latent Q-Barrier shield learns a context representation, latent dynamics, and ensemble cost critic before deployment. Without any parameter updates it infers context from interaction history and filters or softly reweights candidate actions using the remaining budget and predicted future cost. It proves a conditional, error-decomposed barrier-margin result: a Q-Barrier-satisfying action leaves the next latent-budget state with an approximately budget-safe continuation under the learned critic, up to Bellman and latent-prediction errors. Across five safe ICRL benchmarks the shield improves deployment-time reward-safety tradeoffs over a strong baseline after a short context window.

What carries the argument

The latent Q-Barrier shield, which combines a learned latent dynamics model and ensemble cost critic to enforce an explicit barrier condition that checks predicted future cost against the remaining safety budget.

Load-bearing premise

The pre-trained latent dynamics and ensemble cost critic remain sufficiently accurate under OOD deployment shifts to allow reliable filtering of unsafe actions using only the remaining budget and predicted future cost.

What would settle it

On a new safe ICRL benchmark with larger distribution shift, if the shielded policy violates the cost budget more frequently than the baseline or fails to raise return while holding cost, the claimed improvement collapses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Higher return in four of five benchmarks and equal or lower average episode cost in all five after a short context window.
  • A conditional barrier-margin guarantee holds for any Q-Barrier-satisfying action up to explicit error terms.
  • The shield works on top of any frozen pre-trained safe ICRL policy without online adaptation.
  • Explicit action-level cost prediction compensates for the indirect effect of budget information in policy conditioning alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If latent accuracy persists, the same pre-training-plus-shield pattern could apply to continuous-action robotic tasks with hard safety constraints.
  • Replacing the ensemble critic with a single uncertainty-aware network would test whether the method can be made lighter while preserving the margin guarantee.
  • Applying the shield in a setting where the safety budget itself changes mid-episode would expose whether the barrier condition remains valid under non-stationary constraints.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The paper proposes Latent Q-Barrier Shielding for safe in-context reinforcement learning. It pre-trains a context encoder, latent dynamics model, and ensemble cost critic offline. At test time, without parameter updates, the shield infers context from history and filters or reweights actions by checking predicted future cost against the remaining safety budget. It states a conditional theorem decomposing the barrier margin into Bellman and latent-prediction errors, and reports that the method yields higher return than a strong safe-ICRL baseline in four of five benchmarks while matching or lowering average episode cost in all five after a short context window.

Significance. If the conditional barrier result holds with verified error bounds under OOD shifts and the benchmark gains are robust, the approach would supply an explicit, model-based safety filter for ICRL that operates purely at inference time. This addresses a gap between frozen policy conditioning and explicit cost budgeting, with potential applicability to domains requiring deployment-time safety guarantees without online adaptation.

major comments (2)
  1. [Proof of the barrier-margin result and §5 (Experiments)] The barrier-margin theorem (stated in the abstract and detailed in the main text) is explicitly conditional on small Bellman and latent-prediction errors, yet the experimental section provides no measurements or bounds on these errors evaluated on the actual OOD deployment trajectories across the five benchmarks. Without this link, it is impossible to determine whether the reported safety improvements are explained by the theorem or by other factors such as limited distribution shift.
  2. [§5 (Experiments) and abstract] The empirical claim of improved reward-safety tradeoffs rests on benchmark results, but the manuscript supplies no details on the experimental protocol, including number of random seeds, statistical significance testing, exact context-window lengths, cost-budget values, or how the strong safe-ICRL baseline was implemented and tuned.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback. We address each major comment below and commit to incorporating the requested clarifications and analyses in the revised manuscript.

read point-by-point responses

  1. Referee: [Proof of the barrier-margin result and §5 (Experiments)] The barrier-margin theorem (stated in the abstract and detailed in the main text) is explicitly conditional on small Bellman and latent-prediction errors, yet the experimental section provides no measurements or bounds on these errors evaluated on the actual OOD deployment trajectories across the five benchmarks. Without this link, it is impossible to determine whether the reported safety improvements are explained by the theorem or by other factors such as limited distribution shift.

    Authors: We agree that reporting the Bellman and latent-prediction errors on the OOD deployment trajectories would directly strengthen the link between the conditional theorem and the observed safety gains. In the revised manuscript we will add, in Section 5, quantitative measurements of these two error terms computed on the actual test trajectories for each of the five benchmarks, together with a brief discussion of their magnitudes relative to the reported performance improvements. revision: yes

  2. Referee: [§5 (Experiments) and abstract] The empirical claim of improved reward-safety tradeoffs rests on benchmark results, but the manuscript supplies no details on the experimental protocol, including number of random seeds, statistical significance testing, exact context-window lengths, cost-budget values, or how the strong safe-ICRL baseline was implemented and tuned.

    Authors: We acknowledge that several experimental details were omitted. In the revised Section 5 we will explicitly state: the number of random seeds (10), the statistical testing procedure (mean ± std and paired t-tests), the exact context-window lengths used, the per-environment cost-budget values, and a description of the baseline implementation together with its hyperparameter selection protocol. revision: yes

Circularity Check

0 steps flagged

No significant circularity; theorem is conditional error bound, not tautological with fits

full rationale

The paper states a conditional barrier-margin theorem decomposed into Bellman and latent-prediction errors, with empirical results on five benchmarks presented separately. No quoted equations show a prediction or margin reducing by construction to fitted parameters, no self-citation load-bearing the central claim, and no renaming of known results as new derivations. The accuracy-under-OOD assumption is an external premise, not a definitional loop. The derivation chain remains self-contained against the stated conditional result.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review supplies no concrete equations, fitting procedures, or background lemmas; therefore no free parameters, axioms, or invented entities can be extracted.

pith-pipeline@v0.9.1-grok · 5734 in / 1143 out tokens · 26523 ms · 2026-06-30T11:40:22.689148+00:00 · methodology

0 comments
read the original abstract

Safe in-context reinforcement learning (ICRL) adapts online from interaction history without test-time parameter updates while controlling episode cost under a safety budget. Under out-of-distribution (OOD) deployment shifts, pretraining-only safe ICRL can give poor reward-safety tradeoffs because the remaining budget affects behavior only through frozen policy conditioning, not an explicit action-level check against predicted future cost. We propose a latent Q-Barrier shield that learns a context representation, latent dynamics, and an ensemble cost critic before deployment. Without parameter updates, the shield infers context from history and filters or softly reweights candidate actions using the remaining budget and predicted future cost. We prove a conditional, error-decomposed barrier-margin result: a Q-Barrier-satisfying action leaves the next latent-budget state with an approximately budget-safe continuation under the learned critic, up to Bellman and latent-prediction errors. Across five safe ICRL benchmarks, the shield improves deployment-time reward-safety tradeoffs over a strong safe-ICRL baseline: after a short context window, it achieves higher return in four of five benchmarks while matching or lowering average episode cost in all five.

Figures

Figures reproduced from arXiv: 2605.25267 by Amir Moeini, Lu Feng, Minjae Kwon, Shangtong Zhang.

Figure 1
Figure 1. Figure 1: In-context adaptation dynamics (RQ1). Per-episode return (top) and cost (bottom) over in-context episode index k on test-time evaluation tasks. Curves show means over 100 evaluation tasks, with shaded regions denoting standard error of the mean. Q-Barrier generally improves adaptation by reaching high return earlier and reducing cost after a short context window, while SafetyHalfCheetah shows a conservativ… view at source ↗
Figure 2
Figure 2. Figure 2: Reward-cost tradeoffs under budget variation (RQ2). Cumulative reward (top) and average episode cost (bottom) over varying cost budgets on test-time evaluation tasks. Across different budget levels, Q-Barrier achieves the highest cumulative return in four of five environments while maintaining budget compliance in most cases, with violations mostly limited to near-zero budgets. Shaded regions denote standa… view at source ↗
Figure 3
Figure 3. Figure 3: Theorem-aligned diagnostics on OOD evaluation trajectories. We report one-step latent prediction error, induced value perturbation, Bellman upper-bound satisfaction, and positive Bellman residual. Error bars show standard error over rollout-level summaries. We compare the ensemble mean critic with the pessimistic critic used by the deployed Q-barrier. Bellman residuals are plotted on a log scale because th… view at source ↗
Figure 4
Figure 4. Figure 4: Soft versus hard Q-barrier shielding. We compare the soft shield used in the main experiments with a hard truncation variant that selects only from barrier-feasible candidates when possible. The two variants have similar reward–safety profiles in most environments. Hard shielding gives marginal cost reductions in several environments, such as SafeDarkRoom, but these gains can come with small return loss an… view at source ↗
Figure 5
Figure 5. Figure 5: Effect of candidate-set size on OOD adaptation dynamics. We vary the number of sampled candidate actions Ns ∈ {4, 8, 16, 32} used by our Q-barrier shield during deployment. (a) soft filtering, which reweights candidates by their barrier values, and (b) hard filtering, which removes candidates that fail the barrier test. Curves show per-episode return and cost over the in-context OOD evaluation horizon, wit… view at source ↗
Figure 6
Figure 6. Figure 6: Visualization of the environment layouts, directly adapted from Moeini et al. [PITH_FULL_IMAGE:figures/full_fig_p026_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

71 extracted references · 71 canonical work pages

  1. [1]

    Constrained policy optimization

    Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. InProceedings of the International Conference on Machine Learning, 2017

    work page 2017

  2. [2]

    Anysafe: Adapting latent safety filters at runtime via safety constraint parameterization in the latent space

    Sankalp Agrawal, Junwon Seo, Kensuke Nakamura, Ran Tian, and Andrea Bajcsy. Anysafe: Adapting latent safety filters at runtime via safety constraint parameterization in the latent space. IEEE International Conference on Robotics and Automation (ICRA), 2026

    work page 2026

  3. [3]

    Safe reinforcement learning via shielding

    Mohammed Alshiekh, Roderick Bloem, R ¨udiger Ehlers, Bettina K ¨onighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. InAAAI Conference on Artificial Intelligence, 2018

    work page 2018

  4. [4]

    Routledge, 2021

    Eitan Altman.Constrained Markov decision processes. Routledge, 2021

    work page 2021

  5. [5]

    Actsafe: Active exploration with safety constraints for reinforcement learning

    Yarden As, Bhavya Sukhija, Lenart Treven, Carmelo Sferrazza, Stelian Coros, and Andreas Krause. Actsafe: Active exploration with safety constraints for reinforcement learning. InThe Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  6. [6]

    Contextualize me – the case for context in reinforcement learning.Transactions on Machine Learning Research, 2023

    Carolin Benjamins, Theresa Eimer, Frederik Schubert, Aditya Mohan, Sebastian D¨ohler, Andr´e Biedenkapp, Bodo Rosenhahn, Frank Hutter, and Marius Lindauer. Contextualize me – the case for context in reinforcement learning.Transactions on Machine Learning Research, 2023

    work page 2023

  7. [7]

    Safe reinforcement learning via shielding under partial observability

    Steven Carr, Nils Jansen, Sebastian Junges, and Ufuk Topcu. Safe reinforcement learning via shielding under partial observability. InThe Thirty-Seventh AAAI Conference on Artificial Intelligence, 2023

    work page 2023

  8. [8]

    Context-aware safe reinforcement learning for non-stationary environments

    Baiming Chen, Zuxin Liu, Jiacheng Zhu, Mengdi Xu, Wenhao Ding, Liang Li, and Ding Zhao. Context-aware safe reinforcement learning for non-stationary environments. In2021 IEEE International Conference on Robotics and Automation (ICRA), 2021

    work page 2021

  9. [9]

    An adaptive deep RL method for non-stationary environments with piecewise stable context

    Xiaoyu Chen, Xiangming Zhu, Yufeng Zheng, Pushi Zhang, Li Zhao, Wenxue Cheng, Peng Cheng, Yongqiang Xiong, Tao Qin, Jianyu Chen, and Tie-Yan Liu. An adaptive deep RL method for non-stationary environments with piecewise stable context. InAdvances in Neural Information Processing Systems, 2022

    work page 2022

  10. [10]

    Safe and efficient reinforcement learning using disturbance-observer-based control barrier functions

    Yikun Cheng, Pan Zhao, and Naira Hovakimyan. Safe and efficient reinforcement learning using disturbance-observer-based control barrier functions. InProceedings of The 5th Annual Learning for Dynamics and Control Conference, 2023

    work page 2023

  11. [11]

    Tomlin, and Koushil Sreenath

    Jason Choi, Fernando Casta ˜neda, Claire J. Tomlin, and Koushil Sreenath. Reinforcement learning for safety-critical control under model uncertainty, using control lyapunov functions and control barrier functions. InRobotics: Science and Systems, 2020

    work page 2020

  12. [12]

    In-context exploration-exploitation for reinforcement learning.ArXiv Preprint, 2024

    Zhenwen Dai, Federico Tomasi, and Sina Ghiassian. In-context exploration-exploitation for reinforcement learning.ArXiv Preprint, 2024

    work page 2024

  13. [13]

    Fang, Zhuoran Yang, and Vahid Tarokh

    Juncheng Dong, Moyang Guo, Ethan X. Fang, Zhuoran Yang, and Vahid Tarokh. In-context reinforcement learning without optimal action labels. InICML 2024 Workshop on In-Context Learning, 2024

    work page 2024

  14. [14]

    RL2: Fast reinforcement learning via slow reinforcement learning.ArXiv Preprint, 2016

    Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RL2: Fast reinforcement learning via slow reinforcement learning.ArXiv Preprint, 2016. 10

    work page 2016

  15. [15]

    Model-agnostic meta-learning for fast adapta- tion of deep networks

    Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. InProceedings of the International Conference on Machine Learning, 2017

    work page 2017

  16. [16]

    Iterative reachability estimation for safe reinforcement learning

    Milan Ganai, Zheng Gong, Chenning Yu, Sylvia Lee Herbert, and Sicun Gao. Iterative reachability estimation for safe reinforcement learning. InThirty-seventh Conference on Neural Information Processing Systems, 2023

    work page 2023

  17. [17]

    Amago: Scalable in-context reinforcement learning for adaptive agents

    Jake Grigsby, Linxi Fan, and Yuke Zhu. Amago: Scalable in-context reinforcement learning for adaptive agents. InProceedings of the International Conference on Learning Representations, 2024

    work page 2024

  18. [18]

    Amago-2: Breaking the multi-task barrier in meta-reinforcement learning with transformers

    Jake Grigsby, Justin Sasek, Samyak Parajuli, Ikechukwu D Adebi, Amy Zhang, and Yuke Zhu. Amago-2: Breaking the multi-task barrier in meta-reinforcement learning with transformers. In Advances in Neural Information Processing Systems, 2024

    work page 2024

  19. [19]

    Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, R´emi Munos, and Michal Valko

    Jean-Bastien Grill, Florian Strub, Florent Altch´e, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, R´emi Munos, and Michal Valko. Bootstrap your own latent a new approach to self-supervised learning. InProceedings of the 34th Internat...

    work page 2020

  20. [20]

    Enhancing efficiency of safe reinforcement learning via sample manipulation

    Shangding Gu, Laixi Shi, Yuhao Ding, Alois Knoll, Costas Spanos, Adam Wierman, and Ming Jin. Enhancing efficiency of safe reinforcement learning via sample manipulation. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    work page 2024

  21. [21]

    Cost-aware offline safe meta reinforcement learning with robust in-distribution online task adaptation

    Cong Guan, Ruiqi Xue, Ziqian Zhang, Lihe Li, Yi-Chen Li, Lei Yuan, and Yang Yu. Cost-aware offline safe meta reinforcement learning with robust in-distribution online task adaptation. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, 2024

    work page 2024

  22. [22]

    Zero-shot transfer of neural ODEs

    Tyler Ingebrand, Adam Thorpe, and Ufuk Topcu. Zero-shot transfer of neural ODEs. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    work page 2024

  23. [23]

    Zero-shot reinforcement learning via function encoders

    Tyler Ingebrand, Amy Zhang, and Ufuk Topcu. Zero-shot reinforcement learning via function encoders. InProceedings of the 41st International Conference on Machine Learning, 2024

    work page 2024

  24. [24]

    Reinforcement learning with unsupervised auxiliary tasks

    Max Jaderberg, V olodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. InInternational Conference on Learning Representations, 2017

    work page 2017

  25. [25]

    Safety gymnasium: A unified safe reinforcement learning benchmark

    Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong, Josef Dai, and Yaodong Yang. Safety gymnasium: A unified safe reinforcement learning benchmark. InAdvances in Neural Information Processing Systems, 2023

    work page 2023

  26. [26]

    A CMDP-within- online framework for meta-safe reinforcement learning

    Vanshaj Khattar, Yuhao Ding, Bilgehan Sel, Javad Lavaei, and Ming Jin. A CMDP-within- online framework for meta-safe reinforcement learning. InProceedings of the International Conference on Learning Representations, 2023

    work page 2023

  27. [27]

    A survey of zero-shot generalisation in deep reinforcement learning.ArXiv Preprint, 2021

    Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rockt¨aschel. A survey of zero-shot generalisation in deep reinforcement learning.ArXiv Preprint, 2021

    work page 2021

  28. [28]

    Freeman, Jascha Sohl-Dickstein, and J¨urgen Schmidhuber

    Louis Kirsch, James Harrison, C. Freeman, Jascha Sohl-Dickstein, and J¨urgen Schmidhuber. To- wards general-purpose in-context learning agents. InNeurIPS Foundation Models for Decision Making Workshop, 2023

    work page 2023

  29. [29]

    Sim˜ao, Nils Jansen, and Ufuk Topcu

    Cevahir Koprulu, Thiago D. Sim˜ao, Nils Jansen, and Ufuk Topcu. Safety-prioritizing curricula for constrained reinforcement learning. InThe Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  30. [30]

    Can large language models explore in-context?ArXiv Preprint, 2024

    Akshay Krishnamurthy, Keegan Harris, Dylan J Foster, Cyril Zhang, and Aleksandrs Slivkins. Can large language models explore in-context?ArXiv Preprint, 2024. 11

    work page 2024

  31. [31]

    Adaptive shielding for safe reinforcement learning under hidden-parameter dynam- ics shifts

    Minjae Kwon, Tyler Ingebrand, Ufuk Topcu, and Lu Feng. Adaptive shielding for safe re- inforcement learning under hidden-parameter dynamics shifts, 2026. URL https://arxiv. org/abs/2506.11033

    work page arXiv 2026

  32. [32]

    Curl: Contrastive unsupervised represen- tations for reinforcement learning

    Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning. InProceedings of the International Conference on Machine Learning, 2020

    work page 2020

  33. [33]

    In-context reinforcement learning with algorithm distillation

    Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steiger- wald, DJ Strouse, Steven Stenberg Hansen, Angelos Filos, Ethan Brooks, Maxime Gazeau, Himanshu Sahni, Satinder Singh, and V olodymyr Mnih. In-context reinforcement learning with algorithm distillation. InProceedings of the International Conference on Learning Represe...

    work page 2023

  34. [34]

    Supervised pretraining can learn in-context reinforcement learning

    Jonathan Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma Brunskill. Supervised pretraining can learn in-context reinforcement learning. In Advances in Neural Information Processing Systems, 2023

    work page 2023

  35. [35]

    Lars Lindemann, Matthew Cleaveland, Gihyun Shim, and George J. Pappas. Safe planning in dynamic environments using conformal prediction.IEEE Robotics and Automation Letters, 2023

    work page 2023

  36. [36]

    Scalable in-context q-learning

    Jinmei Liu, Fuhong Liu, Zhenhong Sun, Jianye HAO, Huaxiong Li, Bo Wang, Daoyi Dong, Chunlin Chen, and Zhi Wang. Scalable in-context q-learning. InThe Fourteenth International Conference on Learning Representations, 2026

    work page 2026

  37. [37]

    Adapt to environment sudden changes by learning a context-sensitive policy

    Fan-Ming Luo, Shengyi Jiang, Yang Yu, ZongZhang Zhang, and Yi-Feng Zhang. Adapt to environment sudden changes by learning a context-sensitive policy. InProceedings of the AAAI Conference on Artificial Intelligence, 2022

    work page 2022

  38. [38]

    Mesa: Offline meta-rl for safe adaptation and fault tolerance

    Michael Luo, Ashwin Balakrishna, Brijen Thananjeyan, Suraj Nair, Julian Ibarz, Jie Tan, Chelsea Finn, Ion Stoica, and Ken Goldberg. Mesa: Offline meta-rl for safe adaptation and fault tolerance. InAdvances in Neural Information Processing Systems, 2021

    work page 2021

  39. [39]

    A survey of in-context reinforcement learning.ArXiv Preprint, 2025

    Amir Moeini, Jiuqi Wang, Jacob Beck, Ethan Blaser, Shimon Whiteson, Rohan Chandra, and Shangtong Zhang. A survey of in-context reinforcement learning.ArXiv Preprint, 2025

    work page 2025

  40. [40]

    Safe in-context reinforcement learning

    Amir Moeini, Minjae Kwon, Alper Kamil Bozkurt, Yuichi Motai, Rohan Chandra, Lu Feng, and Shangtong Zhang. Safe in-context reinforcement learning. InInternational Conference on Machine Learning (ICML), 2026

    work page 2026

  41. [41]

    Generalizing safety beyond collision- avoidance via latent-space reachability analysis

    Kensuke Nakamura, Lasse Peters, and Andrea Bajcsy. Generalizing safety beyond collision- avoidance via latent-space reachability analysis. InRobotics: Science and Systems XXI, 2025

    work page 2025

  42. [42]

    Vintix: Action model via in-context reinforcement learning

    Andrei Polubarov, Lyubaykin Nikita, Alexander Derevyagin, Ilya Zisman, Denis Tarasov, Alexander Nikulin, and Vladislav Kurenkov. Vintix: Action model via in-context reinforcement learning. InForty-second International Conference on Machine Learning, 2025

    work page 2025

  43. [43]

    Dreaming of many worlds: Learning contextual world models aids zero-shot generalization.Reinforcement Learning Journal, 2024

    Sai Prasanna, Karim Farid, Raghu Rajan, and Andr´e Biedenkapp. Dreaming of many worlds: Learning contextual world models aids zero-shot generalization.Reinforcement Learning Journal, 2024

    work page 2024

  44. [44]

    John Wiley & Sons, 2014

    Martin L Puterman.Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014

    work page 2014

  45. [45]

    Generalization to new sequential decision making tasks with in-context learning.ArXiv Preprint, 2023

    Sharath Chandra Raparthy, Eric Hambro, Robert Kirk, Mikael Henaff, and Roberta Raileanu. Generalization to new sequential decision making tasks with in-context learning.ArXiv Preprint, 2023

    work page 2023

  46. [46]

    Benchmarking safe exploration in deep rein- forcement learning.OpenAI

    Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep rein- forcement learning.OpenAI. https://cdn.openai.com/safexp-short.pdf, 2019

    work page 2019

  47. [47]

    Conformal safety shielding for imperfect-perception agents

    William Scarbro, Calum Imrie, Sinem Getir Yaman, Kavan Fatehi, Corina P ˘as˘areanu, Radu Calinescu, and Ravi Mangal. Conformal safety shielding for imperfect-perception agents. In International Conference on Runtime Verification, 2025. 12

    work page 2025

  48. [48]

    Proximal policy optimization algorithms.ArXiv Preprint, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.ArXiv Preprint, 2017

    work page 2017

  49. [49]

    Pretraining representations for data- efficient reinforcement learning

    Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R Devon Hjelm, Philip Bachman, and Aaron Courville. Pretraining representations for data- efficient reinforcement learning. In A. Beygelzimer, Y . Dauphin, P. Liang, and J. Wortman Vaughan, editors,Advances in Neural Information Processing Systems, 2021

    work page 2021

  50. [50]

    Uncertainty-aware latent safety filters for avoiding out-of-distribution failures.Conference on Robot Learning (CoRL), 2025

    Junwon Seo, Kensuke Nakamura, and Andrea Bajcsy. Uncertainty-aware latent safety filters for avoiding out-of-distribution failures.Conference on Robot Learning (CoRL), 2025

    work page 2025

  51. [51]

    Safe pomdp online planning via shielding

    Shili Sheng, David Parker, and Lu Feng. Safe pomdp online planning via shielding. In2024 IEEE International Conference on Robotics and Automation, 2024

    work page 2024

  52. [52]

    Safe pomdp online plan- ning among dynamic agents via adaptive conformal prediction.IEEE Robotics and Automation Letters, 2024

    Shili Sheng, Pian Yu, David Parker, Marta Kwiatkowska, and Lu Feng. Safe pomdp online plan- ning among dynamic agents via adaptive conformal prediction.IEEE Robotics and Automation Letters, 2024

    work page 2024

  53. [53]

    In-context reinforcement learning for variable action spaces

    Viacheslav Sinii, Alexander Nikulin, Vladislav Kurenkov, Ilya Zisman, and Sergey Kolesnikov. In-context reinforcement learning for variable action spaces. InForty-first International Confer- ence on Machine Learning, 2024

    work page 2024

  54. [54]

    Distilling reinforcement learning algorithms for in-context model-based planning

    Jaehyeon Son, Soochan Lee, and Gunhee Kim. Distilling reinforcement learning algorithms for in-context model-based planning. InProceedings of the International Conference on Learning Representations, 2025

    work page 2025

  55. [55]

    MIT Press, 2018

    Richard S Sutton and Andrew G Barto.Reinforcement Learning: An Introduction (2nd Edition). MIT Press, 2018

    work page 2018

  56. [56]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. InProceedings of the International Conference on Intelligent Robots and Systems, 2012

    work page 2012

  57. [57]

    Safe reinforcement learning in constrained markov decision processes

    Akifumi Wachi and Yanan Sui. Safe reinforcement learning in constrained markov decision processes. InProceedings of the International Conference on Machine Learning, 2020

    work page 2020

  58. [58]

    Learning to reinforcement learn

    Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. ArXiv Preprint, 2016

    work page 2016

  59. [59]

    Transformers can learn temporal difference methods for in-context reinforcement learning

    Jiuqi Wang, Ethan Blaser, Hadi Daneshmand, and Shangtong Zhang. Transformers can learn temporal difference methods for in-context reinforcement learning. InProceedings of the International Conference on Learning Representations, 2025

    work page 2025

  60. [60]

    Towards provable emergence of in-context reinforcement learning

    Jiuqi Wang, Rohan Chandra, and Shangtong Zhang. Towards provable emergence of in-context reinforcement learning. InAdvances in Neural Information Processing Systems, 2025

    work page 2025

  61. [61]

    Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments

    Yixuan Wang, Simon Sinong Zhan, Ruochen Jiao, Zhilu Wang, Wanxin Jin, Zhuoran Yang, Zhaoran Wang, Chao Huang, and Qi Zhu. Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments. InProceedings of the 40th International Conference on Machine Learning, 2023

    work page 2023

  62. [62]

    Barriernet: Differentiable control barrier functions for learning of safe robot control.IEEE Transactions on Robotics, 2023

    Wei Xiao, Tsun-Hsuan Wang, Ramin Hasani, Makram Chahine, Alexander Amini, Xiao Li, and Daniela Rus. Barriernet: Differentiable control barrier functions for learning of safe robot control.IEEE Transactions on Robotics, 2023

    work page 2023

  63. [63]

    Prompting decision transformer for few-shot policy generalization

    Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. InProceedings of the International Conference on Machine Learning, 2022

    work page 2022

  64. [64]

    Efficient safe meta-reinforcement learning: Provable near- optimality and anytime safety

    Siyuan Xu and Minghui Zhu. Efficient safe meta-reinforcement learning: Provable near- optimality and anytime safety. InAdvances in Neural Information Processing Systems, 2025. 13

    work page 2025

  65. [65]

    Single episode policy transfer in reinforcement learning

    Jiachen Yang, Brenden Petersen, Hongyuan Zha, and Daniel Faissol. Single episode policy transfer in reinforcement learning. InInternational Conference on Learning Representations, 2020

    work page 2020

  66. [66]

    Constrained update projection approach to safe policy optimization

    Long Yang, Jiaming Ji, Juntao Dai, Linrui Zhang, Binbin Zhou, Pengfei Li, Yaodong Yang, and Gang Pan. Constrained update projection approach to safe policy optimization. InAdvances in Neural Information Processing Systems, 2022

    work page 2022

  67. [67]

    Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J. Ramadge. Projection-based constrained policy optimization. InInternational Conference on Learning Representations, 2020

    work page 2020

  68. [68]

    Safe reinforcement learning via probabilistic logic shields

    Wen-Chi Yang, Giuseppe Marra, Gavin Rens, and Luc De Raedt. Safe reinforcement learning via probabilistic logic shields. InProceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, 2023

    work page 2023

  69. [69]

    Yiming Zhang, Quan Vuong, and Keith W. Ross. First order constrained optimization in policy space. InProceedings of the 34th International Conference on Neural Information Processing Systems, 2020

    work page 2020

  70. [70]

    Simplified temporal consistency reinforcement learning

    Yi Zhao, Wenshuai Zhao, Rinu Boney, Juho Kannala, and Joni Pajarinen. Simplified temporal consistency reinforcement learning. InProceedings of the 40th International Conference on Machine Learning, 2023

    work page 2023

  71. [71]

    Emergence of in-context reinforcement learning from noise distillation.ArXiv Preprint, 2023

    Ilya Zisman, Vladislav Kurenkov, Alexander Nikulin, Viacheslav Sinii, and Sergey Kolesnikov. Emergence of in-context reinforcement learning from noise distillation.ArXiv Preprint, 2023. A Proofs A.1 Proof of Theorem 1 Before proving the main theorem, we first prove a value error lemma. Under Assumptions 1 and 2, the lemma shows that latent prediction erro...

    work page 2023