Pith. sign in

REVIEW 4 major objections 4 minor 42 references

Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that switching between a reward-maximizing and a cost-minimizing policy, filtered by learned cost Q-values, lets an offline-trained agent satisfy changing safety constraints at deployment without retraining.

desk verdict Useful wrapper for offline safe RL with variable cost limits, but the headline safety count averages over thresholds and flatters the method. read the letter →

arxiv 2412.18946 v2 pith:RYSYDD7H submitted 2024-12-25 cs.LG cs.AI

Yassine Chemingui , Aryan Deshwal , Honghao Wei , Alan Fern , Janardhan Rao Doppa This is my paper · ORCID
classification cs.LGcs.AI
keywords offlinesafereinforcementlearningconstraint-adaptivepolicyswitchingvaryingcostconstraintsQ-functionfilteringsharedrepresentationimplicitQ-learningsafetyguarantees
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Offline safe reinforcement learning usually trains one policy for one fixed cost threshold, so changing the safety budget at deployment means retraining. This paper proposes CAPS, a wrapper around any offline RL algorithm that trains a small set of policies with a shared representation—one reward-maximizing, one cost-minimizing, optionally more—and then, at each test-time state, filters each policy's proposed action through a learned cost Q-function $Q_c$ and picks the highest-reward action that still fits the current cost limit. The central claim is that this simple switching scheme adapts to varying cost constraints without retraining and is safer than existing offline safe RL methods. Across 38 benchmark tasks, CAPS wrapped around IQL satisfies the cost constraint in 34 of 38 tasks and reaches the highest reward among safe methods in 18 of 38 tasks. A theorem bounds expected total cost by $\max\{V^c(s), \kappa-c_{

What carries the argument

The central mechanism is a filter-and-select loop driven by two learned Q-functions. $Q_c$ estimates expected future cost and acts as the safety filter; $Q_r$ estimates expected future reward and acts as the selector; the cost-minimizing policy $\pi_c$ serves as a fallback when no proposed action is safe. Training uses a reduction to standard offline RL: train $Q_r$ and $Q_c$ from the dataset, extract $\pi_r$ from $Q_r$, $\pi_c$ from $Q_c$, and optionally intermediate policies from mixtures $Q_r - \lambda_k Q_c$. A shared neural-network body with one output head per policy transfers features between the reward and cost objectives, and the paper shows this sharing is what makes the cost-minimizing head achieve meaningful reward and the reward head incur lower cost.

What would settle it

Run CAPS on the 38 benchmark tasks with cost thresholds below every threshold used in training, and measure the fraction of episodes whose cumulative cost exceeds the cap; a sharp rise in violations would show the filter admits unsafe actions. More directly, build an MDP with a known large optimal-cost variation $\epsilon$ and check whether realized total cost violates the bound $\max\{V^c(s), \kappa-c_{<t}\} + (T-t)\epsilon$ for a $\kappa$-admissible policy.

Watch

Extended reading notes

Core claim

This paper claims that a finite set of policies with a shared representation, switched by two Q-functions, is enough to handle unknown and varying cost constraints offline. At test time CAPS computes the actions proposed by all policies, keeps only those for which $Q_c(s,a) + c_{\le t} \le \kappa$, and selects the kept action with the largest $Q_r(s,a)$; if none is kept it falls back to the cost-minimizing policy $\pi_c$. The safety theorem states that any such policy is $\kappa$-admissible and its expected remaining cost is at most $\max\{V^c_t(s), \kappa-c_{<t}\} + (T-t)\epsilon$, assuming $Q_c$ is perfect and $\pi_c$ is cost-optimal. The experimental claim is that this recipe is safe on 34/38 tasks with IQL and 30/38 with SAC+BC, and that shared representations are crucial to this performance.

Load-bearing premise

The load-bearing premise is that the learned cost Q-function $Q_c$ accurately estimates future costs and that the cost-minimizing policy $\pi_c$ is truly near cost-optimal; the safety theorem assumes perfect estimation, and the empirical safety margin shrinks if that estimate is wrong.

Editorial extensions

If this is right

  • One model can serve many deployment cost limits: changing $\kappa$ at runtime only changes the filter, not the weights.
  • CAPS is algorithm-agnostic; any offline RL method that returns Q-functions and a policy can be wrapped, so base-algorithm improvements transfer directly.
  • The two-policy CAPS is a useful minimalist default: it removes scalarization hyperparameters and needs only two full offline RL runs.
  • If the environment's optimal-cost variation $\epsilon$ is known, a user can choose $\kappa$ in advance to satisfy a desired expected-cost bound.
  • Training cost stays low relative to per-threshold retraining: CAPS(IQL) trains in 24–33 minutes on the HalfCheetah task versus roughly 154 minutes for CDT, with more heads adding modest overhead.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the safety filter depends on the quality of $Q_c$, a natural extension is a calibration step that measures empirical violation rates across proposed cost limits before deployment and adjusts $\kappa$ or the fallback policy accordingly.
  • The same filter-and-select recipe should generalize to multiple simultaneous constraints by replacing the scalar $Q_c$ with several cost Q-functions and requiring each constraint to pass the filter.
  • A learned fallback policy, rather than the pure cost minimizer, could raise reward under tighter budgets while preserving the guarantee, since the theorem only requires the fallback to be $\kappa$-admissible.
  • The theorem's $(T-t)\epsilon$ term identifies high-stochastic-cost environments as the stress test: deterministic or low-variation domains should show near-tight guarantees, while high-variation domains are where violation monitoring matters most.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes CAPS, a wrapper for offline safe reinforcement learning that trains K policies with a shared representation (reward-only, cost-only, and scalarized trade-offs) and switches among them at test time based on a learned cost Q-function filter and a learned reward Q-function selector. A finite-horizon safety bound is stated under assumptions of a perfect cost Q-function, an optimal-cost fallback policy, and limited optimal-cost variation. Experiments on 38 DSRL tasks compare CAPS(IQL) and CAPS(SAC+BC) with BC, BEAR-Lag, CPQ, COptiDICE, and CDT, with ablations on the number of policies, shared representation, and fitted Q-evaluation. The headline claims are that CAPS(IQL) is safe in 34/38 tasks and achieves the highest reward among safe methods in 18/38 tasks.

Significance. CAPS is a simple and potentially useful contribution: it reduces varying-constraint offline safe RL to standard offline RL, comes with public code, and reports extensive comparisons on 38 tasks with two base algorithms. The ablations on shared representations and number of heads are informative, and the conditional safety analysis is a reasonable starting point. However, the current manuscript does not fully substantiate the headline empirical claims because the main safety metric averages over cost thresholds, and the per-threshold counts in Table 2 are much lower at strict budgets. The theoretical guarantee also depends on an idealization (perfect Qc and cost-optimal πc) without an error-propagation version. These issues are fixable, and if addressed the paper would provide a strong wrapper-based baseline.

major comments (4)
  1. [Section 6.2, Tables 1 and 2] The headline claim that CAPS(IQL) is safe in 34/38 tasks is computed by averaging normalized cost over three distinct cost thresholds and labeling a method safe when this average is ≤ 1. This does not establish that the constraint is satisfied at any individual threshold. Table 2 shows that CAPS(IQL) is safe in only 18/38 tasks at the {5,10} budget, 28/38 at {10,20}, and 33/38 at {20,40}. Thus the 34/38 number conflates thresholds and is contradicted by the paper's own per-threshold counts at strict budgets. Please report normalized costs per threshold and define a task as safe only if it satisfies every evaluated threshold, or revise the abstract and Section 6.2 claims accordingly.
  2. [Section 5, Assumption 2 and Appendix A, Eq. (9)] There is a time-index inconsistency in the proof of Theorem 1. Assumption 2 bounds the variation of V^c_t(s') over next states s' after taking action a at state s and time t, but the next-state values entering the Bellman expansion should be V^c_{t+1}(s'), and the proof of Eq. (9) uses V^c_{t+1}(s') + c(s). As written, the per-support bound 'for any s'' does not follow from the stated assumption. The assumption should be re-indexed to V^c_{t+1}(s') (or the proof otherwise corrected), and the induction step should be checked under that corrected statement.
  3. [Section 5, Theorem 1 and CAPS filter (Eq. 3)] The practical safety guarantee is conditional on a perfectly estimated cost Q-function Qc and a cost-optimal fallback policy πc. The paper says that extending the result to bounded estimation error is straightforward, but no such bound is provided. Since the CAPS filter in Eq. (3) can admit unsafe actions when Qc is inaccurate, the theorem as stated is an ideal-case analysis rather than a guarantee for the implemented system. Please provide a bounded-error version, or explicitly state this limitation and complement it with empirical evidence about the accuracy of the learned Qc.
  4. [Appendix B.3 and Table 2] The CDT comparison is confounded by hand-coded return targets. The manuscript itself notes that CDT requires return and cost targets and that the authors do not provide a clear methodology for selecting these targets; for additional cost configurations the authors interpolated or extrapolated the provided targets. Because Table 2's claim that CAPS 'consistently outperforms CDT across all cost limit configurations' relies on these extra configurations, the comparison is not fully controlled. Please provide a principled target-selection method for CDT or a sensitivity analysis over the target values used.
minor comments (4)
  1. [Section 6.1] The text says the algorithms are evaluated with 'three different target cost threshold configurations' but then lists only two sets, {10, 20, 40} and {20, 40, 80}; please clarify the intended configurations and their mapping to environment categories.
  2. [Section 4.2, Eqs. (5) and (9)] The role of λk for the extreme policies πr and πc is described only in prose; the equation does not show that πr and πc correspond to the limiting cases λ=0 and λ→∞. Please state this explicitly and clarify how the λk values in Table 14 are selected.
  3. [Tables 1 and 3-5] The results are averaged over three seeds and twenty episodes, but no standard deviations or seed-level breakdowns are reported for the normalized rewards and costs. Reporting variance would help assess whether the safety counts and reward differences are stable.
  4. [Appendix B.3] Environment names are inconsistent across tables (for example, 'hardsparse' in Table 10 versus 'HardSparse' elsewhere); please standardize the naming.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CAPS's safety theorem is conditional on explicit assumptions and its empirical claims are evaluated on external benchmarks.

full rationale

The paper's central theoretical result, Theorem 1, is an explicit conditional statement: it assumes a perfectly estimated optimal-cost Q-function, a cost-optimal fallback policy, and the κ-admissibility condition (Assumption 1), then proves a cost bound by induction. The proof does not presuppose the conclusion, and the CAPS decision rule is shown to satisfy κ-admissibility under those stated assumptions rather than being defined to equal the theorem's bound. The empirical evaluation is made against the external DSRL benchmark suite, and the headline numbers (e.g., 34/38 safe tasks) are reported aggregations of measured normalized costs, with per-threshold counts given separately in Table 2; the method is not tuned to fit these benchmark outcomes, and no fitted parameter is later renamed as a prediction. The paper also does not rely on self-citations as load-bearing evidence: references to IQL and SAC+BC point to external algorithms used as base learners, and no uniqueness theorem from the authors' prior work is invoked to force the design choice. Concerns about the realism of the perfect-estimation assumption, the averaging of cost thresholds, or the interpretation of per-threshold safety counts are substantive correctness or evaluation-validity issues, but they are not instances of circular derivation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest primarily on empirical evaluation. The theoretical guarantee depends on assumptions about the cost Q-function accuracy, the optimality of the fallback policy, and limited variability of future optimal costs (optimal-cost variation). The method has several hand-chosen hyperparameters, but no new physical or conceptual entities are introduced.

free parameters (3)
  • scalarization weights λ_k for intermediate policies = λ_k = k / ((K-1)/2) for k=1,...,K-2
    Chosen by a fixed formula in Table 14; they define the reward-cost trade-off of intermediate policies and are not learned from data.
  • number of policies K = 2 (default)
    Selected as the default after ablation; the paper notes CAPS-4 can be slightly better but CAPS-2 is preferred for simplicity and fewer hyperparameters (Section 6.2).
  • IQL expectile τ for value functions = 0.7 for BulletGym, 0.5 for others
    Domain-specific hyperparameter in Table 14; chosen by the authors and affects the learned value estimates.
assumptions (5)
  • ad hoc to paper Assumption 1 (κ-admissibility): for the switching policy, Qc_t(s, π_t(s,c<t)) ≤ max{Vc_t(s), κ-c<t} for all states, times, and accumulated costs.
    Introduced in Section 5; required for the safety bound. It presumes the policy selects actions with low optimal-cost Q-values, which in practice requires accurate Qc and a cost-optimal fallback policy.
  • domain assumption Assumption 2 (optimal-cost variation ε): for every (s,a,t), max_{s'} Vc_t(s') - min_{s'} Vc_t(s') ≤ ε over possible next states.
    Section 5. Limits the variance of future optimal costs; the paper asserts benchmark environments satisfy this with small ε but does not measure ε.
  • domain assumption The cost Q-function Qc is perfectly estimated (in the theorem).
    Section 5 explicitly assumes perfect Qc; the empirical method uses learned Qc, so the guarantee does not directly transfer to experiments.
  • domain assumption The cost-minimizing policy πc achieves the optimal cost Vc_t(s).
    Section 5 states the fallback action has value Vc_t(s) since Qc and Vc are cost optimal; this requires the trained πc to be the true argmin, which is not guaranteed.
  • domain assumption The offline dataset provides adequate coverage of the state-action space for the training policies and Q-functions.
    Implicit in the offline RL formulation (Section 2); if coverage is poor, learned Q-functions and policies may be unreliable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning." pith.science (2026). https://pith.science/paper/RYSYDD7H

@misc{pith2026241218946,
  author       = {Pith},
  title        = {Pith review of: Constraint-Adaptive Policy Switching for Offline Safe Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RYSYDD7H}},
  note         = {Machine review of arXiv:2412.18946}
}
read the original abstract

Offline safe reinforcement learning (OSRL) involves learning a decision-making policy to maximize rewards from a fixed batch of training data to satisfy pre-defined safety constraints. However, adapting to varying safety constraints during deployment without retraining remains an under-explored challenge. To address this challenge, we introduce constraint-adaptive policy switching (CAPS), a wrapper framework around existing offline RL algorithms. During training, CAPS uses offline data to learn multiple policies with a shared representation that optimize different reward and cost trade-offs. During testing, CAPS switches between those policies by selecting at each state the policy that maximizes future rewards among those that satisfy the current cost constraint. Our experiments on 38 tasks from the DSRL benchmark demonstrate that CAPS consistently outperforms existing methods, establishing a strong wrapper-based baseline for OSRL. The code is publicly available at https://github.com/yassineCh/CAPS.

Figures

Figures reproduced from arXiv: 2412.18946 by the authors.

Figure 1
Figure 1. Ablation results for shared architecture vs. indepen [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Visualization of the Safety-Gymnasium environments. [PITH_FULL_IMAGE:figures/full_fig_p020_2.png] view at source ↗
Figure 3
Figure 3. Visualization of the Bullet-Safety-Gym environments. [PITH_FULL_IMAGE:figures/full_fig_p022_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Visualization of the MetaDrive environments. [PITH_FULL_IMAGE:figures/full_fig_p022_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 33 canonical work pages

  1. [1]

    An, G.; Moon, S.; Kim, J.-H.; and Song, H. O. 2021. Uncertainty-based offline reinforcement learning with diversified Q-ensemble. Advances in neural information processing systems, 34: 7436--7447

    work page 2021

  2. [2]

    Bai, C.; Wang, L.; Yang, Z.; Deng, Z.-H.; Garg, A.; Liu, P.; and Wang, Z. 2022. Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning. In International Conference on Learning Representations

    work page 2022

  3. [3]

    Chen, H.; Lu, C.; Ying, C.; Su, H.; and Zhu, J. 2023. Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling. In The Eleventh International Conference on Learning Representations

    work page 2023

  4. [4]

    Figueiredo Prudencio, R.; Maximo, M. R. O. A.; and Colombini, E. L. 2024. A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems. IEEE Transactions on Neural Networks and Learning Systems, 35(8): 10237--10257

    work page 2024

  5. [5]

    Fujimoto, S.; and Gu, S. S. 2021. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34: 20132--20145

    work page 2021

  6. [6]

    Garc a, J.; and Fern \'a ndez, F. 2015. A Comprehensive Curvey on Safe Reinforcement Learning. Journal of Machine Learning Research, 16(1): 1437--1480

    work page 2015

  7. [7]

    Gronauer, S. 2022. Bullet-Safety-Gym: A Framework for Constrained Reinforcement Learning. Technical report, mediaTUM

    work page 2022

  8. [8]

    Gu, S.; Yang, L.; Du, Y.; Chen, G.; Walter, F.; Wang, J.; and Knoll, A. 2024. A Review of Safe Reinforcement Learning: Methods, Theories and Applications. IEEE Transactions on Pattern Analysis and Machine Intelligence

    work page 2024

Show all 42 references
  1. [9]

    Haarnoja, T.; Zhou, A.; Hartikainen, K.; Tucker, G.; Ha, S.; Tan, J.; Kumar, V.; Zhu, H.; Gupta, A.; Abbeel, P.; et al. 2018. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905

    2018 arXiv

  2. [10]

    G.; and Levine, S

    Hansen-Estruch, P.; Kostrikov, I.; Janner, M.; Kuba, J. G.; and Levine, S. 2023. Idql: Implicit Q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573

    2023 arXiv

  3. [11]

    Janner, M.; Li, Q.; and Levine, S. 2021. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34: 1273--1286

    2021

  4. [12]

    H.; Ferguson, C.; Lapedriza, A.; Jones, N.; Gu, S.; and Picard, R

    Jaques, N.; Ghandeharioun, A.; Shen, J. H.; Ferguson, C.; Lapedriza, A.; Jones, N.; Gu, S.; and Picard, R. 2020. Way Off-Policy Batch Deep Reinforcement Learning of Human Preferences in Dialog

    2020

  5. [13]

    Ji, J.; Zhou, J.; Zhang, B.; Dai, J.; Pan, X.; Sun, R.; Huang, W.; Geng, Y.; Liu, M.; and Yang, Y. 2024. OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Research. Journal of Machine Learning Research, 25(285): 1--6

    2024

  6. [14]

    Kidambi, R.; Rajeswaran, A.; Netrapalli, P.; and Joachims, T. 2020. MOReL: Model-Based Offline Reinforcement Learning. In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; and Lin, H., eds., Advances in Neural Information Processing Systems, volume 33, 21810--21823. Curran...

    2020

  7. [15]

    Kostrikov, I.; Nair, A.; and Levine, S. 2022. Offline Reinforcement Learning with Implicit Q-Learning. In International Conference on Learning Representations

    2022

  8. [16]

    Kumar, A.; Fu, J.; Soh, M.; Tucker, G.; and Levine, S. 2019. Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction. Advances in neural information processing systems, 32

    2019

  9. [17]

    Le, H.; Voloshin, C.; and Yue, Y. 2019. Batch policy learning under constraints. In International Conference on Machine Learning, 3703--3712. PMLR

    2019

  10. [18]

    J.; Heess, N.; Precup, D.; Kim, K.-E.; and Guez, A

    Lee, J.; Paduraru, C.; Mankowitz, D. J.; Heess, N.; Precup, D.; Kim, K.-E.; and Guez, A. 2022. CO pti DICE : Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimation. In International Conference on Learning Representations

    2022

  11. [19]

    Levine, S.; Kumar, A.; Tucker, G.; and Fu, J. 2020. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643

    2020 arXiv

  12. [20]

    Li, Q.; Peng, Z.; Feng, L.; Zhang, Q.; Xue, Z.; and Zhou, B. 2022. Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE transactions on pattern analysis and machine intelligence, 45(3): 3461--3475

    2022

  13. [21]

    Li, Y. 2017. Deep reinforcement learning: An overview. arXiv preprint arXiv:1701.07274

    2017 arXiv

  14. [22]

    Lin, Q.; Tang, B.; Wu, Z.; Yu, C.; Mao, S.; Xie, Q.; Wang, X.; and Wang, D. 2023. Safe offline reinforcement learning with real-time budget constraints. In International Conference on Machine Learning, 21127--21152. PMLR

    2023

  15. [23]

    Liu, Z.; Guo, Z.; Lin, H.; Yao, Y.; Zhu, J.; Cen, Z.; Hu, H.; Yu, W.; Zhang, T.; Tan, J.; and Zhao, D. 2024. Datasets and Benchmarks for Offline Safe Reinforcement Learning. Journal of Data-centric Machine Learning Research

    2024

  16. [24]

    Liu, Z.; Guo, Z.; Yao, Y.; Cen, Z.; Yu, W.; Zhang, T.; and Zhao, D. 2023. Constrained decision transformer for offline safe reinforcement learning. In International Conference on Machine Learning, 21611--21630. PMLR

    2023

  17. [25]

    Lyu, J.; Ma, X.; Li, X.; and Lu, Z. 2022. Mildly conservative Q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 35: 1711--1724

    2022

  18. [26]

    A.; Veness, J.; Bellemare, M

    Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.; Fidjeland, A. K.; Ostrovski, G.; et al. 2015. Human-level control through deep reinforcement learning. Nature, 518(7540): 529--533

    2015

  19. [27]

    C.; Fiterau, M.; and Jagannath, J

    Polosky, N.; Da Silva, B. C.; Fiterau, M.; and Jagannath, J. 2022. Constrained offline policy optimization. In International Conference on Machine Learning, 17801--17810. PMLR

    2022

  20. [28]

    Ray, A.; Achiam, J.; and Amodei, D. 2019. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708

    2019 arXiv

  21. [29]

    Rigter, M.; Lacerda, B.; and Hawes, N. 2022. Rambo-rl: Robust adversarial model-based offline reinforcement learning. Advances in neural information processing systems, 35: 16082--16097

    2022

  22. [30]

    Silver, D.; Hubert, T.; Schrittwieser, J.; Antonoglou, I.; Lai, M.; Guez, A.; Lanctot, M.; Sifre, L.; Kumaran, D.; Graepel, T.; et al. 2018. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science, 362(6419): 1140--1144

    2018

  23. [31]

    Wang, K.; Zhao, H.; Luo, X.; Ren, K.; Zhang, W.; and Li, D. 2022. Bootstrapped transformer for offline reinforcement learning. Advances in Neural Information Processing Systems, 35: 34748--34761

    2022

  24. [32]

    Wu, J.; Wu, H.; Qiu, Z.; Wang, J.; and Long, M. 2022. Supported policy optimization for offline reinforcement learning. Advances in Neural Information Processing Systems, 35: 31278--31291

    2022

  25. [33]

    Wu, Y.; Tucker, G.; and Nachum, O. 2020. Behavior Regularized Offline Reinforcement Learning

    2020

  26. [34]

    Xu, H.; Jiang, L.; Jianxiong, L.; and Zhan, X. 2022. A policy-guided imitation approach for offline reinforcement learning. Advances in Neural Information Processing Systems, 35: 4085--4098

    2022

  27. [35]

    Xu, H.; Zhan, X.; and Zhu, X. 2022. Constraints penalized Q-learning for safe offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 8753--8760

    2022

  28. [36]

    Yang, R.; Bai, C.; Ma, X.; Wang, Z.; Zhang, C.; and Han, L. 2022. RORL : Robust Offline Reinforcement Learning via Conservative Smoothing. In Oh, A. H.; Agarwal, A.; Belgrave, D.; and Cho, K., eds., Advances in Neural Information Processing Systems

    2022

  29. [37]

    Yao, Y.; Liu, Z.; Cen, Z.; Zhu, J.; Yu, W.; Zhang, T.; and Zhao, D. 2023. Constraint-Conditioned Policy Optimization for Versatile Safe Reinforcement Learning. In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  30. [38]

    Y.; Levine, S.; Finn, C.; and Ma, T

    Yu, T.; Thomas, G.; Yu, L.; Ermon, S.; Zou, J. Y.; Levine, S.; Finn, C.; and Ma, T. 2020. MOPO: Model-based Offline Policy Optimization. In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; and Lin, H., eds., Advances in Neural Information Processing Systems, volume 33, 14...

    2020

  31. [39]

    Zhang, H.; Xu, W.; and Yu, H. 2023. Policy Expansion for Bridging Offline-to-Online Reinforcement Learning. In The Eleventh International Conference on Learning Representations

    2023

  32. [40]

    E.; Zhan, X.; and Liu, J

    Zheng, Y.; Li, J.; Yu, D.; Yang, Y.; Li, S. E.; Zhan, X.; and Liu, J. 2024. Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model. In The Twelfth International Conference on Learning Representations

    2024

  33. [41]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  34. [42]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.