Pith. sign in

REVIEW 2 major objections 5 minor 73 references

A single conditional diffusion model learns a shared safe policy from multi-task offline data, meeting different cost limits at test time without retraining.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 23:07 UTC pith:N2AO4SUC

load-bearing objection First clean formulation of multi-task offline safe RL plus a workable diffusion planner that actually switches cost thresholds at test time; the MetaDrive numbers hold up, the independence assumption in the guidance is the soft spot. the 2 major comments →

arxiv 2607.03903 v1 pith:N2AO4SUC submitted 2026-07-04 cs.LG

CDCP: Conditional Diffusion Model with Contextual Prompts for Multi-task Offline Safe Reinforcement Learning

Jiayi Guan , Tianle Zhang , Li Shen , Ruiqi Zhang , Ao Zhou , Lusong Li , Guai Chen , Mengjie Li
show 3 more authors
Alois Knoll Xiaodong He Changjun Jiang
This is my paper
classification cs.LG
keywords multi-task offline safe RLconditional diffusion modelclassifier-free guidancecontextual promptscost constraintsMetaDrivegradient loss synchronisation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Multi-task offline safe RL wants one shared policy that maximises average reward across tasks while keeping every task inside its safety cost limit, all from static data alone. Existing methods either ignore costs or suffer accumulating Q-value errors when many tasks and out-of-distribution actions are present. This paper turns the constrained problem into a conditional generation task: a diffusion model is trained by supervised denoising to produce action sequences given history, a textual-plus-trajectory prompt that identifies the task, and remaining discounted reward and cost as conditions. Classifier-free guidance lets the same trained model enforce different cost thresholds simply by changing the cost condition at test time; contextual prompts improve representation of seen and unseen tasks; gradient-loss synchronisation keeps multi-task training stable. On nine MetaDrive driving tasks the resulting policy yields higher mean reward than adapted baselines while keeping mean cost under the limit, and it can switch cost budgets without further training.

Core claim

The multi-task offline safe RL problem (maximise average reward subject to per-task cumulative cost constraints, using only offline trajectories) can be solved by casting it as conditional generation of action sequences with a diffusion model, where classifier-free guidance on remaining reward and cost supplies flexible safety without Q-functions or Lagrangian multipliers.

What carries the argument

Classifier-free guided cost-constraint strategy inside the diffusion reverse process: the noise predictor is conditioned on observation history and contextual prompt always, and on remaining discounted reward and cost with probability 1-p_β; at sampling the two conditional scores are combined with an unconditional score so that different cost thresholds can be realised simply by changing the cost condition after training.

Load-bearing premise

That normalising remaining discounted cost to [0,1] and feeding it as a classifier-free condition is an accurate enough, non-conflicting signal to enforce cumulative (or state-wise) safety across tasks whose raw cost scales and distributions differ.

What would settle it

Retrain or re-evaluate CDCP on the same nine MetaDrive tasks after deliberately mis-normalising or removing the remaining-cost condition; if mean cost then exceeds the stated thresholds while reward stays high, or if test-time threshold changes no longer control cost, the flexible-constraint claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single offline-trained model can serve multiple cost budgets for the same set of tasks by editing only the cost condition at inference.
  • Setting the cost condition to zero converts cumulative cost constraints into state-wise constraints without architectural change.
  • Contextual trajectory-plus-text prompts allow the same model to be adapted to previously unseen tasks from few expert trajectories.
  • Gradient-loss synchronisation removes the need for manual task weighting when training a shared diffusion policy on heterogeneous offline datasets.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same remaining-cost guidance idea could be ported to non-driving multi-task offline settings (robotic manipulation suites, process control) where cost scales also differ across tasks.
  • If the independence assumption between reward and cost conditions is relaxed, a joint reward-cost embedding might further reduce residual cost overshoots observed on a few hard tasks.
  • Because the method never estimates Q-values, it may scale more gracefully than conservative Q-learning variants when the number of simultaneous tasks grows beyond the nine used here.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper defines multi-task offline safe RL (MTOS/CMTOS) as learning a shared policy that maximizes average discounted reward across tasks while satisfying per-task cumulative cost constraints from static multi-task datasets (Definition 1–2, Eq. 6). It recasts the problem as conditional diffusion generation of action sequences (Lemma IV.1, Eqs. 7–9), trains a DDPM-style noise predictor with classifier-free guidance that jointly conditions on remaining discounted reward and cost (Eq. 10, Prop. IV.2 / Eq. 11), injects contextual prompts consisting of short expert trajectory snippets plus a frozen language-model embedding of the task description (Eqs. 12–13), and balances multi-task gradients via a learned weight layer driven by loss norms and decay rates (Eqs. 14–19). On nine MetaDrive driving tasks the method reports higher mean reward (≈270) than five adapted baselines while keeping mean cost near or under a threshold of 25, supports different thresholds at test time without retraining (Table III, Remark IV.3), and shows limited few-shot transfer (Table IV).

Significance. If the empirical claims hold, the work supplies the first concrete formulation and practical algorithm for multi-task offline safe RL, a setting that is practically relevant for autonomous driving and multi-skill robotics where interaction is costly and safety is non-negotiable. The combination of supervised diffusion generation (thereby sidestepping OOD Q-value extrapolation), classifier-free cost guidance that permits post-hoc threshold changes, language-plus-trajectory contextual prompts, and a GradNorm-style loss synchronizer is a coherent engineering package. The experimental suite—nine heterogeneous MetaDrive tasks, three seeds, best-model evaluation over ten episodes, ablations of the three proposed components, few-shot transfer, and a cost-threshold sweep without retraining—is more thorough than is common for early papers in this niche and gives the community a usable baseline and dataset reference.

major comments (2)
  1. The central claim of flexible test-time cost control (Remark IV.3, Table III) rests on two linked modeling choices whose validity is not isolated: (i) feeding the remaining discounted trajectory cost after simple per-task min-max normalization (Eqs. 20–21 plus the free correction η_C) as a CFG condition, and (ii) the derivation of the combined perturbation noise (Prop. IV.2 / Eq. 11 and the Appendix proof) that treats reward and cost conditions as conditionally independent given the noisy action sequence. Figs. 3–4 show that the nine MetaDrive tasks possess markedly different raw cost scales and strong negative reward–cost correlation. If either the normalization fails to produce a commensurate guidance signal or the independence assumption is violated, the guidance direction becomes mis-specified and the model cannot be trusted to stay under a newly chosen threshold while preserving the
  2. All baselines are single-task offline-safe or multi-task offline methods that the authors themselves extend by simple task-ID embeddings or by adding CFG cost conditions (MTCPQ, MTCOptiDICE, MTCDT, CMTDD, CMTDiff). Because no prior multi-task offline safe RL algorithm exists, this is understandable, yet it leaves open whether the observed gains arise from the proposed architecture or merely from the fact that the baselines were never designed for the joint setting. A stronger experimental design would include at least one carefully re-tuned multi-task constrained offline method (e.g., a multi-task Lagrangian or multi-task COptiDICE with shared critics) so that the contribution of diffusion + contextual prompts can be isolated from the contribution of “being the first method purpose-built for the problem.”
minor comments (5)
  1. Notation inconsistency: the cost threshold is written both ¯c_i^τ and ¯C(τ_c); the correction factors η_R, η_C appear only in the practical-algorithm section and are never listed among the free parameters of the method.
  2. Fig. 5 and the ablation curves (Fig. 6) report only mean ± std over three seeds; individual-task learning curves or a table of final per-task costs under the zero-threshold setting would make the safety claim easier to inspect.
  3. The proof of Prop. IV.2 (Appendix) invokes conditional independence of y′, R and C given x_k; a short remark on how strongly this is violated by the MetaDrive data would help readers gauge the approximation quality.
  4. Several acronyms (MTOS, CMTOS, CDCP GradSync, etc.) are introduced without a glossary; a compact notation table would improve readability.
  5. Typos: “contained optimization” (p. 12), “unclassified guidance” (p. 11), “sightly exceeds” (p. 11), and occasional missing articles.

Circularity Check

0 steps flagged

No circularity: standard supervised diffusion reformulation of multi-task offline safe RL with empirical roll-out evaluation; flexible cost conditioning is by design of CFG, not a forced prediction.

full rationale

The paper reformulates the multi-task constrained objective (Defs. 1–2) as conditional generation (Lemma IV.1) and trains a diffusion model by ordinary supervised denoising loss (Eq. 10) on offline trajectories. Classifier-free guidance for reward/cost (Prop. IV.2 / Eq. 11) follows the standard multi-condition derivation under an explicit independence assumption whose proof is self-contained in the appendix; the resulting test-time flexibility (Remarks IV.3–IV.4) is a direct consequence of feeding remaining discounted cost as a free condition after per-task min-max normalization, not a mathematical identity that equates outputs to fitted inputs. Reported rewards/costs (Tables I–III) are measured on held-out environment roll-outs, not reconstructed from the training objective. Free parameters (κ, p_β, η_R, η_C) are ablated or fixed once. Self-citations to the authors’ prior single-task work (VOCE, POCE, etc.) appear only in related-work discussion and do not underwrite any uniqueness claim or load-bearing step of the multi-task derivation. No self-definitional loop, no fitted quantity re-labeled as prediction, and no ansatz smuggled via citation. The derivation chain is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 2 invented entities

The central empirical claim rests on standard CMDP and diffusion-model machinery plus a handful of tunable scalars that control guidance strength and normalization. No new physical entities are postulated; the invented pieces are algorithmic constructs whose only evidence is the MetaDrive tables.

free parameters (5)
  • classifier-free guidance scale κ
    Controls the strength of reward and cost steering in Eq. 11; chosen by hand / ablation.
  • dropout probability p_β
    Bernoulli probability of dropping reward/cost conditions during training (Eq. 10); swept in Fig. 7.
  • reward/cost correction factors η_R, η_C
    Post-hoc scaling applied to normalized conditions at test time (Eqs. 20–21) to compensate for discounting and max-cost mismatch.
  • gradient-loss decay exponent η
    Hyper-parameter of the GradNorm-style weight update (Eq. 19); set to 0.12.
  • low-temperature sampling scale α
    Controls variance of the reverse process; fixed once for all experiments.
axioms (4)
  • domain assumption Standard CMDP formulation of safe RL (Eq. 1) and multi-task average-reward objective (Eq. 2).
    Taken as given from the safe-RL and multi-task RL literature; used to define Definitions 1–2.
  • standard math Diffusion reverse process can be trained by the simplified noise-prediction loss (Eq. 5) and sampled with DDPM.
    Standard result from Ho et al. / Song et al.; invoked without re-derivation.
  • ad hoc to paper Reward and cost conditions are conditionally independent given the noisy action sequence (used in Prop. IV.2).
    Explicit modeling assumption needed to factor the joint guidance term; not independently verified.
  • ad hoc to paper Min-max normalization of remaining discounted return/cost maps every task onto a common [0,1] scale that preserves constraint semantics.
    Eqs. 20–21; required for the flexible test-time cost claim.
invented entities (2)
  • Contextual prompt M(τ) = [trajectory snippet, language embedding] no independent evidence
    purpose: Disambiguate multi-task observations and enable few-shot transfer.
    Constructed for this paper; evidence is only the few-shot table.
  • Gradient-loss synchronization weight layer W no independent evidence
    purpose: Balance per-task losses by both magnitude and decay rate.
    Extension of GradNorm; no external validation beyond the ablation curve.

pith-pipeline@v1.1.0-grok45 · 30422 in / 3096 out tokens · 26313 ms · 2026-07-11T23:07:35.186561+00:00 · methodology

0 comments
read the original abstract

Multi-task offline safe reinforcement learning (RL) promises to learn a shared optimal safe policy from offline data across multiple tasks. This paradigm provides an effective means for the widespread application of RL in multi-task scenarios with high risk and interaction costs. However, the triple challenges of multi-tasking, safety constraints, and out-of-distribution (OOD) actions pose a significant hurdle for existing methods to ensure safety while maximizing reward returns. In this work, we propose a Conditional Diffusion model with Contextual Prompts (CDCP) to address these challenges. Concretely, we first rethink the requirements and challenges in current multi-task decision-making and control scenarios and establish the objectives of multi-task offline safe RL. Subsequently, we transform the multi-task constrained optimization problem into a conditional generation problem using the diffusion model. Based on this, we design a classifier-free guided cost-constraint strategy to provide flexible cost constraints and eliminate extrapolation errors from OOD actions via supervised learning. Additionally, we introduce a novel contextual prompting method to enhance multi-task representation accuracy and adaptability to unseen tasks. A gradient loss synchronization strategy is also introduced to eliminate gradient interference, improving training stability. Finally, extensive experiments demonstrate that the CDCP algorithm exhibits higher performance and safety in multi-task scenarios than the current state-of-the-art baseline methods. It meets different cost constraints without further training, providing a more flexible cost-constraint solution for the multi-task safe RL.

Figures

Figures reproduced from arXiv: 2607.03903 by Alois Knoll, Ao Zhou, Changjun Jiang, Guai Chen, Jiayi Guan, Li Shen, Lusong Li, Mengjie Li, Ruiqi Zhang, Tianle Zhang, Xiaodong He.

Figure 2
Figure 2. Figure 2: The average reward and cost of the algorithms that [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 1
Figure 1. Figure 1: The mean reward and cost for the multi-task offline [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: The radar charts depicting the characteristics of sample data from nine different driving tasks with varying traffic [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The characteristic distribution of sample action-state pairs for nine driving tasks with varying traffic scenarios or densities. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The rewards and costs of the CDCP and the cur [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The reward and cost curves for the ablation experiments [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The mean and variance of rewards and costs for [PITH_FULL_IMAGE:figures/full_fig_p011_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Visualization of driving scenarios with different traffic densities in the [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Visualization of driving scenarios with different traffic densities in the [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Visualization of driving scenarios with different traffic densities in the [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

73 extracted references · 6 linked inside Pith

  1. [1]

    Multi-task reinforcement learning in humans,

    M. S. Tomov, E. Schulz, and S. J. Gershman, “Multi-task reinforcement learning in humans,”Nature Human Behaviour, vol. 5, no. 6, pp. 764– 773, 2021

    2021

  2. [2]

    Curriculum-based asymmetric multi-task reinforcement learning,

    H. Huang, D. Ye, L. Shen, and W. Liu, “Curriculum-based asymmetric multi-task reinforcement learning,”IEEE transactions on pattern anal- ysis and machine intelligence, vol. 45, no. 6, pp. 7258–7269, 2022

    2022

  3. [3]

    Multi-task reinforcement learning with soft modularization,

    R. Yang, H. Xu, Y . WU, and X. Wang, “Multi-task reinforcement learning with soft modularization,” inAdvances in Neural Information Processing Systems, vol. 33, 2020, pp. 4767–4777

    2020

  4. [4]

    Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning,

    T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine, “Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning,” inConference on robot learning. PMLR, 2020, pp. 1094–1100

    2020

  5. [5]

    Lifelong robotic reinforcement learning by retain- ing experiences,

    A. Xie and C. Finn, “Lifelong robotic reinforcement learning by retain- ing experiences,” inConference on Lifelong Learning Agents. PMLR, 2022, pp. 838–855

    2022

  6. [6]

    Multi-task batch reinforcement learning with metric learning,

    J. Li, Q. Vuong, S. Liu, M. Liu, K. Ciosek, H. Christensen, and H. Su, “Multi-task batch reinforcement learning with metric learning,” Advances in Neural Information Processing Systems, vol. 33, pp. 6197– 6210, 2020

    2020

  7. [7]

    A discrete soft actor-critic decision-making strategy with sample filter for freeway autonomous driving,

    J. Guan, G. Chen, J. Huang, Z. Li, L. Xiong, J. Hou, and A. Knoll, “A discrete soft actor-critic decision-making strategy with sample filter for freeway autonomous driving,”IEEE Transactions on Vehicular Technology, vol. 72, no. 2, pp. 2593–2598, 2022

    2022

  8. [8]

    Longitudinal speed control of autonomous vehicle based on a self-adaptive pid of radial basis function neural network,

    L. Nie, J. Guan, C. Lu, H. Zheng, and Z. Yin, “Longitudinal speed control of autonomous vehicle based on a self-adaptive pid of radial basis function neural network,”IET Intelligent Transport Systems, vol. 12, no. 6, pp. 485–494, 2018

    2018

  9. [9]

    Skills regularized task decomposition for multi-task offline reinforcement learning,

    M. Yoo, S. Cho, and H. Woo, “Skills regularized task decomposition for multi-task offline reinforcement learning,” in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 37 432–37 444. [Online]. Available: https://proceedings.neurips.cc/pape...

    2022

  10. [10]

    Gnfactor: Multi-task real robot learning with generalizable neural feature fields,

    Y . Ze, G. Yan, Y .-H. Wu, A. Macaluso, Y . Ge, J. Ye, N. Hansen, L. E. Li, and X. Wang, “Gnfactor: Multi-task real robot learning with generalizable neural feature fields,” inConference on Robot Learning. PMLR, 2023, pp. 284–301

    2023

  11. [11]

    Multi-task learning with attention for end-to-end autonomous driving,

    K. Ishihara, A. Kanervisto, J. Miura, and V . Hautamaki, “Multi-task learning with attention for end-to-end autonomous driving,” inPro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 2902–2911

    2021

  12. [12]

    Aoi-aware resource allocation for platoon-based c-v2x networks via multi-agent multi-task reinforcement learning,

    M. Parvini, M. R. Javan, N. Mokari, B. Abbasi, and E. A. Jorswieck, “Aoi-aware resource allocation for platoon-based c-v2x networks via multi-agent multi-task reinforcement learning,”IEEE Transactions on Vehicular Technology, vol. 72, no. 8, pp. 9880–9896, 2023

    2023

  13. [13]

    A multi-task-learning- based transfer deep reinforcement learning design for autonomic optical networks,

    X. Chen, R. Proietti, C.-Y . Liu, and S. B. Yoo, “A multi-task-learning- based transfer deep reinforcement learning design for autonomic optical networks,”IEEE Journal on Selected Areas in Communications, vol. 39, no. 9, pp. 2878–2889, 2021

    2021

  14. [14]

    Multi-task deep reinforcement learning with popart,

    M. Hessel, H. Soyer, L. Espeholt, W. Czarnecki, S. Schmitt, and H. Van Hasselt, “Multi-task deep reinforcement learning with popart,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, 2019, pp. 3796–3803

    2019

  15. [15]

    A survey of multi-task deep reinforcement learning,

    N. Vithayathil Varghese and Q. H. Mahmoud, “A survey of multi-task deep reinforcement learning,”Electronics, vol. 9, no. 9, p. 1363, 2020

    2020

  16. [16]

    Provably efficient multi-task reinforcement learning with model transfer,

    C. Zhang and Z. Wang, “Provably efficient multi-task reinforcement learning with model transfer,”Advances in Neural Information Process- ing Systems, vol. 34, pp. 19 771–19 783, 2021

    2021

  17. [17]

    Toward trustworthy decision-making for autonomous vehicles: A robust reinforcement learning approach with safety guarantees,

    X. He, W. Huang, and C. Lv, “Toward trustworthy decision-making for autonomous vehicles: A robust reinforcement learning approach with safety guarantees,”Engineering, vol. 33, pp. 77–89, 2024

    2024

  18. [18]

    Residual policy learning facilitates efficient model-free autonomous racing,

    R. Zhang, J. Hou, G. Chen, Z. Li, J. Chen, and A. Knoll, “Residual policy learning facilitates efficient model-free autonomous racing,”IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 11 625–11 632, 2022

    2022

  19. [19]

    Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning,

    Q. Li, Z. Peng, L. Feng, Q. Zhang, Z. Xue, and B. Zhou, “Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning,”IEEE transactions on pattern analysis and machine intelli- gence, vol. 45, no. 3, pp. 3461–3475, 2022

    2022

  20. [20]

    Safe reinforce- ment learning for arm manipulation with constrained markov decision process,

    P. Adjei, N. Tasfi, S. Gomez-Rosero, and M. A. Capretz, “Safe reinforce- ment learning for arm manipulation with constrained markov decision process,”Robotics, vol. 13, no. 4, p. 63, 2024

    2024

  21. [21]

    Safe exploration in reinforcement learning: A generalized formulation and algorithms,

    A. Wachi, W. Hashimoto, X. Shen, and K. Hashimoto, “Safe exploration in reinforcement learning: A generalized formulation and algorithms,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  22. [22]

    Constraints penalized q-learning for safe offline reinforcement learning,

    H. Xu, X. Zhan, and X. Zhu, “Constraints penalized q-learning for safe offline reinforcement learning,”Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 8, pp. 8753–8760, 2022

    2022

  23. [23]

    Poce: Primal policy optimization with conservative estimation for multi-constraint offline reinforcement learning,

    J. Guan, L. Shen, A. Zhou, L. Li, H. Hu, X. He, G. Chen, and C. Jiang, “Poce: Primal policy optimization with conservative estimation for multi-constraint offline reinforcement learning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26 243–26 253

    2024

  24. [24]

    Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation,

    J. Lee, C. Paduraru, D. J. Mankowitz, N. Heess, D. Precup, K.- E. Kim, and A. Guez, “Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation,”arXiv preprint arXiv:2204.08957, 2022

    Pith/arXiv arXiv 2022

  25. [25]

    Constrained offline policy optimization,

    N. Polosky, B. C. Da Silva, M. Fiterau, and J. Jagannath, “Constrained offline policy optimization,” inInternational Conference on Machine Learning. PMLR, 2022, pp. 17 801–17 810

    2022

  26. [26]

    VOCE: Variational optimization with conservative estimation for offline safe reinforcement learning,

    J. Guan, G. Chen, J. Ji, L. Yang, A. Zhou, Z. Li, and changjun jiang, “VOCE: Variational optimization with conservative estimation for offline safe reinforcement learning,” inThirty-seventh Conference on Neural Information Processing Systems, 2023. [Online]. Available: https://openreview.net/forum?id=sIU3WujeSl

    2023

  27. [27]

    Sharing knowledge in multi-task deep reinforcement learning,

    C. D’Eramo, D. Tateo, A. Bonarini, M. Restelli, and J. Peters, “Sharing knowledge in multi-task deep reinforcement learning,” inInternational Conference on Learning Representations, 2019

    2019

  28. [28]

    Optimization of deep rein- forcement learning with hybrid multi-task learning,

    N. V . Varghese and Q. H. Mahmoud, “Optimization of deep rein- forcement learning with hybrid multi-task learning,” in2021 IEEE International Systems Conference (SysCon). IEEE, 2021, pp. 1–8

    2021

  29. [29]

    Conservative data sharing for multi-task offline reinforcement learn- ing,

    T. Yu, A. Kumar, Y . Chebotar, K. Hausman, S. Levine, and C. Finn, “Conservative data sharing for multi-task offline reinforcement learn- ing,”Advances in Neural Information Processing Systems, vol. 34, pp. 11 501–11 516, 2021

    2021

  30. [30]

    Multi-task reinforcement learning with context-based representations,

    S. Sodhani, A. Zhang, and J. Pineau, “Multi-task reinforcement learning with context-based representations,” inInternational Conference on Machine Learning. PMLR, 2021, pp. 9767–9779

    2021

  31. [31]

    Just pick a sign: Optimizing deep multitask models 13 with gradient sign dropout,

    Z. Chen, J. Ngiam, Y . Huang, T. Luong, H. Kretzschmar, Y . Chai, and D. Anguelov, “Just pick a sign: Optimizing deep multitask models 13 with gradient sign dropout,”Advances in Neural Information Processing Systems, vol. 33, pp. 2039–2050, 2020

    2039

  32. [32]

    Conflict-averse gradi- ent descent for multi-task learning,

    B. Liu, X. Liu, X. Jin, P. Stone, and Q. Liu, “Conflict-averse gradi- ent descent for multi-task learning,”Advances in Neural Information Processing Systems, vol. 34, pp. 18 878–18 890, 2021

    2021

  33. [33]

    Multi-task reinforcement learning with soft modularization,

    R. Yang, H. Xu, Y . Wu, and X. Wang, “Multi-task reinforcement learning with soft modularization,”Advances in Neural Information Processing Systems, vol. 33, pp. 4767–4777, 2020

    2020

  34. [34]

    Paco: Parameter- compositional multi-task reinforcement learning,

    L. Sun, H. Zhang, W. Xu, and M. Tomizuka, “Paco: Parameter- compositional multi-task reinforcement learning,”Advances in Neural Information Processing Systems, vol. 35, pp. 21 495–21 507, 2022

    2022

  35. [35]

    Recomposing the reinforcement learning building blocks with hypernetworks,

    E. Sarafian, S. Keynan, and S. Kraus, “Recomposing the reinforcement learning building blocks with hypernetworks,” inInternational Confer- ence on Machine Learning. PMLR, 2021, pp. 9301–9312

    2021

  36. [36]

    Batch policy learning under con- straints,

    H. Le, C. V oloshin, and Y . Yue, “Batch policy learning under con- straints,” inInternational Conference on Machine Learning. PMLR, 2019, pp. 3703–3712

    2019

  37. [37]

    Gendice: Generalized offline estimation of stationary values,

    R. Zhang, B. Dai, L. Li, and D. Schuurmans, “Gendice: Generalized offline estimation of stationary values,” inInternational Conference on Learning Representations, 2019

    2019

  38. [38]

    Coindice: Off-policy confidence interval estimation,

    B. Dai, O. Nachum, Y . Chow, L. Li, C. Szepesv ´ari, and D. Schuur- mans, “Coindice: Off-policy confidence interval estimation,”Advances in neural information processing systems, vol. 33, pp. 9398–9411, 2020

    2020

  39. [39]

    Optidice: Offline policy optimization via stationary distribution correction estimation,

    J. Lee, W. Jeon, B. Lee, J. Pineau, and K.-E. Kim, “Optidice: Offline policy optimization via stationary distribution correction estimation,” in International Conference on Machine Learning. PMLR, 2021, pp. 6120–6130

    2021

  40. [40]

    Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation,

    J. Lee, C. Paduraru, D. J. Mankowitz, N. Heess, D. Precup, K.-E. Kim, and A. Guez, “Coptidice: Offline constrained reinforcement learning via stationary distribution correction estimation,” inInternational Confer- ence on Learning Representations, 2022

    2022

  41. [41]

    Con- strained decision transformer for offline safe reinforcement learning,

    Z. Liu, Z. Guo, Y . Yao, Z. Cen, W. Yu, T. Zhang, and D. Zhao, “Con- strained decision transformer for offline safe reinforcement learning,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 21 611–21 630

    2023

  42. [42]

    Saformer: A conditional sequence model- ing approach to offline safe reinforcement learning,

    Q. Zhang, L. Zhang, H. Xu, L. Shen, B. Wang, Y . Chang, X. Wang, B. Yuan, and D. Tao, “Saformer: A conditional sequence model- ing approach to offline safe reinforcement learning,”arXiv preprint arXiv:2301.12203, 2023

    Pith/arXiv arXiv 2023

  43. [43]

    Scaling up multi-task robotic reinforcement learning,

    D. Kalashnikov, J. Varley, Y . Chebotar, B. Swanson, R. Jonschkowski, C. Finn, S. Levine, and K. Hausman, “Scaling up multi-task robotic reinforcement learning,” in5th Annual Conference on Robot Learning, 2021

    2021

  44. [44]

    Perceiver-actor: A multi-task transformer for robotic manipulation,

    M. Shridhar, L. Manuelli, and D. Fox, “Perceiver-actor: A multi-task transformer for robotic manipulation,” inConference on Robot Learning. PMLR, 2023, pp. 785–799

    2023

  45. [45]

    Effective adapta- tion in multi-task co-training for unified autonomous driving,

    X. Liang, Y . Wu, J. Han, H. Xu, C. Xu, and X. Liang, “Effective adapta- tion in multi-task co-training for unified autonomous driving,”Advances in Neural Information Processing Systems, vol. 35, pp. 19 645–19 658, 2022

    2022

  46. [46]

    Learning shared rep- resentations in multi-task reinforcement learning,

    D. Borsa, T. Graepel, and J. Shawe-Taylor, “Learning shared rep- resentations in multi-task reinforcement learning,”arXiv preprint arXiv:1603.02041, 2016

    Pith/arXiv arXiv 2016

  47. [47]

    Multi-task reinforcement learning with attention-based mixture of experts,

    G. Cheng, L. Dong, W. Cai, and C. Sun, “Multi-task reinforcement learning with attention-based mixture of experts,”IEEE Robotics and Automation Letters, 2023

    2023

  48. [48]

    Shar- ing knowledge in multi-task deep reinforcement learning,

    C. DEramo, D. Tateo, A. Bonarini, M. Restelli, and J. Peters, “Shar- ing knowledge in multi-task deep reinforcement learning,” inEighth International Conference on Learning Representations (ICLR 2020). OpenReview. net, 2020

    2020

  49. [49]

    Impala: Scalable dis- tributed deep-rl with importance weighted actor-learner architectures,

    L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V . Mnih, T. Ward, Y . Doron, V . Firoiu, T. Harley, I. Dunninget al., “Impala: Scalable dis- tributed deep-rl with importance weighted actor-learner architectures,” inInternational conference on machine learning. PMLR, 2018, pp. 1407–1416

    2018

  50. [50]

    Multi-goal reinforcement learning: Challenging robotics environments and request for research,

    M. Plappert, M. Andrychowicz, A. Ray, B. McGrew, B. Baker, G. Pow- ell, J. Schneider, J. Tobin, M. Chociej, P. Welinderet al., “Multi-goal reinforcement learning: Challenging robotics environments and request for research,”arXiv preprint arXiv:1802.09464, 2018

    Pith/arXiv arXiv 2018

  51. [51]

    Asynchronous methods for deep rein- forcement learning,

    V . Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep rein- forcement learning,” inInternational conference on machine learning. PMLR, 2016, pp. 1928–1937

    2016

  52. [52]

    Deterministic policy gradient algorithms,

    D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller, “Deterministic policy gradient algorithms,” inInternational conference on machine learning. Pmlr, 2014, pp. 387–395

    2014

  53. [53]

    Sharing knowledge in multi-task deep reinforcement learning,

    C. D’Eramo, D. Tateo, A. Bonarini, M. Restelli, J. Peterset al., “Sharing knowledge in multi-task deep reinforcement learning,” in8th International Conference on Learning Representations,{ICLR}2020, Addis Ababa, Ethiopia, April 26-30, 2020. International Conference on Learning Representations, ICLR, 2020, pp. 1–11

    2020

  54. [54]

    Multi-game decision transformers,

    K.-H. Lee, O. Nachum, M. S. Yang, L. Lee, D. Freeman, S. Guadarrama, I. Fischer, W. Xu, E. Jang, H. Michalewskiet al., “Multi-game decision transformers,”Advances in Neural Information Processing Systems, vol. 35, pp. 27 921–27 936, 2022

    2022

  55. [55]

    Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning,

    H. He, C. Bai, K. Xu, Z. Yang, W. Zhang, D. Wang, B. Zhao, and X. Li, “Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning,” inThirty-seventh Conference on Neural Information Processing Systems, 2023. [Online]. Available: https://openreview.net/forum?id=fAdMly4ki5

    2023

  56. [56]

    Altman,Constrained Markov decision processes: stochastic modeling

    E. Altman,Constrained Markov decision processes: stochastic modeling. Routledge, 1999

    1999

  57. [57]

    Conservative q-learning for offline reinforcement learning,

    A. Kumar, A. Zhou, G. Tucker, and S. Levine, “Conservative q-learning for offline reinforcement learning,”Advances in Neural Information Processing Systems, vol. 33, pp. 1179–1191, 2020

    2020

  58. [58]

    Uac: Offline reinforcement learning with uncertain action constraint,

    J. Guan, S. Gu, Z. Li, J. Hou, Y . Yang, G. Chen, and C. Jiang, “Uac: Offline reinforcement learning with uncertain action constraint,”IEEE Transactions on Cognitive and Developmental Systems, vol. 16, no. 2, pp. 671–680, 2023

    2023

  59. [59]

    Generative modeling by estimating gradients of the data distribution,

    Y . Song and S. Ermon, “Generative modeling by estimating gradients of the data distribution,”Advances in neural information processing systems, vol. 32, 2019

    2019

  60. [60]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in neural information processing systems, vol. 33, pp. 6840– 6851, 2020

    2020

  61. [61]

    Variational inference: A review for statisticians,

    D. M. Blei, A. Kucukelbir, and J. D. McAuliffe, “Variational inference: A review for statisticians,”Journal of the American statistical Associa- tion, vol. 112, no. 518, pp. 859–877, 2017

    2017

  62. [62]

    Efficient diffusion policies for offline reinforcement learning,

    B. Kang, X. Ma, C. Du, T. Pang, and S. Yan, “Efficient diffusion policies for offline reinforcement learning,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  63. [63]

    Is conditional generative modeling all you need for decision-making?

    A. Ajay, Y . Du, A. Gupta, J. B. Tenenbaum, T. S. Jaakkola, and P. Agrawal, “Is conditional generative modeling all you need for decision-making?” inNeurIPS 2022 Foundation Models for Decision Making Workshop, 2022

    2022

  64. [64]

    Efficient learning of safe driving policy via human-ai copilot optimization,

    Q. Li, Z. Peng, and B. Zhou, “Efficient learning of safe driving policy via human-ai copilot optimization,” inInternational Conference on Learning Representations, 2021

    2021

  65. [65]

    Safe driving via expert guided policy optimization,

    Z. Peng, Q. Li, C. Liu, and B. Zhou, “Safe driving via expert guided policy optimization,” inConference on Robot Learning. PMLR, 2022, pp. 1554–1563

    2022

  66. [66]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” inAdvances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 6840–6851. [Online]. Available: https://proceedings.neurips.cc/paper files/paper/ 2020/file/4c5bcfec8584af0d967f1ab10...

    2020

  67. [67]

    Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks,

    Z. Chen, V . Badrinarayanan, C.-Y . Lee, and A. Rabinovich, “Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks,” inInternational conference on machine learning. PMLR, 2018, pp. 794–803

    2018

  68. [68]

    Datasets and benchmarks for offline safe reinforcement learning,

    Z. Liu, Z. Guo, H. Lin, Y . Yao, J. Zhu, Z. Cen, H. Hu, W. Yu, T. Zhang, J. Tanet al., “Datasets and benchmarks for offline safe reinforcement learning,”arXiv preprint arXiv:2306.09303, 2023

    Pith/arXiv arXiv 2023

  69. [69]

    Learning to simulate self-driven particles system with coordinated policy optimization,

    Z. Peng, Q. Li, K. M. Hui, C. Liu, and B. Zhou, “Learning to simulate self-driven particles system with coordinated policy optimization,”Ad- vances in Neural Information Processing Systems, vol. 34, pp. 10 784– 10 797, 2021

    2021

  70. [70]

    Understanding diffusion models: A unified perspective,

    C. Luo, “Understanding diffusion models: A unified perspective,”arXiv preprint arXiv:2208.11970, 2022

    Pith/arXiv arXiv 2022

  71. [71]

    Denoising diffusion implicit models,

    J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” inInternational Conference on Learning Representations, 2022. 14 VII. SUPPLEMENTARYMATERIAL A. Proof and Discussion the Proposition IV .2 Proposition VII.1.The perturbation noiseˆϵin the interme- diate steps of the denoising process, which takes the state sequenceO(τ c)and task context...

    2022

  72. [72]

    Table 2 outlines the primary network parameters of the model implemented by our CDCP algorithm

    Experimental Setting:The experiment is conducted on a server equipped with L40s, utilizing PyTorch for model construction, training, and testing. Table 2 outlines the primary network parameters of the model implemented by our CDCP algorithm. Furthermore, the code appendix contains detailed configuration information for the testing environment. For instruc...

  73. [73]

    These scenarios include environments such as curves, intersections, T-junctions, and roundabouts

    Tasks for Experimental:Our experiments select nine safety-constrained autonomous driving scenarios encompass- ing diverse driving situations and traffic densities. These scenarios include environments such as curves, intersections, T-junctions, and roundabouts. Regarding traffic density, the tasks cover sparse, medium, and dense traffic flow. Fig. 8 prese...