Pith. sign in

REVIEW 5 major objections 5 minor 63 references

DIAL: Distribution-Informed Adaptive Learning of Multi-Task Constraints for Safety-Critical Systems

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DIAL learns a distribution of safety constraints from multi-task demonstrations and adapts to new tasks by tuning a single risk level.

desk verdict DIAL is a useful, flawed combination paper: the distributional constraint-learning idea is worth engaging, but the algorithm write-up and the λ selection need work before the transfer claim is solid. read the letter →

arxiv 2501.18086 v1 pith:NXJX3DTN submitted 2025-01-30 cs.LG cs.AIcs.ROcs.SYeess.SY

Se-Wook Yoo , Seung-Woo Seo This is my paper · ORCID
classification cs.LGcs.AIcs.ROcs.SYeess.SY
keywords inverseconstrainedreinforcementlearningimitationsafetyconstraintsconditionalvalueatriskmulti-tasktask-agnosticsafeexplorationtransferBetadistribution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes DIAL, a two-stage inverse constrained reinforcement learning method that learns a shared distribution of safety constraints from multi-task demonstrations instead of a single constraint function. In the first stage, a neural network outputs the parameters of a Beta posterior over trajectory feasibility, and the constraint is evaluated through the conditional value at risk (CVaR) at a sampled risk level. In the second stage, the learned risk-sensitive constraint and a task-agnostic safe exploration policy transfer to new tasks by grid-searching the risk level $\lambda$. The paper claims that task-specific constraint definitions are unnecessary and that DIAL attains better safety and success than MaxEnt IRL, ICRL, and shared-constraint baselines in driving and robot-control benchmarks.

What carries the argument

The load-bearing object is the distribution-aware constraint model $f_\phi(\alpha|\tau)$, a neural network that maps a trajectory to the parameters $\alpha=[\alpha_1,\alpha_2]$ of a Beta posterior $q(\zeta(\tau)|\alpha)$ over the probability that the trajectory is feasible. The constraint is evaluated as the conditional value at risk $\Gamma^\lambda_\phi(\tau)=\mathbb{E}_{\alpha\sim f_\phi(\cdot|\tau)}[\mathrm{CVaR}_\lambda]$ at a risk level $\lambda$; the policy side uses an importance-weighted $k$-NN state-entropy estimator $\hat{H}_k(\rho_\theta|\rho_{\bar\theta})$ to encourage task-agnostic safe exploration while staying inside the learned risk bounds. Together they turn the entropy-regularized constrained RL objective in Eq. 1 into the Lagrangian min-max objective in Eq. 11.

What would settle it

Construct a multi-task environment whose feasible region is a mixture of two distinct safe zones (two separated speed-limit regions), train DIAL on demonstrations from both, then inspect the inferred feasibility map and measure per-zone transfer cost-return; the unimodal Beta posterior must collapse onto one zone, and the transferred policy should systematically violate the budget in the other zone.

Watch

Extended reading notes

Core claim

The paper claims that constraints shared across tasks can be learned as a distribution rather than a single function, and that this distribution can be distorted by a CVaR criterion so that adapting to a new task reduces to selecting the risk level $\lambda$. In the safe imitation stage, DIAL uses multi-task expert demonstrations to train $f_\phi(\alpha|\tau)$ and a task-agnostic safe exploration policy; in the safe transfer stage, it fixes the learned constraint, grid-searches $\lambda$, and optimizes the policy against the recovered risk-sensitive constraint with no task-specific cost definition. Experimental results on urban driving and four robot-control environments are presented as evidence that DIAL achieves lower constraint violation rates and comparable or higher rewards than MERL, MECL, and COCL, and that it retains safety behavior when the reward changes.

Load-bearing premise

The whole transfer story depends on the safety requirements across tasks being well captured by a factorized Beta posterior over trajectory feasibility with a CVaR risk level found by grid search; if the real constraints are multimodal or heavy-tailed in a way Beta cannot express, the inferred constraint and the safety transfer claim do not follow.

Editorial extensions

If this is right

  • Agents trained with DIAL can switch to a new task by changing the risk level $\lambda$ alone, without redefining cost functions.
  • Safety constraints learned from multi-task demonstrations transfer to environments with shifted safety conditions while keeping violation rates near the ground-truth budget.
  • Task-agnostic safe exploration lets the policy maintain high state entropy early in transfer, so it finds feasible solutions faster than conservative single-task constraints.
  • Because the constraint is a distribution, DIAL can represent soft constraints such as speed limits and safe distances, and can tune conservatism per task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not pursue it, but the CVaR risk level $\lambda$ could be turned into an online controller that raises $\lambda$ when violations rise and lowers it when the policy is overly conservative.
  • If the Beta assumption holds, the learned constraint becomes a parameterized family of safety thresholds, which would let an operator treat $\lambda$ as a human-interpretable safety dial for autonomous systems.
  • The transfer claim could be tested adversarially by training demonstrations from a bimodal feasible region; a unimodal Beta posterior should miss one mode, and the transferred policy should violate the budget there, a test the paper does not include.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes DIAL, a two-stage method for safe reinforcement learning and imitation learning. In the safe imitation-learning stage, DIAL learns a distribution over trajectory feasibility, modeled as a factorized Beta posterior parameterized by a neural network, and distorts it with conditional value at risk (CVaR) at risk level λ; it also trains a task-agnostic safe-exploration policy using a k-NN state-entropy estimator with a trust-region constraint. In the safe transfer-learning stage, the learned constraint is fixed and used, together with a scalar risk level λ, to constrain a policy that maximizes a newly provided reward. Experiments on urban driving (HighwayEnv) and four robot-control environments compare DIAL with MERL, MECL, and COCL, reporting higher reward-return and lower cost or violation rates. The central claim is that DIAL identifies shared constraints from multi-task demonstrations and adapts to new tasks by adjusting a scalar risk level, without requiring task-specific constraint definitions.

Significance. If the central claim is established, DIAL would be a practically useful contribution: it offers a way to reuse safety information across tasks through a distributional, risk-sensitive constraint model, and it explicitly targets the burden of hand-designing cost functions. The paper is commendable for providing detailed pseudocode for both stages (Algorithms 1 and 2), per-environment hyperparameters in Appendix B, and extensive experiments across five environments. However, the manuscript currently has load-bearing gaps: the policy update in Algorithm 1 does not match the objective in Eq. (11), the constraint update in Eq. (7) is not derived from a well-defined loss, the risk level λ is selected per environment by grid search rather than adapted, and the Beta/CVaR posterior is not directly validated. These issues make the published claims ahead of the evidence, so the work is promising but not yet fully supported.

major comments (5)
  1. [IV-B, Algorithm 1, Eq. (11)] The policy update described in Algorithm 1 does not implement the objective in Eq. (11). Eq. (11) defines the safe-IL objective as min_{κ≥0} max_θ [J_R(θ) + β J_H(θ) − κ J_κ(φ,λ)], where J_R is the expected reward-return and J_κ is the constraint term. Algorithm 1, however, collects only (s,a,s′) transitions and the only θ update is the entropy-gradient step θ ← θ − β ∇_θ J_H(θ) inside the trust-region loop (lines 17–18); no gradient of J_R or J_κ appears. This makes the described safe-IL procedure not directly executable and leaves unclear what objective the reported policy actually optimizes.
  2. [IV-A, Eqs. (4), (7), (8)] The constraint update in Eq. (7) is presented without a derivation. Eq. (4) is the gradient of the maximum-likelihood objective (3) for the feasibility classifier ζ; substituting Γ^λ_φ(τ) = E_{α∼f_φ}[CVaR_λ] for ζ(τ) changes the statistical model, yet no loss function is given whose gradient equals Eq. (7), and the ELBO term L_P in Eq. (8) is added without specifying the joint objective being optimized. A fixed point of Eq. (7) is therefore not shown to recover the true constraint distribution. This is load-bearing for the claim that DIAL 'identifies the shared constraints.'
  3. [IV-A, Table VI, Fig. 9] The scalar risk level λ is not adapted automatically; Table VI fixes it per environment (0.5 for four environments and 0.1 for PointGoal), and the text states that fine-tuning is performed using grid search. Fig. 9 shows that the inferred safe set and the policy's exploratory behavior change qualitatively with λ. Because the paper does not report a selection rule for λ that avoids access to task-specific safety information, the abstract's claim of adaptation 'without requiring task-specific constraint definitions' is not yet substantiated: the experiments as reported do not rule out that the grid-searched λ, rather than the learned distribution, provides the task-specific safety tuning.
  4. [IV-A, Section V-E, Fig. 11] The representational assumption that trajectory feasibility follows a factorized Beta posterior is not validated. The paper assumes q(ζ(τ)|α) = ∏_i q(ζ_i(τ)|α_i) and distorts this posterior with CVaR, but no experiment checks whether the learned posterior is calibrated, whether the Beta family is adequate, or whether the factorized form captures the true multi-constraint dependence. The reported CR/CV metrics can be satisfied by a conservative policy even if the posterior is wrong; Fig. 11 displays histograms and CDFs of environmental data but not posterior fit or calibration. A direct validation—for example, calibration curves, coverage tests, or a comparison against a nonparametric feasibility estimator—is needed to support the 'distribution-informed' claim.
  5. [Algorithm 2, lines 13–15] The safe-TL policy update is inconsistent with the Lagrangian form of Eq. (1). Algorithm 2 updates κ with the constraint violation E_π[ĉ] − ϵ, but the θ update (line 15) contains only the reward and entropy gradients; the term −κ ∇_θ E_π[ĉ] that would enforce the constraint in a Lagrangian update is absent. If this is a typo it should be corrected; otherwise the algorithm does not actually enforce the recovered constraint during transfer.
minor comments (5)
  1. [Section III-B, Eq. (4)] The notation ∇_ζ log ζ(τ) is ambiguous: the gradient is with respect to the network parameters of ζ, not with respect to the function ζ itself. Please clarify the parameterization.
  2. [Section IV-A] The sentence 'We interpret the network’s output as variables sampled from two Gamma distributions' is unclear and appears to conflict with the statement that f_φ outputs the Beta parameters α directly. Please clarify the generative model for α.
  3. [Table I] Table I reports SE and CR without standard deviations even though the text says all metrics are averaged over 20 episodes for 5 seeds. Adding error bars or standard deviations would make the comparisons more informative.
  4. [Algorithm 2, line 15] The entropy term H(π_θ(τ)) in line 15 is written as the ordinary policy entropy, whereas Section IV-B motivates the importance-weighted k-NN estimator of Eq. (10). Please specify which entropy estimator is used in the transfer stage.
  5. [Section IV-B, Eq. (11)] The role of the trust-region constraint D_KL[ρ_θ ∥ ρ_θ̄] ≤ δ is described verbally, but it is not incorporated into Eq. (11) or into the description of the Lagrangian optimization. Please state how the trust-region bound interacts with the Lagrangian objective.

Circularity Check

1 steps flagged · score 5.0 of 10

Per-environment grid search of λ, admitted in Fig. 9b as tuning to match ground truth, makes the safety/constraint-matching result partly a fit rather than a prediction.

  1. fitted input called prediction [Section IV-A (Constraint Inference with Risk-Sensitive Criterion); Section V-E (Fig. 9b); Table VI (Safe TL risk levels)]
    "In the later safe TL stage, where achieving high performance on the target task is crucial, fine-tuning is performed using grid search to determine λ. ... Fig. 9b shows that we can infer a constraint that most closely resembles the ground truth (GT) located on the far left by finely tuning λ in DIAL."

    The scalar λ is the CVaR distortion parameter that directly sets how conservative the recovered constraint is (Eqs. 5–8, Fig. 3). Table VI fixes it per environment (0.5 for Intersection, MountainCar, CartPole, BasicNav; 0.1 for PointGoal), and Section IV-A states that λ is selected by grid search during safe TL. Fig. 9b explicitly admits that the inferred constraint map is made to match the ground-truth safe set by finely tuning λ. Therefore, the reported 'adaptation by adjusting risk levels' and the GT-matching inferred constraints are obtained by fitting λ to the evaluation target, not independently predicted by the learned Beta posterior.

full rationale

The paper's core learning machinery is not circular in the strictest sense: the constraint network is trained on multi-task expert demonstrations through an ICRL-style gradient, and performance is evaluated against external ground-truth costs that are not used for constraint training. There is no load-bearing self-citation chain; the cited ICRL, Bayesian, and entropy-estimation results are external prior work. The one genuine circularity concern is the risk level λ. The method's central transfer mechanism is 'adjusting risk levels within these learned distributions,' but λ is not inferred or automatically adapted; it is set by per-environment grid search (Table VI), and Fig. 9b shows that tuning λ can make the recovered constraint match the ground-truth safe set. This turns part of the safety and constraint-recovery result into a fitted outcome rather than an independent prediction. The paper's own conclusion also lists 'incorporating techniques that automatically optimize or dynamically adjust risk levels' as future work, corroborating that the current λ selection is external to the learned distribution. The unvalidated Beta/CVaR posterior fidelity is a correctness risk, not a circularity, and I do not count it here.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical entities. It introduces a probabilistic model, a Beta posterior with CVaR distortion, and several tuned hyperparameters; those are captured in free_parameters and axioms.

free parameters (5)
  • Risk level lambda (safe TL) = 0.5 for Intersection, MountainCar, CartPole, BasicNav; 0.1 for PointGoal
    Selected by grid search per target environment (Table VI). Controls CVaR distortion and thus how conservative or risky the transferred policy is; this is a task-specific adjustment despite the claim of no task-specific constraints.
  • Entropy coefficient beta = 0.01 (Intersection, MountainCar, CartPole), 1.0 (BasicNav), 0.1 (PointGoal)
    Table V. Hand-tuned per environment; balances exploration bonus against reward and constraint terms.
  • Trust-region threshold delta = 0.1 (Intersection, PointGoal), 0.5 (MountainCar, CartPole), 1.0 (BasicNav)
    Table V. Limits the number of entropy-gradient policy updates; hand-tuned.
  • Beta prior parameters alpha0 = [0.1, 0.9]
    Table IV. KL regularization target for the approximate posterior; mean 0.1 is close to the typical violation budgets, so the prior may encode the expected safety level.
  • Damping scaling factor kappa_d = 10
    Table IV. Adjusts the Lagrange multiplier update to stabilize training; chosen by hand.
assumptions (5)
  • domain assumption Expert demonstrations satisfy all safety requirements in the source environment and are safe for the original task.
    Section IV: 'We assume that while these demonstrations meet all safety requirements in the original environment...'. If demos contain violations, learned constraints inherit them.
  • ad hoc to paper Trajectory feasibility is a random variable in (0,1) following a Beta posterior that factorizes across constraints.
    Section IV-A: 'we choose an ideal Beta distribution as the posterior' and factorize q(zeta(tau)|alpha) as a product over constraints. No justification beyond interval support.
  • domain assumption The IW k-NN entropy estimator (Eq. 10) is a valid, optimizable proxy for the state density entropy of the policy.
    Section IV-B; relies on [57] and [18]. The estimator's variance and bias under RL sampling are not analyzed.
  • standard math The MaxEnt ICRL trajectory distribution (Eq. 2) correctly models demonstration likelihood.
    Adopted from [20]; standard in ICRL, but depends on the reward being known and on the Boltzmann factor.
  • ad hoc to paper A single scalar risk level lambda, fixed during transfer, captures task-specific risk sensitivity across the new environment.
    Section IV and Table VI; lambda is tuned per environment, so this assumption is only as strong as the grid search.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DIAL: Distribution-Informed Adaptive Learning of Multi-Task Constraints for Safety-Critical Systems." pith.science (2026). https://pith.science/paper/NXJX3DTN

@misc{pith2026250118086,
  author       = {Pith},
  title        = {Pith review of: DIAL: Distribution-Informed Adaptive Learning of Multi-Task Constraints for Safety-Critical Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NXJX3DTN}},
  note         = {Machine review of arXiv:2501.18086}
}
read the original abstract

Safe reinforcement learning has traditionally relied on predefined constraint functions to ensure safety in complex real-world tasks, such as autonomous driving. However, defining these functions accurately for varied tasks is a persistent challenge. Recent research highlights the potential of leveraging pre-acquired task-agnostic knowledge to enhance both safety and sample efficiency in related tasks. Building on this insight, we propose a novel method to learn shared constraint distributions across multiple tasks. Our approach identifies the shared constraints through imitation learning and then adapts to new tasks by adjusting risk levels within these learned distributions. This adaptability addresses variations in risk sensitivity stemming from expert-specific biases, ensuring consistent adherence to general safety principles even with imperfect demonstrations. Our method can be applied to control and navigation domains, including multi-task and meta-task scenarios, accommodating constraints such as maintaining safe distances or adhering to speed limits. Experimental results validate the efficacy of our approach, demonstrating superior safety performance and success rates compared to baselines, all without requiring task-specific constraint definitions. These findings underscore the versatility and practicality of our method across a wide range of real-world tasks.

Figures

Figures reproduced from arXiv: 2501.18086 by the authors.

Figure 1
Figure 1. Comparison of standard ICRL (a) and proposed ICRL with DIAL (b). The left side shows the problem each approach addresses. In (a), the black [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Architecture of safe TL stage in DIAL. The constraint model is used [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Difference between CVaR and mean in Beta distribution. We set [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Unsigned intersection environments in urban driving. The agent controls a red car, guiding it toward its destination by following a yellow arrow, while [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Robot control environments that aim to perform target tasks while ensuring safety. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Comparison of RR and CV for urban driving tasks based on the number of expert trajectories. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Learning curves of DIAL for each constraint during safe IL. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Visualization of the changes in the constraint distribution for DIAL during safe IL. [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Visualization of state visitation frequencies and inferred constraints. Each map is normalized between 0 and 1, and darker blue indicates higher values. [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Learning curves of DIAL showing environmental data on safety and [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: Visualization of the distribution of environmental data related to safety requirements during a single episode executed by an agent trained with DIAL. [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]
Figure 12
Figure 12. Figure 12: Visualization of changes in state visitation maps during safe TL [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]
Figure 13
Figure 13. Figure 13: Visualization of explicit reward for each environment. Darker blue represents higher values for each map. [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: Learning curves for auto-tuning safety weight. [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 42 canonical work pages

  1. [46]

    Uncertainty-aware constraint inference in inverse constrained reinforcement learning,

    S. Xu and G. Liu, “Uncertainty-aware constraint inference in inverse constrained reinforcement learning,” in The Twelfth International Con- ference on Learning Representations , 2023

    work page 2023

  2. [52]

    Bayesian methods for constraint inference in reinforcement learning,

    D. Papadimitriou, U. Anwar, and D. S. Brown, “Bayesian methods for constraint inference in reinforcement learning,”Transactions on Machine Learning Research, 2024

    work page 2024

  3. [1]

    Cic: Contrastive intrinsic control for unsupervised skill discovery,

    M. Laskin, H. Liu, X. B. Peng, D. Yarats, A. Rajeswaran, and P. Abbeel, “Cic: Contrastive intrinsic control for unsupervised skill discovery,” in Deep RL Workshop NeurIPS 2021 , 2021. 15 TABLE V HYPERPARAMETERS FOR EACH ENVIRONMENT IN SAFE IL Hyperparameters Intersection MountainCar CartPole BasicNav PointGoal Notation Environmental Steps 1.5 × 105 5 × 10...

    work page 2021

  4. [2]

    Urlb: Unsupervised reinforcement learning benchmark,

    M. Laskin, D. Yarats, H. Liu, K. Lee, A. Zhan, K. Lu, C. Cang, L. Pinto, and P. Abbeel, “Urlb: Unsupervised reinforcement learning benchmark,” in 35th Conference on Neural Information Processing Sys- tems (NeurIPS) . Neural Information Processing Systems Foundation, 2021

    work page 2021

  5. [3]

    Unsupervised reinforcement learning in multiple environments,

    M. Mutti, M. Mancassola, and M. Restelli, “Unsupervised reinforcement learning in multiple environments,” in Proceedings of the AAAI Confer- ence on Artificial Intelligence , vol. 36, no. 7, 2022, pp. 7850–7858

    work page 2022

  6. [4]

    R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction . MIT press, 2018

    2018

  7. [5]

    Efficient training of artificial neural networks for autonomous navigation,

    D. A. Pomerleau, “Efficient training of artificial neural networks for autonomous navigation,” Neural computation, vol. 3, no. 1, pp. 88–97, 1991

    1991

  8. [6]

    Learning from demonstration,

    S. Schaal, “Learning from demonstration,” Advances in neural informa- tion processing systems , vol. 9, 1996

    work page 1996

Show all 63 references
  1. [7]

    A survey of robot learning from demonstration,

    B. D. Argall, S. Chernova, M. Veloso, and B. Browning, “A survey of robot learning from demonstration,” Robotics and autonomous systems , vol. 57, no. 5, pp. 469–483, 2009

    2009

  2. [8]

    Model-agnostic meta-learning for fast adaptation of deep networks,

    C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in International conference on machine learning. PMLR, 2017, pp. 1126–1135

    2017

  3. [9]

    Never give up: Learning directed exploration strategies,

    A. P. Badia, P. Sprechmann, A. Vitvitskyi, D. Guo, B. Piot, S. Kaptur- owski, O. Tieleman, M. Arjovsky, A. Pritzel, A. Bolt et al. , “Never give up: Learning directed exploration strategies,” in International Conference on Learning Representations , 2019

    2019

  4. [10]

    Novelty search in repre- sentational space for sample efficient exploration,

    R. Y . Tao, V . Franc ¸ois-Lavet, and J. Pineau, “Novelty search in repre- sentational space for sample efficient exploration,” Advances in Neural Information Processing Systems , vol. 33, pp. 8114–8126, 2020

    2020

  5. [11]

    State entropy maximization with random encoders for efficient exploration,

    Y . Seo, L. Chen, J. Shin, H. Lee, P. Abbeel, and K. Lee, “State entropy maximization with random encoders for efficient exploration,” in International Conference on Machine Learning . PMLR, 2021, pp. 9443–9454

    2021

  6. [12]

    A comprehensive survey on safe reinforce- ment learning,

    J. Garcıa and F. Fern ´andez, “A comprehensive survey on safe reinforce- ment learning,” Journal of Machine Learning Research , vol. 16, no. 1, pp. 1437–1480, 2015

    2015

  7. [13]

    Learning to run a power network challenge for training topology controllers,

    A. Marot, B. Donnot, C. Romero, B. Donon, M. Lerousseau, L. Veyrin- Forrer, and I. Guyon, “Learning to run a power network challenge for training topology controllers,” Electric Power Systems Research , vol. 189, p. 106635, 2020

    2020

  8. [14]

    Challenges of real-world reinforcement learning: definitions, benchmarks and analysis,

    G. Dulac-Arnold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester, “Challenges of real-world reinforcement learning: definitions, benchmarks and analysis,” Machine Learning, vol. 110, no. 9, pp. 2419–2468, 2021

    2021

  9. [15]

    Constrained policy optimization,

    J. Achiam, D. Held, A. Tamar, and P. Abbeel, “Constrained policy optimization,” in International conference on machine learning. PMLR, 2017, pp. 22–31

    2017

  10. [16]

    Safe reinforcement learning in constrained markov decision processes,

    A. Wachi and Y . Sui, “Safe reinforcement learning in constrained markov decision processes,” in International Conference on Machine Learning . PMLR, 2020, pp. 9797–9806

    2020

  11. [17]

    Density constrained reinforcement learning,

    Z. Qin, Y . Chen, and C. Fan, “Density constrained reinforcement learning,” in International Conference on Machine Learning . PMLR, 2021, pp. 8682–8692

    2021

  12. [18]

    Cem: Constrained entropy maximization for task-agnostic safe exploration,

    Q. Yang and M. T. Spaan, “Cem: Constrained entropy maximization for task-agnostic safe exploration,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 37, no. 9, 2023, pp. 10 798–10 806

    2023

  13. [19]

    Learning constraints from demon- strations,

    G. Chou, D. Berenson, and N. Ozay, “Learning constraints from demon- strations,” in Algorithmic Foundations of Robotics XIII: Proceedings of the 13th Workshop on the Algorithmic Foundations of Robotics 13 . Springer, 2020, pp. 228–245

    2020

  14. [20]

    Inverse constrained re- inforcement learning,

    S. Malik, U. Anwar, A. Aghasi, and A. Ahmed, “Inverse constrained re- inforcement learning,” in International conference on machine learning. PMLR, 2021, pp. 7390–7399

    2021

  15. [21]

    Learning shared safety constraints from multi-task demonstrations,

    K. Kim, G. Swamy, Z. Liu, D. Zhao, S. Choudhury, and S. Z. Wu, “Learning shared safety constraints from multi-task demonstrations,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  16. [22]

    Conditional value-at-risk for elliptical distributions,

    V . Khokhlov, “Conditional value-at-risk for elliptical distributions,” Evropsk`y ˇcasopis ekonomiky a managementu , vol. 2, no. 6, pp. 70–79, 2016

    2016

  17. [23]

    Worst cases policy gradients,

    Y . C. Tang, J. Zhang, and R. Salakhutdinov, “Worst cases policy gradients,” in Proceedings of the Conference on Robot Learning, ser. Proceedings of Machine Learning Research, L. P. Kaelbling, D. Kragic, and K. Sugiura, Eds., vol. 100. PMLR, 30 Oct–01 Nov 2020, pp. 1078–1093. ...

    2020

  18. [24]

    Wcsac: Worst-case soft actor critic for safety-constrained reinforcement learn- ing,

    Q. Yang, T. D. Sim ˜ao, S. H. Tindemans, and M. T. Spaan, “Wcsac: Worst-case soft actor critic for safety-constrained reinforcement learn- ing,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 35, no. 12, 2021, pp. 10 639–10 646. 16

    2021

  19. [25]

    Task-agnostic exploration in reinforce- ment learning,

    X. Zhang, Y . Ma, and A. Singla, “Task-agnostic exploration in reinforce- ment learning,” Advances in Neural Information Processing Systems , vol. 33, pp. 11 734–11 743, 2020

    2020

  20. [26]

    Learning safety constraints from demonstrations with unknown re- wards,

    D. Lindner, X. Chen, S. Tschiatschek, K. Hofmann, and A. Krause, “Learning safety constraints from demonstrations with unknown re- wards,” in International Conference on Artificial Intelligence and Statis- tics. PMLR, 2024, pp. 2386–2394

    2024

  21. [27]

    Train hard, fight easy: Robust meta reinforcement learning,

    I. Greenberg, S. Mannor, G. Chechik, and E. Meirom, “Train hard, fight easy: Robust meta reinforcement learning,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  22. [28]

    Incentivizing exploration in reinforcement learning with deep predictive models,

    B. C. Stadie, S. Levine, and P. Abbeel, “Incentivizing exploration in reinforcement learning with deep predictive models,” arXiv preprint arXiv:1507.00814, 2015

    2015 arXiv

  23. [29]

    Reward-free exploration for reinforcement learning,

    C. Jin, A. Krishnamurthy, M. Simchowitz, and T. Yu, “Reward-free exploration for reinforcement learning,” in International Conference on Machine Learning. PMLR, 2020, pp. 4870–4879

    2020

  24. [30]

    Provably efficient maximum entropy exploration,

    E. Hazan, S. Kakade, K. Singh, and A. Van Soest, “Provably efficient maximum entropy exploration,” in International Conference on Machine Learning. PMLR, 2019, pp. 2681–2691

    2019

  25. [31]

    Efficient exploration via state marginal matching,

    L. Lee, B. Eysenbach, E. Parisotto, E. Xing, S. Levine, and R. Salakhut- dinov, “Efficient exploration via state marginal matching,”arXiv preprint arXiv:1906.05274, 2019

    1906 arXiv

  26. [32]

    Constrained cross-entropy method for safe reinforcement learning,

    M. Wen and U. Topcu, “Constrained cross-entropy method for safe reinforcement learning,” Advances in Neural Information Processing Systems, vol. 31, 2018

    2018

  27. [33]

    Learning to fly,

    C. Sammut, S. Hurst, D. Kedzier, and D. Michie, “Learning to fly,” in Machine Learning Proceedings 1992 . Elsevier, 1992, pp. 385–393

    1992

  28. [34]

    G. M. Hayes and J. Demiris, A robot controller using learning by imitation. University of Edinburgh, Department of Artificial Intelligence Edinburgh, UK, 1994

    1994

  29. [35]

    Apprenticeship learning via inverse rein- forcement learning,

    P. Abbeel and A. Y . Ng, “Apprenticeship learning via inverse rein- forcement learning,” in Proceedings of the twenty-first international conference on Machine learning , 2004, p. 1

    2004

  30. [36]

    Generative adversarial imitation learning,

    J. Ho and S. Ermon, “Generative adversarial imitation learning,” Ad- vances in neural information processing systems , vol. 29, 2016

    2016

  31. [37]

    Learning robust rewards with adverserial inverse reinforcement learning,

    J. Fu, K. Luo, and S. Levine, “Learning robust rewards with adverserial inverse reinforcement learning,” in International Conference on Learn- ing Representations, 2018

    2018

  32. [38]

    Iq-learn: Inverse soft-q learning for imitation,

    D. Garg, S. Chakraborty, C. Cundy, J. Song, and S. Ermon, “Iq-learn: Inverse soft-q learning for imitation,” Advances in Neural Information Processing Systems, vol. 34, pp. 4028–4039, 2021

    2021

  33. [39]

    Maximum entropy inverse reinforcement learning

    B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey et al., “Maximum entropy inverse reinforcement learning.” in Aaai, vol. 8. Chicago, IL, USA, 2008, pp. 1433–1438

    2008

  34. [40]

    Bridging the gap between imitation learning and inverse reinforcement learning,

    B. Piot, M. Geist, and O. Pietquin, “Bridging the gap between imitation learning and inverse reinforcement learning,” IEEE transactions on neural networks and learning systems , vol. 28, no. 8, pp. 1814–1826, 2016

    2016

  35. [41]

    A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models,

    C. Finn, P. Christiano, P. Abbeel, and S. Levine, “A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models,” arXiv preprint arXiv:1611.03852 , 2016

    2016 arXiv

  36. [42]

    Maximum likelihood constraint inference for inverse reinforcement learning,

    D. R. Scobee and S. S. Sastry, “Maximum likelihood constraint inference for inverse reinforcement learning,” arXiv preprint arXiv:1909.05477 , 2019

    1909 arXiv

  37. [43]

    Learning constraints from demon- strations with grid and parametric representations,

    G. Chou, D. Berenson, and N. Ozay, “Learning constraints from demon- strations with grid and parametric representations,” The International Journal of Robotics Research, vol. 40, no. 10-11, pp. 1255–1283, 2021

    2021

  38. [44]

    Gaussian process constraint learning for scalable chance-constrained motion planning from demon- strations,

    G. Chou, H. Wang, and D. Berenson, “Gaussian process constraint learning for scalable chance-constrained motion planning from demon- strations,” IEEE Robotics and Automation Letters , vol. 7, no. 2, pp. 3827–3834, 2022

    2022

  39. [45]

    Learning soft constraints from constrained expert demonstrations,

    A. Gaurav, K. Rezaee, G. Liu, and P. Poupart, “Learning soft constraints from constrained expert demonstrations,” in The Eleventh International Conference on Learning Representations , 2023

    2023

  40. [47]

    Confidence aware inverse constrained reinforcement learning,

    S. G. Subramanian, G. Liu, M. Elmahgiubi, K. Rezaee, and P. Poupart, “Confidence aware inverse constrained reinforcement learning,” arXiv preprint arXiv:2406.16782, 2024

    2024 arXiv

  41. [48]

    Inverse constraint learning and gen- eralization by transferable reward decomposition,

    J. Jang, M. Song, and D. Park, “Inverse constraint learning and gen- eralization by transferable reward decomposition,” IEEE Robotics and Automation Letters, vol. 9, no. 1, pp. 279–286, 2023

    2023

  42. [49]

    Altman, Constrained Markov decision processes

    E. Altman, Constrained Markov decision processes . Routledge, 2021

    2021

  43. [50]

    Benchmarking safe exploration in deep reinforcement learning,

    A. Ray, J. Achiam, and D. Amodei, “Benchmarking safe exploration in deep reinforcement learning,” arXiv preprint arXiv:1910.01708 , vol. 7, no. 1, p. 2, 2019

    1910 arXiv

  44. [51]

    Approxi- mate robust control of uncertain dynamical systems,

    E. Leurent, Y . Blanco, D. Efimov, and O.-A. Maillard, “Approxi- mate robust control of uncertain dynamical systems,” arXiv preprint arXiv:1903.00220, 2019

    1903 arXiv

  45. [53]

    Risk-sensitive inverse reinforcement learning via coherent risk models

    A. Majumdar, S. Singh, A. Mandlekar, and M. Pavone, “Risk-sensitive inverse reinforcement learning via coherent risk models.” in Robotics: science and systems , vol. 16, 2017, p. 117

    2017

  46. [54]

    Benchmarking constraint inference in inverse reinforcement learning,

    G. Liu, Y . Luo, A. Gaurav, K. Rezaee, and P. Poupart, “Benchmarking constraint inference in inverse reinforcement learning,” in The Eleventh International Conference on Learning Representations , 2023

    2023

  47. [55]

    Near- est neighbor estimates of entropy,

    H. Singh, N. Misra, V . Hnizdo, A. Fedorowicz, and E. Demchuk, “Near- est neighbor estimates of entropy,” American journal of mathematical and management sciences , vol. 23, no. 3-4, pp. 301–321, 2003

    2003

  48. [56]

    Particle based probability density fusion with differential shannon entropy criterion,

    J. Ajgl and M. ˇSimandl, “Particle based probability density fusion with differential shannon entropy criterion,” in 14th International Conference on Information Fusion . IEEE, 2011, pp. 1–8

    2011

  49. [57]

    Task-agnostic exploration via policy gradient of a non-parametric state entropy estimate,

    M. Mutti, L. Pratissoli, and M. Restelli, “Task-agnostic exploration via policy gradient of a non-parametric state entropy estimate,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 35, no. 10, 2021, pp. 9028–9036

    2021

  50. [58]

    Reward constrained policy optimization,

    C. Tessler, D. J. Mankowitz, and S. Mannor, “Reward constrained policy optimization,” arXiv preprint arXiv:1805.11074 , 2018

    2018 arXiv

  51. [59]

    An environment for autonomous driving decision- making,

    E. Leurent et al. , “An environment for autonomous driving decision- making,” 2018

    2018

  52. [60]

    Openai gym,

    G. Brockman, V . Cheung, L. Pettersson, J. Schneider, J. Schul- man, J. Tang, and W. Zaremba, “Openai gym,” arXiv preprint arXiv:1606.01540, 2016

    2016 arXiv

  53. [61]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , 2014

    2014 arXiv

  54. [62]

    Constrained differential optimization,

    J. Platt and A. Barr, “Constrained differential optimization,” in Neural Information Processing Systems , 1987

    1987

  55. [63]

    Controlled text gen- eration as continuous optimization with multiple constraints,

    S. Kumar, E. Malmi, A. Severyn, and Y . Tsvetkov, “Controlled text gen- eration as continuous optimization with multiple constraints,” Advances in Neural Information Processing Systems , vol. 34, pp. 14 542–14 554, 2021. Se-Wook Yoo (Member, IEEE) received the B.S. degree in e...

    2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.