Pith. sign in

REVIEW 3 major objections 5 minor 39 references

TraCeS: Learning Per-Timestep Constraint-Violation Credit from Sparse Trajectory-Level Labels

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Whole-trajectory safe/unsafe labels suffice to learn per-step safety credit and train safe policies.

desk verdict A solid, incremental safe-RL method for learning constraints from binary trajectory labels, but the advertised theory is thinner than the abstract suggests and the safety claim is average-cost rather than a guarantee. read the letter →

arxiv 2504.12557 v3 pith:EVEVLUX3 submitted 2025-04-17 cs.LG cs.AI

Siow Meng Low , Ze Gong , Akshat Kumar This is my paper · ORCID
classification cs.LGcs.AI
keywords safereinforcementlearningconstraintinferencecreditassignmenttrajectory-levelsafetylabelssummaryvectorPPO-Lagrangiansparsefeedbackcontinuouscontrol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether an agent can learn to be safe when the only safety feedback is a binary label attached to a whole trajectory, and when the underlying cost function and budget are unknown. It proposes TraCeS, which trains a per-timestep credit score for each state-action pair so that the product of the scores equals the model's predicted probability that the trajectory is safe. That score is used as a per-step pseudo-cost, and the safe-reinforcement-learning problem is reformulated as maximizing reward subject to a lower bound on the discounted sum of log-scores. The paper reports that on twelve continuous-control tasks with hidden constraints, policies trained this way are safe in all twelve and achieve total return close to an oracle method that knows the true cost and budget. If correct, this means coarse approvals and rejections of whole rollouts are enough supervision to train safe control policies.

What carries the argument

The load-bearing object is the Safety Summary Vector (SSV) model. A recurrent network $f_w$ compresses the trajectory prefix into a summary vector $h_{t+1}=f_w(s_t,a_t,h_t)$, and a decoder compares $h_t$ with $h_{t+1}$ to output the per-step multiplicative safety score $\hat P^\Delta_t$. By the telescoping identity of Proposition 3.3, $\prod_{t=0}^{T-1} \hat P^\Delta_t = \hat P(\psi=1\mid\tau)$, so taking logs converts the product into a sum of per-step pseudo-costs. Jensen's inequality then turns the desired safe-trajectory rate $\mathbb{E}[\Psi(\tau)]\ge d$ into the tractable constraint $\mathbb{E}[\sum_t \gamma^t \log \hat P^\Delta_t]\ge \log d$, which PPO-Lagrangian can enforce. A decoder that outputs a negative lognormal distribution for each log-score lets the method use the coefficient of variation of the total score to choose which trajectories are worth labeling, reducing the human feedback needed.

What would settle it

Take the final trained policy, run 1,000 new trajectories, and compare the model's predicted safety probability with the true safe/unsafe label. If trajectories the model calls almost certainly safe turn out to be unsafe much more often than the model predicts, the learned credit is not faithful. A sharper test uses a task where an early violation can be repaired by later actions: the true label would be safe once the total cost falls back below budget, but the model's safety probability can only stay the same or decrease, so it must either miscredit the repair or force the policy to be overly conservative.

Watch

Extended reading notes

Core claim

The paper's central claim is that per-timestep safety credit can be recovered from sparse, trajectory-level binary labels by factorizing the estimated probability that a trajectory is safe. If the model outputs one multiplicative score $\hat P^\Delta_t$ per step, the product over the trajectory telescopes to $\hat P(\psi=1\mid\tau)$, the estimated probability that the whole trajectory is safe, and each score is interpreted as the change in safety caused by that step. TraCeS turns these scores into a discounted-sum constraint $\mathbb{E}[\sum_t \gamma^t \log \hat P^\Delta_t] \ge \log d$, a Jensen lower bound on requiring at least a fraction $d$ of trajectories to be safe, and solves the resulting program with PPO-Lagrangian, a standard constrained-RL method. In twelve continuous-control benchmarks where both cost function and budget are kept hidden, the paper reports that the policy trained with its SSV model is safe on every task and that total return stays close to PPO-Lagrangian with full knowledge, while using fewer labeled trajectories than the RLSF baseline in several domains.

Load-bearing premise

The load-bearing premise is that the model's estimate of the probability that a trajectory is safe stays accurate on the states and actions the policy visits during online training, even though the model was trained beforehand on a fixed dataset and the policy's behavior keeps changing; if the model overestimates safety in those new situations, the constraint it enforces is weaker than the intended one.

Editorial extensions

If this is right

  • Users who cannot provide a cost function, a budget, or per-state costs can still obtain a constrained policy from whole-trajectory yes/no labels.
  • Because the safety scores factorize, standard constrained-RL solvers such as PPO-Lagrangian need only a state augmented with a summary vector; no new safe-RL algorithm is required.
  • The inferred credit spikes when accumulated true cost crosses the hidden budget, so the method localizes the moment an unknown constraint is violated.
  • Selecting trajectories for labeling by coefficient of variation cuts the number of labels needed without substantially changing final reward or safety.
  • Continual retraining on in-distribution trajectories keeps the safety estimate usable as the policy distribution shifts during online learning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the same factorized score could serve as a diagnostic, since the log-score spike identifies the first moment an unknown threshold is crossed; that localization is useful for auditing demonstration data even without policy optimization.
  • Inference: because the reformulated constraint is a Jensen lower bound, TraCeS is conservative when the model is well calibrated, but a natural safeguard is to re-check predicted safety probabilities against true labels on held-out rollouts and retrain when calibration drifts.
  • Inference: the monotone non-increasing form of $\hat P(\psi=1\mid\tau)$ matches constraints with non-negative costs where violations cannot be undone; reversible violations would need a different decomposition that allows the safety probability to rise again.
  • Inference: the selective-feedback rule suggests a general recipe for other human-in-the-loop settings: label only trajectories with high model uncertainty, and per-step credit assignment from coarse labels remains reliable.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes TraCeS, a method for learning per-timestep safety credit from binary trajectory-level labels when both the cost function and budget are unknown. The method factorizes an estimated trajectory safety probability into per-step multiplicative scores (Proposition 3.3), trains these scores with a binary cross-entropy loss, and reformulates the constrained RL problem using a discounted sum of log safety scores (Eq. 8). The authors claim that the resulting policy is safe across all twelve evaluated continuous-control tasks with return close to an oracle PPO-Lagrangian baseline. Theoretical support is claimed through Lemma 3.4 and a Jensen-based lower bound, and empirical evaluation is conducted on MuJoCo, Safety Gymnasium, and Bullet Safety Gym tasks.

Significance. If the claims hold, the paper addresses a practically important setting: safe RL without a known cost function or budget, relying only on sparse binary trajectory feedback. The proposed factorization is simple and the integration with PPO-Lagrangian is straightforward, making the method potentially useful for applications where dense safety feedback is unavailable. The paper also includes a credit-assignment visualization that could aid interpretability. However, the theoretical contribution is limited to an asymptotic consistency argument and an algebraic identity; the promised approximation-gap analysis is absent. The empirical support is suggestive but not as strong as the headline claims: several tasks have average costs close to the budget, and the reward gap relative to the oracle baseline is substantial on some tasks. The comparison with RLSF is also confounded by RLSF's assumption of a known budget. Overall, the idea is promising but the current manuscript does not fully substantiate its central safety and credit-assignment claims.

major comments (3)
  1. [Section 3.2.2, Lemma 3.4 and Eqs. (7)-(10)] The safety guarantee is only asymptotic and does not address distribution shift. The proof of Lemma 3.4 argues that as the training sample size tends to infinity, the learned model's probability estimate approximates the true labeling function. However, the constraint in Eq. (10) is evaluated under the current policy's rollout distribution, which differs from the offline pretraining distribution and changes during continual learning. No finite-sample bound or calibration evidence is provided for E_{πθ}[P̂] vs. E_{πθ}[Ψ] under this shift. Furthermore, the abstract promises a theoretical analysis of the approximation gap, but the manuscript contains no such finite-sample or distribution-shift analysis; the lemmas are algebraic identities and an asymptotic consistency statement. This is a load-bearing issue because the safety of the final policy depends on the surrogate constraint being a valid lower bound for the true constraint under the online distribution.
  2. [Section 3.2.1, Proposition 3.3 and Eqs. (4)-(6)] The per-timestep safety scores are not identifiable from the available supervision. The factorization in Eq. (5) is a tautology: P̂^Δ_t is defined as the ratio of subsegment safety probabilities, so the product equals the trajectory-level probability by construction. Training with the binary cross-entropy loss in Eq. (6) constrains only the product, not the individual factors. Section 2.2 states that a trajectory segment receives a single binary label, and the experimental description says labels are given to a variable-length trajectory segment without clarifying whether prefixes are labeled separately. Consequently, infinitely many decompositions achieve the same loss, and the interpretation of P̂^Δ_t as the per-timestep cost contribution is an inductive-bias assumption rather than a learned, identifiable quantity. The credit-assignment analysis in Figures 3 and 4 cannot rule out that the observed spikes are artifacts of the chosen architecture and the monotonicity assumption (Assumption 3.6).
  3. [Section 4, Tables 1 and 2] The claim that 'the policy trained using our SSV model is safe across all twelve tasks' is not fully supported by the reported statistics. Several tasks have mean costs within one standard deviation of the budget of 25: Walker2d reports 24.1 ± 3.9, BallRun 24.7 ± 0.8, and CarRun 23.3 ± 1.1. For Walker2d, the mean plus one standard deviation exceeds the budget, so a nontrivial fraction of evaluation seeds likely violate the constraint. Additionally, the reward gap relative to PPO-Lagrangian is substantial on some tasks (HalfCheetah 2372 vs. 3008, a 21% decrease; Walker2d 2211 vs. 2682, an 18% decrease), so the statement 'total return close to PPO-Lagrangian' overstates the match. The comparison with RLSF is also not apples-to-apples because RLSF assumes a known budget; the paper should temper the empirical claims accordingly.
minor comments (5)
  1. [Section 2.2] The sentence 'trajectory is only labeled safe if the true cost incurred is above the threshold' appears to be a typo; it should read 'below or equal to the threshold' to match the definition of Ψ(τ)=1 when cost is within budget.
  2. [Title and Section 1] The acronym TraCeS is defined differently in the arXiv abstract ('Trajectory-based Constraint Estimation for Safety') and in Section 1 ('trajectory based credit assignment for safe RL'). Please use one consistent definition.
  3. [Eq. (6)] Equation (6) is missing a closing parenthesis: the expectation should be written as E_{τi∼D}[ψi log P̂(ψi=1|τi) + (1−ψi) log(1−P̂(ψi=1|τi))].
  4. [Section 4 and Figure 3] The caption and legend of Figure 3 should explicitly define the green curve, which the text describes as the normalized inferred cost from the SSV model; currently the caption mentions only the red and blue curves.
  5. [Appendix C.2, Eq. (12)] The coefficient of variation formula uses X for the lognormal random variables, but the text says the safety score log P̂^Δ_t is sampled from the distribution; please clarify the relationship between X and log P̂^Δ_t so the notation is consistent.

Circularity Check

2 steps flagged · score 6.0 of 10

Per-timestep credit is a definitional reparameterization of the fitted trajectory-level safety probability; the validation of credit assignment reuses the same threshold labels, and the promised approximation-gap analysis is absent.

  1. self definitional [Section 3.2.1, Proposition 3.3, Eqs. (2)-(5)]
    "ˆP ∆ 0 ≜ ˆP (ψ0:0 = 1|τ0:0) 1 and ˆP ∆ t ≜ ˆP (ψ0:t = 1|τ0:t) ˆP (ψ0:t−1 = 1|τ0:t−1) , 1≤t≤T− 1 (4) ... ˆP (ψ0:T−1 = 1|τ0:T−1) ≜ Y T−1 t=0 ˆP ∆ t (5)"

    The per-timestep score P̂Δt is defined to be the ratio of two trajectory-level probability estimates, so the product in Eq. (5) is exactly the single trajectory probability in Eq. (2) restated; the factors telescope and no new per-step information is introduced. The paper presents this factorization as 'safety credit assignment' that assigns each timestep's impact, but the only quantity fitted to the sparse trajectory labels is the trajectory-level P̂(ψ=1|τ) via the BCE loss in Eq. (6). Unless labels are provided for many prefixes of the same trajectory, infinitely many per-step assignments produce the same BCE loss, and the paper gives no identifiability argument.

  2. fitted input called prediction [Section 4, 'Credit Assignment' and experiment setup]
    "In our experiments, a script implementing the true labeling function is used to label the trajectories. ... It can be observed from the figure that the inferred cost typically peaks when the accumulated cost reaches 25, the ground-truth budget, and there are other spikes closely following the true non-zero cost at other timesteps."

    The SSV model is trained on labels generated by a script that implements the true labeling function, i.e., a threshold on cumulative true cost at the ground-truth budget of 25. The credit-assignment evaluation then reports that the inferred per-step cost peaks exactly when the accumulated cost reaches that same threshold, and that its spikes track the true costs. Because the per-step scores are defined as ratios of the fitted trajectory-level safety probabilities (Eq. 4), and that classifier was optimized against these threshold-based labels, this agreement is a consistency check between the fitted model and its own label-generating rule, not an independent test of per-step credit from sparse trajectory labels.

full rationale

The central definitional issue is in Proposition 3.3: the per-timestep safety credit P̂Δt is defined as the ratio of consecutive trajectory-level probability estimates, so Eq. (5) is a tautological telescoping identity with Eq. (2). Consequently, the method's headline capability — learning per-timestep violation credit from sparse trajectory-level labels — is, at the level of the formulation, a reparameterization of the trajectory-level classifier fitted to those labels. The remaining RL machinery (Lemma 3.4, Eqs. 8-10) is a sequence of algebraic manipulations: substituting the product identity into a trajectory-level constraint, applying an asymptotic MLE consistency argument, and taking a Jensen lower bound. These steps are not themselves circular, but they inherit the definitional content of P̂Δt. The safety claim therefore rests on whether the trajectory-level probabilities are accurate under the online policy's distribution; the paper provides only an asymptotic consistency sketch and no finite-sample or distribution-shift gap analysis. I flag the abstract's promise of 'a theoretical analysis of the approximation gap' as unsupported: the body contains no such finite-sample or off-policy analysis, only the consistency assertion in Lemma 3.4 and the Jensen bound. The empirical comparison to PPO-Lagrangian is against externally evaluated true costs and is not circular; however, the credit-assignment figures (Figures 3 and 4) reuse the same threshold-based label generator, so they do not independently validate the per-step attribution. There is no load-bearing self-citation chain. Overall, one central 'prediction' — the per-timestep credit — reduces by construction to the fitted trajectory-level probability, while the safe-RL outcome retains independent empirical content, warranting a partial-circularity score of 6.

Assumptions & free parameters 1 free parameters · 4 assumptions · 1 invented entities

The method's contribution is concentrated in the learned safety score; that score is defined by a tautological factorization, so the meaningful content rests on the convergence, monotonicity, and threshold-labeling assumptions above.

free parameters (1)
  • d = 0.9
    Threshold in Equation 7 for the minimum expected fraction of safe trajectories; hand-set in Section 3.2.2. Lower values would permit more violations; higher values (e.g., 1) are noted as overly conservative.
assumptions (4)
  • domain assumption The estimated trajectory safety probability converges to the true labeling function as training data grows (Lemma 3.4)
    Used to equate E[P-hat] with E[Psi]; no finite-sample bound is given, and neural network training is not guaranteed to achieve global MLE consistency.
  • domain assumption Safety probability is non-increasing as a trajectory is extended (Assumption 3.6), equivalent to non-negative costs
    Needed so that P-hat-Delta-t lies in [0,1] and the product in Proposition 3.3 is a proper probability. The paper argues it is standard in safe RL benchmarks, but does not enforce it explicitly in the non-CV architecture.
  • domain assumption The true labeling function has the threshold form: a trajectory is safe iff total ground-truth cost is at most the budget
    The entire experiment protocol (script labeling, pre-training, evaluation) assumes this structure; real human feedback may not satisfy it.
  • domain assumption The offline DSRL dataset is sufficiently representative for pre-training the constraint model before online RL
    Pre-training to 95% accuracy is stated without analysis of coverage or distribution shift.
invented entities (1)
  • Per-timestep multiplicative safety score P-hat-Delta-t independent evidence
    purpose: Serves as pseudo-cost in the reformulated constrained RL program (Equations 2 to 10)
    The score is compared qualitatively to ground-truth cost spikes in Figures 3 and 4, providing a falsifiable but not quantitative external handle; it is not directly benchmarked per-step against ground truth.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TraCeS: Learning Per-Timestep Constraint-Violation Credit from Sparse Trajectory-Level Labels." pith.science (2026). https://pith.science/paper/EVEVLUX3

@misc{pith2026250412557,
  author       = {Pith},
  title        = {Pith review of: TraCeS: Learning Per-Timestep Constraint-Violation Credit from Sparse Trajectory-Level Labels},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EVEVLUX3}},
  note         = {Machine review of arXiv:2504.12557}
}
read the original abstract

Ensuring safe behavior in reinforcement learning (RL) is challenging when safety constraints are implicit and cannot be densely measured. In many settings, supervision is limited to coarse approvals or rejections of whole trajectories (e.g., whether a rollout remained within an unknown safety threshold). We propose TraCeS (Trajectory-based Constraint Estimation for Safety), a method for learning per-timestep violation credit from such sparse trajectory-level labels. TraCeS trains a sequential violation estimator whose per-step credits factorize the predicted probability that a trajectory has not yet violated the constraint, and integrates this learned signal into constrained policy optimization. The method requires neither a known cost function nor a known threshold, and remains compatible with standard continuous-control algorithms. We provide a theoretical analysis of the approximation gap introduced by the learning objective, and demonstrate empirically that TraCeS improves constraint satisfaction and feedback efficiency over baselines across multiple continuous-control benchmarks, including long-horizon tasks and settings with noisy or inconsistent labels.

Figures

Figures reproduced from arXiv: 2504.12557 by the authors.

Figure 1
Figure 1. Safety Model Architectures: (a) Cost & Budget model; (b) Safety Summary Vector Model [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Distribution of Safety Score (Walker2d) 0 200 400 600 800 1000 0 20 40 Cumulative Cost Ant Cumulative Cost True Cost 0 200 400 600 800 1000 0 20 40 Walker2d 0 20 40 60 80 100 0 20 40 BallRun 0 50 100 150 200 0 20 40 CarRun 0 200 400 600 800 1000 Timestep 0 20 40 Cumulative Cost Cumulative Cost Inferred Cost 0 200 400 600 800 1000 Timestep 0 20 40 0 20 40 60 80 100 Timestep 0 20 40 0 50 100 150 200 Timestep 0 20 40 … view at source ↗
Figure 3
Figure 3. Sample Trajectories from Four Tasks the existing notable work which can serve as baselines include: (1) PPO-Lagrangian, an oracle baseline where cost is directly observed and budget is known; (2) RLSF (Chirra et al., 2024), cost is learned from human feedback but budget is known. These two baselines are compared with our two proposed methods where both budget and cost are learned: (3) Cost Budget Model; and (4) Safe… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Credit Assignment within Flat Region and around Critical Region [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Training Curve for Circle Tasks 0.0 0.2 0.4 0.6 0.8 1.0 1e7 4000 2000 0 2000 Return SafetyAntVelocity-v1 0.0 0.2 0.4 0.6 0.8 1.0 1e7 1000 0 1000 2000 3000 SafetyHalfCheetahVelocity-v1 0.0 0.2 0.4 0.6 0.8 1.0 1e7 500 0 500 1000 1500 2000 2500 SafetyHopperVelocity-v1 0.0…
Figure 6
Figure 6. Figure 6: Training Curve for MuJoCo Velocity Tasks [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Training Curve for Bullet Run Tasks 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 Total Cost Incurred 0.0 0.2 0.4 0.6 0.8 1.0 Distribution - P(Safe Traj) SafetyAntVelocity-v1 [PITH_FULL_IMAGE:figures/f…
Figure 8
Figure 8. Figure 8: Distribution of Safety Score (Ant) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 Total Cost Incurred 0.0 0.2 0.4 0.6 0.8 1.0 Distribution - P(Safe Traj) SafetyBallRun-v0 [PITH_FULL_IMAGE:figures/full_f…
Figure 9
Figure 9. Figure 9: Distribution of Safety Score (BallRun) 14 [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Distribution of Safety Score (CarRun) (-25, -20) (-20, -15) (-15, -10) (-10, -5) (-5, 0) (0, 5) (5, 10) (10, 15) (15, 20) (20, 25) Window Positions 0 1 2 3 4 5 6 7 8 Ratio (Window Ave / Traj Ave) SafetyAntVelocity-v1 [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Ratio of Average Logscore within 5-Timestep Window to Overall Average Logscore [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Ratio of Average Logscore within 5-Timestep Window to Overall Average Logscore [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: Ratio of Average Logscore within 5-Timestep Window to Overall Average Logscore [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 25 canonical work pages

  1. [1]

    Constrained policy optimization

    Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pp.\ 22--31. PMLR, 2017

    work page 2017

  2. [2]

    Constrained markov decision processes with total cost criteria: Lagrangian approach and dual linear program

    Eitan Altman. Constrained markov decision processes with total cost criteria: Lagrangian approach and dual linear program. Mathematical methods of operations research, 48 0 (3): 0 387--417, 1998

    work page 1998

  3. [3]

    Pattern recognition and machine learning, volume 4

    Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006

    2006

  4. [4]

    Safety through feedback in constrained RL

    Shashank Reddy Chirra, Pradeep Varakantham, and Praveen Paruchuri. Safety through feedback in constrained RL . In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=WSsht66fbC

    work page 2024

  5. [5]

    Learning constraints from demonstrations, 2019

    Glen Chou, Dmitry Berenson, and Necmiye Ozay. Learning constraints from demonstrations, 2019

    work page 2019

  6. [6]

    Learning parametric constraints in high dimensions from demonstrations

    Glen Chou, Necmiye Ozay, and Dmitry Berenson. Learning parametric constraints in high dimensions from demonstrations. In Proceedings of the Conference on Robot Learning, pp.\ 1211--1230, 2020

    work page 2020

  7. [7]

    Risk-constrained reinforcement learning with percentile risk criteria

    Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18 0 (167): 0 1--51, 2018

    2018

  8. [8]

    Deep learning, volume 1

    Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016

    2016

Show all 39 references
  1. [9]

    Bullet-safety-gym: A framework for constrained reinforcement learning

    Sven Gronauer. Bullet-safety-gym: A framework for constrained reinforcement learning. Technical report, mediaTUM, 2022

    2022

  2. [10]

    Learning to walk in the real world with minimal human effort

    Sehoon Ha, Peng Xu, Zhenyu Tan, Sergey Levine, and Jie Tan. Learning to walk in the real world with minimal human effort. In Conference on Robot Learning, pp.\ 1110--1120. PMLR, 2021

    2021

  3. [11]

    The elements of statistical learning: data mining, inference, and prediction, 2017

    Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction, 2017

    2017

  4. [12]

    Safety gymnasium: A unified safe reinforcement learning benchmark

    Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong, Josef Dai, and Yaodong Yang. Safety gymnasium: A unified safe reinforcement learning benchmark. In Thirty-seventh Conference on Neural Information Processing Systems Datasets...

    2023

  5. [13]

    Omnisafe: An infrastructure for accelerating safe reinforcement learning research

    Jiaming Ji, Jiayi Zhou, Borong Zhang, Juntao Dai, Xuehai Pan, Ruiyang Sun, Weidong Huang, Yiran Geng, Mickel Liu, and Yaodong Yang. Omnisafe: An infrastructure for accelerating safe reinforcement learning research. Journal of Machine Learning Research, 25 0 (285): 0 1--6, 2024...

    2024

  6. [14]

    Deep reinforcement learning for autonomous driving: A survey

    B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick P \'e rez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 23 0 (6): 0 4909--4926, 2021

    2021

  7. [15]

    Penalizing side effects using stepwise relative reachability

    Victoria Krakovna, Laurent Orseau, Ramana Kumar, Miljan Martic, and Shane Legg. Penalizing side effects using stepwise relative reachability. arXiv preprint arXiv:1806.01186, 2018

    2018 arXiv

  8. [16]

    Steven M. LaValle. Planning Algorithms. Cambridge University Press, 2006

    2006

  9. [17]

    Benchmarking constraint inference in inverse reinforcement learning

    Guiliang Liu, Yudong Luo, Ashish Gaurav, Kasra Rezaee, and Pascal Poupart. Benchmarking constraint inference in inverse reinforcement learning. In International Conference on Learning Representations, 2023

    2023

  10. [18]

    Datasets and benchmarks for offline safe reinforcement learning

    Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu, Tingnan Zhang, Jie Tan, and Ding Zhao. Datasets and benchmarks for offline safe reinforcement learning. Journal of Data-centric Machine Learning Research, 2024

    2024

  11. [19]

    Inverse constrained reinforcement learning

    Shehryar Malik, Usman Anwar, Alireza Aghasi, and Ali Ahmed. Inverse constrained reinforcement learning. In International conference on machine learning, pp.\ 7390--7399. PMLR, 2021

    2021

  12. [20]

    Algorithms for inverse reinforcement learning

    Andrew Y Ng, Stuart Russell, et al. Algorithms for inverse reinforcement learning. In Icml, volume 1, pp.\ 2, 2000

    2000

  13. [21]

    Puterman

    Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., 1st edition, 1994

    1994

  14. [22]

    Benchmarking safe exploration in deep reinforcement learning

    Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708, 7 0 (1): 0 2, 2019

    1910 arXiv

  15. [23]

    Catastrophic forgetting, rehearsal and pseudorehearsal

    Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7 0 (2): 0 123--146, 1995

    1995

  16. [24]

    Avoiding negative side effects due to incomplete knowledge of AI systems

    Sandhya Saisubramanian, Shlomo Zilberstein, and Ece Kamar. Avoiding negative side effects due to incomplete knowledge of AI systems. AI Mag. , 42 0 (4): 0 62--71, 2021

    2021

  17. [25]

    Avoiding negative side effects of autonomous systems in the open world

    Sandhya Saisubramanian, Ece Kamar, and Shlomo Zilberstein. Avoiding negative side effects of autonomous systems in the open world. J. Artif. Intell. Res., 74: 0 143--177, 2022

    2022

  18. [26]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  19. [27]

    Preferences implicit in the state of the world

    R Shah and D Krasheninnikov. Preferences implicit in the state of the world. In International Conference on Learning Representations (ICLR), 2019

    2019

  20. [28]

    Responsive safety in reinforcement learning by pid lagrangian methods

    Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pp.\ 9133--9143. PMLR, 2020

    2020

  21. [29]

    Introduction to reinforcement learning, 1998

    Richard S Sutton, Andrew G Barto, et al. Introduction to reinforcement learning, 1998

    1998

  22. [30]

    Reward constrained policy optimization

    Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. In International Conference on Learning Representations, 2018

    2018

  23. [31]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp.\ 5026--5033. IEEE, 2012. doi:10.1109/IROS.2012.6386109

    2012

  24. [32]

    Avoiding side effects in complex environments

    Alex Turner, Neale Ratzlaff, and Prasad Tadepalli. Avoiding side effects in complex environments. Advances in Neural Information Processing Systems, 33: 0 21406--21415, 2020 a

    2020

  25. [33]

    Conservative agency via attainable utility preservation

    Alexander Matt Turner, Dylan Hadfield-Menell, and Prasad Tadepalli. Conservative agency via attainable utility preservation. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pp.\ 385--391, 2020 b

    2020

  26. [34]

    Minimax-regret querying on side effects for safe optimality in factored markov decision processes

    Shun Zhang, Edmund H Durfee, and Satinder Singh. Minimax-regret querying on side effects for safe optimality in factored markov decision processes. In IJCAI, pp.\ 4867--4873, 2018

    2018

  27. [35]

    Deep reinforcement learning for power system applications: An overview

    Zidong Zhang, Dongxia Zhang, and Robert C Qiu. Deep reinforcement learning for power system applications: An overview. CSEE Journal of Power and Energy Systems, 6 0 (1): 0 213--225, 2019

    2019

  28. [36]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  29. [37]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  30. [38]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  31. [39]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.