REVIEW 3 major objections 6 minor 13 references
Offline Safe Reinforcement Learning Using Trajectory Classification
T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Offline safe RL can be reduced to binary trajectory classification, with one contrastive loss replacing min-max constrained optimization.
desk verdict TraC is a practical trajectory-classification method for offline safe RL with strong DSRL results, but its safety guarantee rests on heuristic labels rather than the objective, and per-environment tuning needs scrutiny. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the contrastive trajectory classifier (TraC). It trains the policy with the loss $$L(\pi_\$\theta$, D) = -\mathbb{E}_{\tau \sim D}\left[ \lambda_d w_\tau y_\tau \log\$\sigma$(\psi(\tau,\pi_\$\theta$)) + \lambda_u w_\tau (1-y_\tau)\log(1-\$\sigma$(\psi(\tau,\pi_\$\theta$))) \right],$$ where $y_\tau$ is the desirability label and $w_\tau$ weights trajectories by normalized return. The score function $$\psi(\tau,\pi)=\sum_{t=0}^T \gamma^t \$\alpha$ \log \frac{\pi(a_t|s_t)}{\pi_{\mathrm{ref}}(a_t|s_t)}$$ carries the argument: it is built from the maximum-entropy identity $A^*_r(s,a)=\alpha\log(\pi^*(a|s)/\pi_{\mathrm{ref}}(a|s))$, so high score means high discounted advantage, which the paper shows is proportional to the log-probability of the trajectory under $\pi^*$. The reference policy $\pi_{\mathrm{ref}}$ is pretrained by behavior cloning on the whole dataset and is what keeps the learned policy near the data, replacing the KL constraint in Eq. (1).
What would settle it
Take a DSRL task, train TraC with the paper's hyperparameters, then evaluate the learned policy under a cost threshold that was not used when building the desirable/undesirable split; if the policy's average normalized cost exceeds 1 while a large fraction of its rolled-out trajectories are still classified as desirable, the assumption that classification score enforces the CMDP constraint is falsified.
Extended reading notes
Core claim
The paper's central claim is that constrained offline RL can be reframed as a contrastive trajectory classification problem. From a pre-collected dataset, the algorithm separates trajectories into weighted desirable and undesirable classes, then trains a policy through the loss in Eq. (11), which is a sigmoid binary classifier over the trajectory score $\psi(\tau, \pi) = \sum_t \gamma^t \alpha \log(\pi/\pi_{\mathrm{ref}})$. The score function is derived from the maximum-entropy identity $A^*_r(s,a) = \alpha \log(\pi^*(a|s)/\pi_{\mathrm{ref}}(a|s))$, so a trajectory's desirability score is the discounted sum of advantages over the behavioral reference policy. The paper argues that this loss implicitly captures the reward objective, the safety constraint, and behavior regularization from Eq. (1), while avoiding the min-max optimization used by prior methods. Empirically, the paper reports that TraC achieves normalized cost below the threshold on all BulletGym and MetaDrive tasks and most SafetyGym tasks, with reward matching or exceeding seven baselines.
Load-bearing premise
The load-bearing premise is that optimizing the classification loss, trained only on labels derived from the offline dataset, also satisfies the original cost constraint on states and transitions that were never seen during training.
Editorial extensions
If this is right
- Trajectory-level classification removes the need for per-step cost constraints and min-max optimization, the two sources of conservatism and instability the paper identifies in earlier methods.
- Because the score function is anchored to a behavior-cloned reference policy, the learned policy stays near the dataset and avoids distribution drift without a separate KL projection step.
- The same framework extends to preference-based settings: the paper shows the classification objective is closely connected to reinforcement learning from human feedback, so any source of desirable/undesirable labels could replace hand-constructed trajectory splits.
- On the DSRL benchmark's 38 tasks, the trained policy satisfies the cost threshold on all BulletGym and MetaDrive tasks and most SafetyGym tasks while matching or beating baselines on reward.
Reading between the lines
- Beyond the paper, if the safety signal really lives only in the trajectory labels, then adjusting the cost threshold should require re-labelling the same offline dataset and re-running TraC, not re-learning value or cost functions; this is testable with the paper's own code and data.
- Beyond the paper, the method's guarantees are bounded by dataset coverage: on states never seen in the offline data, the classifier can assign high scores to trajectories that would violate the cost constraint, so deployment-time monitoring of cost would be a natural safeguard.
- Beyond the paper, the RLHF connection suggests TraC could be turned into a direct human-preference learner by replacing the hand-built reward-based labels with human judgements, which would avoid reward modeling entirely.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes TraC (Trajectory Classification), a two-phase approach for offline safe RL. In the first phase, the offline dataset is partitioned into desirable trajectories (safe, high-return) and undesirable trajectories (unsafe, or safe but low-return) using pre-specified percentiles x% and y% of the safe trajectories. In the second phase, a policy is trained by minimizing a contrastive classification loss (Eq. 11) in which the logit is a score function ψ(τ,π) defined as the discounted sum of the max-entropy advantage A*_r (Eq. 7), which depends on the ratio π/π_ref. The authors claim this loss implicitly captures the CMDP objective, the safety constraint, and the behavioral-regularization term of Eq. (1), and that the resulting policies outperform prior offline safe RL baselines on the DSRL benchmark across 38 tasks in SafetyGym, BulletSafetyGym, and MetaDrive, as well as showing ablations over several hyperparameters.
Significance. The paper addresses a real problem—offline safe RL—and proposes a surprisingly simple alternative to min-max and Q-learning baselines. Its strengths are the breadth of the empirical evaluation (38 tasks, 3 cost thresholds, 3 seeds) and the detailed ablations (x%, y%, δ, η, α, π_ref). If the reported results hold, TraC would be a useful practical tool. However, the paper's theoretical framing overclaims: the score function contains no cost term, and the safety guarantee rests entirely on the heuristic label construction. The lack of a formal connection between Eq. (11) and the constrained objective in Eq. (1) is a substantive gap that must be addressed, either by analysis or by reframing the method as a heuristic with empirical safety validation.
major comments (3)
- [Section 4.2, Eq. (11)] The claim that the loss in Eq. (11) 'implicitly captures the objective, the constraints, and behavior regularization' of Eq. (1) is not justified. The score ψ(τ,π*) in Eq. (7) is defined via the unconstrained reward advantage A*_r from Eq. (6), so the objective contains no cost term; safety information enters only through the binary labels constructed in Sec. 4.1 (top x% of safe trajectories by reward vs. bottom y% plus all unsafe). The paper offers no derivation showing that optimizing Eq. (11) yields a policy satisfying the cumulative cost constraint, and nothing prevents a policy from violating the constraint on states where the classifier generalizes poorly. Since the central claim is the simultaneous achievement of high reward and safety, this is a load-bearing gap. The authors should either prove a connection to the constrained objective (e.g., via a Lagrangian or a cost-aware score) or explicitly reframe TraC as a heuristic whose safety must be empirically established per task.
- [Equations (8)-(10)] The 'score function justification' is imprecise. Equation (10) gives log p(τ;π*) = Σ [A*_r(st,at) + log π_ref(at|st)] up to terms independent of the policy, but the text then says the log-probability is 'directly proportional' to the score Σ γ^t A*_r(st,at). This discards the log π_ref terms, which are not constant when π_ref is a learned behavior-cloning policy, and also discards the discount factors and temperature α that appear in Eq. (7). The relation is therefore not direct proportionality. Please correct the statement or state the exact relationship.
- [Table 4] The main results rely on environment-specific hyperparameters: x% and y% differ across all three domains (50%/0%, 50%/50%, 25%/0%), as do the segment length ratio and η. The sensitivity analyses in Figures 4-5 and Table 2 cover only a few tasks, so the possibility remains that the reported superiority over baselines is partly due to per-environment tuning that baselines were not given. The paper should report results with a single fixed hyperparameter setting, or provide a comprehensive sensitivity analysis across all 38 tasks, to support the claim of general robustness.
minor comments (6)
- [Section 4.1] The notation 'x, y' used in the text conflicts with 'x%, y%' in Figures 4 and 5; please define the notation consistently.
- [Section 4.2 (after Eq. 4)] The constraint λd Nd/(λu Nu)=η is introduced without motivation; a sentence explaining its role would help the reader understand the balancing mechanism.
- [Section 4.2] The symbol f is reused with different meanings: in Eq. (4) it is a scalar trajectory score, while in the TraC formulation it becomes the binary likelihood p(Y=y|τ,π). Please differentiate the notation to avoid confusion.
- [Appendix A.3 / Table 5] Table 5 in the appendix is extremely wide and difficult to read; splitting it into three tables, one per environment, would improve readability.
- [Abstract and Section 4.2] The abstract claims a theoretical connection to human-feedback paradigms, but Section 4.2 only briefly mentions CPL and KTO; please expand the connection or soften the abstract.
- [Appendix B.2] The learning curves in Figures 13-17 are not discussed in the text; a short paragraph summarizing typical convergence behavior would improve the appendix.
Circularity Check
No significant circularity: TraC is a supervised surrogate trained on hand-built reward/cost labels and judged on held-out DSRL returns; the unproven Eq. (11)-to-Eq. (1) link and the log-pi_ref omission in Eq. (10) are soundness gaps, not circular reductions.
full rationale
The claimed derivation chain is not circular. Section 4.1 constructs desirable and undesirable trajectories directly from cumulative reward and cost in the offline data (top x% safe by return vs. bottom y% safe plus unsafe), with weights from Eqs. (2)-(3). The TraC loss (Eq. 11) then trains the policy to score those labels; it is a classification objective, not an algebraic restatement of the constrained CMDP (Eq. 1). The paper asserts rather than proves that the loss 'implicitly captures the objective, the constraints, and behavior regularization' (Sec. 4.2); that missing equivalence is a validity gap, but it is not circular because Eq. (11) is not obtained from Eq. (1) by substituting the same quantities. Similarly, the score-function justification (Eqs. 8-10) attempts to link the advantage-based score to trajectory log-probability; its treatment of the trajectory-dependent log pi_ref term is technically sloppy, but the score is not defined as the training target and the benchmark results are not forced by construction. The only author-overlapping reference (Hoang, Mai, and Varakantham 2024) appears in related work and is not load-bearing; the regret-preference and CPL citations are external. TraC is evaluated against independent DSRL environments with reported normalized reward/cost, making the central claim externally falsifiable rather than equivalent to its inputs.
Assumptions & free parameters
free parameters (7)
- x% (desirable percentile) =
50% (BulletGym), 50% (SafetyGym), 25% (MetaDrive)
- y% (undesirable safe percentile) =
0% (BulletGym), 50% (SafetyGym), 0% (MetaDrive)
- δ (weight lower bound) =
0.7
- η (balancing factor) =
0.25 (BulletGym), 0.5 (SafetyGym), 0.25 (MetaDrive)
- α (temperature) =
0.2 (BulletGym), 0.1 (SafetyGym), 0.2 (MetaDrive)
- segment length ratio =
1.0 (BulletGym), 0.75 (SafetyGym), 0.25 (MetaDrive)
- discount factor γ =
0.99 (BulletGym, SafetyGym), 1.0 (MetaDrive)
assumptions (4)
- domain assumption The regret-based preference model P[τ+≻τ−] = exp(Σ γ^t A*_r(s+,a+)) / (exp(...) + exp(...))
- domain assumption Maximum entropy RL identity A*_r(s,a) = α log(π*(a|s)/π_ref(a|s))
- standard math The transition probabilities cancel when comparing trajectory likelihoods under different policies
- ad hoc to paper Optimizing the classification loss (Eq. 11) yields a policy that solves the CMDP constrained objective (Eq. 1)
Cite this review
Pith. "Pith review of Offline Safe Reinforcement Learning Using Trajectory Classification." pith.science (2026). https://pith.science/paper/37LYZANL
@misc{pith2026241215429,
author = {Pith},
title = {Pith review of: Offline Safe Reinforcement Learning Using Trajectory Classification},
year = {2026},
howpublished = {\url{https://pith.science/paper/37LYZANL}},
note = {Machine review of arXiv:2412.15429}
}
read the original abstract
Offline safe reinforcement learning (RL) has emerged as a promising approach for learning safe behaviors without engaging in risky online interactions with the environment. Most existing methods in offline safe RL rely on cost constraints at each time step (derived from global cost constraints) and this can result in either overly conservative policies or violation of safety constraints. In this paper, we propose to learn a policy that generates desirable trajectories and avoids undesirable trajectories. To be specific, we first partition the pre-collected dataset of state-action trajectories into desirable and undesirable subsets. Intuitively, the desirable set contains high reward and safe trajectories, and undesirable set contains unsafe trajectories and low-reward safe trajectories. Second, we learn a policy that generates desirable trajectories and avoids undesirable trajectories, where (un)desirability scores are provided by a classifier learnt from the dataset of desirable and undesirable trajectories. This approach bypasses the computational complexity and stability issues of a min-max objective that is employed in existing methods. Theoretically, we also show our approach's strong connections to existing learning paradigms involving human feedback. Finally, we extensively evaluate our method using the DSRL benchmark for offline safe RL. Empirically, our method outperforms competitive baselines, achieving higher rewards and better constraint satisfaction across a wide variety of benchmark tasks.
Figures
Figures from the paper (13 more)
Reference graph
Works this paper leans on
-
[2]
In these tasks, agents must reach a goal while avoiding hazards, with task names formatted as {Agent}{Task}{Difficulty}. Additionally, SafetyGymnasium includes five velocity- constrained tasks for the agents, Ant, HalfCheetah, Hopper, Walker2d, and Swimmer. Figure 7 provides a visualization of these tasks in SafetyGymnasium. • BulletSafetyGym is a suite o...
work page 2004
-
[5]
Information asymmetry in KL-regularized RL
Information asymmetry in KL-regularized RL. arXiv preprint arXiv:1905.01240. Garcıa, J.; and Fern´andez, F
work page Pith review arXiv 1905
-
[8]
IEEE transactions on pattern analysis and machine intelligence, 45(3): 3461–3475
Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning. IEEE transactions on pattern analysis and machine intelligence, 45(3): 3461–3475. Liu, Z.; Guo, Z.; Lin, H.; Yao, Y .; Zhu, J.; Cen, Z.; Hu, H.; Yu, W.; Zhang, T.; Tan, J.; et al. 2023a. Datasets and benchmarks for offline safe reinforcement learning. arXiv preprint a...
-
[9]
arXiv preprint arXiv:1910.01708, 7(1):
Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708, 7(1):
arXiv 1910
-
[11]
Model-free safe control for zero-violation reinforcement learning. In CoRL. Zheng, Y .; Li, J.; Yu, D.; Yang, Y .; Li, S. E.; Zhan, X.; and Liu, J. 2024a. Safe offline reinforcement learning with feasibility-guided diffusion model. arXiv preprint arXiv:2401.10700. Zheng, Y .; Li, J.; Yu, D.; Yang, Y .; Li, S. E.; Zhan, X.; and Liu, J. 2024b. Safe Offline ...
-
[2008]
In Aaai, vol- ume 8, 1433–1438
Maximum entropy inverse reinforcement learning. In Aaai, vol- ume 8, 1433–1438. Chicago, IL, USA. A Experimental Details This section outlines the experimental details necessary for reproducing the experiments and the results reported in our paper. A.1 Task Description We conducted experiments using the well-established DSRL benchmark (Liu et al. 2023a), ...
work page 2019
-
[2018]
arXiv preprint arXiv:1805.11074
Reward con- strained policy optimization. arXiv preprint arXiv:1805.11074. Xu, H.; Zhan, X.; and Zhu, X
-
[2019]
Off-policy deep reinforcement learning without exploration. In ICML, 2052–2062. PMLR. Galashov, A.; Jayakumar, S. M.; Hasenclever, L.; Tirumala, D.; Schwarz, J.; Desjardins, G.; Czarnecki, W. M.; Teh, Y . W.; Pascanu, R.; and Heess, N
Show all 13 references
-
[2020]
arXiv preprint arXiv:2004.07219
D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219. Fujimoto, S.; Meger, D.; and Precup, D
2004 arXiv
-
[2021]
arXiv preprint arXiv:2101.10895
A primal-dual ap- proach to constrained markov decision processes. arXiv preprint arXiv:2101.10895. Chow, Y .; Ghavamzadeh, M.; Janson, L.; and Pavone, M
-
[2022]
arXiv preprint arXiv:2205.10330
A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205.10330. Guan, J.; Chen, G.; Ji, J.; Yang, L.; Li, Z.; et al
-
[2023]
arXiv preprint arXiv:2305.09304
Omnisafe: An infrastruc- ture for accelerating safe reinforcement learning research. arXiv preprint arXiv:2305.09304. Kiran, B. R.; Sobh, I.; Talpaert, V .; Mannion, P.; Al Sallab, A. A.; Yogamani, S.; and P´erez, P
-
[2024]
arXiv preprint arXiv:2402.01306
Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306. Fu, J.; Kumar, A.; Nachum, O.; Tucker, G.; and Levine, S
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.