{"id":"90698761-9209-46aa-bb9a-3162b2ea8318","arxiv_id":"2412.15429","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"TraC trains an offline safe RL policy by classifying trajectories as desirable (safe, high-reward) versus undesirable (unsafe or low-reward) using a logistic loss on a policy-ratio score.","lead":"This paper proposes a new offline safe reinforcement learning method, TraC, that trains a policy by classifying past trajectories as desirable or undesirable, avoiding unstable min-max optimization. The authors report that it achieves higher rewards and better safety constraint satisfaction than existing baselines across the DSRL benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never shows that optimizing Eq. (11) solves the constrained CMDP: the score uses an unconstrained reward advantage, so safety rests entirely on heuristic labels.","rationale":"The reader's weakest assumption was exactly that optimizing Eq. (11) with hand-constructed labels is assumed, without proof, to solve the constrained CMDP objective of Eq. (1). My stress-test sharpens this by pointing to the specific mechanism: the score in Eq. (7) is derived from the unconstrained reward advantage A*_r, not from any cost-aware or constrained advantage, so the loss has no built-in safety term. The concrete tabular test would settle whether this theoretical gap translates into actual constraint violation. Because the reader already assigned CONDITIONAL on this basis, my read does not change the verdict; it reinforces the need for the condition to be met before the central claim is accepted.","tokens_in":19435,"tokens_out":5504,"duration_ms":55601,"concrete_test":"Build a minimal tabular CMDP (e.g., two states, two actions per state) where the unconstrained max-entropy optimal policy violates the cost threshold l but a safe constrained optimum exists. Generate an offline dataset from a stochastic behavior policy, apply the Sec. 4.1 label construction with the paper's x and y, then optimize Eq. (11) exactly with a tabular softmax policy until convergence. If the resulting policy's expected discounted cost exceeds l (or differs from the true solution of Eq. (1)), the claimed implicit capture of constraints is false. As a complementary analytic check, write the KKT conditions of Eq. (11) for a tabular policy and compare them with the KKT conditions of Eq. (1); if the former can hold while the cost constraint is violated, the gap is demonstrated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central safety claim depends on the assertion in Sec. 4.2 that the TraC loss 'implicitly captures the objective, the constraints, and behavior regularization' of Eq. (1). No derivation supports this. The score in Eq. (7) is defined through A*_r, the advantage of the unconstrained reward-optimal policy in max-entropy RL (Eq. 6), and contains no cost term. The only safety information enters through the heuristic labels of Sec. 4.1 (top x% of safe trajectories by reward vs. bottom y% of safe trajectories plus all unsafe ones). The 'score function justification' (Eqs. 8-10) proves only that a high score makes a trajectory more probable under the unconstrained optimal policy; it says nothing about cost. Moreover, Eq. (10) drops the trajectory-dependent sum of log pi_ref terms when claiming the log-probability is 'directly proportional' to the score, so even the likelihood interpretation is imprecise. Thus nothing prevents a policy that optimizes Eq. (11) from violating the cost constraint on states or actions where the classifier generalizes poorly, and the claimed bridge from classification to the CMDP solution is unproven. Since the headline result is safety plus reward, this is a load-bearing gap, not merely a missing formal nicety.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes TraC (Trajectory Classification), a two-phase approach for offline safe RL. In the first phase, the offline dataset is partitioned into desirable trajectories (safe, high-return) and undesirable trajectories (unsafe, or safe but low-return) using pre-specified percentiles x% and y% of the safe trajectories. In the second phase, a policy is trained by minimizing a contrastive classification loss (Eq. 11) in which the logit is a score function ψ(τ,π) defined as the discounted sum of the max-entropy advantage A*_r (Eq. 7), which depends on the ratio π/π_ref. The authors claim this loss implicitly captures the CMDP objective, the safety constraint, and the behavioral-regularization term of Eq. (1), and that the resulting policies outperform prior offline safe RL baselines on the DSRL benchmark across 38 tasks in SafetyGym, BulletSafetyGym, and MetaDrive, as well as showing ablations over several hyperparameters.","tokens_in":19733,"tokens_out":5071,"duration_ms":39705,"significance":"The paper addresses a real problem—offline safe RL—and proposes a surprisingly simple alternative to min-max and Q-learning baselines. Its strengths are the breadth of the empirical evaluation (38 tasks, 3 cost thresholds, 3 seeds) and the detailed ablations (x%, y%, δ, η, α, π_ref). If the reported results hold, TraC would be a useful practical tool. However, the paper's theoretical framing overclaims: the score function contains no cost term, and the safety guarantee rests entirely on the heuristic label construction. The lack of a formal connection between Eq. (11) and the constrained objective in Eq. (1) is a substantive gap that must be addressed, either by analysis or by reframing the method as a heuristic with empirical safety validation.","major_comments":[{"comment":"The claim that the loss in Eq. (11) 'implicitly captures the objective, the constraints, and behavior regularization' of Eq. (1) is not justified. The score ψ(τ,π*) in Eq. (7) is defined via the unconstrained reward advantage A*_r from Eq. (6), so the objective contains no cost term; safety information enters only through the binary labels constructed in Sec. 4.1 (top x% of safe trajectories by reward vs. bottom y% plus all unsafe). The paper offers no derivation showing that optimizing Eq. (11) yields a policy satisfying the cumulative cost constraint, and nothing prevents a policy from violating the constraint on states where the classifier generalizes poorly. Since the central claim is the simultaneous achievement of high reward and safety, this is a load-bearing gap. The authors should either prove a connection to the constrained objective (e.g., via a Lagrangian or a cost-aware score) or explicitly reframe TraC as a heuristic whose safety must be empirically established per task.","section":"Section 4.2, Eq. (11)"},{"comment":"The 'score function justification' is imprecise. Equation (10) gives log p(τ;π*) = Σ [A*_r(st,at) + log π_ref(at|st)] up to terms independent of the policy, but the text then says the log-probability is 'directly proportional' to the score Σ γ^t A*_r(st,at). This discards the log π_ref terms, which are not constant when π_ref is a learned behavior-cloning policy, and also discards the discount factors and temperature α that appear in Eq. (7). The relation is therefore not direct proportionality. Please correct the statement or state the exact relationship.","section":"Equations (8)-(10)"},{"comment":"The main results rely on environment-specific hyperparameters: x% and y% differ across all three domains (50%/0%, 50%/50%, 25%/0%), as do the segment length ratio and η. The sensitivity analyses in Figures 4-5 and Table 2 cover only a few tasks, so the possibility remains that the reported superiority over baselines is partly due to per-environment tuning that baselines were not given. The paper should report results with a single fixed hyperparameter setting, or provide a comprehensive sensitivity analysis across all 38 tasks, to support the claim of general robustness.","section":"Table 4"}],"minor_comments":[{"comment":"The notation 'x, y' used in the text conflicts with 'x%, y%' in Figures 4 and 5; please define the notation consistently.","section":"Section 4.1"},{"comment":"The constraint λd Nd/(λu Nu)=η is introduced without motivation; a sentence explaining its role would help the reader understand the balancing mechanism.","section":"Section 4.2 (after Eq. 4)"},{"comment":"The symbol f is reused with different meanings: in Eq. (4) it is a scalar trajectory score, while in the TraC formulation it becomes the binary likelihood p(Y=y|τ,π). Please differentiate the notation to avoid confusion.","section":"Section 4.2"},{"comment":"Table 5 in the appendix is extremely wide and difficult to read; splitting it into three tables, one per environment, would improve readability.","section":"Appendix A.3 / Table 5"},{"comment":"The abstract claims a theoretical connection to human-feedback paradigms, but Section 4.2 only briefly mentions CPL and KTO; please expand the connection or soften the abstract.","section":"Abstract and Section 4.2"},{"comment":"The learning curves in Figures 13-17 are not discussed in the text; a short paragraph summarizing typical convergence behavior would improve the appendix.","section":"Appendix B.2"}],"recommendation":"major_revision","confidential_remarks":"The paper does not mention any public code release, which would be important for reproducibility of a benchmark-based empirical paper. The fit with the journal's scope is appropriate, but the novelty over CPL (Hejna et al., 2024) and KTO (Ethayarajh et al., 2024) is incremental and should be stated more carefully. The main concern remains the missing safety guarantee, which I have raised as a major comment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: TraC is a simple, practical offline safe RL method that swaps min-max optimization for a binary trajectory classifier, and it backs this up with a broad DSRL evaluation. The main caveat is that safety lives in the hand-constructed labels, not in the objective, and the paper never proves otherwise.\n\nWhat's actually new: applying the contrastive preference-style score ψ(τ,π)=Σ γ^t α log(π/π_ref) to offline safe RL at trajectory level, with desirable/undesirable sets built from reward and cost. That is a close cousin of CPL and of the authors' earlier 'imitate the good avoid the bad' work, but the offline safe RL packaging is distinct enough to be useful. The method is refreshingly simple: no min-max, one policy network, and the instability of log-loss on undesirable trajectories is avoided via the sigmoid.\n\nWhat it does well: the empirical work is the main asset. 38 tasks across three environments, three cost thresholds, three seeds, and per-task results in the appendix. The ablations on x%, y%, δ, η, and the πref pretraining show the method is reasonably robust and that the reference policy matters. The safety/reward tradeoff looks favorable against strong baselines, especially in BulletGym and MetaDrive.\n\nSoft spots: the stress-test note is on target. Equation 7's score contains no cost term; all safety information enters through the heuristic top-x%/bottom-y% split. The assertion in Sec 4.2 that the loss 'implicitly captures the objective, the constraints, and behavior regularization' is not derived, and the likelihood justification in Eq 10 drops the Σ log π_ref term, so even that hand-wavy argument is imprecise. This is a real gap, but not fatal: the empirical claim stands on its own, and the authors should either add a formal safety argument or soften the theoretical language.\n\nSecond, hyperparameters are environment-specific (x%, y%, α, η, segment ratio all differ per domain). Without a described validation procedure, there is a selection-on-the-test-set risk. That should be clarified in a revision.\n\nThird, no significance tests; many per-task standard deviations overlap, so the aggregate 'highest reward' claim is shakier than it looks.\n\nBottom line: this paper deserves a serious referee. The idea is practical, the eval is broad, and the flaws are addressable. I'd ask for code release, a validation-based hyperparameter strategy, and a rewritten Sec 4.2 that either proves a safety bound or explicitly presents the method as a heuristic with empirical support.","headline":"TraC is a practical trajectory-classification method for offline safe RL with strong DSRL results, but its safety guarantee rests on heuristic labels rather than the objective, and per-environment tuning needs scrutiny.","tokens_in":20272,"tokens_out":2505,"would_cite":true,"duration_ms":22482,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Offline safe RL can be reduced to binary trajectory classification, with one contrastive loss replacing min-max constrained optimization.","keywords":["offline safe reinforcement learning","trajectory classification","constrained Markov decision process","contrastive preference learning","maximum entropy reinforcement learning","behavior cloning reference policy","DSRL benchmark"],"falsifier":"Take a DSRL task, train TraC with the paper's hyperparameters, then evaluate the learned policy under a cost threshold that was not used when building the desirable/undesirable split; if the policy's average normalized cost exceeds 1 while a large fraction of its rolled-out trajectories are still classified as desirable, the assumption that classification score enforces the CMDP constraint is falsified.","tokens_in":19209,"feed_emoji":"🛡️","tokens_out":9032,"duration_ms":71548,"temperature":0.7,"pith_summary":"Offline safe RL normally couples reward maximization to a constraint, usually via Lagrangian or min-max training that can be unstable and conservative. This paper proposes instead to sort the logged trajectories into 'desirable' and 'undesirable' sets and to train the policy by a binary classification loss over those labels. The central proposal is that the trajectory score $\\psi(\\tau,\\pi)=\\sum_t \\gamma^t \\alpha \\log(\\pi/\\pi_{\\mathrm{ref}})$ makes the classifier simultaneously encode reward, safety, and closeness to the behavioral policy, so the constrained objective in Eq. (1) can be replaced by one loss. The authors support the proposal with a theoretical link to maximum-entropy RL and regret-based preference models, and with DSRL experiments across 38 tasks where TraC reports high reward with normalized cost under 1 on all BulletGym and MetaDrive tasks. If the claim holds, safe policy learning from logged data becomes noticeably simpler: one policy network and a trajectory-labeling step, with no online rollouts and no min-max game.","feed_headline":"Trajectory labels alone make offline RL safe without min-max training","feed_subtitle":"A binary split of logged trajectories into desirable and undesirable sets turns safe RL into a classification problem.","key_machinery":"The load-bearing object is the contrastive trajectory classifier (TraC). It trains the policy with the loss $$L(\\pi_\\$\\theta$, D) = -\\mathbb{E}_{\\tau \\sim D}\\left[ \\lambda_d w_\\tau y_\\tau \\log\\$\\sigma$(\\psi(\\tau,\\pi_\\$\\theta$)) + \\lambda_u w_\\tau (1-y_\\tau)\\log(1-\\$\\sigma$(\\psi(\\tau,\\pi_\\$\\theta$))) \\right],$$ where $y_\\tau$ is the desirability label and $w_\\tau$ weights trajectories by normalized return. The score function $$\\psi(\\tau,\\pi)=\\sum_{t=0}^T \\gamma^t \\$\\alpha$ \\log \\frac{\\pi(a_t|s_t)}{\\pi_{\\mathrm{ref}}(a_t|s_t)}$$ carries the argument: it is built from the maximum-entropy identity $A^*_r(s,a)=\\alpha\\log(\\pi^*(a|s)/\\pi_{\\mathrm{ref}}(a|s))$, so high score means high discounted advantage, which the paper shows is proportional to the log-probability of the trajectory under $\\pi^*$. The reference policy $\\pi_{\\mathrm{ref}}$ is pretrained by behavior cloning on the whole dataset and is what keeps the learned policy near the data, replacing the KL constraint in Eq. (1).","core_discovery":"The paper's central claim is that constrained offline RL can be reframed as a contrastive trajectory classification problem. From a pre-collected dataset, the algorithm separates trajectories into weighted desirable and undesirable classes, then trains a policy through the loss in Eq. (11), which is a sigmoid binary classifier over the trajectory score $\\psi(\\tau, \\pi) = \\sum_t \\gamma^t \\alpha \\log(\\pi/\\pi_{\\mathrm{ref}})$. The score function is derived from the maximum-entropy identity $A^*_r(s,a) = \\alpha \\log(\\pi^*(a|s)/\\pi_{\\mathrm{ref}}(a|s))$, so a trajectory's desirability score is the discounted sum of advantages over the behavioral reference policy. The paper argues that this loss implicitly captures the reward objective, the safety constraint, and behavior regularization from Eq. (1), while avoiding the min-max optimization used by prior methods. Empirically, the paper reports that TraC achieves normalized cost below the threshold on all BulletGym and MetaDrive tasks and most SafetyGym tasks, with reward matching or exceeding seven baselines.","pith_inferences":["Beyond the paper, if the safety signal really lives only in the trajectory labels, then adjusting the cost threshold should require re-labelling the same offline dataset and re-running TraC, not re-learning value or cost functions; this is testable with the paper's own code and data.","Beyond the paper, the method's guarantees are bounded by dataset coverage: on states never seen in the offline data, the classifier can assign high scores to trajectories that would violate the cost constraint, so deployment-time monitoring of cost would be a natural safeguard.","Beyond the paper, the RLHF connection suggests TraC could be turned into a direct human-preference learner by replacing the hand-built reward-based labels with human judgements, which would avoid reward modeling entirely."],"forward_implications":["Trajectory-level classification removes the need for per-step cost constraints and min-max optimization, the two sources of conservatism and instability the paper identifies in earlier methods.","Because the score function is anchored to a behavior-cloned reference policy, the learned policy stays near the dataset and avoids distribution drift without a separate KL projection step.","The same framework extends to preference-based settings: the paper shows the classification objective is closely connected to reinforcement learning from human feedback, so any source of desirable/undesirable labels could replace hand-constructed trajectory splits.","On the DSRL benchmark's 38 tasks, the trained policy satisfies the cost threshold on all BulletGym and MetaDrive tasks and most SafetyGym tasks while matching or beating baselines on reward."],"supporting_citations":[{"why":"contributes the DSRL benchmark, the 38 offline datasets, and the normalized reward/cost metrics the method is evaluated on.","marker":"(Liu et al. 2023a)"},{"why":"supplies the contrastive preference learning paradigm that the paper ties its classifier to.","marker":"(Hejna et al. 2024)"},{"why":"provides the regret-based preference model whose advantage-score form the trajectory score function builds on.","marker":"(Knox et al. 2024)"},{"why":"motivates the lambda_d/lambda_u balance constraints that weight desirable versus undesirable trajectories in the loss.","marker":"(Ethayarajh et al. 2024)"},{"why":"gives the maximum entropy principle behind the advantage-as-log-ratio identity.","marker":"(Ziebart et al. 2008)"},{"why":"formalizes maximum causal entropy, the justification for the score function.","marker":"(Ziebart 2010)"},{"why":"supplies the noise-contrastive estimation perspective used to view the trajectory score as a classifier score.","marker":"(Gutmann and Hyvärinen 2010)"},{"why":"is the CPQ baseline and representative per-step constrained-Q method that TraC is compared against.","marker":"(Xu, Zhan, and Zhu 2022)"},{"why":"is the COptiDICE baseline and distribution-correction approach that TraC is compared against.","marker":"(Lee et al. 2022)"},{"why":"is the CDT baseline, the sequence-modeling approach whose high rewards come from unsafe tasks in the comparison.","marker":"(Liu et al. 2023b)"}],"fun_headline_variants":["Safe offline RL by classifying good from bad trajectories","Trajectory classification eliminates min-max in offline RL","Offline RL safety via trajectory labels, not min-max","Classify trajectories to learn safe offline policies","Turn offline RL safety into a classification problem"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that optimizing the classification loss, trained only on labels derived from the offline dataset, also satisfies the original cost constraint on states and transitions that were never seen during training.","fun_headline_variants_meta":{"raw":{"variants":["Safe offline RL by classifying good from bad trajectories","Trajectory classification eliminates min-max in offline RL","Offline RL safety via trajectory labels, not min-max","Classify trajectories to learn safe offline policies","Turn offline RL safety into a classification problem"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000209,"raw_usage":{"total_tokens":1436,"prompt_tokens":1001,"completion_tokens":435,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":363}},"tokens_in":617,"tokens_out":435,"duration_ms":3954,"temperature":1.0,"reasoning_tokens":363,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:26:02.231909+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a DSRL task, train TraC with the paper's hyperparameters, then evaluate the learned policy under a cost threshold that was not used when building the desirable/undesirable split; if the policy's average normalized cost exceeds 1 while a large fraction of its rolled-out trajectories are still classified as desirable, the assumption that classification score enforces the CMDP constraint is falsified.","supporting_citations":[],"review_version":1}