{"id":"c2a7ae11-03bc-4429-996f-8984493e2b1e","arxiv_id":"2504.12557","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"TraCeS factorizes a learned trajectory safety probability into per-timestep multiplicative scores and uses their log-sum as a constraint in PPO-Lagrangian, producing safe continuous-control policies without known costs or budgets.","lead":"The paper proposes TraCeS, which learns per-step safety scores from only whole-trajectory safe/unsafe labels, then uses them to steer reinforcement learning agents toward safe behavior. A generalist might care because many real systems cannot provide per-action safety costs, only coarse approval or rejection of entire rollouts.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 3.4's safety guarantee depends on identifiability of per-step credits and on generalization under policy shift; neither is established, and the promised approximation-gap analysis is absent.","rationale":"The reader's weakest assumption correctly identifies the convergence and distribution-shift step in Lemma 3.4. I agree that this is a load-bearing link, but I would sharpen it: the more fundamental gap is identifiability of the per-step factorization from trajectory-level labels, together with the missing approximation-gap analysis promised in the abstract. If per-step credits are not identifiable, then even perfect MLE convergence on the training distribution does not yield the correct P̂Δt, and the reformulated constraint can be satisfied by a model whose credit assignment is wrong. The paper's empirical credit-assignment figures (Figure 3 and Figure 4) provide some evidence that the learned scores align with true cost spikes, but these are selected trajectories and do not settle identifiability. The concern does not warrant rejection because the method is clearly implemented and evaluated on standard benchmarks, the reported mean costs are below budget on all tasks, and the flaws are addressable with additional experiments and a released implementation. Therefore the conditional verdict stands, with the condition expanded to include identifiability checks and an approximation-gap evaluation.","tokens_in":16421,"tokens_out":9366,"duration_ms":107874,"concrete_test":"Run the SSV training on a synthetic CMDP with known per-step costs, where the only supervision is one binary label per full trajectory (no prefix labels). Compare the inferred P̂Δt against the true cost at each timestep (e.g., rank correlation or localization of the violation step). If the inferred credit does not reliably identify the true violation timestep across trajectories, then the per-timestep credit claim is not supported by trajectory-level labels alone. Additionally, on the final policies for all twelve tasks, report E_{πθ}[Π P̂Δt] and E_{πθ}[Ψ(τ)] on the held-out evaluation trajectories; if the undiscounted product constraint is violated while the discounted-log surrogate is satisfied, the reformulation in Eq. (10) is not conservative with respect to the intended safety constraint.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central safety claim rests on Lemma 3.4, which replaces the trajectory-level constraint E[Ψ(τ)] ≥ d with the surrogate E[Σ γ^t log P̂Δt] ≥ log d. Three gaps compound at this step. First, the proof only argues that P̂(ψ|τ) converges to Ψ(τ) on the training distribution as the dataset grows; the constraint is evaluated under the online policy's rollout distribution, and the paper gives no bound or calibration evidence for E_{πθ}[P̂] − E_{πθ}[Ψ] under that shift. Second, Proposition 3.3 and Eq. (6) define the per-step factors P̂Δt only through the product matching whole-trajectory labels. Unless the dataset provides labels for many prefixes of the same trajectory, infinitely many per-step credit assignments produce the same BCE loss, so the 'per-timestep violation credit' is not identifiable from a single binary label per trajectory segment. Section 2.2 states only that a trajectory segment receives one label, and the experiments' mention of labeling 'a variable-length trajectory segment' does not clarify whether prefixes are labeled. Third, the abstract promises a theoretical analysis of the approximation gap, but the text contains no such finite-sample or distribution-shift analysis; the lemmas are algebraic identities and an asymptotic consistency argument. The empirical support is also weaker than the headline: costs like Walker2d at 24.1 ± 3.9 are within one standard deviation of the budget, and rewards are 13–21% below PPO-Lagrangian on several tasks, so 'safe across all twelve tasks' is an average-cost statement rather than a demonstrated safety guarantee.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TraCeS, a method for learning per-timestep safety credit from binary trajectory-level labels when both the cost function and budget are unknown. The method factorizes an estimated trajectory safety probability into per-step multiplicative scores (Proposition 3.3), trains these scores with a binary cross-entropy loss, and reformulates the constrained RL problem using a discounted sum of log safety scores (Eq. 8). The authors claim that the resulting policy is safe across all twelve evaluated continuous-control tasks with return close to an oracle PPO-Lagrangian baseline. Theoretical support is claimed through Lemma 3.4 and a Jensen-based lower bound, and empirical evaluation is conducted on MuJoCo, Safety Gymnasium, and Bullet Safety Gym tasks.","tokens_in":16725,"tokens_out":7106,"duration_ms":68331,"significance":"If the claims hold, the paper addresses a practically important setting: safe RL without a known cost function or budget, relying only on sparse binary trajectory feedback. The proposed factorization is simple and the integration with PPO-Lagrangian is straightforward, making the method potentially useful for applications where dense safety feedback is unavailable. The paper also includes a credit-assignment visualization that could aid interpretability. However, the theoretical contribution is limited to an asymptotic consistency argument and an algebraic identity; the promised approximation-gap analysis is absent. The empirical support is suggestive but not as strong as the headline claims: several tasks have average costs close to the budget, and the reward gap relative to the oracle baseline is substantial on some tasks. The comparison with RLSF is also confounded by RLSF's assumption of a known budget. Overall, the idea is promising but the current manuscript does not fully substantiate its central safety and credit-assignment claims.","major_comments":[{"comment":"The safety guarantee is only asymptotic and does not address distribution shift. The proof of Lemma 3.4 argues that as the training sample size tends to infinity, the learned model's probability estimate approximates the true labeling function. However, the constraint in Eq. (10) is evaluated under the current policy's rollout distribution, which differs from the offline pretraining distribution and changes during continual learning. No finite-sample bound or calibration evidence is provided for E_{πθ}[P̂] vs. E_{πθ}[Ψ] under this shift. Furthermore, the abstract promises a theoretical analysis of the approximation gap, but the manuscript contains no such finite-sample or distribution-shift analysis; the lemmas are algebraic identities and an asymptotic consistency statement. This is a load-bearing issue because the safety of the final policy depends on the surrogate constraint being a valid lower bound for the true constraint under the online distribution.","section":"Section 3.2.2, Lemma 3.4 and Eqs. (7)-(10)"},{"comment":"The per-timestep safety scores are not identifiable from the available supervision. The factorization in Eq. (5) is a tautology: P̂^Δ_t is defined as the ratio of subsegment safety probabilities, so the product equals the trajectory-level probability by construction. Training with the binary cross-entropy loss in Eq. (6) constrains only the product, not the individual factors. Section 2.2 states that a trajectory segment receives a single binary label, and the experimental description says labels are given to a variable-length trajectory segment without clarifying whether prefixes are labeled separately. Consequently, infinitely many decompositions achieve the same loss, and the interpretation of P̂^Δ_t as the per-timestep cost contribution is an inductive-bias assumption rather than a learned, identifiable quantity. The credit-assignment analysis in Figures 3 and 4 cannot rule out that the observed spikes are artifacts of the chosen architecture and the monotonicity assumption (Assumption 3.6).","section":"Section 3.2.1, Proposition 3.3 and Eqs. (4)-(6)"},{"comment":"The claim that 'the policy trained using our SSV model is safe across all twelve tasks' is not fully supported by the reported statistics. Several tasks have mean costs within one standard deviation of the budget of 25: Walker2d reports 24.1 ± 3.9, BallRun 24.7 ± 0.8, and CarRun 23.3 ± 1.1. For Walker2d, the mean plus one standard deviation exceeds the budget, so a nontrivial fraction of evaluation seeds likely violate the constraint. Additionally, the reward gap relative to PPO-Lagrangian is substantial on some tasks (HalfCheetah 2372 vs. 3008, a 21% decrease; Walker2d 2211 vs. 2682, an 18% decrease), so the statement 'total return close to PPO-Lagrangian' overstates the match. The comparison with RLSF is also not apples-to-apples because RLSF assumes a known budget; the paper should temper the empirical claims accordingly.","section":"Section 4, Tables 1 and 2"}],"minor_comments":[{"comment":"The sentence 'trajectory is only labeled safe if the true cost incurred is above the threshold' appears to be a typo; it should read 'below or equal to the threshold' to match the definition of Ψ(τ)=1 when cost is within budget.","section":"Section 2.2"},{"comment":"The acronym TraCeS is defined differently in the arXiv abstract ('Trajectory-based Constraint Estimation for Safety') and in Section 1 ('trajectory based credit assignment for safe RL'). Please use one consistent definition.","section":"Title and Section 1"},{"comment":"Equation (6) is missing a closing parenthesis: the expectation should be written as E_{τi∼D}[ψi log P̂(ψi=1|τi) + (1−ψi) log(1−P̂(ψi=1|τi))].","section":"Eq. (6)"},{"comment":"The caption and legend of Figure 3 should explicitly define the green curve, which the text describes as the normalized inferred cost from the SSV model; currently the caption mentions only the red and blue curves.","section":"Section 4 and Figure 3"},{"comment":"The coefficient of variation formula uses X for the lognormal random variables, but the text says the safety score log P̂^Δ_t is sampled from the distribution; please clarify the relationship between X and log P̂^Δ_t so the notation is consistent.","section":"Appendix C.2, Eq. (12)"}],"recommendation":"major_revision","confidential_remarks":"The paper is a workshop submission and would benefit from a clearer statement of novelty relative to RLSF. The main concern is that the theoretical claims in the abstract are not delivered in the body, and the identifiability of the per-step credit is not established. If revised, the authors should either provide a finite-sample or distribution-shift bound, or explicitly reframe the contribution as an empirical method and soften the theoretical claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a serious read if you work on safe RL with sparse feedback. The concrete contribution is real: TraCeS removes the two things RLSF still needs (known budget and number of constraints) and learns per-step safety credit from whole-trajectory binary labels. The architecture – a summary vector plus a decoder that outputs per-timestep log-safety scores – is sensible and the empirical scope is respectable: twelve continuous-control tasks, comparisons to an oracle and to RLSF, and a selective-feedback variant that cuts labeling load. The credit-assignment plots (Figures 3 and 4) do show the inferred scores peaking near the true budget crossing, which is more than many cost-learning papers show.\n\nThe paper is honest about its central algebra: Proposition 3.3 is a telescoping product identity, not a deep theorem. That is fine; the value is in the pipeline around it. The Jensen step in the reformulation is directionally correct. The real soft spot is the theory. Lemma 3.4 is just an asymptotic MLE argument with no finite-sample bound and no treatment of distribution shift from the offline pretraining distribution to the online policy rollouts. The identifiability concern is also real: a single binary label per trajectory cannot pin down per-step factors, and infinitely many decompositions give the same BCE loss. The paper does not discuss this. And the v2 abstract promises a theoretical analysis of the approximation gap; the full text does not deliver it. That is a mismatch that should be fixed.\n\nThe empirical safety claim is stated too strongly. Costs like Walker2d 24.1 ± 3.9, BallRun 24.7 ± 0.8, and AntRun 21.4 ± 1.9 sit within one standard deviation of the budget, so “safe across all twelve tasks” is an average-cost statement, not a demonstrated safety guarantee. Rewards are also 13–21% below PPO-Lagrangian on several MuJoCo tasks, which is acceptable but should be described as a trade-off. No code is released, which makes the quantitative results harder to verify.\n\nA serious referee would be able to improve this paper: the experiments and the architecture are worth preserving, but the abstract, the safety language, and the theoretical claims need revision. I would send it to peer review at workshop level, and I would cite it in my own work on learning constraints from binary feedback, with the caveat that the per-step credits are a useful heuristic rather than an identified quantity.","headline":"A solid, incremental safe-RL method for learning constraints from binary trajectory labels, but the advertised theory is thinner than the abstract suggests and the safety claim is average-cost rather than a guarantee.","tokens_in":17276,"tokens_out":2650,"would_cite":true,"duration_ms":29942,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Whole-trajectory safe/unsafe labels suffice to learn per-step safety credit and train safe policies.","keywords":["safe reinforcement learning","constraint inference","credit assignment","trajectory-level safety labels","safety summary vector","PPO-Lagrangian","sparse feedback","continuous control"],"falsifier":"Take the final trained policy, run 1,000 new trajectories, and compare the model's predicted safety probability with the true safe/unsafe label. If trajectories the model calls almost certainly safe turn out to be unsafe much more often than the model predicts, the learned credit is not faithful. A sharper test uses a task where an early violation can be repaired by later actions: the true label would be safe once the total cost falls back below budget, but the model's safety probability can only stay the same or decrease, so it must either miscredit the repair or force the policy to be overly conservative.","tokens_in":16209,"feed_emoji":"🛡️","tokens_out":12480,"duration_ms":108611,"temperature":0.7,"pith_summary":"This paper asks whether an agent can learn to be safe when the only safety feedback is a binary label attached to a whole trajectory, and when the underlying cost function and budget are unknown. It proposes TraCeS, which trains a per-timestep credit score for each state-action pair so that the product of the scores equals the model's predicted probability that the trajectory is safe. That score is used as a per-step pseudo-cost, and the safe-reinforcement-learning problem is reformulated as maximizing reward subject to a lower bound on the discounted sum of log-scores. The paper reports that on twelve continuous-control tasks with hidden constraints, policies trained this way are safe in all twelve and achieve total return close to an oracle method that knows the true cost and budget. If correct, this means coarse approvals and rejections of whole rollouts are enough supervision to train safe control policies.","feed_headline":"Coarse safe/unsafe labels can train safe control policies","feed_subtitle":"TraCeS learns per-step violation credit from trajectory-level binary feedback and matches oracle PPO-Lagrangian on 12 tasks.","key_machinery":"The load-bearing object is the Safety Summary Vector (SSV) model. A recurrent network $f_w$ compresses the trajectory prefix into a summary vector $h_{t+1}=f_w(s_t,a_t,h_t)$, and a decoder compares $h_t$ with $h_{t+1}$ to output the per-step multiplicative safety score $\\hat P^\\Delta_t$. By the telescoping identity of Proposition 3.3, $\\prod_{t=0}^{T-1} \\hat P^\\Delta_t = \\hat P(\\psi=1\\mid\\tau)$, so taking logs converts the product into a sum of per-step pseudo-costs. Jensen's inequality then turns the desired safe-trajectory rate $\\mathbb{E}[\\Psi(\\tau)]\\ge d$ into the tractable constraint $\\mathbb{E}[\\sum_t \\gamma^t \\log \\hat P^\\Delta_t]\\ge \\log d$, which PPO-Lagrangian can enforce. A decoder that outputs a negative lognormal distribution for each log-score lets the method use the coefficient of variation of the total score to choose which trajectories are worth labeling, reducing the human feedback needed.","core_discovery":"The paper's central claim is that per-timestep safety credit can be recovered from sparse, trajectory-level binary labels by factorizing the estimated probability that a trajectory is safe. If the model outputs one multiplicative score $\\hat P^\\Delta_t$ per step, the product over the trajectory telescopes to $\\hat P(\\psi=1\\mid\\tau)$, the estimated probability that the whole trajectory is safe, and each score is interpreted as the change in safety caused by that step. TraCeS turns these scores into a discounted-sum constraint $\\mathbb{E}[\\sum_t \\gamma^t \\log \\hat P^\\Delta_t] \\ge \\log d$, a Jensen lower bound on requiring at least a fraction $d$ of trajectories to be safe, and solves the resulting program with PPO-Lagrangian, a standard constrained-RL method. In twelve continuous-control benchmarks where both cost function and budget are kept hidden, the paper reports that the policy trained with its SSV model is safe on every task and that total return stays close to PPO-Lagrangian with full knowledge, while using fewer labeled trajectories than the RLSF baseline in several domains.","pith_inferences":["Inference: the same factorized score could serve as a diagnostic, since the log-score spike identifies the first moment an unknown threshold is crossed; that localization is useful for auditing demonstration data even without policy optimization.","Inference: because the reformulated constraint is a Jensen lower bound, TraCeS is conservative when the model is well calibrated, but a natural safeguard is to re-check predicted safety probabilities against true labels on held-out rollouts and retrain when calibration drifts.","Inference: the monotone non-increasing form of $\\hat P(\\psi=1\\mid\\tau)$ matches constraints with non-negative costs where violations cannot be undone; reversible violations would need a different decomposition that allows the safety probability to rise again.","Inference: the selective-feedback rule suggests a general recipe for other human-in-the-loop settings: label only trajectories with high model uncertainty, and per-step credit assignment from coarse labels remains reliable."],"forward_implications":["Users who cannot provide a cost function, a budget, or per-state costs can still obtain a constrained policy from whole-trajectory yes/no labels.","Because the safety scores factorize, standard constrained-RL solvers such as PPO-Lagrangian need only a state augmented with a summary vector; no new safe-RL algorithm is required.","The inferred credit spikes when accumulated true cost crosses the hidden budget, so the method localizes the moment an unknown constraint is violated.","Selecting trajectories for labeling by coefficient of variation cuts the number of labels needed without substantially changing final reward or safety.","Continual retraining on in-distribution trajectories keeps the safety estimate usable as the policy distribution shifts during online learning."],"supporting_citations":[{"why":"Supplies the PPO objective that TraCeS extends into a Lagrangian constrained optimization.","marker":"Schulman et al., 2017"},{"why":"Defines PPO-Lagrangian, the oracle baseline and the concrete safe-RL solver TraCeS builds on.","marker":"Tessler et al., 2018"},{"why":"RLSF is the closest baseline: it learns safety credit from trajectory feedback but still requires a known safety budget.","marker":"Chirra et al., 2024"},{"why":"Safety Gymnasium defines the circle benchmark tasks and the binary-cost, budget-25 evaluation protocol.","marker":"Ji et al., 2023"},{"why":"Bullet Safety Gym supplies the four run tasks and the non-negative cost convention used in experiments.","marker":"Gronauer, 2022"},{"why":"Provides the PPO-Lagrangian implementation used for both the baselines and TraCeS.","marker":"Ji et al., 2024"},{"why":"Supplies the offline safe-RL dataset used to pre-train the constraint models.","marker":"Liu et al., 2024"},{"why":"Grounds the binary cross-entropy loss as maximum likelihood, the convergence argument behind Lemma 3.4.","marker":"Bishop & Nasrabadi, 2006"}],"fun_headline_variants":["Per-step safety credit from sparse trajectory-level labels","TraCeS turns coarse trajectory verdicts into per-step safety credit","Recover per-timestep safety credit from sparse trajectory-level labels","Step-wise safety from whole-trajectory approvals and rejections","No cost function needed: TraCeS learns per-step violation from trajectory labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the model's estimate of the probability that a trajectory is safe stays accurate on the states and actions the policy visits during online training, even though the model was trained beforehand on a fixed dataset and the policy's behavior keeps changing; if the model overestimates safety in those new situations, the constraint it enforces is weaker than the intended one.","fun_headline_variants_meta":{"raw":{"variants":["Per-step safety credit from sparse trajectory-level labels","TraCeS turns coarse trajectory verdicts into per-step safety credit","Recover per-timestep safety credit from sparse trajectory-level labels","Step-wise safety from whole-trajectory approvals and rejections","No cost function needed: TraCeS learns per-step violation from trajectory labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000663,"raw_usage":{"total_tokens":3035,"prompt_tokens":956,"completion_tokens":2079,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":1990}},"tokens_in":572,"tokens_out":2079,"duration_ms":15056,"temperature":1.0,"reasoning_tokens":1990,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:29:09.705882+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the final trained policy, run 1,000 new trajectories, and compare the model's predicted safety probability with the true safe/unsafe label. If trajectories the model calls almost certainly safe turn out to be unsafe much more often than the model predicts, the learned credit is not faithful. A sharper test uses a task where an early violation can be repaired by later actions: the true label would be safe once the total cost falls back below budget, but the model's safety probability can only stay the same or decrease, so it must either miscredit the repair or force the policy to be overly conservative.","supporting_citations":[{"cited_title":"Reward constrained policy optimization","cited_arxiv_id":null,"evidence_quote":"Defines PPO-Lagrangian, the oracle baseline and the concrete safe-RL solver TraCeS builds on."},{"cited_title":"Safety through feedback in constrained RL","cited_arxiv_id":null,"evidence_quote":"RLSF is the closest baseline: it learns safety credit from trajectory feedback but still requires a known safety budget."}],"review_version":1}