{"id":"c37702b7-d422-4681-ac7e-b71e8d1376e5","arxiv_id":"2412.08920","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A text-trajectory contrastive model with per-step cost assignment reduces safety violations in reinforcement learning agents under natural language constraints.","lead":"This paper trains a model to read a safety rule in plain language, watch a robot's movement, and give it a live safety score instead of only a final pass or fail. The authors show that robots trained with these predicted scores break rules less often than robots trained with the usual final penalty.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The cost assignment loss in Eq. (11) is unsupervised: \\hat{C}(y) is never trained against the true violation label, so the dense per-step costs may be arbitrary and the reported improvement over ground-truth cost may reflect generic dense penalty rather than textual comprehension.","rationale":"The reader's weakest_assumption focuses on the dependency on manually designed violation-checking functions and template-generated constraints, which is a valid concern about generalization and about the claim of replacing manually designed cost functions. I agree that this limits the evidence for free-form use. However, the single most load-bearing concern for the paper's central empirical claim is internal: the cost assignment component, which is credited with the main performance gains, is trained with a loss that has no ground-truth target for the episodic cost. This makes the dense cost predictions formally degenerate and potentially arbitrary. The paper does not provide any quantitative validation that the learned per-step costs correlate with actual risk; only a qualitative case study is shown. If this degeneracy is real, the headline result (CP mode beating GC mode) could be produced by an unconstrained dense cost signal rather than by textual comprehension, undermining the interpretation of the experiments. This is not an attack on the authors' integrity; it is a precise technical flaw in the loss design that can be tested. The reader did not identify this issue, hence 'partial' agreement. My recommended verdict remains CONDITIONAL, matching the reader's overall assessment, but the condition should be expanded to require either a supervised target for \\hat{C}(y) or clear evidence that the learned dense costs are calibrated to constraint risk. Since I do not propose a different final verdict, 'UNCHANGED' is appropriate.","tokens_in":23164,"tokens_out":10652,"duration_ms":123052,"concrete_test":"Retrain TTCT with a modified cost assignment loss that includes a supervised target: add a binary cross-entropy (or MSE) term training \\hat{C}(y) against the true violation label (1 if the trajectory violates y, 0 otherwise), while keeping all other components and hyperparameters identical. Then compare (a) whether the learned \\hat{C}(y) values separate violating from non-violating trajectories on a held-out set, and (b) whether the downstream policy violation rates change substantially. If the original \\hat{C}(y) is uninformative or if the policy results depend on adding the supervised term, the original CA component is degenerate and the reported improvement over ground-truth cost must be re-interpreted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that policies trained with TTCT's predicted cost achieve lower violation rates than those trained with the standard cost function, and that this follows from TTCT's ability to comprehend textual constraints and trajectories. The cost assignment component (Section 4.2) is presented as a key ingredient that provides dense, meaningful per-step costs. However, the loss in Eq. (11), L_CA = E[(\\sum_{t=1}^{T-1} \\hat{c}(s_t,a_t,y,\\tau_t) - \\hat{C}(y))^2], only enforces consistency between the sum of per-step costs and the episodic-cost prediction \\hat{C}(y). There is no supervised term tying \\hat{C}(y) to the actual violation indicator, which is available in the offline dataset. The pseudocode in Figure 9 confirms this: EpCost is computed from the text embedding, and loss_CA is only the MSE between EpCost and sum_cost. The global minimum of L_CA can therefore be achieved by any constant prediction (e.g., \\hat{C}(y)=0 and all \\hat{c}_t=0), so the per-step costs are unconstrained by violation status. In test time (Eq. 13), these arbitrary dense costs are used whenever the alignment score is below the threshold, meaning the policy is optimized against a cost signal that could be unrelated to the textual constraint. The ablation study comparing full TTCT to TTCT without the CA component (Section 6.3) indicates that the component is necessary for the best results, but if its training is degenerate, the observed improvement may be caused by an uncalibrated dense penalty rather than by a genuine decomposition of the constraint. This directly threatens the paper's claim that TTCT 'effectively comprehends textual constraint and trajectory' and that the lower violation rate is due to that comprehension. The qualitative case study in Appendix B.2 does not resolve this because no quantitative calibration of the learned costs against risk is provided.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TTCT, a framework that uses natural language constraints as both the task specification and the training signal for safe reinforcement learning. It has two main components: a text-trajectory alignment component trained with contrastive and within-trajectory losses to predict whether a trajectory violates a given textual constraint, and a cost assignment component that decomposes an episodic cost prediction into per-step costs. TTCT is trained on offline data with template-generated constraints and then used as the cost provider in several safe RL algorithms. Experiments on Hazard-World-Grid and SafetyGoal show that policies trained with TTCT's predicted costs achieve lower violation rates than those trained with ground-truth cost functions, and a zero-shot transfer experiment to LavaWall is reported.","tokens_in":23458,"tokens_out":5744,"duration_ms":57966,"significance":"If validated, the work addresses a practical limitation of safe RL: the need for manually designed cost functions. The dual use of text as both constraint and supervision is a clean idea, and the paper includes a reasonable set of experiments covering multiple algorithms, ablations, and a zero-shot transfer. The reported AUC of 0.98 for violation prediction is strong. However, the central claim that the cost assignment component provides meaningful dense costs is undermined by the absence of a supervised training signal for the episodic cost prediction, and the lack of an empirical comparison with the closest prior methods [14, 27] leaves the contribution's advantage unquantified.","major_comments":[{"comment":"The cost assignment loss L_CA is a consistency loss between the predicted episodic cost \\hat{C}(y) and the sum of per-step costs. Since \\hat{C}(y) is computed from the text embedding only and is never trained against the true episodic cost (0 or 1) that is available in the offline dataset, the global minimum includes the trivial solution \\hat{C}(y)=0 and all \\hat{c}_t=0. The dense costs used in Eq. (13) are therefore not anchored to violation status. Please add a supervised loss for \\hat{C}(y) against the true episodic cost and report the resulting cost calibration, or otherwise provide quantitative evidence that the per-step costs correlate with actual violation risk.","section":"Section 4.2, Eq. (11)"},{"comment":"The experimental section does not compare TTCT with the closest prior natural-language cost methods, namely [14] (Safe RL with natural language constraints) and [27] (Safe RL with free-form natural language constraints and pre-trained language models). Because the paper's central claim is that TTCT outperforms the standard cost function and prior approaches, a direct comparison on the same tasks is necessary to support this claim.","section":"Section 6 (Baselines, Table 5)"},{"comment":"All constraints are generated from predefined templates (Appendix A.1) and require hand-written violation checkers. The paper claims to handle 'total free-form and complex textual constraints' (Conclusion), but the experiments do not test with human-written, unseen constraint phrasings. The zero-shot transfer to LavaWall also uses constraints from the same template distribution. Please clarify the scope and include an evaluation with more diverse constraints to support the generalization claim.","section":"Appendix A.1, Conclusion"},{"comment":"Equation (13) uses a single threshold beta determined from the ROC curve of the full-trajectory violation prediction (Appendix B.1). During policy training, the same threshold is applied to prefixes of trajectories at every time step, even though the alignment component was trained on complete trajectories up to the violation time. The appropriateness of this threshold for prefixes is not analyzed. Please provide an analysis or ablation of the threshold choice for intermediate time steps.","section":"Section 5, Eq. (13), Appendix B.1"}],"minor_comments":[{"comment":"The text states that the final embedding H_T is used as the representation for the entire trajectory, but the cost assignment component in Section 4.2 also uses all intermediate embeddings H_t; this distinction could be clarified.","section":"Section 4.1"},{"comment":"The caption says 'The blue bars are our proposed cost prediction (CP) mode performance and the orange bars are the ground-truth cost (GC) mode performance,' but the explanation of the black dashed line is given only in the text; adding it to the caption would aid readability.","section":"Figure 3 caption"},{"comment":"The header 'PPO' appears without a corresponding 'GC'/'CP' split, and the table's shaded cells are not visible in the text; please ensure the table is self-contained.","section":"Table 5"},{"comment":"The threshold beta is described as the best cutoff value of the ROC curve, but the ROC curve appears to be computed on the test set; please specify whether the threshold is chosen on a validation set to avoid optimistic bias.","section":"Appendix B.1"},{"comment":"The paper mentions 'up to 4.0x' lower violation rates in the introduction, but the reported numbers in Table 5 show ratios around 2-3x; please reconcile these figures.","section":"Introduction"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly written and the idea is interesting. The main concern is whether the cost assignment component truly learns meaningful per-step costs, given the unsupervised consistency loss. The empirical comparison with prior work [14, 27] is essential. The template-generated constraints limit the generality of the claims. If the authors can fix the cost assignment supervision and add the missing comparison, the paper could be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper's alignment component is a real contribution, but the cost assignment component is not supervised by any violation label, so the paper's best result—lower violation rates than ground-truth cost—may just be a generic dense-penalty effect, not the result of textual comprehension.\n\nThe problem setup is genuinely new: trajectory-level textual constraints with multiple entities, temporal order, and quantitative logic, and the paper is right that prior work only handles single state/entity constraints. The contrastive text-trajectory alignment with the within-trajectory loss is a sensible way to get a violation predictor, and the numbers are believable: AUC 0.98, and the zero-shot transfer to LavaWall is a nice extra. The authors also test across four safe RL algorithms and include ablations, which is more than most papers in this area do.\n\nThe soft spots are real. First, no empirical comparison with Yang et al. [14] or Lou et al. [27], the two closest prior methods. The paper argues those can't handle trajectory-level constraints, but a direct comparison would clarify what the new machinery buys. Second, the constraints are all template-generated and the violation checkers are hand-written, so the 'free-form' claim is overstated; the real bottleneck is whether the checkers transfer to new constraints. Third—and this is the one I'd focus on in review—the cost assignment loss L_CA in Eq. (11) only enforces consistency between the predicted episodic cost and the sum of per-step costs. \\hat{C}(y) is never trained against the actual violation indicator, and the paper explicitly stops gradients from L_CA into the encoders. So a constant prediction is a global optimum of that term. The attention scores are trained by the WT loss, so the per-step costs are probably correlated with risk, but the cost magnitudes are uncalibrated. The ablation comparing full TTCT to TTCT without CA doesn't rule out that any dense penalty would help. Add a dense-penalty baseline, or supervise \\hat{C}(y) with the violation labels you already have, before claiming the CA component understands the constraint.\n\nI'd send this to review—the problem and the alignment approach deserve attention—but the authors should be asked to address the CA supervision and add the missing baselines.","headline":"The alignment part is a real step forward, but the cost assignment component is trained without any violation label, so the dense-cost advantage may be generic rather than semantic.","tokens_in":24080,"tokens_out":2924,"would_cite":false,"duration_ms":33073,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces TTCT, which turns natural-language safety constraints into per-step costs for reinforcement learning, and reports that policies trained on its predicted costs violate constraints less often than policies trained on…","keywords":["safe reinforcement learning","natural language constraints","trajectory-level textual constraints","contrastive learning","text-trajectory alignment","cost assignment","zero-shot transfer","constrained Markov decision process"],"falsifier":"Train TTCT only on the paper's template-generated constraints, then evaluate it on a held-out set of genuinely free-form natural-language constraints written by humans (not templates) with manually verified trajectory labels; if violation-prediction AUC drops well below 0.98 or the trained policy's average episodic cost is no longer lower than the ground-truth-cost policy's, the replacement claim and zero-shot generalization are refuted.","tokens_in":22938,"feed_emoji":"🤖","tokens_out":7929,"duration_ms":74601,"temperature":0.7,"pith_summary":"Safe reinforcement learning usually requires a human expert to encode each safety constraint as a cost function, which is brittle and task-specific. This paper argues that constraints can be given as complex, trajectory-level natural-language sentences—such as 'after stepping through water, your shoes can't touch lava'—and that the same text can serve both as the constraint and as the training signal that replaces the cost function. The proposed TTCT model embeds trajectories and constraint texts into a shared space, predicts a violation when their cosine similarity crosses a threshold, and decomposes the episodic violation cost into per-state-action costs to give the agent dense feedback. In experiments on a 2D grid task and a 3D robot-navigation task, agents trained with TTCT's predicted costs achieve lower average episodic cost (up to 4.0x lower violation rates) than agents trained with ground-truth cost functions, with comparable rewards, and the model transfers zero-shot to a constraint-shift environment.","feed_headline":"Natural-language constraints beat hand-coded costs in safe RL","feed_subtitle":"TTCT predicts violations from text and trajectory (AUC 0.98) and cuts violation rates up to 4x.","key_machinery":"The central mechanism is the text–trajectory alignment carried by TTCT: a causal Transformer encodes the state-action sequence into embeddings $H_1,\\dots,H_T$, a language model encodes the textual constraint into $L$, and scaled cosine similarity $\\operatorname{sim}(\\tau,y)=e^\\alpha H_T^{\\top}L/(\\|H_T\\|\\|L\\|)$ measures whether a trajectory $\\tau$ violates constraint $y$. Three losses train it end-to-end: a multimodal contrastive (MC) KL loss that pulls matching trajectory-text pairs together and pushes non-matching pairs apart, a within-trajectory (WT) loss that makes the final time step's embedding maximally similar to the text while earlier steps are dissimilar, and a cost-assignment (CA) loss that forces the sum of per-step predicted costs $\\sum_{t=1}^{T-1}\\hat{c}(s_t,a_t,y,\\tau_t)$ to match the predicted episodic cost $\\hat{C}(y)$. At deployment, a violation is flagged when $\\operatorname{sim}(\\tau,y)\\ge\\beta$, where $\\beta$ is the best ROC threshold, and non-violation steps receive the decomposed cost. This turns sparse, end-of-episode safety feedback into dense per-action cost while keeping the manual cost function out of the loop.","core_discovery":"On the paper's own terms, the central claim is that a single learned model, TTCT, can replace the manually designed cost function for safe RL with natural-language constraints. TTCT aligns a causal-transformer trajectory encoder with a pretrained language model through multimodal contrastive learning and a within-trajectory loss, so that the final trajectory embedding sits close to the text embedding of a constraint the trajectory violates; a violation is declared when the scaled cosine similarity exceeds a threshold chosen from the ROC curve. A cost-assignment head then uses attention scores between each state-action embedding and the text embedding to spread the episodic violation cost across time steps, giving non-violation actions informative cost signals. The paper reports violation-prediction accuracy and AUC of 0.98, and shows that across PPO-Lagrangian, CPPO_PID, and FOCOPS, policies using TTCT's predicted cost reach lower average episodic cost than the same algorithms with ground-truth cost while keeping reward comparable, and that the trained TTCT transfers without fine-tuning to a new environment with different hazards.","pith_inferences":["The paper leaves implicit that the same trajectory-text alignment could serve as an offline safety auditor: because it labels full trajectories with AUC 0.98, it could flag dangerous rollouts in a dataset without ever running a new policy.","A stronger test of the zero-shot claim would evaluate TTCT on genuinely free-form constraints written by humans rather than the template-generated sentences used in training; the current test split comes from the same template distribution, so out-of-template generalization remains unmeasured.","The attention-based cost assignment suggests a built-in per-step risk attribution; comparing those attribution scores against human judgments or ground-truth risk gradients would be a natural next experiment."],"forward_implications":["A single TTCT trained once can provide cost signals for any textual constraint in the same environment, so users only need to write the constraint in natural language instead of designing a cost function for each one.","Because the episodic cost is decomposed into per-state-action costs, agents learn which early behaviors escalate risk even though the underlying constraint is only violated at the final step.","Across three safe-RL algorithms (PPO-Lagrangian, CPPO_PID, FOCOPS), replacing the ground-truth cost with TTCT's predicted cost yields lower average episodic cost and comparable reward, so the benefit is not tied to one policy optimizer.","Zero-shot transfer holds for constraint-shift: a TTCT trained in Hazard-World-Grid, applied to LavaWall without fine-tuning, keeps a violation rate comparable to a policy trained with the ground-truth cost."],"supporting_citations":[{"why":"Provides the prior safe-RL-with-natural-language-constraints setting and the Hazard-World-Grid environment that TTCT builds on and compares against.","marker":"[14]"},{"why":"Supplies the multimodal contrastive alignment method that TTCT adapts to trajectory-text pairs.","marker":"[20]"},{"why":"The pretrained language model used as the text encoder to produce the constraint embedding.","marker":"[18]"},{"why":"The causal Transformer architecture used as the trajectory encoder to capture temporal dependencies in state-action sequences.","marker":"[16]"},{"why":"Motivates the temporal credit-assignment idea of decomposing an episodic outcome into per-step costs.","marker":"[30]"},{"why":"PPO-Lagrangian baseline whose ground-truth-cost mode is the comparison for TTCT's cost-prediction mode.","marker":"[6]"},{"why":"CPPO_PID baseline used to compare ground-truth versus predicted cost in the main experiments.","marker":"[8]"},{"why":"FOCOPS baseline used to compare ground-truth versus predicted cost in the main experiments.","marker":"[9]"},{"why":"Provides the 3D robot-navigation benchmark used as a main evaluation task.","marker":"[37]"},{"why":"Provides the LavaWall environment used for the zero-shot transfer experiment.","marker":"[40]"}],"fun_headline_variants":["Text-driven safety: TTCT replaces hand-coded costs in RL","From text to trajectory: AI learns safety constraints without manual costs","Language model predicts RL violations, beats manual cost design","Zero-shot safety: TTCT transfers text constraints to new RL tasks","Safe RL gets text-savvy: TTCT outdoes manual cost functions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"TTCT is trained on labels produced by hand-written violation-checking functions applied to template-generated constraint sentences, so its ability to replace manual cost design rests on those checkers being available and accurate for every constraint and on the templates being representative of real constraints.","fun_headline_variants_meta":{"raw":{"variants":["Text-driven safety: TTCT replaces hand-coded costs in RL","From text to trajectory: AI learns safety constraints without manual costs","Language model predicts RL violations, beats manual cost design","Zero-shot safety: TTCT transfers text constraints to new RL tasks","Safe RL gets text-savvy: TTCT outdoes manual cost functions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000209,"raw_usage":{"total_tokens":1392,"prompt_tokens":914,"completion_tokens":478,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":389}},"tokens_in":530,"tokens_out":478,"duration_ms":5222,"temperature":1.0,"reasoning_tokens":389,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:25:14.984518+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train TTCT only on the paper's template-generated constraints, then evaluate it on a held-out set of genuinely free-form natural-language constraints written by humans (not templates) with manually verified trajectory labels; if violation-prediction AUC drops well below 0.98 or the trained policy's average episodic cost is no longer lower than the ground-truth-cost policy's, the replacement claim and zero-shot generalization are refuted.","supporting_citations":[{"cited_title":"Safe reinforcement learning with natural language constraints,","cited_arxiv_id":null,"evidence_quote":"Provides the prior safe-RL-with-natural-language-constraints setting and the Hazard-World-Grid environment that TTCT builds on and compares against."},{"cited_title":"Learning transferable visual models from natural language supervi- sion,","cited_arxiv_id":null,"evidence_quote":"Supplies the multimodal contrastive alignment method that TTCT adapts to trajectory-text pairs."},{"cited_title":"Responsive safety in reinforcement learning by pid lagrangian methods,","cited_arxiv_id":null,"evidence_quote":"CPPO_PID baseline used to compare ground-truth versus predicted cost in the main experiments."},{"cited_title":"First order constrained optimization in policy space,","cited_arxiv_id":null,"evidence_quote":"FOCOPS baseline used to compare ground-truth versus predicted cost in the main experiments."},{"cited_title":"Safety gymnasium: A unified safe reinforcement learning benchmark,","cited_arxiv_id":null,"evidence_quote":"Provides the 3D robot-navigation benchmark used as a main evaluation task."},{"cited_title":"Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks,","cited_arxiv_id":null,"evidence_quote":"Provides the LavaWall environment used for the zero-shot transfer experiment."}],"review_version":1}