{"id":"f4509a8d-6ed0-4a73-b3ae-cffe9e6ed268","arxiv_id":"2607.26820","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Recast predicts the turn distribution of future multi-turn LLM safety failures from dual-scale trajectory evidence, catching 88.3% of failures 2.41 turns early at 12.3% false alarms.","lead":"Recast forecasts when a multi-turn LLM dialogue will turn unsafe, before any single turn clearly breaks the rules. It matters because agent-style chat can hide harmful goals across many harmless-looking steps, and today’s filters mostly react after the damage is already visible.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Headline forecasting metrics are tied to the authors’ own LLM-augmented trajectory pipeline; transfer of EWR/FAR/MLT beyond that construction is not shown.","rationale":"The paper is internally coherent as a systems/ML method: dual-scale retrieval, compositional state/transition, causal temporal head, and broad ablations are well specified, and online ASR gains are modest and complementary to guardrails (45.9% vs THRD 45.7%). There is no mathematical contradiction. The load-bearing hinge is exactly the reader’s weakest assumption—faithfulness of pipeline-derived t⋆ and hard-benign supervision for time-to-risk learning. I sharpen only the measurement gap: the 88.3%/2.41/12.3% suite is not shown under attack structure or annotation processes outside the training construction, whereas seen/unseen splits are reported only for ASR. That keeps correctness_risk medium and the verdict CONDITIONAL pending external trajectory corpora, human t⋆ adjudication, and code/data release. No stronger independent flaw (e.g. broken objective, non-causal leakage in the encoder) is evident from the text. Stress-test therefore leaves the reader’s CONDITIONAL call unchanged.","tokens_in":21627,"tokens_out":653,"duration_ms":47598,"concrete_test":"Freeze the released Recast checkpoint and recompute Table 2 (NLL, Expected-MAE, EWR, FAR, MLT at H=3, τa=0.5) on a fresh set of multi-turn trajectories from methods excluded from training construction (e.g. ICON, X-Teaming, Crescendo/FITD or human red-team logs), with t⋆ labeled by independent human annotators—not the pipeline LLM judge. If EWR falls below ~80% or FAR exceeds ~15% at comparable MLT, the headline forecasting claim does not transfer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (88.3% EWR, 2.41 MLT, 12.3% FAR, NLL 0.45 / Expected-MAE 0.31 at H=3) is a supervised time-to-first-risk result. Every training and (for Table 2) evaluation label depends on t⋆ and on hard-benign twins from the four-stage pipeline in Appendix “Construction of Training Dataset”: public ActorAttack/Red Queen/CKA-Agent trajectories → length expansion → LLM identification of failure-support turns → LLM rewrite into safe counterparts → LLM harm judgment, with only a 1,000-response human spot-check (~86% agreement). Cross-attack evidence in Tables 5–6 is ASR reduction only; Table 2-style distributional and early-warning metrics are not reported on attack families or labeling processes outside that pipeline (ICON/X-Teaming, Crescendo, FITD, human red-team logs). If the model mainly fits pipeline-regularized progression and rewrite artifacts rather than genuine compositional risk dynamics, the headline forecasting numbers overstate live transfer—the exact condition the strongest claim needs.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes Recast, a black-box trajectory-level safety risk forecasting framework for multi-turn LLM interactions. Rather than classifying whether the current turn is unsafe, Recast estimates a discrete time-to-first-risk distribution π_t over a horizon H from the observed trajectory τ_≤t. It retrieves dual-scale evidence (short-term windowed progression and long-term memory queries), builds a compositional risk state s_t and transition r_t (Eqs. 5–6), and feeds the causal sequence of (s_i, r_i) into a temporal encoder whose final state predicts π_t (Eqs. 7–8). Training uses NLL on first-failure buckets plus ordinal, ranking, and benign auxiliaries. On a 10k-trajectory corpus spanning 7 risk categories, Recast reports NLL 0.45, Expected-MAE 0.31, EWR 88.3%, FAR 12.3%, and MLT 2.41 turns (H=3). Online, early warnings cut average multi-turn ASR from 71.5% to 45.9% (40.4% with a guardrail) across three target models and four attack families, two of which are held out of training, with ~1.4% latency overhead.","tokens_in":21962,"tokens_out":1879,"duration_ms":48701,"significance":"If the forecasting results transfer beyond the authors’ construction pipeline, the work is a genuine shift from reactive multi-turn moderation to proactive risk anticipation—an important and timely problem as LLMs become agents. Strengths include a clean problem formulation (time-to-risk distribution), a structured inductive bias (compositional state + transitions rather than a flat sequence model), thorough ablations and sensitivity sweeps (horizon, length, threshold), parameter-matched sequence baselines (Table 7), seen/unseen attack splits for defense (Tables 5–6), negligible latency, and interpretable case analysis. The comparison showing lower FAR than GRU/Transformer forecasters at matched capacity is particularly useful. These contributions would matter to the LLM safety and agent-safety communities even if some evaluation scope must be tightened.","major_comments":[{"comment":"Load-bearing evaluation coupling for the headline claim (Abstract/Table 2). NLL, Expected-MAE, EWR, FAR, and MLT are supervised metrics that depend on first-failure labels t⋆ and on hard-benign twins from the four-stage LLM-assisted pipeline (Appendix, Construction of Training Dataset: public ActorAttack/Red Queen/CKA-Agent → length expansion → support-turn ID → safe rewrite → harm judgment; ~86% agreement on a 1,000-response spot-check). Tables 5–6 show ASR reduction on unseen ICON/X-Teaming, but do not report Table 2-style distributional or early-warning metrics on trajectories whose attack structure and labeling process lie outside that pipeline (e.g., ICON/X-Teaming, Crescendo, FITD, or human red-team logs with independently annotated t⋆). Without that, it remains open whether 88.3%/12.3%/2.41 mainly reflect pipeline-regularized progression patterns. Please either (i) annotate first-","section":"Table 2; Appendix Construction of Training Dataset; Tables 5–6"},{"comment":"Definition and operationalization of t⋆ at test time for online defense are underspecified relative to the claim that warnings precede “safety failures.” §2 defines t⋆ as the first turn exhibiting a predefined safety risk, and Appendix metrics define success as warning before the annotated failure turn with interaction termination. For AdvBench/SORRY-Bench multi-turn runs (RQ1), it is unclear who/what annotates the failure turn (automatic judge? policy checklist? same LLM harm judge as training?), whether that annotator matches training labels, and how partial or borderline assists are treated. Inconsistent t⋆ between training and ASR evaluation would inflate ΔASR. Please state the exact failure criterion, judge model/prompt, and agreement with human labels on the attack evaluation set.","section":"§2 Problem Formulation; §4 RQ1; Appendix Details of Evaluation Metrics"},{"comment":"Auxiliary losses L_ord, L_rank, and L_ben are named in Eq. (10) and said to enforce monotonic escalation, cross-sample ranking, and benign false-alarm reduction, but their mathematical forms, sampling of pairs, and λ weights are not given in the main text or appendix. These terms directly shape FAR and the risk-score calibration used for τ_a. Please fully specify the three losses, hyperparameters, and an ablation removing each auxiliary (analogous to Table 4’s architectural ablations) so the forecasting results are reproducible and the contribution of supervision design is clear.","section":"§3 Training objectives, Eq. (10)"},{"comment":"Defense comparison protocol (Table 1). Recast terminates the dialogue when R_t > τ_a, which can prevent later turns from ever being generated; guardrail and history-aware baselines appear to filter or judge per turn without the same early-stop coupling. If baselines are not given an equivalent early-intervention option (or if ASR is counted only when a full attack completes), the comparison mixes forecasting-triggered termination with turn-level filtering. Please equalize the intervention interface (e.g., allow all methods to halt on their risk signal) or report both “halt-on-warning ASR” and “full-trajectory judge ASR” so ΔASR is attributable to better risk signals rather than protocol asymmetry.","section":"Table 1; §4 RQ1 Early warning defense"}],"minor_comments":[{"comment":"Eq. (2) writes LocalTr_w on [v_i; v_i−v_{i−1}; v_{i+1}−v_i] for i=1..t−1. Boundary handling for i=1 and the use of v_{i+1} inside a causal “historical” path should be clarified (is v_{i+1} only within the past window, or is there leakage?).","section":"§3 Eq. (2)"},{"comment":"Hyperparameters K (memory queries), window radius w, and λ_ord/λ_rank/λ_ben are free parameters listed only partially; a small table of defaults would aid reproduction.","section":"§4 Hyperparameters and device"},{"comment":"Figure 1 / Figure 2 text is dense; expanding the caption to define e_loc, e_mem, s_t, r_t, and π_t in one place would help readers who skim figures first.","section":"Figure 2"},{"comment":"Related Work cites several 2025–2026 arXiv items appropriately; ensure camera-ready versions and venue names are updated where available, and that ICON (Lin et al. 2026) is consistently treated as excluded from training in all tables.","section":"§5 Related Work; Tables 5–6"},{"comment":"Minor typography: missing spaces in several compound phrases in the Abstract PDF text (“Aslargelanguagemodels”, “toaddressthislimitation”); clean for the camera-ready.","section":"Abstract"},{"comment":"Category-wise FAR varies widely (Misinformation 2.4% vs Privacy 24.6% in Table 2); a brief discussion of calibration or threshold-per-category would strengthen the deployment claims in RQ2.","section":"Table 2; RQ2"}],"recommendation":"major_revision","confidential_remarks":"The core idea is publishable and the experimental surface is broader than many multi-turn safety papers. The skeptic’s concern about pipeline-tied forecasting metrics is the main reason I chose major_revision rather than minor_revision: it directly underwrites the Abstract’s strongest numbers. If the authors add first-failure-annotated EWR/FAR on ICON/X-Teaming (or an external human multi-turn set) and fully specify the auxiliary losses plus the t⋆ judge, I would expect this to clear a second round. Scope fit for a serious ML/safety venue is good; not a reject on novelty or soundness of the method itself."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a real method paper, not a rebrand of LlamaGuard. They treat multi-turn risk as predicting a discrete time-to-first-failure distribution from the black-box transcript, and they ship a full stack—dual-scale retrieval, compositional state plus transition, causal encoder—plus the usual sweeps.\n\nWhat is new is the objective and the packaging. Most multi-turn defenses still score the observed dialogue. Recast asks whether and when the trajectory will tip, and reports NLL/Expected-MAE plus EWR/FAR/MLT (88.3% / 12.3% / 2.41 turns at H=3). Architecture is sensible: short-term window dynamics, long-term memory queries, explicit cross-path product in the state, signed deltas as transitions. Ablations move the right way; latency is ~48 ms (~1.4%). Online, it cuts average ASR 71.5→45.9, and pairs with a guardrail to 40.4—complementary, not magic. Seen vs unseen attack families (ICON, X-Teaming) show transfer on ASR, which is the right check to attempt. Citations cover the multi-turn attack and defense line fairly.\n\nSoft spot, in proportion: the stress-test lands. Table 2-style forecasting metrics are supervised on first-failure labels and hard-benign twins from their four-stage LLM pipeline (public attacks → length expand → support-turn ID → rewrite → harm judge, ~86% human agreement on a 1k spot-check). Cross-attack tables report ASR reduction, not the full distributional/early-warning suite on outside labeling processes. So the strongest claim is “works on this constructed distribution,” not yet “works on live novel red-team logs.” No code/data release in the text. Standalone defense is roughly THRD-level; the win is lead-time forecasting, not crushing every baseline on ASR.\n\nNo load-bearing math contradiction—just evaluation coupling and free knobs (H, τ_a, lambdas). For people building agent guardrails or multi-turn red-team eval, this is worth a careful read. I would send it to referees; I would want external corpora and released artifacts before treating 88.3/12.3 as a deployment number.","headline":"Clean reframing of multi-turn safety as time-to-first-risk forecasting, with a coherent system and broad tables—but the headline EWR/FAR numbers live on the authors’ own LLM-built trajectory corpus.","tokens_in":22608,"tokens_out":593,"would_cite":true,"duration_ms":21895,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Recast forecasts multi-turn LLM safety failures before they occur by modeling how risk composes and evolves across dialogue trajectories.","keywords":["multi-turn LLM safety","trajectory-level risk forecasting","jailbreak defense","compositional risk evolution","early warning","black-box agents","time-to-risk prediction"],"falsifier":"Run Recast unchanged on live or fully held-out multi-turn attacks whose structure, length, and surface topics differ from the training sources and benign rewrites; if early-warning rate collapses, false-alarm rate spikes, or mean lead time falls near zero while NLL/Expected-MAE worsen, the central forecasting claim fails.","tokens_in":22452,"feed_emoji":"⚠️","tokens_out":978,"duration_ms":26296,"temperature":0.7,"pith_summary":"As language models become multi-turn agents, harmful goals can be split across many turns that look safe alone and only become dangerous when composed. Existing guards mostly judge the current reply after risk has already shown up. This paper argues that safety should instead be treated as a forecasting problem: given the trajectory so far, predict whether and when a failure will appear in future turns. It introduces Recast, which retrieves short- and long-term risk evidence, builds a compositional risk state and its turn-to-turn changes, and uses a causal temporal encoder to output a distribution over when risk will emerge. On seven risk categories it reports warning on 88.3% of future failures about 2.41 turns early at a 12.3% false-alarm rate, and when used online it cuts multi-turn attack success substantially. A sympathetic reader cares because early, timed warnings open a window for preemptive intervention instead of only blocking already-unsafe outputs.","feed_headline":"Warns on 88% of multi-turn jailbreaks 2.4 turns early","feed_subtitle":"Recast forecasts when dialogue risk will emerge, cutting attack success while keeping false alarms near 12%.","key_machinery":"Recast: dual-scale historical evidence retrieval (short-term windowed progression plus long-term memory queries), compositional risk state and transition modeling (current short/long evidence and their element-wise interaction, plus signed deltas), and a causal temporal encoder that maps the state-transition sequence to a discrete time-to-risk distribution over a horizon H.","core_discovery":"The paper claims that multi-turn safety failures are trajectory-level phenomena that can be forecast from pre-failure dialogue: by retrieving dual-scale risk evidence, composing a structured risk state with its transitions, and encoding that evolution causally, a model can predict the distribution of future risk-emergence turns well enough to warn on most failures before they materialize while keeping false alarms moderate.","pith_inferences":["If compositional trajectory forecasting works, agent memory and tool-use logs could be folded into the same dual-scale state so risk is predicted across actions, not only chat turns.","Category-wise gaps (e.g., harder cybersecurity timing, higher privacy false alarms) suggest calibrated per-domain thresholds or mixture heads rather than one global risk score.","Adversaries who know the forecaster may deliberately flatten or delay compositional signals; stress-testing adaptive multi-turn attackers against the early-warning loop is a natural next measurement.","The same time-to-risk head could supervise safer training or intervention policies that steer trajectories away from high near-horizon mass before any refusal is needed."],"forward_implications":["Safeguards can move from reactive turn-level violation detection to proactive trajectory-level early warning with a few turns of lead time.","Combining Recast-style forecasts with existing guardrail models further reduces multi-turn jailbreak success beyond either alone.","Forecasting horizon and warning threshold become tunable knobs trading coverage, lead time, and false alarms for deployment.","Risk evolution patterns transfer across multiple harm categories and to some attack families held out of training.","Negligible per-turn latency (about 1.4% of typical model response time) makes online trajectory monitoring practical."],"fun_headline_variants":["Recast forecasts 88% of multi-turn safety failures 2.4 turns ahead","Predicts trajectory-level LLM risks 2.4 turns before failures hit","Dual-scale evidence forecasts 88% of emerging multi-turn risks early","Catches 88% of latent multi-turn jailbreaks with 2.4-turn lead time","Models risk evolution to warn on most multi-turn failures pre-violation"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The first-failure labels and hard-benign twins built by the authors’ multi-stage pipeline from public multi-turn attacks are faithful enough proxies for real latent risk evolution that supervised time-to-risk training will transfer beyond those curated patterns.","fun_headline_variants_meta":{"raw":{"variants":["Recast forecasts 88% of multi-turn safety failures 2.4 turns ahead","Predicts trajectory-level LLM risks 2.4 turns before failures hit","Dual-scale evidence forecasts 88% of emerging multi-turn risks early","Catches 88% of latent multi-turn jailbreaks with 2.4-turn lead time","Models risk evolution to warn on most multi-turn failures pre-violation"]},"model":"grok-4.5","effort":"low","cost_usd":0.004441,"raw_usage":{"total_tokens":1336,"prompt_tokens":785,"num_sources_used":0,"completion_tokens":92,"cost_in_usd_ticks":44408000,"prompt_tokens_details":{"text_tokens":785,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":459,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":785,"tokens_out":92,"duration_ms":9713,"temperature":1.0,"reasoning_tokens":459,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-30T20:07:55.494105+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run Recast unchanged on live or fully held-out multi-turn attacks whose structure, length, and surface topics differ from the training sources and benign rewrites; if early-warning rate collapses, false-alarm rate spikes, or mean lead time falls near zero while NLL/Expected-MAE worsen, the central forecasting claim fails.","supporting_citations":[],"review_version":1}