{"id":"75de9d88-3e7b-42bd-900e-3a3c76c1e530","arxiv_id":"2606.00003","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LLMs sometimes regain safe behavior after multi-turn jailbreaks, and this recovery can be measured with turn-level safety trajectories and metrics such as misalignment length and recovery duration.","lead":"This paper proposes a way to track how an LLM’s safety changes turn-by-turn after a jailbreak and to measure whether the model returns to safe behavior on its own. It offers a practical lens for AI safety teams who care about recovery after failure, not only about never failing.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Label noise and dialogue-vs-turn mismatch can fabricate or erase recovery events, so the reported self-recovery rates rest on an unvalidated proxy.","rationale":"The Reader correctly isolates the weakest assumption: Guard flags (and the all-safe conversation rule) are treated as faithful enough for recovery statistics despite ~70–77% agreement and the dialogue/turn mismatch. That assumption is load-bearing because every reported recovery count, average length, and category comparison is a direct function of those binary transitions; the methodological framing of trajectories is sound, but the empirical claim that the model “can self-recover” rests on them. The paper itself flags the issue as exploratory and calls for better labels, so the concern does not overturn the contribution; it simply confirms that CONDITIONAL (accept method + exploratory patterns once label fidelity and data release are tightened) remains the right verdict. No stronger internal inconsistency appears: the single-model, short-dialogue, student-red-team design is already scoped as preliminary. The concrete test above is the minimal check that would settle whether the proxy noise actually moves the headline numbers.","tokens_in":10466,"tokens_out":620,"duration_ms":7090,"concrete_test":"On the 597 dialogues, obtain independent turn-level human safe/unsafe labels for every model response (or a stratified sample of all dialogues that Guard flags as containing ≥1 recovery). Recompute Tables 1–2 and the top-5 category rows of Table 3 using only human labels; if the fraction of conversations with recoveries shifts by >30% relative or the ranking of Misalignment Length / Recovery Duration across the five retained categories changes, the self-recovery rates and risk-specific claims are not robust to the proxy.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that Minerva-7B exhibits intrinsic self-recovery after multi-turn jailbreaks, quantified by safety trajectories and metrics (Misalignment Length, Recovery Duration, absolute vs temporary recovery) derived from Llama Guard flags (Tables 1–3, §4–5). Recovery is defined as the first subsequent safe model response after an unsafe one (§4.2). Section 3 reports only 77.08% (3-1B) / 69.54% (3-8B) agreement with organizer ground truth, plus an explicit granularity mismatch: human labels are dialogue-level (overall jailbreak success) while Guard is turn-level, and conversations are declared safe only if every response is safe. Because recovery events are sparse (14.6% of dialogues under 3-1B; 3.2% multiple), even modest false-positive/false-negative rates at the critical transition turns can create spurious recoveries or hide real ones. The paper treats the resulting counts as evidence of intrinsic recoverability and risk-category differences, yet never validates that the detected transitions match human turn-level safety judgments. Without that, the quantitative recovery statistics are not known to reflect true alignment dynamics.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper argues that LLM safety research should examine intrinsic self-recovery after misalignment (rather than only stronger initial alignment or filters). It introduces safety trajectories that plot turn-level safe/unsafe flags from a content moderator (Llama Guard) against dialogue turns, defines recovery as the first subsequent safe model response after an unsafe one (absolute if held to dialogue end; temporary otherwise), and distinguishes upper/lower recovery paths depending on the safety of the intervening user prompt. Metrics include Misalignment Length and Recovery Duration. These are applied to a filtered set of 597 multi-turn adversarial dialogues (2454 turns) collected via a student red-teaming challenge against Minerva-7B-instruct-v1.0, yielding descriptive statistics (roughly one-third of conversations unsafe; recoveries in 11–15%; average misalignment ~1.6–2.1 turns) plus comparisons across Llama Guard 3-1B vs 3-8B and across risk categories from the MLCommons taxonomy.","tokens_in":10790,"tokens_out":1395,"duration_ms":28759,"significance":"If self-recovery is a genuine, measurable property of aligned models under multi-turn adversarial pressure, the trajectory-centric framing and the two core metrics supply a useful additional axis for safety evaluation beyond static success/failure rates. The work correctly highlights that multi-turn jailbreaks can erode constraints incrementally and that post-failure dynamics matter for risk. Concrete strengths include the real (if student-generated) multi-turn adversarial corpus with technique annotations, the explicit sensitivity check across two Guard sizes, the open interactive visualizer, and the clear operational definitions of trajectories and recovery events. These elements make the methodological proposal reusable even if the particular numerical rates for Minerva-7B prove noisy. The quantitative claims themselves remain preliminary and hinge on the fidelity of the automatic labels.","major_comments":[{"comment":"Section 3 reports only 77.08% (Llama Guard 3-1B) and 69.54% (3-8B) agreement with organizer ground truth, together with an explicit granularity mismatch: human labels are dialogue-level (overall jailbreak success) while Guard labels are turn-level, and a conversation is declared safe under Guard only if every model response is safe. Recovery events (Tables 1–2) are defined precisely by transitions at individual turns and are sparse (14.6% of conversations under 3-1B; only 3.2% with multiple recoveries). Modest false-positive or false-negative rates at the critical transition turns can therefore fabricate or erase the very events that the recovery statistics count. Without turn-level human validation of those transitions (or a sensitivity analysis that flips labels at the observed error rate), the reported Misalignment Length, Recovery Duration, and risk-category differences cannot be tak","section":"Section 3, Tables 1–2"},{"comment":"Section 4.2 and Figure 2 carefully distinguish upper versus lower recovery paths according to whether the user prompt at the recovery turn is itself unsafe. The subsequent quantitative analysis in Section 5 and Tables 1–3 collapses these paths into aggregate counts and averages. Because the claim is that recovery is intrinsic (i.e., occurs without external intervention), the frequency of recovery under continued adversarial prompts versus under benign prompts is load-bearing; the paper should report the path-conditioned statistics or justify why the distinction can be ignored for the metrics that are presented.","section":"Section 4.2, Figure 2, Section 5"},{"comment":"Table 3 retains risk categories only when each exhibits at least five recovery events under both Guard models, then ranks recoverability by the joint pattern of shorter Misalignment Length and longer Recovery Duration. With absolute recovery counts as low as 5–9 for several retained categories and with large discrepancies between the two evaluators, the ranking (Violent Crimes / Hate / Non-Violent Crimes stronger; Indiscriminate Weapons / Privacy weaker) rests on very small samples and is sensitive to the free threshold of five events. Confidence intervals or a bootstrap over label uncertainty are needed before category-specific claims can be treated as robust.","section":"Section 5.3, Table 3"}],"minor_comments":[{"comment":"Figure 1 caption and surrounding text refer to “bold arrows” representing trends, yet the textual description of ascending/descending arrows is slightly inconsistent with the claim that a descending arrow (resistance) is absent; a short legend inside the figure would remove ambiguity.","section":"Figure 1"},{"comment":"The conversation-level aggregation rule (“safe only if every response is safe”) is stated clearly in Section 3 but is never listed among the free parameters or subjected to an ablation (e.g., majority vote or last-k-turns). A one-sentence sensitivity check would strengthen transparency.","section":"Section 3"},{"comment":"Average conversation length remains short (~8–10 turns) because participants were not incentivized to prolong misalignment. The limitation is acknowledged, yet the metrics Recovery Duration and absolute-versus-temporary recovery are most informative on longer trajectories; the planned future challenge design should be mentioned already in the dataset section so readers can calibrate expectations.","section":"Section 5.1"},{"comment":"The paper cites its own concurrent taxonomy paper [26] for the 50 attack techniques; a brief summary table of the seven families (or a pointer to an appendix) would make the present manuscript more self-contained.","section":"Section 3"},{"comment":"Minor wording: “self-recoverability” and “recoverability” are used interchangeably; pick one term after first definition. Also, “LlG” abbreviation appears only in Table 3 header and is never expanded.","section":"Throughout / Table 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is explicitly positioned as a workshop paper (AAAI’26 WS37) and repeatedly labels its own results “preliminary.” For a full journal the validation gap on the Guard labels would be decisive; for a workshop the methodological contribution and the open visualizer may still be of interest after the major revisions above. The student red-teaming origin of the data is a double-edged sword: realistic multi-turn attacks, but also short dialogues and possible selection effects from the two-hour window."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful bit here is the framing, not the rates. They treat multi-turn alignment as a trajectory of turn-level safety flags, define recovery as the first safe model response after an unsafe one, distinguish absolute vs temporary recovery and upper vs lower recovery paths (safe vs unsafe user prompt at the recovery turn), and introduce simple descriptive metrics: Misalignment Length and Recovery Duration. That is a clean shift away from one-shot refusal rates and static success/failure, and it is new relative to the multi-turn jailbreak and Llama Guard literature they cite.\n\nThey collected a real student red-team set (597 dialogues, Minerva-7B, Italian-heavy, manual attack-success and technique labels), ship an interactive visualizer, and report transparent descriptive tables: ~33% unsafe conversations, ~11–15% with at least one recovery, short average misalignment (~1.6–2.1 turns), and some risk-category variation under two Guard sizes. They are explicit that this is preliminary and that Guard choice moves the numbers. The methodology section is clear and the invented terms are operational, not decorative.\n\nThe soft spot the stress-test flags is real and load-bearing for the quantitative claims. Guard–human agreement is only 77% (1B) / 70% (8B), and they map dialogue-level human labels onto turn-level Guard flags by requiring every response safe. Recovery events are sparse, so modest label noise at the transition turns can invent or erase recoveries. They never validate the detected transitions against turn-level human judgments. Dialogues are also short (participants switched tasks once they “won”), single target model, no uncertainty, no code/data release yet. Risk-category patterns are therefore suggestive only.\n\nThis is for people who already work on multi-turn safety evaluation or product monitoring and want a vocabulary for post-failure dynamics. It is not a claim of a universal self-recovery law. I would send it to peer review: the method is worth refereeing, the limitations are stated, and tightening labels, length, and release would make a solid workshop or short paper. I would cite the trajectory idea if I am writing on multi-turn robustness; I would not cite the recovery percentages as established facts.","headline":"Useful trajectory framing for multi-turn safety recovery, but the recovery rates rest on noisy Guard labels and short student dialogues, so treat the numbers as exploratory.","tokens_in":11381,"tokens_out":555,"would_cite":true,"duration_ms":11379,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"LLMs can regain alignment after jailbreak failures, and that recovery can be measured turn by turn.","keywords":["LLM safety","alignment","self-recovery","safety trajectories","jailbreaking","content moderation","multi-turn dialogues","misalignment length"],"falsifier":"Re-label the same multi-turn dialogues with high-agreement turn-level human safety annotations (or a clearly superior moderator) and recompute misalignment length and recovery duration; if recovery events largely disappear or the metrics reverse, the self-recovery claim collapses under the original evaluator.","tokens_in":11350,"feed_emoji":"🔄","tokens_out":603,"duration_ms":6483,"temperature":0.7,"pith_summary":"Most safety work tries to keep language models from ever going off the rails. This paper asks a different question: once a model has already produced unsafe replies under multi-turn adversarial pressure, can it pull itself back into alignment without an external fix? The authors treat each user-assistant exchange as a safety trajectory, mark when the model slips and when it returns, and introduce simple recovery metrics such as how long the bad stretch lasts and how long the restored safe stretch holds. Using a red-teaming dataset of multi-turn attacks against a 7B instruction model and Llama Guard as the safety evaluator, they show that recovery events do occur, that many of them are temporary, and that recovery strength varies by risk category and by which moderation model is used. The point is not that current systems are already safe, but that self-recovery itself is a measurable dimension of safety behavior that can be studied and, later, improved.","feed_headline":"LLMs can self-recover after jailbreak failures","feed_subtitle":"Turn-by-turn safety trajectories measure how long misalignment lasts and how recovery holds","key_machinery":"Safety trajectories: sequential plots of safe/unsafe flags for each user prompt and model response across dialogue turns, from which recovery is defined as the first safe model response after an unsafe one, with paths distinguished by whether the recovering-turn prompt itself is safe or unsafe.","core_discovery":"After misalignment caused by multi-turn adversarial prompting, an LLM can exhibit intrinsic self-recovery of alignment without external intervention, and this behavior can be modeled and quantified by representing dialogues as turn-level safety trajectories and extracting recovery events, misalignment length, recovery duration, and absolute versus temporary recovery.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["LLMs self-recover alignment after jailbreak misalignment","Safety trajectories quantify LLM recovery from multi-turn attacks","After jailbreaks, models regain alignment without external fixes","Turn-level paths reveal temporary vs lasting LLM self-recovery","Intrinsic recovery detected in adversarial multi-turn dialogues"],"cache_read_input_tokens":1920,"weakest_assumption_plain":"That automatic turn-level safe/unsafe labels from a content-moderation model are accurate enough proxies for true alignment to support recovery statistics, even though they only partially agree with human ground truth and use a different labeling grain than the dialogue-level annotations.","fun_headline_variants_meta":{"raw":{"variants":["LLMs self-recover alignment after jailbreak misalignment","Safety trajectories quantify LLM recovery from multi-turn attacks","After jailbreaks, models regain alignment without external fixes","Turn-level paths reveal temporary vs lasting LLM self-recovery","Intrinsic recovery detected in adversarial multi-turn dialogues"]},"model":"grok-4.5","effort":"low","cost_usd":0.001548,"raw_usage":{"total_tokens":806,"prompt_tokens":726,"num_sources_used":0,"completion_tokens":80,"cost_in_usd_ticks":15480000,"prompt_tokens_details":{"text_tokens":726,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":0,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":726,"tokens_out":80,"duration_ms":1174,"temperature":1.0,"reasoning_tokens":0,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T18:49:58.202805+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-label the same multi-turn dialogues with high-agreement turn-level human safety annotations (or a clearly superior moderator) and recompute misalignment length and recovery duration; if recovery events largely disappear or the metrics reverse, the self-recovery claim collapses under the original evaluator.","supporting_citations":[],"review_version":1}