Pith. sign in

REVIEW 4 major objections 6 minor 86 references

Recast forecasts multi-turn LLM safety failures before they occur by modeling how risk composes and evolves across dialogue trajectories.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 20:07 UTC pith:Q6SIJB67

load-bearing objection Clean reframing of multi-turn safety as time-to-first-risk forecasting, with a coherent system and broad tables—but the headline EWR/FAR numbers live on the authors’ own LLM-built trajectory corpus. the 4 major comments →

arxiv 2607.26820 v1 pith:Q6SIJB67 submitted 2026-07-29 cs.LG cs.CR

Forecasting Trajectory-Level Safety Risks in Black-Box Multi-Turn Interactions

Shi Lin , Peng Qian , Dinghao Liu , Renjie Sun , Sifan Wu , Dezhang Kong , Chenpei Wang , Xun Wang This is my paper
classification cs.LG cs.CR
keywords multi-turn LLM safetytrajectory-level risk forecastingjailbreak defensecompositional risk evolutionearly warningblack-box agentstime-to-risk prediction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

As language models become multi-turn agents, harmful goals can be split across many turns that look safe alone and only become dangerous when composed. Existing guards mostly judge the current reply after risk has already shown up. This paper argues that safety should instead be treated as a forecasting problem: given the trajectory so far, predict whether and when a failure will appear in future turns. It introduces Recast, which retrieves short- and long-term risk evidence, builds a compositional risk state and its turn-to-turn changes, and uses a causal temporal encoder to output a distribution over when risk will emerge. On seven risk categories it reports warning on 88.3% of future failures about 2.41 turns early at a 12.3% false-alarm rate, and when used online it cuts multi-turn attack success substantially. A sympathetic reader cares because early, timed warnings open a window for preemptive intervention instead of only blocking already-unsafe outputs.

Core claim

The paper claims that multi-turn safety failures are trajectory-level phenomena that can be forecast from pre-failure dialogue: by retrieving dual-scale risk evidence, composing a structured risk state with its transitions, and encoding that evolution causally, a model can predict the distribution of future risk-emergence turns well enough to warn on most failures before they materialize while keeping false alarms moderate.

What carries the argument

Recast: dual-scale historical evidence retrieval (short-term windowed progression plus long-term memory queries), compositional risk state and transition modeling (current short/long evidence and their element-wise interaction, plus signed deltas), and a causal temporal encoder that maps the state-transition sequence to a discrete time-to-risk distribution over a horizon H.

Load-bearing premise

The first-failure labels and hard-benign twins built by the authors’ multi-stage pipeline from public multi-turn attacks are faithful enough proxies for real latent risk evolution that supervised time-to-risk training will transfer beyond those curated patterns.

What would settle it

Run Recast unchanged on live or fully held-out multi-turn attacks whose structure, length, and surface topics differ from the training sources and benign rewrites; if early-warning rate collapses, false-alarm rate spikes, or mean lead time falls near zero while NLL/Expected-MAE worsen, the central forecasting claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Safeguards can move from reactive turn-level violation detection to proactive trajectory-level early warning with a few turns of lead time.
  • Combining Recast-style forecasts with existing guardrail models further reduces multi-turn jailbreak success beyond either alone.
  • Forecasting horizon and warning threshold become tunable knobs trading coverage, lead time, and false alarms for deployment.
  • Risk evolution patterns transfer across multiple harm categories and to some attack families held out of training.
  • Negligible per-turn latency (about 1.4% of typical model response time) makes online trajectory monitoring practical.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If compositional trajectory forecasting works, agent memory and tool-use logs could be folded into the same dual-scale state so risk is predicted across actions, not only chat turns.
  • Category-wise gaps (e.g., harder cybersecurity timing, higher privacy false alarms) suggest calibrated per-domain thresholds or mixture heads rather than one global risk score.
  • Adversaries who know the forecaster may deliberately flatten or delay compositional signals; stress-testing adaptive multi-turn attackers against the early-warning loop is a natural next measurement.
  • The same time-to-risk head could supervise safer training or intervention policies that steer trajectories away from high near-horizon mass before any refusal is needed.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Recast, a black-box trajectory-level safety risk forecasting framework for multi-turn LLM interactions. Rather than classifying whether the current turn is unsafe, Recast estimates a discrete time-to-first-risk distribution π_t over a horizon H from the observed trajectory τ_≤t. It retrieves dual-scale evidence (short-term windowed progression and long-term memory queries), builds a compositional risk state s_t and transition r_t (Eqs. 5–6), and feeds the causal sequence of (s_i, r_i) into a temporal encoder whose final state predicts π_t (Eqs. 7–8). Training uses NLL on first-failure buckets plus ordinal, ranking, and benign auxiliaries. On a 10k-trajectory corpus spanning 7 risk categories, Recast reports NLL 0.45, Expected-MAE 0.31, EWR 88.3%, FAR 12.3%, and MLT 2.41 turns (H=3). Online, early warnings cut average multi-turn ASR from 71.5% to 45.9% (40.4% with a guardrail) across three target models and four attack families, two of which are held out of training, with ~1.4% latency overhead.

Significance. If the forecasting results transfer beyond the authors’ construction pipeline, the work is a genuine shift from reactive multi-turn moderation to proactive risk anticipation—an important and timely problem as LLMs become agents. Strengths include a clean problem formulation (time-to-risk distribution), a structured inductive bias (compositional state + transitions rather than a flat sequence model), thorough ablations and sensitivity sweeps (horizon, length, threshold), parameter-matched sequence baselines (Table 7), seen/unseen attack splits for defense (Tables 5–6), negligible latency, and interpretable case analysis. The comparison showing lower FAR than GRU/Transformer forecasters at matched capacity is particularly useful. These contributions would matter to the LLM safety and agent-safety communities even if some evaluation scope must be tightened.

major comments (4)
  1. [Table 2; Appendix Construction of Training Dataset; Tables 5–6] Load-bearing evaluation coupling for the headline claim (Abstract/Table 2). NLL, Expected-MAE, EWR, FAR, and MLT are supervised metrics that depend on first-failure labels t⋆ and on hard-benign twins from the four-stage LLM-assisted pipeline (Appendix, Construction of Training Dataset: public ActorAttack/Red Queen/CKA-Agent → length expansion → support-turn ID → safe rewrite → harm judgment; ~86% agreement on a 1,000-response spot-check). Tables 5–6 show ASR reduction on unseen ICON/X-Teaming, but do not report Table 2-style distributional or early-warning metrics on trajectories whose attack structure and labeling process lie outside that pipeline (e.g., ICON/X-Teaming, Crescendo, FITD, or human red-team logs with independently annotated t⋆). Without that, it remains open whether 88.3%/12.3%/2.41 mainly reflect pipeline-regularized progression patterns. Please either (i) annotate first-
  2. [§2 Problem Formulation; §4 RQ1; Appendix Details of Evaluation Metrics] Definition and operationalization of t⋆ at test time for online defense are underspecified relative to the claim that warnings precede “safety failures.” §2 defines t⋆ as the first turn exhibiting a predefined safety risk, and Appendix metrics define success as warning before the annotated failure turn with interaction termination. For AdvBench/SORRY-Bench multi-turn runs (RQ1), it is unclear who/what annotates the failure turn (automatic judge? policy checklist? same LLM harm judge as training?), whether that annotator matches training labels, and how partial or borderline assists are treated. Inconsistent t⋆ between training and ASR evaluation would inflate ΔASR. Please state the exact failure criterion, judge model/prompt, and agreement with human labels on the attack evaluation set.
  3. [§3 Training objectives, Eq. (10)] Auxiliary losses L_ord, L_rank, and L_ben are named in Eq. (10) and said to enforce monotonic escalation, cross-sample ranking, and benign false-alarm reduction, but their mathematical forms, sampling of pairs, and λ weights are not given in the main text or appendix. These terms directly shape FAR and the risk-score calibration used for τ_a. Please fully specify the three losses, hyperparameters, and an ablation removing each auxiliary (analogous to Table 4’s architectural ablations) so the forecasting results are reproducible and the contribution of supervision design is clear.
  4. [Table 1; §4 RQ1 Early warning defense] Defense comparison protocol (Table 1). Recast terminates the dialogue when R_t > τ_a, which can prevent later turns from ever being generated; guardrail and history-aware baselines appear to filter or judge per turn without the same early-stop coupling. If baselines are not given an equivalent early-intervention option (or if ASR is counted only when a full attack completes), the comparison mixes forecasting-triggered termination with turn-level filtering. Please equalize the intervention interface (e.g., allow all methods to halt on their risk signal) or report both “halt-on-warning ASR” and “full-trajectory judge ASR” so ΔASR is attributable to better risk signals rather than protocol asymmetry.
minor comments (6)
  1. [§3 Eq. (2)] Eq. (2) writes LocalTr_w on [v_i; v_i−v_{i−1}; v_{i+1}−v_i] for i=1..t−1. Boundary handling for i=1 and the use of v_{i+1} inside a causal “historical” path should be clarified (is v_{i+1} only within the past window, or is there leakage?).
  2. [§4 Hyperparameters and device] Hyperparameters K (memory queries), window radius w, and λ_ord/λ_rank/λ_ben are free parameters listed only partially; a small table of defaults would aid reproduction.
  3. [Figure 2] Figure 1 / Figure 2 text is dense; expanding the caption to define e_loc, e_mem, s_t, r_t, and π_t in one place would help readers who skim figures first.
  4. [§5 Related Work; Tables 5–6] Related Work cites several 2025–2026 arXiv items appropriately; ensure camera-ready versions and venue names are updated where available, and that ICON (Lin et al. 2026) is consistently treated as excluded from training in all tables.
  5. [Abstract] Minor typography: missing spaces in several compound phrases in the Abstract PDF text (“Aslargelanguagemodels”, “toaddressthislimitation”); clean for the camera-ready.
  6. [Table 2; RQ2] Category-wise FAR varies widely (Misinformation 2.4% vs Privacy 24.6% in Table 2); a brief discussion of calibration or threshold-per-category would strengthen the deployment claims in RQ2.

Circularity Check

0 steps flagged

No derivation circularity: Recast is standard supervised time-to-risk forecasting; headline metrics are empirical hold-out results, not quantities forced by definition or self-citation.

full rationale

The paper's load-bearing chain is (i) dual-scale retrieval and compositional state/transition features, (ii) a causal temporal encoder producing π_t over discrete time-to-first-risk buckets, (iii) NLL training on labeled first-failure turns plus auxiliary ordinal/rank/benign losses, and (iv) empirical evaluation (NLL, Expected-MAE, EWR/FAR/MLT, ASR reduction). None of these steps reduces algebraically to its inputs: y_t = min(t★−t, H+1) is external supervision from annotated trajectories, not a quantity defined from π_t; the network is optimized to match those labels and then scored on held-out trajectories and (for ASR) unseen attack families. There is no fitted scalar renamed as a prediction, no uniqueness theorem imported from the authors to forbid alternatives, and no ansatz smuggled in via self-citation that forces the forecasting equations. Self-citations (e.g., ICON as an excluded attack baseline; prior Lin/Kong survey work) are ordinary related-work or evaluation references and are not load-bearing premises of the method. Concerns about LLM-assisted dataset construction and label fidelity affect external validity and transfer, not circularity of the claimed derivation. Per the analyzer rules this is an honest non-finding: score 0, no circular steps.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 3 invented entities

The central empirical claim rests on standard sequence-modeling machinery plus several modeling and data assumptions: that frozen text embeddings plus learned positional/turn structure suffice; that short- and long-term retrieved evidence composed with elementwise interaction and explicit Δ-transitions carry forecastable risk signal; that a fixed horizon H and threshold τ_a define operational warnings; and that author-constructed harmful/benign trajectories with annotated t★ are valid supervision for ‘true’ future safety failures in black-box chat.

free parameters (5)
  • forecasting_horizon_H = 3 (main tables)
    Discretizes time-to-risk and defines π_t(>H); main results fix H=3 after a sweep showing accuracy/warning tradeoffs.
  • warning_threshold_tau_a = not fixed in main text; case study uses 0.5
    Converts risk score R_t=1-π_t(>H) into binary early warnings; EWR/FAR/MLT and online ASR depend on this operating point (sweep 0.3–0.7).
  • loss_weights_lambda_ord_rank_ben = unspecified numeric values
    Balance NLL against ordinal escalation, ranking, and benign false-alarm losses; affect calibration of FAR vs EWR.
  • memory_queries_K_and_window_w = unspecified
    Control long-term memory slots and short-term attention radius in evidence retrieval; capacity/inductive bias knobs.
  • optimizer_and_train_hparams = lr=4e-5, wd=1e-2, epochs=24, seeds={42,43,44}
    AdamW lr 4e-5, weight decay 1e-2, batch 64, 24 epochs, three seeds—standard fit controls for reported averages.
axioms (6)
  • domain assumption Black-box textual trajectory τ≤t alone determines a learnable conditional distribution over future first safety-risk turn without model internals.
    Stated in §2 Problem Formulation; load-bearing for the entire safeguard setting.
  • domain assumption Multi-turn safety risk is compositional risk evolution: benign fragments can be reactivated and recomposed into pre-violation configurations whose state transitions forecast failure.
    Core modeling thesis in §1 and §3; justifies dual-scale retrieval and s_t / r_t design.
  • domain assumption A single annotated first-risk turn t★ per harmful trajectory is a well-defined supervision target for ‘safety failure’ across seven risk categories.
    Defines y_t and all forecasting metrics; depends on dataset annotation pipeline in Appendix.
  • ad hoc to paper Frozen bge-large-en-v1.5 turn embeddings plus learned positional embeddings are adequate features for risk-relevant semantics.
    Experimental setup; no comparison to end-to-end fine-tuned encoders in main ablations.
  • standard math Standard multi-head attention, windowed/causal transformers, and softmax classification heads are valid sequence learners for this task.
    Equations (1)–(8); conventional deep learning background.
  • ad hoc to paper Hard benign trajectories obtained by LLM rewrite of failure-supporting turns remain topically matched but truly non-failing, so FAR is meaningful.
    Stage 3 dataset construction; critical for claimed 12.3% FAR.
invented entities (3)
  • Compositional risk state s_t no independent evidence
    purpose: Structured encoding of short-term evidence, long-term memory evidence, and their elementwise conjunction at turn t.
    Architectural latent, not an externally measured physical quantity; defined in Eq. (5).
  • Risk transition representation r_t no independent evidence
    purpose: Encode signed deltas of local/memory/composition features between consecutive turns for dynamics.
    Defined in Eq. (6); usefulness supported only by ablation within this paper.
  • Dual-scale cross-path evidence (e_loc_t, e_mem_t) with K memory queries no independent evidence
    purpose: Retrieve complementary short-horizon dynamics and reactivated long-horizon factors conditioned on current turn.
    Method-specific retrieval construct (Eqs. 2–4); no external measurement protocol outside Recast.

pith-pipeline@v1.2.0-daily-grok45 · 25636 in / 4260 out tokens · 88408 ms · 2026-07-30T20:07:55.494105+00:00 · methodology

0 comments
read the original abstract

As large language models (LLMs) evolve from standalone assistants into autonomous agents, ensuring their safety requires shifting beyond pointwise risk assessment to understand how risks emerge and unfold over long-horizon trajectories. In multi-turn interactions, malicious intent can be decomposed across seemingly harmless turns and gradually reconstructed through interaction trajectories, eventually resulting in safety failures. Existing safeguards remain largely reactive, detecting manifested violations while lacking the ability to predict latent risk evolution and enable preemptive prevention. To address this limitation, we propose Recast, a safety risk forecasting framework that advances LLM safeguarding beyond turn-level violation detection to trajectory-level risk prediction. Recast first retrieves risk-relevant evidence from both short-term dialogue progression and long-term historical context via a dual-scale trajectory view. It then models compositional risk evolution by capturing the current risk configuration and its temporal dynamics. Finally, a causal temporal encoder learns latent risk evolution patterns and predicts the distribution of future risk emergence turns. Extensive experiments across 7 risk categories show that Recast predicts 88.3% of future safety failures with an average lead time of 2.41 turns, while maintaining a false alarm rate of 12.3%, showcasing the effectiveness of trajectory-level forecasting in identifying emerging risks before safety violations occur.

Figures

Figures reproduced from arXiv: 2607.26820 by Chenpei Wang, Dezhang Kong, Dinghao Liu, Peng Qian, Renjie Sun, Shi Lin, Sifan Wu, Xun Wang.

Figure 1
Figure 1. Figure 1: Existing safeguards: detect manifested risks. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Recast consists of three components: (1) risk-relevant historical evidence retrieval for capturing short- and long-term trajectory dependencies; (2) compositional risk state and transition modeling that characterizes risk composition and temporal dynamics; and (3) future risk distribution forecasting for predicting risk emergence from pre-failure evolution patterns. encode the combined text into a shared s… view at source ↗
Figure 5
Figure 5. Figure 5: Warning threshold sensitivity. Model Variant Risk Forecasting Early Warning NLL Expected-MAE EWR (%) FAR (%) Recast 0.45 0.31 88.3 12.3 w/o Short-term progression 0.50(↑ 11.1%) 0.36(↑ 16.1%) 81.3(↓ 7.9%) 19.4(↑ 57.7%) w/o Long-term context 0.52(↑ 15.6%) 0.37(↑ 19.4%) 80.8(↓ 8.5%) 18.3(↑ 48.8%) w/o Cross-path evidence 0.51(↑ 13.3%) 0.38(↑ 22.6%) 83.5(↓ 5.4%) 19.5(↑ 58.5%) w/o Risk state transition 0.53(↑ 17… view at source ↗
Figure 4
Figure 4. Figure 4: Performance across dialogue length groups. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Risk separation between harmful and benign tra [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Risk evolution along a real-world interaction. [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Comparison with direct trajectory forecasting baselines. [PITH_FULL_IMAGE:figures/full_fig_p012_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Analysis of the vehicle theft case [PITH_FULL_IMAGE:figures/full_fig_p013_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Prompt used for trajectory expansion, failure-supporting turn identification, hard benign trajectory construction, and [PITH_FULL_IMAGE:figures/full_fig_p014_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

86 extracted references · 19 linked inside Pith

  1. [2]

    Anthropic. 2023. Introducing Claude. https://www.anthropic.com/news/introducing-claude

    2023

  2. [3]

    J.; and Wong, E

    Chao, P.; Robey, A.; Dobriban, E.; Hassani, H.; Pappas, G. J.; and Wong, E. 2025. Jailbreaking black box large language models in twenty queries. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), 23--42. IEEE

    2025

  3. [4]

    Ding, P.; Kuang, J.; Ma, D.; Cao, X.; Xian, Y.; Chen, J.; and Huang, S. 2024. A wolf in sheep’s clothing: Generalized nested jailbreak prompts can fool large language models easily. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2136--2153

    2024

  4. [5]

    N.; Padmakumar, A.; Rebedea, T.; Varghese, J

    Ghosh, S.; Varshney, P.; Sreedhar, M. N.; Padmakumar, A.; Rebedea, T.; Varghese, J. R.; and Parisien, C. 2025. Aegis2. 0: A diverse ai safety dataset and risks taxonomy for alignment of llm guardrails. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies ...

    2025

  5. [9]

    Y.; Lambert, N.; Choi, Y.; and Dziri, N

    Han, S.; Rao, K.; Ettinger, A.; Jiang, L.; Lin, B. Y.; Lambert, N.; Choi, Y.; and Dziri, N. 2024. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. Advances in neural information processing systems, 37: 8093--8131

    2024

  6. [11]

    Jiang, Y.; Aggarwal, K.; Laud, T.; Munir, K.; Pujara, J.; and Mukherjee, S. 2025. Red queen: Exposing latent multi-turn risks in large language models. In Findings of the Association for Computational Linguistics: ACL 2025, 25554--25591

    2025

  7. [13]

    Li, N.; Han, Z.; Steneker, I.; Primack, W.; Goodside, R.; Zhang, H.; Wang, Z.; Menghini, C.; and Yue, S. 2025. LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet. In Red Teaming GenAI: What Can We Learn from Adversaries?

    2025

  8. [17]

    Liu, X.; Xu, N.; Chen, M.; and Xiao, C. 2024 b . Autodan: Generating stealthy jailbreak prompts on aligned large language models. In International Conference on Learning Representations, volume 2024, 56174--56194

    2024

  9. [18]

    Liu, X.; Yu, H.; Zhang, H.; Xu, Y.; Lei, X.; Lai, H.; Gu, Y.; Ding, H.; Men, K.; Yang, K.; et al. 2024 c . Agentbench: Evaluating llms as agents. In International Conference on Learning Representations, volume 2024, 52989--53046

    2024

  10. [20]

    Mehrotra, A.; Zampetakis, M.; Kassianik, P.; Nelson, B.; Anderson, H.; Singer, Y.; and Karbasi, A. 2024. Tree of attacks: Jailbreaking black-box llms automatically. Advances in Neural Information Processing Systems, 37: 61065--61105

    2024

  11. [21]

    Peng, K.; Ding, L.; Zhong, Q.; Shen, L.; Liu, X.; Zhang, M.; Ouyang, Y.; and Tao, D. 2023. Towards making the most of chatgpt for machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2023, 5622--5633

    2023

  12. [22]

    Perez, E.; Huang, S.; Song, F.; Cai, T.; Ring, R.; Aslanides, J.; Glaese, A.; McAleese, N.; and Irving, G. 2022. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 3419--3448

    2022

  13. [23]

    Qin, C.; Zhang, A.; Zhang, Z.; Chen, J.; Yasunaga, M.; and Yang, D. 2023. Is ChatGPT a general-purpose natural language processing task solver? In Proceedings of the 2023 conference on empirical methods in natural language processing, 1339--1384

    2023

  14. [24]

    R.; Palangi, H.; Chang, K.-W.; Choi, Y.; and Gabriel, S

    Rahman, S.; Jiang, L.; Shiffer, J.; Liu, G.; Issaka, S.; Parvez, M. R.; Palangi, H.; Chang, K.-W.; Choi, Y.; and Gabriel, S. 2025. X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents. In Second Conference on Language Modeling

    2025

  15. [25]

    Ren, Q.; Li, H.; Liu, D.; Xie, Z.; Lu, X.; Qiao, Y.; Sha, L.; Yan, J.; Ma, L.; and Shao, J. 2025. Llms know their vulnerabilities: Uncover safety gaps through natural distribution shifts. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 24763--24785

    2025

  16. [26]

    Russinovich, M.; Salem, A.; and Eldan, R. 2025. Great, now write an article about that: The crescendo Multi-Turn LLM jailbreak attack. In 34th USENIX Security Symposium (USENIX Security 25), 2421--2440

    2025

  17. [27]

    Schick, T.; Dwivedi-Yu, J.; Dess \` , R.; Raileanu, R.; Lomeli, M.; Hambro, E.; Zettlemoyer, L.; Cancedda, N.; and Scialom, T. 2023. Toolformer: Language models can teach themselves to use tools. Advances in neural information processing systems, 36: 68539--68551

    2023

  18. [28]

    Shen, X.; Wei, R.; Niu, P.; Wang, H.; Wu, R.; Chien, E.; Li, B.; Chen, P.-Y.; and Li, P. 2026. One Turn Too Late: Response-Aware Defense Against Hidden Malicious Intent in Multi-Turn Dialogue. In Second Workshop on Agents in the Wild: Safety, Security, and Beyond

    2026

  19. [30]

    Wei, A.; Haghtalab, N.; and Steinhardt, J. 2023. Jailbroken: How does llm safety training fail? Advances in neural information processing systems, 36: 80079--80110

    2023

  20. [31]

    V.; Zhou, D.; et al

    Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; Zhou, D.; et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 24824--24837

    2022

  21. [32]

    Weng, Z.; Jin, X.; Jia, J.; and Zhang, X. 2025. Foot-in-the-door: A multi-turn jailbreak for llms. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 1939--1950

    2025

  22. [33]

    Wu, Q.; Bansal, G.; Zhang, J.; Wu, Y.; Li, B.; Zhu, E.; Jiang, L.; Zhang, X.; Zhang, S.; Liu, J.; et al. 2024. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. In ICLR 2024 Workshop on Large Language Model (LLM) Agents

    2024

  23. [34]

    Xie, T.; Qi, X.; Zeng, Y.; Huang, Y.; Sehwag, U.; Huang, K.; He, L.; Wei, B.; Li, D.; Sheng, Y.; et al. 2025. Sorry-bench: Systematically evaluating large language model safety refusal. In International Conference on Learning Representations, volume 2025, 59937--59973

    2025

  24. [37]

    Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; and Cao, Y. 2023. REACT: SYNERGIZING REASONING AND ACTING IN LANGUAGE MODELS. In 11th International Conference on Learning Representations, ICLR 2023

    2023

  25. [38]

    Yu, J.; Lin, X.; Yu, Z.; and Xing, X. 2024. LLM-Fuzzer : Scaling Assessment of Large Language Model Jailbreaks. In 33rd USENIX Security Symposium (USENIX Security 24), 4657--4674. Philadelphia, PA: USENIX Association. ISBN 978-1-939133-44-1

    2024

  26. [40]

    Zeng, Y.; Wu, Y.; Zhang, X.; Wang, H.; and Wu, Q. 2024 b . AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks. In Neurips Safe Generative AI Workshop 2024

    2024

  27. [41]

    B.; and Gan, C

    Zhang, S.; Chen, Z.; Shen, Y.; Ding, M.; Tenenbaum, J. B.; and Gan, C. 2023. Planning with Large Language Models for Code Generation. In The Eleventh International Conference on Learning Representations

    2023

  28. [42]

    Zhang, Y.; Ding, L.; Zhang, L.; and Tao, D. 2025. Intention analysis makes llms a good jailbreak defender. In Proceedings of the 31st International Conference on Computational Linguistics, 2947--2968

    2025

  29. [47]

    https://www.anthropic.com/news/introducing-claude , year=

    Introducing Claude , author=. https://www.anthropic.com/news/introducing-claude , year=

  30. [48]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    Pith/arXiv arXiv

  31. [49]

    arXiv preprint arXiv:2412.19437 , year=

    Deepseek-v3 technical report , author=. arXiv preprint arXiv:2412.19437 , year=

    Pith/arXiv arXiv

  32. [50]

    arXiv preprint arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    Pith/arXiv arXiv

  33. [51]

    arXiv preprint arXiv:2406.12793 , year=

    Chatglm: A family of large language models from glm-130b to glm-4 all tools , author=. arXiv preprint arXiv:2406.12793 , year=

    Pith/arXiv arXiv

  34. [52]

    Proceedings of the 2023 conference on empirical methods in natural language processing , pages=

    Is ChatGPT a general-purpose natural language processing task solver? , author=. Proceedings of the 2023 conference on empirical methods in natural language processing , pages=

    2023

  35. [53]

    arXiv preprint arXiv:2302.10198 , year=

    Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert , author=. arXiv preprint arXiv:2302.10198 , year=

    Pith/arXiv arXiv

  36. [54]

    The Eleventh International Conference on Learning Representations , year=

    Planning with Large Language Models for Code Generation , author=. The Eleventh International Conference on Learning Representations , year=

  37. [55]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    Towards making the most of chatgpt for machine translation , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    2023

  38. [56]

    Advances in neural information processing systems , volume=

    Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in neural information processing systems , volume=

  39. [57]

    Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    A wolf in sheep’s clothing: Generalized nested jailbreak prompts can fool large language models easily , author=. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    2024

  40. [58]

    Neurips Safe Generative AI Workshop 2024 , year=

    DeepInception: Hypnotize Large Language Model to Be Jailbreaker , author=. Neurips Safe Generative AI Workshop 2024 , year=

    2024

  41. [59]

    arXiv preprint arXiv:2506.19676 , year=

    A survey of llm-driven ai agent communication: Protocols, security risks, and defense countermeasures , author=. arXiv preprint arXiv:2506.19676 , year=

    arXiv

  42. [60]

    arXiv preprint arXiv:2303.08774 , year=

    Gpt-4 technical report , author=. arXiv preprint arXiv:2303.08774 , year=

    Pith/arXiv arXiv

  43. [61]

    arXiv preprint arXiv:2307.09288 , year=

    Llama 2: Open foundation and fine-tuned chat models , author=. arXiv preprint arXiv:2307.09288 , year=

    Pith/arXiv arXiv

  44. [62]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    Pith/arXiv arXiv

  45. [63]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=

  46. [64]

    arXiv preprint arXiv:2212.08073 , year=

    Constitutional ai: Harmlessness from ai feedback , author=. arXiv preprint arXiv:2212.08073 , year=

    Pith/arXiv arXiv

  47. [65]

    Advances in neural information processing systems , volume=

    Direct preference optimization: Your language model is secretly a reward model , author=. Advances in neural information processing systems , volume=

  48. [66]

    11th International Conference on Learning Representations, ICLR 2023 , year=

    REACT: SYNERGIZING REASONING AND ACTING IN LANGUAGE MODELS , author=. 11th International Conference on Learning Representations, ICLR 2023 , year=

    2023

  49. [67]

    Advances in neural information processing systems , volume=

    Toolformer: Language models can teach themselves to use tools , author=. Advances in neural information processing systems , volume=

  50. [68]

    ICLR 2024 Workshop on Large Language Model (LLM) Agents , year=

    AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation , author=. ICLR 2024 Workshop on Large Language Model (LLM) Agents , year=

    2024

  51. [69]

    International Conference on Learning Representations , volume=

    Agentbench: Evaluating llms as agents , author=. International Conference on Learning Representations , volume=

  52. [70]

    arXiv preprint arXiv:2307.15043 , year=

    Universal and transferable adversarial attacks on aligned language models , author=. arXiv preprint arXiv:2307.15043 , year=

    Pith/arXiv arXiv

  53. [71]

    Advances in neural information processing systems , volume=

    Jailbroken: How does llm safety training fail? , author=. Advances in neural information processing systems , volume=

  54. [72]

    2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) , pages=

    Jailbreaking black box large language models in twenty queries , author=. 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) , pages=. 2025 , organization=

    2025

  55. [73]

    Advances in Neural Information Processing Systems , volume=

    Tree of attacks: Jailbreaking black-box llms automatically , author=. Advances in Neural Information Processing Systems , volume=

  56. [74]

    International Conference on Learning Representations , volume=

    Autodan: Generating stealthy jailbreak prompts on aligned large language models , author=. International Conference on Learning Representations , volume=

  57. [75]

    33rd USENIX Security Symposium (USENIX Security 24) , year =

    Jiahao Yu and Xingwei Lin and Zheng Yu and Xinyu Xing , title =. 33rd USENIX Security Symposium (USENIX Security 24) , year =

  58. [76]

    34th USENIX Security Symposium (USENIX Security 25) , pages=

    Great, now write an article about that: The crescendo Multi-Turn LLM jailbreak attack , author=. 34th USENIX Security Symposium (USENIX Security 25) , pages=

  59. [77]

    arXiv preprint arXiv:2405.05610 , year=

    Chain of attack: a semantic-driven contextual multi-turn attacker for llm , author=. arXiv preprint arXiv:2405.05610 , year=

    Pith/arXiv arXiv

  60. [78]

    Red Teaming GenAI: What Can We Learn from Adversaries? , year=

    LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet , author=. Red Teaming GenAI: What Can We Learn from Adversaries? , year=

  61. [79]

    Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    Red queen: Exposing latent multi-turn risks in large language models , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=

    2025

  62. [80]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Llms know their vulnerabilities: Uncover safety gaps through natural distribution shifts , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  63. [81]

    Second Conference on Language Modeling , year=

    X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents , author=. Second Conference on Language Modeling , year=

  64. [82]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Foot-in-the-door: A multi-turn jailbreak for llms , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  65. [83]

    arXiv preprint arXiv:2601.20903 , year=

    ICON: Intent-Context Coupling for Efficient Multi-Turn Jailbreak Attack , author=. arXiv preprint arXiv:2601.20903 , year=

    arXiv

  66. [84]

    arXiv preprint arXiv:2503.10619 , year=

    Tempest: Autonomous multi-turn jailbreaking of large language models with tree search , author=. arXiv preprint arXiv:2503.10619 , year=

    Pith/arXiv arXiv

  67. [85]

    arXiv preprint arXiv:2312.06674 , year=

    Llama guard: Llm-based input-output safeguard for human-ai conversations , author=. arXiv preprint arXiv:2312.06674 , year=

    Pith/arXiv arXiv

  68. [86]

    Advances in neural information processing systems , volume=

    Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms , author=. Advances in neural information processing systems , volume=

  69. [87]

    arXiv preprint arXiv:2407.21772 , year=

    Shieldgemma: Generative ai content moderation based on gemma , author=. arXiv preprint arXiv:2407.21772 , year=

    Pith/arXiv arXiv

  70. [88]

    arXiv preprint arXiv:2510.14276 , year=

    Qwen3guard technical report , author=. arXiv preprint arXiv:2510.14276 , year=

    Pith/arXiv arXiv

  71. [89]

    Neurips Safe Generative AI Workshop 2024 , year=

    AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks , author=. Neurips Safe Generative AI Workshop 2024 , year=

    2024

  72. [90]

    Proceedings of the 31st International Conference on Computational Linguistics , pages=

    Intention analysis makes llms a good jailbreak defender , author=. Proceedings of the 31st International Conference on Computational Linguistics , pages=

  73. [91]

    arXiv preprint arXiv:2509.26345 , year=

    SafeBehavior: Simulating Human-Like Multistage Reasoning to Mitigate Jailbreak Attacks in Large Language Models , author=. arXiv preprint arXiv:2509.26345 , year=

    arXiv

  74. [92]

    Second Workshop on Agents in the Wild: Safety, Security, and Beyond , year=

    One Turn Too Late: Response-Aware Defense Against Hidden Malicious Intent in Multi-Turn Dialogue , author=. Second Workshop on Agents in the Wild: Safety, Security, and Beyond , year=

  75. [93]

    arXiv preprint arXiv:2606.01738 , year=

    THRD: A Training-Free Multi-Turn Defense Framework for Jailbreak Attacks on Large Language Models , author=. arXiv preprint arXiv:2606.01738 , year=

    Pith/arXiv arXiv

  76. [94]

    0: A diverse ai safety dataset and risks taxonomy for alignment of llm guardrails , author=

    Aegis2. 0: A diverse ai safety dataset and risks taxonomy for alignment of llm guardrails , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    2025

  77. [95]

    arXiv preprint arXiv:2402.04249 , year=

    Harmbench: A standardized evaluation framework for automated red teaming and robust refusal , author=. arXiv preprint arXiv:2402.04249 , year=

    Pith/arXiv arXiv

  78. [96]

    Advances in Neural Information Processing Systems , volume=

    Jailbreakbench: An open robustness benchmark for jailbreaking large language models , author=. Advances in Neural Information Processing Systems , volume=

  79. [97]

    Advances in Neural Information Processing Systems , volume=

    A strongreject for empty jailbreaks , author=. Advances in Neural Information Processing Systems , volume=

  80. [98]

    International Conference on Learning Representations , volume=

    Sorry-bench: Systematically evaluating large language model safety refusal , author=. International Conference on Learning Representations , volume=

Showing first 80 references.