Pith. sign in

REVIEW 4 major objections 4 minor 34 references

Multi-turn LLM risk accumulates as a trajectory property, and a session-level scorer trained only on synthetic conversations can detect human-authored escalation attacks with AUROC 0.929.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 12:23 UTC pith:CET45QBV

load-bearing objection A genuinely useful taxonomy and benchmark with unusually honest caveats, but the headline transfer claim is provenance-confounded and the code isn't out yet. the 4 major comments →

arxiv 2607.19361 v1 pith:CET45QBV submitted 2026-06-03 cs.CL cs.AI

Stateful Guardrails for Multi-Turn LLM Systems: A Conversational Risk Accumulation Framework

Sanjay Mishra , Divya Chukkapalli , Ganesh R. Naik This is my paper
classification cs.CL cs.AI
keywords LLM safetymulti-turn conversationsconversational risk accumulationsession-layer guardrailssemantic driftinformation accumulation graphcompliance gradientdomain-adversarial training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that current guardrails are turn-level and therefore blind to harm that only emerges over a conversation: fragmented assembly of a prohibited request, gradual compliance erosion, and information build-up from repeated disclosures. It formalizes this failure class as Conversational Risk Accumulation and proposes a session-layer framework that tracks three signals — semantic drift from an anchored intent, a sensitivity-weighted information accumulation graph, and a compliance-gradient trend — fused into a single CRA score. The main model is CRA-Net DA, a compact recurrent network trained with adversarial objectives to ignore threat-family surface cues. The paper reports that, after calibration against benign chat, CRA-Net DA flags essentially all risky sessions on its benchmark at a 1% benign false-positive rate, and that training only on synthetic sessions transfers to human-authored attacks with AUROC 0.929. A sympathetic reader would care because it makes a concrete case that session-state monitoring can be lightweight, interpretable, and deployable alongside existing turn-level filters.

Core claim

The central discovery is a demonstration that multi-turn safety failures can be scored as trajectory events rather than single-turn events, using three complementary signals. The paper defines CRA as a session in which every turn passes a turn-level filter but a stateful scoring function can detect accumulated risk, and it operationalizes this with a semantic drift monitor, an information accumulation graph, and a compliance-gradient detector. CRA-Net DA, a roughly 52K-parameter GRU with gradient-reversal heads for session length and threat-family invariance, is the primary fusion model. On CRA-Bench v0.2 — LLM-paraphrased, five threat families, length-matched sessions — it achieves TPR 1.00

What carries the argument

CRA-Net DA: a compact two-layer GRU (hidden width 128, roughly 5.2×10^4 parameters) that consumes per-turn features — normalized semantic drift (S1), an information accumulation index from the sensitivity-weighted entity graph (S2), the negated slope of a windowed refusal trend (S3), plus length and turn-index features — and emits a session risk score. Two gradient-reversal adversarial heads force the encoder representation to be uninformative about session length and threat family, which removes template-memorization and suppresses false alarms on benign chat. A convex fusion of the three normalized signals is retained as an interpretable diagnostic decomposition rather than the primary det

Load-bearing premise

The framework's central claim depends on the engineered trajectory signals — especially the compliance-gradient S3, which is computed from a keyword refusal proxy — measuring genuine conversational risk dynamics rather than artifacts of the synthetic templates; the paper's own audit reports that this proxy agrees with a guard model's labels on only 11.5% of windows (κ = 0.00).

What would settle it

Evaluate CRA-Net DA zero-shot on a length-matched human corpus of fragmentation-style attacks in which the assistant never shows declining refusal behavior; if AUROC drops to about 0.5, the compliance-gradient mechanism is not carrying the transfer claim. Alternatively, replace the keyword-based S3 with a trained refusal classifier and re-run Human-CRA-Transfer; a large score shift would show the reported numbers depend on the unreliable proxy.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Turn-level filters can be complemented by a session layer that scores the trajectory and emits per-turn alerts, giving operators time-to-detection rather than post-hoc transcript review.
  • If the synthetic-to-human transfer holds, organizations can bootstrap session-risk detectors from generated benchmarks without large labeled human corpora.
  • Benign-anchored calibration — choosing thresholds to constrain false alarms on ordinary chat before measuring detection — becomes a necessary step for deploying such scorers.
  • The leave-one-family-out diagnostics imply that detectors trained on a few threat families should not be expected to generalize to unseen accumulation mechanisms; family coverage needs to scale with deployment scope.
  • Interpretable decision certificates tied to the three sub-signals can give auditors a policy-safe explanation of why a session was flagged.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The S3 compliance-gradient signal is the least trusted leg: the paper's own audit reports that the keyword refusal proxy agrees with a guard model's labels on only 11.5% of windows (κ = 0.00). If a trained refusal classifier replaces the proxy and the results hold, the framework is robust; if they weaken, the headline claims may be carried more by template structure than by genuine compliance dyna
  • The same trajectory machinery could be applied to agentic tool-use traces, treating tool calls and data access as first-class turns; the paper sketches this but does not evaluate it, so the benefit is untested.
  • A natural next experiment is to generate CRA sessions that hold S1 and S2 constant while varying only S3, and vice versa, to isolate which signal carries each family's signal; the current ablations do not fully separate them.
  • The 1% benign-FPR result is measured on a particular open-chat corpus; real RAG or agent traffic may have different score distributions, so the operating point should be recalibrated per deployment, as the paper itself notes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Conversational Risk Accumulation (CRA) as a failure class in which multi-turn dialogues become unsafe through gradual intent drift, fragmented assembly of prohibited instructions, or sensitivity buildup, even though each turn appears benign. It defines three trajectory signals (semantic drift S1, an Information Accumulation Graph index S2, and a compliance-gradient signal S3), combines them either through convex fusion or through a learned GRU-based CRA-Net with length- and family-adversarial training (CRA-Net DA), and releases CRA-Bench v0.1/v0.2 plus a five-family expansion. The central empirical claims are (i) within-distribution performance on CRA-Bench, culminating in TPR=1.000 at ≤1% benign FPR for CRA-Net DA after benign-anchored calibration (Table 16), and (ii) synthetic-to-human transfer: CRA-Net DA trained only on CRA-Bench reaches AUROC 0.929 on Human-CRA-Transfer (Table 17). The paper is unusually explicit about its limitations, including the CoSafe length confound, the inversion of the IAG signal on public-figure benign twins, and the failure of its S3 keyword proxy validation.

Significance. If the transfer result survives a provenance control, the paper makes a useful contribution: it gives a reproducible session-layer scoring architecture, a public benchmark with session-level splits, per-turn time-to-detection, and an honest evaluation protocol that exposes template memorization and length confounds. The release of code, result JSONs, and fixed-seed generation scripts is a real strength, as are the LOFO diagnostics and the explicit separation of within-distribution, paraphrase-stress, and human-transfer evidence. However, the scientific weight of the paper rests on the Human-CRA-Transfer claim and on the validity of the S3 signal, and both currently have load-bearing gaps. The CRA-Bench numbers alone are self-referential because the labels are produced by the same template machinery that instantiates the authors' definition of CRA; thus the human-transfer result is the main external anchor, and it must be made robust to corpus-provenance confounds.

major comments (4)
  1. [§8.10, Eq. (6)] The Human-CRA-Transfer result (AUROC 0.919–0.929, Table 17) is confounded by provenance: positives are CoSafe red-team sessions and negatives are ShareGPT organic chats, matched only on length. The per-turn feature vector in Eq. (6) includes response length ℓ_t, refusal probability r_t, and moderation score c_t, any of which can separate CoSafe from ShareGPT independently of accumulation. The paper does not provide a source-matched control or a turn-order-scrambling experiment. Please add controls such as (i) permuting the order of user turns within Human-CRA-Transfer sessions and re-measuring AUROC, and (ii) replacing the ShareGPT negatives with non-escalating multi-turn CoSafe sessions (or vice versa) to show the model is not separating corpora. Without this, the headline synthetic-to-real transfer claim is not established.
  2. [§4.3, Appendix 13.2] The S3 compliance-gradient signal fails its own validation: Cohen's κ=0.00 and raw agreement 11.5% with Llama Guard 3 on 200 windows. Since S3 is one of the three core mechanisms and is the main driver of the Behavioral Conditioning family results (Tables 11 and 14), the current evidence does not show that S3 measures refusal softening; the conditioning-family AUROC may reflect template-specific keyword behavior rather than genuine compliance dynamics. Re-run the central results with a validated refusal classifier, or at least report all headline tables with S3 ablated and quantify the delta. The manuscript already contains partial ablations (e.g., CRA-convex\S3 and CRA-Net noS2/S3), but the full CRA-Net DA numbers on Human-CRA-Transfer are not reported without S3.
  3. [§6.2, §8.9, Table 16] CRA-Bench labels are generated by a template generator designed to instantiate the authors' own CRA definition, and the operating thresholds and hyperparameters are tuned on validation splits from the same generator. This creates a self-referential loop for the within-distribution claims, including the TPR=1.000 at ≤1% benign FPR in Table 16. The paraphrase stress test and the human-transfer subset partly break the loop, but the paper should provide either an independent human annotation study on CRA-Bench or another external validation anchor before using the within-distribution operating point as a production recommendation. At minimum, clearly state that Table 16's TPR applies to synthetic template sessions, not to human-authored attacks.
  4. [§3 footnote, Tables 11 and 14] The IAG/S2 signal inverts on benign public-figure twins: per-family AUROC for Aggregation is near zero for all baseline methods that rely on S2 (Tables 11 and 14), because public historical figures trigger more NER hits than fictional targets. This is acknowledged in the Table 3 footnote, but it means the 'most structurally novel' component S2 does not, by itself, measure sensitivity accumulation. CRA-Net recovers aggregation by learned compensation, but this raises a transfer risk: the compensation may be specific to the benchmark's entity-choice asymmetry. Please provide an IAG-only analysis on a corpus with real sensitive entities, or explicitly reposition S2 as an unvalidated placeholder in the contribution claims.
minor comments (4)
  1. [§8.5, §9.7] The benign-FPR estimates rely on ShareGPT filtered with unsafe-keyword exclusion. This filter is not a gold-standard benign label; please state the filtering heuristic and its limitations more prominently, since a false-alarm rate on 'benign traffic' is only as good as the benign proxy.
  2. [Table 10] The row 'CRA-Net noS2/S3 (drift-only)' lists AUROC 1.000 with sFPR 1.000, which is a degenerate saturated ranking. Consider moving this row to a footnote or clearly marking it as a degenerate configuration in the table itself, not only in the table caption.
  3. [§7 / Figures 2–10] The synthetic signal illustrations are appropriately labeled as non-empirical, but the Figures are low-resolution placeholders in the submitted PDF. Re-render them at publication quality and ensure axis labels are legible.
  4. [§4.6, §8.8] The default λ_fam changes from 0.3 (three-family) to 0.1 (five-family) with retrospective justification. This is defensible, but the selection of λ_fam should be reported as a hyperparameter sweep over validation, not as a fixed default, to avoid the appearance of post hoc tuning.

Circularity Check

0 steps flagged

No significant circularity: the benchmark, validation, and human-transfer numbers are held out and no equation reduces to a fitted input.

full rationale

The derivation chain is not circular. The CRA definition (Sec. 3) is an operational taxonomy rather than a fitted quantity, and the benchmark labels are generated from templates, but the model is trained on a 60% split and all headline AUROC/TPR numbers are reported on a held-out 20% test split (Sec. 8.2), with thresholds calibrated either on a mixed validation split or on external ShareGPT sessions (Sec. 8.9, Table 16). No equation in Sec. 4 reduces to a fitted constant: CRA(t) is a fixed convex combination of S1-S3, and CRA-Net parameters are learned by BCE; AUROC is threshold-independent. The synthetic-to-human transfer claim (Sec. 8.10) is an external check: CRA-Net DA is trained only on CRA-Bench v0.2 and evaluated zero-shot on 750 CoSafe positives + 222 ShareGPT negatives with equal user-turn count, with no human-corpus threshold fitting. The disclosed S3 proxy validation (Appendix 13.2, Cohen's kappa=0.00, 11.5% agreement) and the potential provenance confound in Human-CRA-Transfer are validity/correctness concerns, not definitional reductions; they do not make any reported number equal to its input by construction. There are no load-bearing self-citations, and the illustrative Sec. 7 results are explicitly labeled as non-empirical heuristic simulations. Everything load-bearing is either held out or externally sourced, so the central claims have independent empirical content.

Axiom & Free-Parameter Ledger

7 free parameters · 6 axioms · 4 invented entities

The central empirical contribution rests on a small set of hand-chosen signal parameters, a template-generated ground-truth benchmark, and the assumption that embeddings/NER/refusal proxies capture risk-relevant dynamics. The paper is transparent about many of these, but they are still assumptions, and the invented constructs (CRA taxonomy, IAG, S3) lack independent validation outside the authors' own benchmark.

free parameters (7)
  • Fusion weights α,β,γ = defaults 0.35,0.45,0.20; v0.2 grid peak α=0,β=0.05,γ=0.95
    Chosen by hand from synthetic trajectory statistics and later by validation grid search; the paper describes them as deployment-tunable policy configuration.
  • IAG sensitivity tier increments = 0.05,0.10,0.20,0.35
    Hand-chosen roughly doubling values mirroring NIST SP 800-60 confidentiality impact; no data fit, and the paper recommends a ±50% sensitivity sweep for new corpora.
  • IAG coverage exponent p = p=2
    Chosen to make attribute coverage super-additive; no empirical fitting reported.
  • S3 sliding window width w = 6 turns (cap 32)
    Selected as a latency/robustness trade-off; the paper derives a lower bound L_cond ≥ w but does not fit w to data.
  • S3 saturation gain κ = not specified in text
    Gain in σ(κ S3) normalization; no fitted value reported in the main text.
  • CRA-Net hyperparameters λ, λ_fam, ρ = λ=0.05; λ_fam=0.3 (3-family) / 0.1 (5-family); ρ=1
    Tuned on validation/LOFO diagnostics; the paper explicitly states λ_fam must be re-tuned per family pool size.
  • Operating threshold θ = e.g., θ=0.492 (CRA-Net DA, benign-anchored, 5-family); θ=0.697 (v0.2 TPR-0.90)
    Calibrated on a mixed validation split or on ShareGPT benign FPR; standard practice, but the reported TPR/sFPR numbers depend on it.
axioms (6)
  • domain assumption NIST SP 800-60 confidentiality categorisation is a valid basis for assigning attribute sensitivity weights in the IAG.
    Section 4.2 maps sensitivity tiers to numeric increments; no independent evidence is given that these specific increments reflect conversational risk.
  • domain assumption Cosine distance in all-MiniLM-L6-v2 embedding space measures semantic drift and opposition of session intent.
    Section 4.1 uses cos distance for S1; the paper provides no validation that this embedding captures policy-relevant intent drift.
  • domain assumption spaCy en_core_web_sm named-entity recognition is sufficient for the entity/attribute extraction underlying S2.
    Section 4.2 pins the reference implementation to spaCy; the paper's own aggregation results show that sparse NER hits invert the signal for fictional vs public-figure targets.
  • ad hoc to paper Template-generated sessions with labels assigned by the generator constitute ground-truth CRA risk.
    Section 6.2 and Section 8: CRA-Bench positives/negatives are produced by the authors' own templates; no external annotation establishes that these sessions represent real-world CRA.
  • domain assumption CoSafe gradual-escalation sessions are representative human-authored CRA positives, and filtered 3-turn ShareGPT chats are benign negatives.
    Section 8.10 / Table 17: the human-transfer corpus mixes two different data sources; source identity is a potential shortcut independent of CRA content.
  • standard math The OLS slope estimator of binary refusal indicators is a consistent estimator of compliance trend under linear trend plus bounded noise.
    Remark 1 in Section 5 invokes Gauss-Markov unbiasedness; the paper itself notes real refusal trajectories are non-linear, so the formal statement applies only to the idealized linear case.
invented entities (4)
  • Conversational Risk Accumulation (CRA) threat taxonomy no independent evidence
    purpose: To name and classify multi-turn safety failures (fragmentation, behavioral conditioning, aggregation leakage, intent drift, context poisoning).
    The five types are instantiated by the authors' own template generator; the paper's LOFO diagnostics show poor held-out-family generalization, so the taxonomy is not independently confirmed as a set of distinct mechanisms.
  • Information Accumulation Graph (IAG) no independent evidence
    purpose: To track sensitivity-weighted entity disclosures across turns as a graph.
    S2 in isolation inverts on the aggregation family (AUROC 0.159 on v0.1), and convex fusion using S2 is near chance; no isolated external validation supports the graph as a useful risk signal.
  • Compliance Gradient Detector (S3) no independent evidence
    purpose: To detect declining refusal/hedge behavior as a signature of behavioral conditioning.
    The paper's own proxy validation reports κ=0.00 and 11.5% agreement with Llama Guard 3 labels (Appendix 13.2), so the detector has no validated external handle.
  • Semantic Drift Monitor (S1) no independent evidence
    purpose: To detect displacement from an anchored session intent.
    S1-only baselines are near chance or below chance on CRA-Bench v0.1 (AUROC 0.553) and on length-matched human data, so the signal lacks independent demonstrated utility by itself.

pith-pipeline@v1.3.0-alltime-deepseek · 35207 in / 13503 out tokens · 132919 ms · 2026-08-02T12:23:21.630038+00:00 · methodology

0 comments
read the original abstract

Most safety guardrails for large language models (LLMs) evaluate each prompt-response pair in isolation, which misses failures that arise only over a dialogue as benign turns compose into harm. We term this Conversational Risk Accumulation (CRA): gradual intent drift, fragmented assembly of prohibited instructions, and sensitivity build-up from repeated disclosures. We propose a session-layer CRA Framework that tracks three trajectory signals: semantic drift from a session anchor, a sensitivity-weighted information accumulation graph over extracted entities, and a compliance-gradient signal capturing increasing willingness to comply. For scoring, we provide (i) an unsupervised convex fusion for attribution and ablations, and (ii) CRA-Net DA, a compact learned trajectory model trained with family-adversarial objectives to reduce length and topic-coverage confounds. To benchmark CRA, we release CRA-Bench v0.1 (1,200 eight-turn sessions across three threat families with topic-matched benign twins), CRA-Bench v0.2 (LLM-paraphrased variants to reduce template artifacts), and an extended 5-family set (2,000 sessions adding persona priming and context stuffing). We introduce a trajectory-native evaluation protocol with session-level splits, mixed-set threshold calibration, Trajectory AUROC, turns-to-detection, calibrated false-positive metrics, bootstrap confidence intervals, leave-one-family-out diagnostic stress tests, and synthetic-to-human transfer checks. Claims focus on within-distribution session scoring on CRA-Bench and human-transfer subsets.

Figures

Figures reproduced from arXiv: 2607.19361 by Divya Chukkapalli, Ganesh R. Naik, Sanjay Mishra.

Figure 1
Figure 1. Figure 1: Illustration of the CRA failure mode: turn-level risk scores can remain below a per-turn [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Composite CRA score trajectories across three [PITH_FULL_IMAGE:figures/full_fig_p017_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Sub-signal decomposition for the synthetic benign trajectory. All three signals remain low and stable, with no accumulation pattern present [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Sub-signal decomposition for the synthetic fragmentation scenario. The information￾accumulation signal S2 drives the composite score with a near-monotonic rise consistent with Re￾mark 1 (i), while the drift and compliance-gradient signals remain subdued. correlation matrix for the pooled signal set across all scenarios, and [PITH_FULL_IMAGE:figures/full_fig_p018_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Sub-signal decomposition for the synthetic behavioral conditioning scenario. The compliance-gradient signal S3 is the dominant contributor and shows a delayed but steady rise consistent with the windowed detection latency in Eq. (10) [PITH_FULL_IMAGE:figures/full_fig_p019_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Latency–window trade-off for behavioral conditioning ( [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Mean composite CRA by scenario (synthetic trajectories). Error bars show ±1 standard deviation. The separation is consistent with theoretical predictions for each CRA type; it is not an empirical discrimination result [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Faceted CRA score curves (one panel per synthetic scenario) on a shared vertical axis, illustrating the qualitatively distinct trajectory shapes produced by each CRA type. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Pearson correlation matrix for pooled signal rows across all three [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Scatter plot of the information-accumulation signal versus the composite CRA score, [PITH_FULL_IMAGE:figures/full_fig_p021_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Reference deployment pattern: CRA runs as a stateful session layer alongside existing [PITH_FULL_IMAGE:figures/full_fig_p036_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 9 linked inside Pith

  1. [1]

    CoSafe: A collection of multi-turn LLM conversations for safety evaluation

    Asap7772. CoSafe: A collection of multi-turn LLM conversations for safety evaluation. Hug- gingFace Dataset, 2024. URLhttps://huggingface.co/datasets/Asap7772/cosafe_all_ rollouts

    2024

  2. [2]

    Guide for mapping types of information and information systems to security categories

    National Institute of Standards and Technology. Guide for mapping types of information and information systems to security categories. Technical Report SP 800-60 Vol. 1 Rev. 1, NIST,

  3. [3]

    HarmBench: A standardized evaluation framework for automated red teaming and robust refusal

    Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. HarmBench: A standardized evaluation framework for automated red teaming and robust refusal. InProceedings of the International Conference on Machine Learning (ICML),

  4. [4]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. Universal and transferable adver- sarial attacks on aligned language models. InProceedings of the 41st International Conference on Machine Learning (ICML 2024), volume 235 ofProceedings of Machine Learning Research. PMLR, 2024. arXiv:2307.15043

    Pith/arXiv arXiv 2024

  5. [5]

    Universal adver- sarial triggers for attacking and analyzing NLP

    Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. Universal adver- sarial triggers for attacking and analyzing NLP. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCNLP), pages 2153–2162. Association for Compu...

    2019

  6. [6]

    Do Anything Now

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. “Do Anything Now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models. In Proceedings of the 2024 ACM SIGSAC Conference on Computer and Communications Security (CCS ’24), pages 1–15, New York, NY, USA, 2024. ACM. doi: 10.1145/3658644.3670388

    arXiv 2024

  7. [7]

    Jailbroken: How does LLM safety training fail? InAdvances in Neural Information Processing Systems, volume 36, pages 71988– 72006

    Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does LLM safety training fail? InAdvances in Neural Information Processing Systems, volume 36, pages 71988– 72006. Curran Associates, Inc., 2023

    2023

  8. [8]

    Ignore previous prompt: Attack techniques for language mod- els

    Fabio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language mod- els. InProceedings of the Workshop on Machine Learning Safety at NeurIPS 2022, 2022. arXiv:2211.09527

    Pith/arXiv arXiv 2022

  9. [9]

    Not what you’ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection

    Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection. InProceedings of the 16th ACM Workshop on Artificial In- telligence and Security (AISec ’23), pages 79–90, New York, NY, USA, 2023. ACM. doi: 10...

    arXiv 2023

  10. [10]

    Llama Guard: LLM-based input-output safeguard for human-AI conver- sations.arXiv preprint arXiv:2312.06674, 2023

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tung, et al. Llama Guard: LLM-based input-output safeguard for human-AI conver- sations.arXiv preprint arXiv:2312.06674, 2023. URLhttps://arxiv.org/abs/2312.06674

    Pith/arXiv arXiv 2023

  11. [11]

    Qwen3Guard: A multilingual safety guardrail model for LLM deployments

    Qwen Team. Qwen3Guard: A multilingual safety guardrail model for LLM deployments. arXiv preprint arXiv:2510.14276, 2025. URLhttps://arxiv.org/abs/2510.14276

    Pith/arXiv arXiv 2025

  12. [12]

    CoSafe: Evaluating large language model safety in multi-turn dialogue coreference

    Erxin Yu, Jing Li, Ming Liao, Siqi Wang, Gao Zuchen, Fei Mi, and Lanqing Hong. CoSafe: Evaluating large language model safety in multi-turn dialogue coreference. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 17494–17508, 2024. URLhttps://aclanthology.org/2024.emnlp-main.968/

    2024

  13. [13]

    AgentDoG: A diagnostic guardrail framework for AI agent safety and security, 2026

    Yu Li, Haoyu Luo, Yuejin Xie, Jiapeng Gu, Yuhan Wang, Yanwei Fu, Yujiu Yang, Jing Shao, Xia Hu, and Dongrui Liu. AgentDoG: A diagnostic guardrail framework for AI agent safety and security, 2026. URLhttps://arxiv.org/abs/2601.18491. Code:https: //github.com/AI45Lab/AgentDoG

    Pith/arXiv arXiv 2026

  14. [14]

    TraceSafe: A sys- tematic assessment of LLM guardrails on multi-step tool-calling trajectories, 2026

    Yen-Shan Chen, Sian-Yao Huang, Cheng-Lin Yang, and Yun-Nung Chen. TraceSafe: A sys- tematic assessment of LLM guardrails on multi-step tool-calling trajectories, 2026. URL https://arxiv.org/abs/2604.07223

    Pith/arXiv arXiv 2026

  15. [15]

    Unsupervised domain adaptation by backpropagation

    Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. InProceedings of the 32nd International Conference on Machine Learning (ICML), pages 1180–1189, 2015. URLhttps://proceedings.mlr.press/v37/ganin15.html

    2015

  16. [16]

    Deep CORAL: Correlation alignment for deep domain adap- tation

    Baochen Sun and Kate Saenko. Deep CORAL: Correlation alignment for deep domain adap- tation. InProceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 443–450, 2016. arXiv:1607.01759

    Pith/arXiv arXiv 2016

  17. [17]

    Artificial intelligence risk management frame- work (AI RMF 1.0)

    National Institute of Standards and Technology. Artificial intelligence risk management frame- work (AI RMF 1.0). Technical Report AI 100-1, NIST, 2023. URLhttps://nvlpubs.nist .gov/nistpubs/ai/NIST.AI.100-1.pdf

    2023

  18. [18]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedba...

    2022

  19. [19]

    Deep reinforcement learning from human preferences.Advances in Neural Information Processing Systems, 30, 2017

    Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences.Advances in Neural Information Processing Systems, 30, 2017

    2017

  20. [20]

    Claude’s model specification

    Anthropic. Claude’s model specification. Technical report, Anthropic, 2024. URLhttps: //www.anthropic.com/research/model-specification

    2024

  21. [21]

    Constitutional AI: Harmlessness from AI feedback.arXiv preprint arXiv:2212.08073, 2022

    Yuntao Bai et al. Constitutional AI: Harmlessness from AI feedback.arXiv preprint arXiv:2212.08073, 2022

    Pith/arXiv arXiv 2022

  22. [22]

    Latanya Sweeney.k-anonymity: A model for protecting privacy.International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 10(05):557–570, 2002. 44

    2002

  23. [23]

    Ninghui Li, Tiancheng Li, and Suresh Venkatasubramanian.t-closeness: Privacy beyondk- anonymity andl-diversity.IEEE Transactions on Knowledge and Data Engineering, 22(4): 445–456, 2010

    2010

  24. [24]

    Calibrating noise to sen- sitivity in private data analysis

    Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sen- sitivity in private data analysis. InTheory of Cryptography: Third Theory of Cryptography Conference (TCC 2006), volume 3876 ofLecture Notes in Computer Science, pages 265–284. Springer, 2006. doi: 10.1007/11681878_14

    doi:10.1007/11681878_14 2006

  25. [25]

    Now Publishers, 2014

    Cynthia Dwork and Aaron Roth.The Algorithmic Foundations of Differential Privacy. Now Publishers, 2014

    2014

  26. [26]

    Pearson, Dietrich A

    Nils Homer, Szabolcs Szelinger, Margot Redman, David Duggan, Waibhav Tembe, Jill Muehling, John V. Pearson, Dietrich A. Stephan, Stanley F. Nelson, and David W. Craig. Resolving individuals contributing trace amounts of DNA to highly complex mixtures using high-density SNP genotyping microarrays.PLoS Genetics, 4(8):e1000167, 2008

    2008

  27. [27]

    Robust de-anonymization of large sparse datasets

    Arvind Narayanan and Vitaly Shmatikov. Robust de-anonymization of large sparse datasets. InProceedings of the 2008 IEEE Symposium on Security and Privacy, pages 111–125, 2008

    2008

  28. [28]

    Williams and Steve Young

    Jason D. Williams and Steve Young. Partially observable Markov decision processes for spoken dialog systems. InComputer Speech & Language, volume 21, pages 393–422. Elsevier, 2007

    2007

  29. [29]

    Williams

    Steve Young, Milica Gašić, Blaise Thomson, and Jason D. Williams. POMDP-based statistical spoken dialog systems: A review.Proceedings of the IEEE, 101(5):1160–1179, 2013

    2013

  30. [30]

    Williams

    Matthew Henderson, Blaise Thomson, and Jason D. Williams. Word-based dialog state track- ing with recurrent neural networks. InProceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 292–299, 2014

    2014

  31. [31]

    Neural belief tracker: Data-driven dialogue state tracking

    Nikola Mrkšić, Diarmuid Ó Séaghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. Neural belief tracker: Data-driven dialogue state tracking. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1777–1788, 2017

    2017

  32. [32]

    Tod-BERT: Pre-trained natural language understanding for task-oriented dialogue

    Chien-Sheng Wu, Steven Sohn, Jing Chen, Hongyuan Lee, Semih Zhang, Jason Williams, Wen-tau Wang, and Caiming Yu. Tod-BERT: Pre-trained natural language understanding for task-oriented dialogue. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 917–929, 2020

    2020

  33. [2008]

    URLhttps://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication80 0-60v1r1.pdf

  34. [2024]

    URLhttps://arxiv.org/abs/2405.01470. 45

    Pith/arXiv arXiv