Pith. sign in

REVIEW 5 major objections 6 minor 69 references

A frozen LLM can be defended against multi-turn attacks by treating user intent as a latent variable and softmax-reweighting sampled intent hypotheses with a learned critic.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 13:12 UTC pith:2M623PTW

load-bearing objection The method and theory are credible, but the main empirical claim is undercut by the fact that the safety judge used to train the critic is the same judge used to evaluate it — so DSR gains may partly reflect gaming that judge rather than real safety. the 5 major comments →

arxiv 2607.20472 v1 pith:2M623PTW submitted 2026-05-24 cs.AI

Robust Critics: Defending LLMs Against Multi-Turn Attacks

Roman Belaire , Arunesh Sinha , Pradeep Varakantham This is my paper
classification cs.AI MSC 68T5090C4068T05
keywords multi-turn adversarial dialoguelatent intent inferencecritic-guided samplinginference-time alignmentexponential tiltingpolicy improvementover-refusalLLM safety
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that multi-turn dialogue safety is fundamentally an intent-inference problem, not a fixed refusal rule, and proposes Dialogue Critic Guided Sampling (DCGS) to solve it at inference time. At each turn, the frozen base model proposes K natural-language hypotheses about the user's goal; a learned critic scores each; one hypothesis is sampled via softmax and conditions the final response. The authors prove that this reweighting approximates exponential tilting of the base policy and guarantees improvement in expected critic value for any finite candidate pool, a property group-relative objectives lack. On four adversarial dialogue benchmarks, DCGS reports higher defense success than fine-tuned and training-free baselines while preserving goal completion on benign queries, and it transfers to larger API models without fine-tuning. A sympathetic reader would care because the method offers a lightweight, black-box-compatible defense against gradually escalating attacks without retraining the generator.

Core claim

On the paper's own terms, the central discovery is that the ambiguity between a harmful attack and a benign request can be resolved by treating the partner's intent as a latent variable and making the refusal/response decision an inference-time sampling problem. At each turn the frozen base model proposes K natural-language intent hypotheses; a learned action-value critic scores each; one hypothesis is drawn via softmax and used to condition response generation. The paper proves this softmax resampling converges to the exponentially tilted policy π_+(z|s) ∝ π_ref(z|s) exp(Q_θ(s,z)) and that for any finite K ≥ 2 with unequal critic values, the expected critic value of the resampled policy is

What carries the argument

The load-bearing object is the learned action-value critic Q_θ coupled with softmax resampling. The paper defines Q_θ as a linear head on a frozen LLM's last hidden layer, trained by temporal-difference learning against rewards that sum an LLM-judge helpfulness signal and a guardrail harmlessness signal. The identity doing the work is the Boltzmann/exponential-tilting update from entropy-regularized RL, π_ref(z|s) exp(Q_θ(s,z)/α) / Z(s), which DCGS approximates by drawing K proposals and resampling one with softmax weights — a sampling-importance-resampling scheme. A regret-augmented critic δ subtracts a pessimistic baseline r_min(s) before the same TD backup, blending expected value with ro

Load-bearing premise

The real-world claim rests on the untested assumption that the learned critic Q_θ, trained on simulator interactions with rewards from an LLM judge and a guardrail model, accurately estimates true future helpfulness and harmlessness; if that estimate is biased or the simulator's adversary differs from real ones, the reweighting improves the wrong quantity.

What would settle it

Replace the simulator's attacker with human-written or adaptively generated multi-turn attacks on a held-out set and re-measure defense success and goal completion: if the DSR advantage over the frozen base model disappears or goal completion collapses on benign escalations, the central empirical claim fails. For the theory, any state with two candidate intents of different critic value where softmax resampling yields lower mean Q_θ than the base policy would contradict Proposition 1.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • A defense against multi-turn attacks can be applied to any frozen or API-only LLM without fine-tuning; the paper reports improved defense success and comparable goal completion on frontier models.
  • For any finite candidate pool of size at least two with two differently valued intents, softmax reweighting by the critic gives strict improvement in expected critic value over the base policy, so the method has a per-turn guarantee, not just an asymptotic one.
  • Group-relative objectives such as pairwise preference models do not define a consistent density ratio, so they lack the policy-improvement property DCGS claims; this positions critic-guided sampling against those training methods.
  • Because the token-level critic uses post-hoc masking, it can identify which specific tokens in a candidate response drive harm or helpfulness, enabling more precise rejection or rewording.
  • On the four evaluated dialogue benchmarks, DCGS generalizes across datasets better than fine-tuned adversarial-training or over-refusal baselines while maintaining high goal completion.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The finite-sample guarantee is over the learned Q_θ, not true return; a natural consequence is that a miscalibrated or biased reward model would be amplified by the softmax, so the method's practical ceiling tracks critic quality — a point the paper does not develop.
  • Because the proof is dialogue-agnostic, the same reweighting recipe should transfer to verifiable-reward domains the paper names (code completion, mathematics), where candidate solutions could be scored by a learned value critic instead of intent hypotheses.
  • The empirical claims depend on a simulator whose attacker escalates in a prescribed pattern; replacing it with real multi-turn attack transcripts or adaptive human attackers would be a sharper test of whether the defense succeeds outside the training distribution.
  • Injecting fixed benign and malicious intents into every candidate set suggests a possible failure mode: if the critic learns to always favor one injected hypothesis, the K candidates become decoration; testing with K=1 and only injected intents would reveal whether the softmax is doing the work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes Dialogue Critic Guided Sampling (DCGS), an inference-time defense for multi-turn adversarial dialogue. The agent first samples K candidate latent intent hypotheses from a frozen base LLM, scores them with a learned action-value critic (including a regret-augmented variant, RDCGS), and resamples via softmax. A second-stage critic assigns token-level marginal contributions to score candidate responses. The authors prove Lemma 1 (asymptotic weak convergence to an exponentially tilted policy) and Proposition 1 (finite-K expected Qθ improvement), and report experiments on CARES-18k, WildJailbreak, RedBench, and Harmbench, claiming superior Defense Success Rate (DSR) over robust baselines and transferability to a frontier model.

Significance. If the empirical claims hold, the framework is a practical, black-box, inference-time defense that can improve safety without fine-tuning the actor LLM, and the transferability result is interesting. The mathematical results are correct but modest: Proposition 1 is a standard property of softmax reweighting (log-sum-exp convexity) and does not by itself ensure improvement in actual safety or task success. The paper's main contributions are the two-stage critic architecture and the regret-augmented objective, plus a relatively thorough experiment appendix with code release and prompt details. However, the central evaluation is undermined by the overlap between the training reward and the reported safety metric, the missing base-model baseline in the main table, and the apparent selection of hyperparameters on the evaluation benchmarks.

major comments (5)
  1. [Abstract, §1 Contribution 2, §4.1.4 Eq. (13)] The abstract claims that the reweighting 'guarantees improvement in expected return.' Proposition 1 actually guarantees improvement only in expected Qθ, the learned critic's value. Since Qθ is trained from a specific reward model (Eq. 5), a biased or misspecified Qθ can improve its own score while degrading true safety or helpfulness. The theoretical result is an identity that holds for any scoring function Qθ and carries no safety guarantee by itself. Please restate the claim to be about expected Qθ, and provide evidence that Qθ is calibrated to true return (e.g., correlation with independent human judgments or a different judge).
  2. [§4.1.1 Eq. (5), §5.1, Table 1] The evaluation metric DSR is the same harmlessness reward used to train the critic. Section 4.1.1 defines Rharmless via a guardrail model (e.g., LlamaGuard) and Section 5.1 states the simulator 'judges the DCGS utterance according to Section 4.1.1.' Consequently, DCGS is optimized to maximize exactly the scalar reported as the safety metric in Table 1, while the baselines (CAT, DCR, TPO, SmoothLLM) are not. This creates a real risk that reported DSR improvements reflect reward hacking against LlamaGuard rather than robust defense. Please re-evaluate with an independent judge (a different guardrail model, human evaluation, or an established safety benchmark) and report the results; also report the agreement between the training judge and the independent judge.
  3. [Table 1] The main results table lacks the unmodified base model. The base agent in Table 1 is Zephyr-7b-beta, but no 'Zephyr-7b-beta (no DCGS)' row appears—only GPT-4o Naïve is shown as a reference. Without the base-model baseline, the central claim that DCGS improves robustness over its own frozen base policy is not directly supported. Please add the base model for each architecture used (e.g., Zephyr-7b-beta without any defense) to Table 1 and, if applicable, to Table 3.
  4. [Appendix C.3] The hyperparameters K and β are selected via a search reported in C.3, with no mention of a held-out validation split. Since the search is described as finding the values that perform best on the benchmarks that are later reported in Tables 1–3, this is effectively tuning on the test set. This can inflate the reported results relative to a fair comparison. Please specify the validation procedure (e.g., a separate validation split from the training data, or nested cross-validation) and report results with the pre-specified hyperparameters.
  5. [§5.2, Table 1 vs Table 2] Table 1 reports no error bars, confidence intervals, or significance tests for the central DSR/GCR comparisons, while Table 2 does report variance. Given that Table 2's variances are on the order of ±2–6 percentage points, several differences in Table 1 (e.g., RDCGS 62.5 vs CAT 65.0 on CARES-18K DSR) could be within noise. Please provide error bars or statistical tests for all main results, or state clearly which differences are significant.
minor comments (6)
  1. [Table 3] Typo: 'RDGCS' and 'VDGCS' should be 'RDCGS' and 'VDCGS' to match the method names.
  2. [Eq. (14)] The denominator uses ∥h(gt)∥, but 'gt' is not defined; presumably this should be ∥h(zt)∥.
  3. [§5 intro] The text defines 'goal completion rate (GSR)' but the tables and later text use 'GCR'; please use one abbreviation consistently.
  4. [Table 2 caption] The caption reports '±variance' but does not state whether this is standard deviation or standard error; please clarify.
  5. [Checklist Q4 / Appendix] The checklist claims 'algorithm pseudocode' is provided, but the appendix contains prompts and hyperparameters, not a pseudocode listing. Either add the pseudocode or remove that claim from the checklist justification.
  6. [§5.1] The simulator is a central component of the evaluation, yet it is not validated against real multi-turn attacks or human-annotated dialogue. Even if the proposed method is not the culprit, a short validation of the simulator (e.g., comparison with a small set of manually written multi-turn attacks) would strengthen external validity.

Circularity Check

1 steps flagged

Main empirical claim is partially circular: DSR is measured by the same guardrail reward used to train the critic, so the reported improvement largely reflects optimization of the training objective.

specific steps
  1. fitted input called prediction [Section 4.1.1 (Eq. 5), Section 4.1.2 (Eq. 7), Section 5.1, Section 5 (DSR definition)]
    "Harmlessness reward Rharmless(s,a)∈{0,1} is positive when the agent's response is safe, as assessed by a guardrail model (e.g., LlamaGuard [19]). ... The simulator ... judges the DCGS utterance according to Section 4.1.1. ... A defense is successful if an agent does not respond harmfully and also does not attempt to help the adversarial agent."

    The critic Qθ is trained via Eq. (7) on TD targets built from r(s,z)=R_helpful+R_harmless (Eq. 5), where R_harmless is LlamaGuard's safety judgment. For adversarial episodes R_helpful=0, so the training reward is exactly the harmlessness judgment. Section 5.1 says the simulator judges DCGS utterances 'according to Section 4.1.1,' and DSR is defined as not responding harmfully. Hence the paper's headline empirical claim—DCGS outperforms baselines on DSR—is essentially the statement that a policy trained to maximize a LlamaGuard-based reward scores higher on that same LlamaGuard-based metric than baselines that do not optimize it. This is a fitted-input-called-prediction pattern: the evaluation metric is the training objective renamed, so it does not independently validate robustness.

full rationale

Proposition 1 and Lemma 1 are standard soft-policy-improvement identities: for any fixed function Qθ, softmax reweighting of K i.i.d. samples from πref has expectation under the reweighted policy at least the expectation under πref, and converges to exponential tilting as K→∞. These statements are self-contained and do not depend on how Qθ is obtained, so the theoretical contribution is not circular. The self-citations ([3] for the regret objective, [4] for token credit assignment) are not uniqueness theorems, and the relevant equations (8)-(10) and (14)-(16) are fully specified in this paper; they do not force the paper's conclusions. The circular element is in the empirical validation. Eq. (5) defines the training reward r=R_helpful+R_harmless, with R_harmless judged by LlamaGuard; Eq. (7) trains Qθ on this reward; Section 5.1 says the simulator 'judges the DCGS utterance according to Section 4.1.1'; and DSR is defined as not responding harmfully. For adversarial episodes R_helpful=0, so r=R_harmless and DSR is essentially the same scalar used as the TD target. The headline result—DCGS outperforms baselines on adversarial dialogue—therefore reduces, in large part, to the fact that DCGS optimizes the guardrail score while baselines do not. This is a training/evaluation overlap (reward hacking) rather than a purely logical circularity in the proofs, but it makes the main empirical claim partially circular. The paper also does not provide an independent judge or a base-policy (Zephyr) naive baseline in Table 1, which would be needed to separate genuine robustness from optimization of the metric.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 1 invented entities

The two tuned hyperparameters β and K are the main free parameters; γ and reward weights are conventional choices. The framework's validity requires the listed domain assumptions about intent representation, reward decomposition, reward-model fidelity, and the simulator being a faithful proxy for real adversaries.

free parameters (5)
  • β (robustness temperature) = 0.2
    Grid search over {0.1,...,0.9} on the evaluation datasets; controls the blend of Qϕ and regret critic in Eq. (10).
  • K (candidate pool size) = 5
    Search over {2,3,5,7,10}; affects both intent and response sampling (Eqs. 4 and 17).
  • γ (discount factor) = 0.99
    Set in Appendix C.2; affects TD targets in Eqs. (7) and (16).
  • k (top-token masking window) = not reported
    The token-level critic masks at most two of the k highest-attention tokens (Eq. 15); the value of k is never given.
  • Reward weights on Rhelpful and Rharmless = 1.0 each
    Additive combination in Eq. (5); the relative scale against the Sim term in Eq. (14) is not optimized.
axioms (6)
  • domain assumption State = full conversation history; dialogue transitions are Markovian (Eq. 1)
    The MDP formulation assumes the full history is a sufficient statistic for both the partner model and the agent's decisions.
  • domain assumption User intent is a latent variable representable as a token sequence and proposable by the base LLM (Eq. 2)
    The method assumes sampled natural-language hypotheses can cover the true intent space, including adversarial and benign regimes.
  • domain assumption Reward is an additive sum of binary helpfulness and harmlessness signals (Eq. 5)
    The reward model assumes these two binary terms capture all relevant objectives and that Rhelpful is zero for adversarial partners.
  • domain assumption LLM judge and LlamaGuard provide reliable reward signals for TD learning (Eq. 7)
    Critic training relies on these model-based reward labels; no human validation of the reward labels is presented.
  • standard math Law of large numbers, continuous mapping theorem, and soft policy improvement results
    Lemma 1 and Proposition 1 use these standard results (Appendix A).
  • domain assumption Token-level marginal contributions can be estimated by leave-one/two-out masking (Eq. 15)
    The token critic assumes masking at most two high-attention tokens identifies the causal contribution of token subsets to reward.
invented entities (1)
  • latent intent hypothesis z (token sequence) no independent evidence
    purpose: Represents the user's unobserved goal; conditions the response policy and is selected via critic reweighting.
    The paper postulates that intent is representable as sampled natural-language tokens from the base LLM, but gives no external evidence that the sampled hypotheses cover the true intent space.

pith-pipeline@v1.3.0-alltime-deepseek · 23235 in / 15091 out tokens · 165903 ms · 2026-08-02T13:12:32.509431+00:00 · methodology

0 comments
read the original abstract

When a user asks a language model something harmful, is it a genuine attack or a misunderstood but well-meaning question? This ambiguity is one of the central challenges of LLM safety. A model that assumes the worst harms legitimate users; one that assumes the best is easily exploited. The problem is compounded in multi-turn dialogue, where an attacker's true intent may only reveal itself gradually across many exchanges, yet existing safety frameworks apply a contextual bandit treatment, ignoring the trajectory of the conversation. To that end, we propose Dialogue Critic Guided Sampling (DCGS), a framework that addresses this by inferring user intent at every turn of dialogue. Instead of applying a fixed rule about what is or is not safe, DCGS learns what the user's intent is likely to be based on the full conversational history and generates responses accordingly. Formally, we model adversarial dialogue as a Markov Decision Process and learn value and regret-based critics at both the individual token and utterance (full response) levels, scoring candidate responses via an action-value critic. We prove that this inference-time reweighting approximates exponential tilting of the base policy, guaranteeing improvement in expected return for any finite candidate pool, a property that group-relative objectives do not exhibit. Evaluated on CARES-18k, WildJailbreak, Redbench, and Harmbench, DCGS outperforms strong robust baselines and frontier models on adversarial dialogue tasks. DCGS also transfers to frontier models, improving their robustness without fine-tuning.

Figures

Figures reproduced from arXiv: 2607.20472 by Arunesh Sinha, Pradeep Varakantham, Roman Belaire.

Figure 1
Figure 1. Figure 1: Critic-weighted resampling. The frozen base LLM generates [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Token-level reward attribution methods for VCGS in CARES-18k. Assigning rewards [PITH_FULL_IMAGE:figures/full_fig_p017_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Survival analysis (percentage of conversations in which, by each turn, the robust agent [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

69 extracted references · 4 canonical work pages · 3 internal anchors

  1. [1]

    Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

    Pith/arXiv arXiv doi:10.48550/arxiv.2204.05862 2022

  2. [2]

    Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, K...

    Pith/arXiv arXiv doi:10.48550/arxiv.2212.08073 2022

  3. [3]

    Regret-based defense in adversarial reinforcement learning

    Roman Belaire, Pradeep Varakantham, Thanh Hong Nguyen, and David Lo. Regret-based defense in adversarial reinforcement learning. In Mehdi Dastani, Jaime Simão Sichman, Natasha Alechina, and Virginia Dignum, editors,Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2024, Auckland, New Zealand, May 6-10, 202...

    arXiv 2024

  4. [4]

    Automatic LLM Red Teaming

    Roman Belaire, Arunesh Sinha, and Pradeep Varakantham. Automatic LLM red teaming. CoRR, abs/2508.04451, 2025. doi: 10.48550/ARXIV .2508.04451. URL https://doi.org/ 10.48550/arXiv.2508.04451

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2508.04451 2025

  5. [5]

    Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons.Biometrika, 39(3/4):324–345, 1952. ISSN 00063444, 14643510. URLhttp://www.jstor.org/stable/2334029

    arXiv 1952

  6. [6]

    The art of saying no: Contextual noncompliance in language models.Advances in Neural Information Processing Systems, 37:49706–49748, 2024

    Faeze Brahman, Sachin Kumar, Vidhisha Balachandran, Pradeep Dasigi, Valentina Pyatkin, Abhilasha Ravichander, Sarah Wiegreffe, Nouha Dziri, Khyathi Chandu, Jack Hessel, et al. The art of saying no: Contextual noncompliance in language models.Advances in Neural Information Processing Systems, 37:49706–49748, 2024

    2024

  7. [7]

    Scans: Mitigating the exaggerated safety for llms via safety-conscious activation steering

    Zouying Cao, Yifei Yang, and Hai Zhao. Scans: Mitigating the exaggerated safety for llms via safety-conscious activation steering. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23523–23531, 2025

    2025

  8. [8]

    Yixin Cheng, Markos Georgopoulos, V olkan Cevher, and Grigorios G. Chrysos. Leveraging the context through multi-round interactions for jailbreaking attacks.CoRR, abs/2402.09177, 2024. URLhttps://doi.org/10.48550/arXiv.2402.09177

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.09177 2024

  9. [9]

    Christiano, Jan Leike, Tom B

    Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V . N. Vishwanathan, and Roman Gar- nett, editors,Advances in Neural Information Processing Systems 30: Annual Conference on Neura...

    2017

  10. [10]

    Or-bench: An over-refusal benchmark for large language models.arXiv preprint arXiv:2405.20947, 2024

    Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. Or-bench: An over-refusal benchmark for large language models.arXiv preprint arXiv:2405.20947, 2024. 10

    Pith/arXiv arXiv 2024

  11. [11]

    Just enough shifts: Mitigating over-refusal in aligned language models with targeted representation fine-tuning

    Mahavir Dabas, Si Chen, Charles Fleming, Ming Jin, and Ruoxi Jia. Just enough shifts: Mitigating over-refusal in aligned language models with targeted representation fine-tuning. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors,Forty-second International Conference on Ma...

    2025

  12. [12]

    Safe RLHF: safe reinforcement learning from human feedback

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: safe reinforcement learning from human feedback. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11,

    2024

  13. [13]

    Redbench: A universal dataset for com- prehensive red teaming of large language models.CoRR, abs/2601.03699, 2026

    Quy-Anh Dang, Chris Ngo, and Truong-Son Hy. Redbench: A universal dataset for com- prehensive red teaming of large language models.CoRR, abs/2601.03699, 2026. doi: 10.48550/ARXIV .2601.03699. URLhttps://doi.org/10.48550/arXiv.2601.03699

    Pith/arXiv arXiv doi:10.48550/arxiv.2601.03699 2026

  14. [14]

    Reward-augmented decoding: Efficient controlled text gen- eration with a unidirectional reward model

    Haikang Deng and Colin Raffel. Reward-augmented decoding: Efficient controlled text gen- eration with a unidirectional reward model. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, EMNLP 2023, Singapore, December 6-10, 2023, pages 11781–11791. Association for C...

    doi:10.18653/v1/2023.emnlp-main.721 2023

  15. [15]

    Safety Alignment in NLP Tasks: Weakly Aligned Summarization as an In-Context Attack

    Yu Fu, Yufei Li, Wen Xiao, Cong Liu, and Yue Dong. Safety alignment in nlp tasks: Weakly aligned summarization as an in-context attack.Annual Meeting of the Association for Computa- tional Linguistics, 2023. doi: 10.48550/arXiv.2312.06924

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2312.06924 2023

  16. [16]

    Adversarial policies: Attacking deep reinforcement learning.arXiv preprint arXiv:1905.10615, 2019

    Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adversarial policies: Attacking deep reinforcement learning.arXiv preprint arXiv:1905.10615, 2019

    Pith/arXiv arXiv 1905

  17. [17]

    Mtsa: Multi-turn safety alignment for llms through multi-round red-teaming

    Weiyang Guo, Jing Li, Wenya Wang, Yu Li, Daojing He, Jun Yu, and Min Zhang. Mtsa: Multi-turn safety alignment for llms through multi-round red-teaming. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 26424–26442, 2025

    2025

  18. [18]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer G. Dy and Andreas Krause, editors,Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, Proceedings ...

    2018

  19. [19]

    Llama guard: Llm-based input-output safeguard for human-ai conversations.CoRR, abs/2312.06674,

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Madian Khabsa. Llama guard: Llm-based input-output safeguard for human-ai conversations.CoRR, abs/2312.06674,

    Pith/arXiv arXiv

  20. [20]

    Approximately optimal approximate reinforcement learning

    Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. InProceedings of the nineteenth international conference on machine learning, pages 267–274, 2002

    2002

  21. [21]

    Adversarial suffix filtering: a defense pipeline for llms

    David Khachaturov and Robert Mullins. Adversarial suffix filtering: a defense pipeline for llms. arXiv preprint arXiv:2505.09602, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    Critic-guided decoding for controlled text generation

    Minbeom Kim, Hwanhee Lee, Kang Min Yoo, Joonsuk Park, Hwaran Lee, and Kyomin Jung. Critic-guided decoding for controlled text generation. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors,Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, Findings of ACL, pages 4598–4612. Association for...

    doi:10.18653/v1/2023.findings-acl.281 2023

  23. [23]

    Evaluating large language models in theory of mind tasks.Proceedings of the National Academy of Sciences, 121(45):e2405460121, 2024

    Michal Kosinski. Evaluating large language models in theory of mind tasks.Proceedings of the National Academy of Sciences, 121(45):e2405460121, 2024

    2024

  24. [24]

    Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum

    Tejas D. Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Gar- nett, editors,Advances in Neural Information Processing Systems 29: Annual Conference on Neura...

    2016

  25. [25]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V . Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Ha- jishi...

    Pith/arXiv arXiv

  26. [26]

    Test-time preference optimization: On-the-fly alignment via iterative textual feedback.arXiv preprint arXiv:2501.12895, 2025

    Yafu Li, Xuyang Hu, Xiaoye Qu, Linjie Li, and Yu Cheng. Test-time preference optimization: On-the-fly alignment via iterative textual feedback.arXiv preprint arXiv:2501.12895, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Efficient adversarial train- ing without attacking: Worst-case-aware robust reinforcement learning.Advances in neural information processing systems, 35:22547–22561, 2022

    Yongyuan Liang, Yanchao Sun, Ruijie Zheng, and Furong Huang. Efficient adversarial train- ing without attacking: Worst-case-aware robust reinforcement learning.Advances in neural information processing systems, 35:22547–22561, 2022

    2022

  28. [28]

    URL https://doi.org/10.48550/arXiv.2411

    doi: 10.48550/ARXIV .2411.15124. URL https://doi.org/10.48550/arXiv.2411. 15124

    Pith/arXiv arXiv doi:10.48550/arxiv.2411.15124

  29. [29]

    Evolutionary guided decoding: Iterative value refinement for llms.arXiv preprint arXiv: 2503.02368, 2025

    Zhenhua Liu, Lijun Li, Ruizhe Chen, Yuxian Jiang, Tong Zhu, Zhaochen Su, Wenliang Chen, and Jing Shao. Evolutionary guided decoding: Iterative value refinement for llms.arXiv preprint arXiv: 2503.02368, 2025

    Pith/arXiv arXiv 2025

  30. [30]

    Inference-time language model alignment via integrated value guidance

    Zhixuan Liu, Zhanhui Zhou, Yuanfu Wang, Chao Yang, and Yu Qiao. Inference-time language model alignment via integrated value guidance. In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen, editors,Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, Findings of ACL, pages 4181–4195. Association...

    doi:10.18653/v1/2024.findings-emnlp.242 2024

  31. [31]

    Wildbench: Benchmarking llms with challenging tasks from real users in the wild

    Bill Yuchen Lin, Yuntian Deng, Khyathi Raghavi Chandu, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking llms with challenging tasks from real users in the wild. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. U...

    2025

  32. [32]

    Forsyth, and Dan Hendrycks

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David A. Forsyth, and Dan Hendrycks. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024...

    2024

  33. [33]

    Controlled decoding from language models

    Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, and Ahmad Beirami. Controlled decoding from language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine A. Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenka...

    2024

  34. [34]

    Discern truth from falsehood: Reducing over-refusal via contrastive refinement.CoRR, abs/2603.03323, 2026

    Yuxiao Lu, Lin Xu, Yang Sun, Wenjun Li, and Jie Shi. Discern truth from falsehood: Reducing over-refusal via contrastive refinement.CoRR, abs/2603.03323, 2026. doi: 10.48550/ARXIV . 2603.03323. URLhttps://doi.org/10.48550/arXiv.2603.03323

    work page doi:10.48550/arxiv.2603.03323 2026

  35. [35]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schul- man, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Pe- ter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language mod- els to follow instructions with hum...

    2022

  36. [36]

    Robust adversarial reinforcement learning

    Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. InInternational conference on machine learning, pages 2817–2826. PMLR, 2017

    2017

  37. [37]

    Nguyen, Krishnaram Kenthapadi, and Hal Daumé III

    Huy Nghiem, Swetasudha Panda, Devashish Khatwani, Huy V . Nguyen, Krishnaram Kenthapadi, and Hal Daumé III. Balancing safety and helpfulness in healthcare AI assistants through iterative preference alignment.CoRR, abs/2512.04210, 2025. doi: 10.48550/ARXIV .2512.04210. URL https://doi.org/10.48550/arXiv.2512.04210

    doi:10.48550/arxiv.2512.04210 2025

  38. [38]

    Rizwan Parvez, Hamid Palangi, Kai-Wei Chang, Yejin Choi, and Saadia Gabriel

    Salman Rahman, Liwei Jiang, James Shiffer, Genglin Liu, Sheriff Issaka, Md. Rizwan Parvez, Hamid Palangi, Kai-Wei Chang, Yejin Choi, and Saadia Gabriel. X-teaming: Multi-turn jailbreaks and defenses with adaptive multi-agents. 2025. doi: 10.48550/ARXIV .2504.13203. URLhttps://doi.org/10.48550/arXiv.2504.13203

    Pith/arXiv arXiv doi:10.48550/arxiv.2504.13203 2025

  39. [39]

    Smoothllm: Defending large language models against jailbreaking attacks.arXiv preprint arXiv:2310.03684, 2023

    Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. Smoothllm: Defending large language models against jailbreaking attacks.arXiv preprint arXiv:2310.03684, 2023

    Pith/arXiv arXiv 2023

  40. [40]

    Manning, Stefano Ermon, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors,Advances in Neural Information Processing Systems 36: Annual Conference o...

    2023

  41. [41]

    Xstest: A test suite for identifying exaggerated safety behaviours in large language models.North American Chapter of the Association for Computational Linguistics, 2023

    Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models.North American Chapter of the Association for Computational Linguistics, 2023. doi: 10.48550/arXiv.2308.01263

    Pith/arXiv arXiv doi:10.48550/arxiv.2308.01263 2023

  42. [42]

    High- dimensional continuous control using generalized advantage estimation

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. InProceedings of the International Conference on Learning Representations (ICLR), 2016

    2016

  43. [43]

    Great, now write an article about that: The crescendo multi-turn LLM jailbreak attack.CoRR, abs/2404.01833, 2024

    Mark Russinovich, Ahmed Salem, and Ronen Eldan. Great, now write an article about that: The crescendo multi-turn LLM jailbreak attack.CoRR, abs/2404.01833, 2024. URL https: //doi.org/10.48550/arXiv.2404.01833

    Pith/arXiv arXiv doi:10.48550/arxiv.2404.01833 2024

  44. [44]

    do anything now

    Xinyue Shen, Z. Chen, M. Backes, Yun Shen, and Yang Zhang. "do anything now": Character- izing and evaluating in-the-wild jailbreak prompts on large language models.Conference on Computer and Communications Security, 2023. doi: 10.48550/arXiv.2308.03825

    Pith/arXiv arXiv doi:10.48550/arxiv.2308.03825 2023

  45. [45]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto.Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 1998

    1998

  46. [46]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y .K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URLhttps://arxiv.org/abs/2402.03300

    Pith/arXiv arXiv 2024

  47. [47]

    Surgical, cheap, and flexi- ble: Mitigating false refusal in language models via single vector ablation.arXiv preprint arXiv:2410.03415, 2024

    Xinpeng Wang, Chengzhi Hu, Paul Röttger, and Barbara Plank. Surgical, cheap, and flexi- ble: Mitigating false refusal in language models via single vector ablation.arXiv preprint arXiv:2410.03415, 2024. 13

    Pith/arXiv arXiv 2024

  48. [48]

    Speculative safety-aware decoding

    Xuekang Wang, Shengyu Zhu, and Xueqi Cheng. Speculative safety-aware decoding. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4-9, 2025, pages 12827–12841. Association for Computational Li...

    doi:10.18653/v1/2025.emnlp-main.648 2025

  49. [49]

    van der Vaart.Asymptotic Statistics

    Aad W. van der Vaart.Asymptotic Statistics. Cambridge University Press, 1998

    1998

  50. [50]

    The dialog state tracking challenge series: A review.Dialogue & Discourse, 7(3):4–33, 2016

    Jason D Williams, Antoine Raux, and Matthew Henderson. The dialog state tracking challenge series: A review.Dialogue & Discourse, 7(3):4–33, 2016

    2016

  51. [51]

    Efficient adversarial training in llms with continuous attacks.Advances in Neural Information Processing Systems, 37:1502–1530, 2024

    Sophie Xhonneux, Alessandro Sordoni, Stephan Günnemann, Gauthier Gidel, and Leo Schwinn. Efficient adversarial training in llms with continuous attacks.Advances in Neural Information Processing Systems, 37:1502–1530, 2024

    2024

  52. [52]

    Assessing the brittleness of safety alignment via pruning and low-rank modifications

    Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. In Ruslan Salakhutdinov, Zico Kolter, Katherine A. Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors,Fort...

    2024

  53. [53]

    URLhttps://proceedings.mlr.press/v235/wei24f.html

  54. [54]

    Holistic automated red teaming for large language models through top-down test case generation and multi-turn interaction

    Jinchuan Zhang, Yan Zhou, Yaxin Liu, Ziming Li, and Songlin Hu. Holistic automated red teaming for large language models through top-down test case generation and multi-turn interaction. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, pages 13711–13736. Association for Computational Linguistics,

    2024

  55. [55]

    Safety is not only about refusal: Reasoning-enhanced fine-tuning for interpretable LLM safety

    Yuyou Zhang, Miao Li, William Han, Yihang Yao, Zhepeng Cen, and Ding Zhao. Safety is not only about refusal: Reasoning-enhanced fine-tuning for interpretable LLM safety. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics: ACL 2025, pages 18727–18746, Vienna, Au...

    doi:10.18653/v1/2025.findings-acl.960 2025

  56. [56]

    Preference-grounded token-level guidance for language model fine-tuning

    Shentao Yang, Shujian Zhang, Congying Xia, Yihao Feng, Caiming Xiong, and Mingyuan Zhou. Preference-grounded token-level guidance for language model fine-tuning. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, edi- tors,Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information...

    2023

  57. [57]

    The Anatomy of Conversational Scams: A Topic-Based Red Teaming Analysis of Multi-Turn Interactions in LLMs

    Xiangzhe Yuan, Zhenhao Zhang, Haoming Tang, and Siying Hu. The anatomy of conversa- tional scams: A topic-based red teaming analysis of multi-turn interactions in llms.CoRR, abs/2601.03134, 2026. doi: 10.48550/ARXIV .2601.03134. URL https://doi.org/10. 48550/arXiv.2601.03134

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2601.03134 2026

  58. [58]

    Archer: Training language model agents via hierarchical multi-turn RL

    Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn RL. InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URLhttps://openreview.net/forum?id=b6rA0kAHT1. 14

    2024

  59. [59]

    URLhttps://aclanthology.org/2024.emnlp-main.760

    2024

  60. [61]

    Zhehao Zhang, Weijie Xu, Fanyou Wu, and Chandan K. Reddy. Falsereject: A resource for improving contextual safety and mitigating over-refusals in llms via structured reasoning. CoRR, abs/2505.08054, 2025. doi: 10.48550/ARXIV .2505.08054. URL https://doi.org/ 10.48550/arXiv.2505.08054

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.08054 2025

  61. [62]

    On prompt-driven safeguarding for large language models

    Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. On prompt-driven safeguarding for large language models. InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URLhttps://openreview.net/forum?id=ugxGpOEkox

    2024

  62. [64]

    Maximum entropy inverse reinforcement learning

    Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. InAaai, volume 8, pages 1433–1438. Chicago, IL, USA, 2008. 15 Table 3: DCGS methods apply to any base model. We train DCGS critics with each of three families of base models and evaluate them on the experimental datasets, and observe that ...

    2008

  63. [65]

    You do not have access to the user’s true or hidden goals; infer only from the dialogue

    Phase 1 (benign assumption): "You are a reasoning module in an assistant that models user desires and intent. You do not have access to the user’s true or hidden goals; infer only from the dialogue. Given the dialogue history, build a list of {n candidates} high-level action candidates one item at a time. Dialogue History: {formatted history} Current cand...

  64. [66]

    Given the dialogue history, build a list of {n candidates} high-level action candidates one item at a time

    Phase 1 (adversarial assumption): "You are a reasoning module in an assistant that models user desires and intent. Given the dialogue history, build a list of {n candidates} high-level action candidates one item at a time. Dialogue History: {formatted history} Current candidate list: {existing candidates} Generate ONLY candidate number {next candidate ind...

  65. [67]

    Generate a response based on the following understanding of user intent (from the high-level policy)

    Phase 2 (CARES environment): "You are a healthcare assistant. Generate a response based on the following understanding of user intent (from the high-level policy). Understanding of User Intent: {belief context} Dialogue History: {formatted history} OUTPUT FORMAT: Structure your response exactly as follows: 1. Start with [RESPONSE] 2. Write your natural la...

  66. [68]

    [Yes]" is generally preferable to

    Phase 2 (other environments): "You are a helpful assistant. Generate {n candidates} diverse response candidates based on the following understanding of user intent. Understanding of User Intent: {belief context} Dialogue History: {formatted history} Generate exactly {n candidates} diverse response candidates. Each candidate should take a meaningfully dist...

  67. [69]

    • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or ...

    2025

  68. [2023]

    doi: 10.48550/ARXIV .2312.06674

    Pith/arXiv arXiv doi:10.48550/arxiv.2312.06674

  69. [2024]

    URLhttps://openreview.net/forum?id=TyFrPOKYXw

    OpenReview.net, 2024. URLhttps://openreview.net/forum?id=TyFrPOKYXw

    2024