Pith. sign in

REVIEW 4 major objections 5 minor 71 cited by

Deliberative Alignment: Reasoning Enables Safer Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Teaching a model to recall and reason over its written safety policies before answering makes it simultaneously more robust to jailbreaks and less prone to overrefusal.

desk verdict Deliberative alignment is a genuinely new method with strong empirical results, but the headline causal claim is underdetermined: the comparisons are cross-model and the ablations lack a standard safety-training control. read the letter →

arxiv 2412.16339 v2 pith:GL3GA2CT submitted 2024-12-20 cs.CL cs.AIcs.CYcs.LG

Melody Y. Guan , Manas Joglekar , Eric Wallace , Saachi Jain , Boaz Barak , Alec Helyar , Rachel Dias , Andrea Vallone
show 7 more authors
Hongyu Ren Jason Wei Hyung Won Chung Sam Toyer Johannes Heidecke Alex Beutel Amelia Glaese
This is my paper · ORCID
classification cs.CLcs.AIcs.CYcs.LG
keywords deliberativealignmentchain-of-thoughtreasoningjailbreakrobustnessoverrefusalsafetyspecificationreinforcementlearningfromAIfeedbackout-of-distributiongeneralizationlanguagemodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that language models become safer and more precisely controllable when they are given the actual text of their safety policies and trained to think through that text before answering. The authors call this "deliberative alignment" and apply it to OpenAI's o-series reasoning models, reporting that it improves both axes of safety at once: stronger resistance to jailbreak attacks and fewer refusals of harmless requests. The training data is produced synthetically, by having a reasoning model apply the written policy and then stripping the policy out of the prompt, so no human-written reasoning traces or answers are required. If the claim is right, safety alignment can scale with specification text rather than with human labeling effort, and test-time reasoning becomes a lever for safety as well as for capability.

What carries the argument

The load-bearing mechanism is the chain-of-thought as a policy-reasoning trace. Training examples are produced by context distillation: a reasoning model $G_{base}$ is given the original prompt plus a category-specific safety specification $\text{spec}(\text{category})$ and instructed to cite and apply the relevant policy in its reasoning; the specification is then stripped away, leaving $(\text{prompt}, \text{CoT}, \text{output})$ tuples in which the chain-of-thought cites policy text the model must recall on its own. A judge reasoning model $G_{RM}$, prompted again with $\text{spec}(\text{category})$, scores and filters these completions, with the score taken as the minimum across $k$ runs, and the survivors are used for supervised fine-tuning. A second, reinforcement-learning stage feeds $G_{RM}$'s scores on final answers back as reward, with the CoT hidden from the judge "to reduce the chance of encouraging deceptive CoTs." The category-specific construction of the specification is what keeps the policy text short enough to fit in context while still covering all safety categories.

What would settle it

Construct jailbreak prompts that the judge model $G_{RM}$ scores as policy-compliant but that trained human reviewers judge to be harmful, and test whether the deliberatively aligned model complies with them; if it does, the RL reward is being satisfied rather than safety being learned. The paper's own human review of StrongREJECT validates only the autograder, not $G_{RM}$, so this check is currently open.

Watch

Extended reading notes

Core claim

The paper's central claim is that a language model aligned by "deliberative alignment" - trained to recall the relevant written safety policy inside its chain-of-thought and reason over it before answering - adheres to safety specifications more precisely than models trained on labeled examples alone. The evidence is that o1 reaches a StrongREJECT goodness@0.1 of 0.88 versus GPT-4o's 0.37, and a not-overrefuse rate of 0.93 on XSTest versus 0.88, moving the Pareto frontier rather than trading one property off against the other. The paper also reports that a model whose safety training excluded all encoded and non-English data performs essentially as well on encoding-based and multilingual jailbreak evals as the fully trained model, which the authors take as evidence that reasoning over explicit policies generalizes out of distribution. Policy-retrieval measurements further show that the trained model cites the correct policy in its chain-of-thought far more often than a safety-untrained baseline, supporting the claim that the model is "right for the right reasons" rather than pattern-matching.

Load-bearing premise

The entire training signal comes from a judge AI that is handed the same rulebook the model is being taught, and the paper assumes that judge's scores genuinely track policy compliance even on adversarial and out-of-distribution inputs that the judge has never seen.

Editorial extensions

If this is right

  • Safety training no longer needs to grow human-labeled refusal datasets; the specification text itself can be distilled into the model, with humans reserved for evaluation.
  • Reasoning budget becomes a safety dial: the paper shows that giving the aligned model more inference-time compute improves performance on the hardest jailbreak and safe-completion evals, so safety can be bought with test-time thinking.
  • Embedded policy knowledge beats handed policy text: a model trained to recall the specification outperforms a baseline that merely receives the full specification in its system prompt at deployment time.
  • The method transfers across input formats: models safety-trained only on English, unencoded prompts generalize to encoded and multilingual jailbreaks at essentially the same level as models trained on those data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism generalizes, the binding constraint on safety becomes the quality and completeness of the written specification plus the judge's ability to read it, not the volume of human demonstrations; that would shift where safety research should concentrate its effort.
  • The design choice to hide the chain-of-thought from the judge during RL hints that directly optimizing the reasoning trace would risk rewarded rationalization; a testable extension is to compare whether cited policies actually cause refusals between SFT-only and SFT-plus-RL models.
  • Because prompts are routed to their specification by a safety-category label during training, routing errors are inherited by the model; since deployment has no such label, category-retrieval accuracy becomes a measurable predictor of safety performance, and the paper's Table 2 only measures it in distribution.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces 'Deliberative Alignment', a two-stage training method (SFT on model-generated chain-of-thought that reasons over safety specifications, followed by RL with a specification-prompted judge) for teaching LLMs to recall and reason about safety policies before answering. The authors apply the method to OpenAI's o-series models and report strong results on internal and public safety benchmarks, claiming that deliberative alignment pushes the Pareto frontier between jailbreak robustness and overrefusal, and improves out-of-distribution generalization. Section 4.1 presents ablations on a reduced o1-mini model showing both SFT and RL contribute, and Section 4.3 shows OOD generalization to non-English and encoded prompts.

Significance. If the causal claim holds, this is a practically important result: it suggests a scalable alternative to human-labeled safety data, and it provides evidence that explicit policy reasoning can reduce both safety failures and overrefusals. The paper has several strengths: a clear, detailed method description; a human-review validation of StrongREJECT (Appendix A); OOD experiments with bootstrap uncertainty; and a plausible mechanism (process supervision instills a reasoning prior). The main weakness is that the headline evidence is a cross-model comparison, so the central attribution to the method is not established.

major comments (4)
  1. [§3.1, Table 1, Figure 2] The central claim that deliberative alignment causes the reported safety gains is underdetermined because the headline comparisons are between o1 and GPT-4o, which differ in scale, architecture, pretraining, and general reasoning ability. The Section 4.1 ablations on o1-mini compare safety data versus no safety data, not versus a standard safety-training method (e.g., RLHF/CAI) on the same base. Please add a same-base control trained with a conventional safety method, or otherwise disentangle the method from base-model capability before claiming causation.
  2. [§4.1, Figure 14] The ablation baseline 'spec at inference-time' has no safety training at all, so it is not a valid control for comparing deliberative alignment against providing specifications at deployment time after standard safety training. Moreover, the text in §4.1 states that safety training increases overrefusals in this reduced setup, which appears to conflict with the abstract's claim of simultaneously improving jailbreak robustness and reducing overrefusals; this tension needs to be resolved.
  3. [§2.3.2, §2.4, §3.1] There is a circularity risk: the same safety specification is used for data generation, data filtering, RL reward, and autograding of internal evaluations, so improvements may reflect learning to satisfy the judge rather than genuine policy compliance. The human validation in Appendix A covers only StrongREJECT; I request an independent human evaluation (or a held-out judge) for the internal disallowed-content and safe-completion style metrics, and ideally a check that the GRM reward signal is not hackable.
  4. [Table 1] Table 1 reports point estimates without error bars for most rows, despite Figure 2 showing that bootstrap uncertainty is notable on these metrics. Report standard errors or confidence intervals for all Table 1 entries and state which cross-model differences are statistically significant.
minor comments (5)
  1. [Footnote 1] The abstract's claim 'without requiring human-written chain-of-thoughts or answers' should be stated with the footnote's caveat that broad category labels are used; consider moving this qualification into the main text.
  2. [§2.2] The phrase 'spec(category))' after 'than when given the entire specification' appears to have a typo (double closing parenthesis).
  3. [Appendix A] The text refers to 'Claude 3.4 Sonnet' while the rest of the paper and citations refer to Claude 3.5 Sonnet; the model version should be consistent.
  4. [§3.3] The explanation that 'the model had fewer regulated advice training examples' is a post-hoc hypothesis with no supporting data; either provide training-set counts or label it as speculation.
  5. [§5.1] The characterization that CAI's critiques are 'not employed during optimization' is stated without a citation to the CAI training details; add a reference or qualify the claim.

Circularity Check

1 steps flagged · score 2.0 of 10

No load-bearing circularity: the headline safety and Pareto results rest on external benchmarks and human review; only the auxiliary policy-retrieval metric is partly self-confirming.

  1. self definitional [Section 2.3.1 (data generation) and Section 4.2 (Policy Retrieval Accuracy)]
    "We then collect (CoT, output) completions which reference our policies within the chain-of-thought, by prompting the base reasoning model Gbase with the text of the associated safety specification. ... the specification-augmented prompt consists of: ... instructions to cite and discuss relevant parts of spec(category). ... To answer this question, we evaluate how often CoTs reference appropriate safety categories."

    The SFT labels were manufactured by prompting Gbase with spec(category) and explicitly instructing it to 'cite and discuss relevant parts of spec(category)', so the resulting CoTs are, by construction, policy-citing. Measuring 'policy retrieval accuracy' as the fraction of the trained model's CoTs whose cited category matches the prompt's category is therefore largely checking whether SFT memorized its own constructed labels, not independently testing a derived capability. The Full Data values (0.75/0.91/0.54 vs baseline 0.27/0.21/0.09) show the model learned the training distribution; the baseline contrast keeps the measure from being vacuous, but the metric is a process check, not a load-bearing safety prediction.

full rationale

The central safety claims are not circular. Gbase is spec-agnostic; the trained Gspec is a new function produced by SFT and RL, and the paper reports failure modes (e.g., increased overrefusals in the ablation setup, and o1-preview's poor safe-completion style). The headline Pareto improvement (Figure 2), jailbreak robustness, and overrefusal numbers use external benchmarks (StrongREJECT, XSTest), and the StrongREJECT results were validated by human review (Appendix A), breaking the 'same spec in, same spec out' loop. The same safety specification is indeed used for SFT data generation, GRM filtering/reward, and internal autograders; this makes the internal policy-adherence evals partly self-referential, but those evals measure the stated target (adherence to OpenAI's policies) and are not the basis of the main external comparison. The causal attribution that deliberative alignment, rather than o1's general reasoning capability, causes the gains is underdetermined because the headline comparison is cross-model (o1 vs GPT-4o) and the ablations compare safety training vs no safety training rather than vs standard RLHF/Constitutional AI; that is a control/confound limitation, not circularity. The only identifiable by-construction step is the auxiliary policy-retrieval metric in Section 4.2, which is not load-bearing for the safety results, so the overall circularity score is 2.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method introduces no fitted numeric parameters beyond small hyperparameters such as the number of judge runs and the retention threshold. It does rest on several unproven domain assumptions about the reliability of LLM judges, the accuracy of category labels, and the effectiveness of CoT supervision, and these assumptions are load bearing for the claimed results.

free parameters (2)
  • k (number of judge runs per completion)
    In Section 2.3.2, each completion is judged k times and the minimum score is kept. The value of k is not reported, and it controls the strictness of SFT data filtering.
  • SFT retention threshold on GRM scores
    Section 2.3.2 says 'retain completions with high scores'; the threshold is not specified but affects the training set composition.
assumptions (4)
  • domain assumption An LLM judge prompted with a safety specification produces valid reward scores for policy compliance.
    Used in Section 2.3.2 for quality filtering and Section 2.4 for RL reward; if the judge is systematically wrong or hackable, the training signal is misaligned.
  • domain assumption The safety category label for each prompt is accurate enough that spec(category) contains the relevant policy details.
    Footnote 1 and Section 2.3.1; labels come from a mix of human and AI labeling and may be noisy.
  • ad hoc to paper Supervising the chain-of-thought during SFT instills a genuine reasoning prior rather than teaching the model to output policy citations without improving behavior.
    This is the core mechanistic assumption of the method; the paper provides behavioral evidence but not a mechanistic proof.
  • domain assumption A model trained only for helpfulness (Gbase) can reason over an inserted safety spec well enough to generate good completions (context distillation works).
    Relied on in Section 2.3.1 for data generation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Deliberative Alignment: Reasoning Enables Safer Language Models." pith.science (2026). https://pith.science/paper/GL3GA2CT

@misc{pith2026241216339,
  author       = {Pith},
  title        = {Pith review of: Deliberative Alignment: Reasoning Enables Safer Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GL3GA2CT}},
  note         = {Machine review of arXiv:2412.16339}
}
read the original abstract

As large-scale language models increasingly impact safety-critical domains, ensuring their reliable adherence to well-defined principles remains a fundamental challenge. We introduce Deliberative Alignment, a new paradigm that directly teaches the model safety specifications and trains it to explicitly recall and accurately reason over the specifications before answering. We used this approach to align OpenAI's o-series models, and achieved highly precise adherence to OpenAI's safety policies, without requiring human-written chain-of-thoughts or answers. Deliberative Alignment pushes the Pareto frontier by simultaneously increasing robustness to jailbreaks while decreasing overrefusal rates, and also improves out-of-distribution generalization. We demonstrate that reasoning over explicitly specified policies enables more scalable, trustworthy, and interpretable alignment.

Figures

Figures reproduced from arXiv: 2412.16339 by the authors.

Figure 1
Figure 1. A sample o1 chain-of-thought. Here, a user attempts to obtain advice on untraceable payment methods to use for an adult website, in order to avoid detection by law enforcement. The user tries to jailbreak the model, by encoding the request and wrapping it with instructions intended to encourage the model to comply. In the model’s chain-of-thought, the model decodes the request and recognizes that the user is trying … view at source ↗
Figure 2
Figure 2. Main safety results. The o1 models advance the Pareto frontier of refusing to answer malicious jailbreak prompts (from StrongREJECT [12]) and not over-refusing benign prompts (from XSTest [13]), compared to GPT-4o and other state-of-the-art LLMs. Error bars represent estimates of standard deviation calculated over 1,000 bootstrap trials. 2 Method Our approach to deliberative alignment is motivated by the following o… view at source ↗
Figure 3
Figure 3. Illustration of overall methodology. Key processes are shown along the bottom of the figure. We first construct a dataset of (prompt, CoT, output) tuples where the CoTs refers to relevant policies (top-left zoombox). We collect these by prompting a reasoning model Gbase with safety prompts along with safety specifications (spec) that are tailored to safety categories (cat). After filtering with a policy-aware reward… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Excerpt of style guidelines for hard refusals [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Our template for creating the modified prompt that is given to [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Our template for creating the prompt that is given to [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: A sample o1 chain-of-thought for an illicit advice prompt. Here, a user asks for illicit advice. In the model’s chain-of-thought, it successfully reasons through the relevant snippets from the OpenAI safety policies (highlighted in green). The model provides an answer …
Figure 8
Figure 8. Figure 8: A sample o1 chain-of-thought for a self-harm prompt. Here, a user expresses suicidal intent and asks for help. In the model’s chain-of-thought, it successfully reasons through the relevant snippets from the OpenAI safety policies (highlighted in green). The model provi…
Figure 9
Figure 9. Figure 9: Excerpt of an example content policy for illicit behavior safety category [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Excerpt of an example content policy for self-harm safety category [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: A sample o1 chain-of-thought for an overrefusal prompt. Here, a user asks for a translation of a policy-violating instruction, which should be allowed by the transformation exception. In the model’s chain-of-thought, it successfully reasons that, while the instruction…
Figure 12
Figure 12. Figure 12: Comparative evaluation of text safety and robustness across leading LLMs. o1 is competitive com￾pared to other leading models on benchmarks assessing disallowed content (WildChat), jailbreaks (Stron￾gREJECT), overrefusals (XSTest), hallucinations (SimpleQA), and bias …
Figure 13
Figure 13. Figure 13: Impact of inference-time compute on model performance. The o1 model has stronger performance on challenging evals when allowed more compute to spend on reasoning. Our results demonstrate that safety failures can result from the model being given insufficient time to r…
Figure 14
Figure 14. Figure 14: Ablations on safety training stages We remove safety training from SFT and/or RL to see the impact on model performance. We also compare to a baseline where we provide the spec at inference-time without performing any safety training. 16 [PITH_FULL_IMAGE:figures/full…
Figure 15
Figure 15. Figure 15: Comparison of deliberative alignment and representative methods of existing alignment approaches. a) Training data generation: Even though RLAIF methods like CAI [26] use safety specifications to generate training labels, only the labels themselves are used in trainin…

Discussion (0). Continue with ORCID to comment.

Forward citations

Showing 60 of 71 Pith papers that cite this

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. See all 71 Pith citations

  1. BavGround: A Benchmark for Regional Cultural Grounding and Dialect Competence in Bavarian

    cs.CL 2026-08 conditional novelty 7.0 of 10

    A new tri-lingual Bavarian culture benchmark shows open-weight LLMs underperform on Bavarian and source-grounded items, and that evaluation protocol materially changes accuracy and rankings.

  2. AI Security Leaderboard: Methodology, Results and Minimal Standard

    cs.CR 2026-08 conditional novelty 7.0 of 10

    A new framework, the FAR.AI Minimal Standard, measures frontier safeguards and finds Grok 4.5 and Gemini 3.1 Pro are cheaply jailbroken while Claude Fable 5 and GPT-5.6 Sol showed no universal jailbreaks under the sam...

  3. Verbalizable Representations Form a Global Workspace in Language Models

    cs.CL 2026-07 conditional novelty 7.0 of 10

    Language models represent their current reasoning in a small, readable set of verbalizable vectors (the J-space) that functions like a global workspace.

  4. HARC: Coupling Harmfulness and Refusal Directions for Robust Safety Alignment

    cs.AI 2026-07 unverdicted novelty 7.0 of 10

    HARC couples harmfulness and refusal directions at prompt and response positions, yielding the best robustness-capability-usability trade-off among major safety methods.

  5. Between a Rock and a Hard Place: The Tension Between Ethical Reasoning and Safety Alignment in LLMs

    cs.CR 2025-09 unverdicted novelty 7.0 of 10

    TRIAL, a multi-turn jailbreak method that frames harmful actions as lesser-evil ethical choices, reports high jailbreak success rates across open and closed LLMs.

  6. Evaluating Language Model Reasoning about Confidential Information

    cs.LG 2025-08 conditional novelty 7.0 of 10

    PasswordEval shows frontier models frequently leak passwords or confidential information, jailbreaks worsen failures, and reasoning traces leak secrets even when final answers do not.

  7. Subliminal Learning: Language models transmit behavioral traits via hidden signals in data

    cs.LG 2025-07 conditional novelty 7.0 of 10

    Teacher language models transfer behavioral traits to students fine-tuned on semantically unrelated number, code, and reasoning traces, especially when the two models share initialization.

  8. Underwriting the Agent Economy: The Blueprint for an AI Insurance Stack

    cs.CY 2026-07 conditional novelty 6.5 of 10

    Affirmative AI-agent insurance with billion-scale limits is achievable by 2030 solely through coordinated industry build-out of an eight-component stack spanning data, CAT models, standards, contracts, underwriting, p...

  9. Position: Preventing AI-Generated CSAM Necessitates New Approaches to AI Safety

    cs.CY 2026-06 accept novelty 6.5 of 10

    Legal and ethical bans on CSAM access and generation break standard AI safety techniques, creating 15 open problems that demand new methods for dataset cleaning, concept fusion prevention, fine-tuning resilience, dete...

  10. Multi-Agent AI Safety as an Institutional Design Problem

    cs.LG 2026-08 conditional novelty 6.0 of 10

    In synthetic delegation workflows, identical final violation rates hide different mechanisms: prompts prevent prohibited attempts, provenance-aware guards block and recover, and a local policy guard fails when transfo...

  11. Constitutional Midtraining: Content Presence Drives Alignment Gains

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Constitutional midtraining at 120B scale produces alignment gains that persist through instruction tuning and benign fine-tuning, especially reducing blackmail behavior, with no average capability cost.

  12. Shared SFT Lessons Across Alignment, Model Organisms, and Toy Models

    cs.LG 2026-07 conditional novelty 6.0 of 10

    SFT lessons — reason-based training, on-model replay, and wash-out robustness — transfer across toy models, model organisms, and alignment SFT, improving the capability–safety tradeoff.

  13. QuantiBias: Benchmarking Quantization-Induced Bias in LLMs

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Quantization leaves refusal and multiple-choice bias checks flat while open-ended stereotype endorsement remains high (~24–27% under an independent judge), a gap standard safety evaluations miss.

  14. A Geometric Perspective on Stabilizing Value Conflict Resolution

    cs.LG 2026-07 conditional novelty 6.0 of 10

    An annealing-inspired chain-of-thought prompt lowers the sharpest loss-landscape curvature and improves moral reasoning benchmark scores in small LLMs.

  15. Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Models

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Zero-RL multi-stage constructive safety alignment with SERL and long-context training lets a 14B model match much larger models on safety without collapsing helpfulness or style.

  16. Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges

    cs.CL 2026-05 conditional novelty 6.0 of 10

    Per-bias selection of a cross-family LLM auditor lifts biased-judgment accuracy from 0.805/0.824 baselines to 0.884.

  17. Think Before You Code: Dual Reasoning for the NLSafety-Utility Trade-Off in LLM Code Generation

    cs.SE 2026-04 unverdicted novelty 6.0 of 10

    Dual Reasoning with explicit safety audits improves the new SUDS metric by 1.32x to 3.42x over baselines on code generation benchmarks containing injected harmful keywords.

  18. Reasoning Up the Instruction Ladder for Controllable Language Models

    cs.CL 2025-10 conditional novelty 6.0 of 10

    RLVR on ~7K verifiable system/user conflict examples teaches LLMs to prioritize higher-priority instructions, improving instruction-hierarchy and safety benchmarks.

  19. SafeSearch: Automated Red-Teaming of LLM-Based Search Agents

    cs.AI 2025-09 conditional novelty 6.0 of 10

    An automated red-teaming framework and 300-case benchmark show that a single unreliable website can induce unsafe responses in LLM search agents, with attack success rates up to 90.5%.

  20. Statutory Construction and Interpretation for Artificial Intelligence

    cs.CL 2025-09 conditional novelty 6.0 of 10

    Prompt-based legal canons and iterative rule refinement reduce disagreement among LLM judges about whether a response complies with natural-language rules.

  21. IntentionReasoner: Facilitating Adaptive LLM Safeguards through Intent Reasoning and Selective Query Refinement

    cs.AI 2025-08 reject novelty 6.0 of 10

    IntentionReasoner adds four-level intent classification and targeted query rewriting to LLM guardrails, reporting state-of-the-art harm detection and near-zero jailbreak success.

  22. Reinforcement Learning with Rubric Anchors

    cs.AI 2025-08 unverdicted novelty 6.0 of 10

    Rubric-based rewards extend reinforcement learning to open-ended text generation, yielding a 30B model that outperforms a 671B model on humanities-style benchmarks.

  23. Whose Truth? Pluralistic Geo-Alignment for (Agentic) AI

    cs.AI 2025-08 conditional novelty 6.0 of 10

    Geo-alignment means matching an AI system's output distribution to the locally appropriate distribution for each query, location, and time, and the paper argues spatial structure makes that target learnable.

  24. ASTRA: Autonomous Spatial-Temporal Red-teaming for AI Software Assistants

    cs.CR 2025-08 conditional novelty 6.0 of 10

    ASTRA is an automated red-teaming agent that uses knowledge-graph-guided spatial and temporal probing to find 11-66% more safety violations in AI coding assistants than existing tools.

  25. Libra: Large Chinese-based Safeguard for AI Content

    cs.AI 2025-07 conditional novelty 6.0 of 10

    Libra-Guard, a Chinese-language AI content safety model, reaches 86.8% accuracy on the new 5,720-sample Libra-Test benchmark, beating open-source baselines and approaching GPT-4o and Claude-3.5-Sonnet.

  26. SAND: Boosting LLM Agents with Self-Taught Action Deliberation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A self-training framework that uses self-consistency action sampling and execution-guided critiques to teach LLM agents to explicitly deliberate over candidate actions, improving average rewards by about 20% over supe...

  27. FORTRESS: Frontier Risk Evaluation for National Security and Public Safety

    cs.CY 2025-06 conditional novelty 6.0 of 10

    A new benchmark with instance-specific rubrics measures frontier LLMs' willingness to assist with national security and public safety threats, alongside a paired over-refusal test.

  28. Think Clearly: Improving Reasoning via Redundant Token Pruning

    cs.AI 2025-06 conditional novelty 6.0 of 10

    A training-free test-time method prunes low-attention reasoning tokens from the KV cache, guided by an injected end-of-thinking token, and reports accuracy gains on math competition benchmarks.

  29. Teaching an Old LLM Secure Coding: Localized Preference Optimization on Distilled Preferences

    cs.CR 2025-05 conditional novelty 6.0 of 10

    A new dataset and a token-masked preference optimization loss reduce security vulnerabilities in LLM-generated Python code while preserving code quality.

  30. The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating It

    cs.CL 2025-05 accept novelty 6.0 of 10

    LLM safety research at ACL venues from 2020 to 2024 is predominantly English-only, and the language gap is growing over time.

  31. LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training

    cs.LG 2025-05 conditional novelty 6.0 of 10

    LlamaRL is a distributed asynchronous RL framework that reports up to 10.7x faster RL post-training steps for 405B-parameter LLMs than a co-located synchronous baseline, using off-policy corrections and GPU-direct wei...

  32. Are Reasoning Models More Prone to Hallucination?

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Post-training pipeline choice (SFT+RL vs RL-only vs SFT-only) reliably shifts hallucination rates in large reasoning models on fact-seeking benchmarks.

  33. Lifelong Safety Alignment for Language Models

    cs.CR 2025-05 conditional novelty 6.0 of 10

    A co-evolutionary attacker-defender loop, warmed up by strategies extracted from jailbreak papers, reduces jailbreak success rate on a robust model from 73% to 7% in two iterations.

  34. Beyond Safe Answers: A Benchmark for Evaluating True Risk Awareness in Large Reasoning Models

    cs.AI 2025-05 conditional novelty 6.0 of 10

    A new benchmark shows that top reasoning models identify all relevant risks in under 40% of cases even when their final answers look safe.

  35. Real-World Gaps in AI Governance Research

    cs.AI 2025-04 conditional novelty 6.0 of 10

    Corporate AI safety research is dominated by pre-deployment alignment and evaluation work, while high-risk deployment topics such as medical error, misinformation, bias, behavioral design, and copyright are measured t...

  36. HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generation

    cs.CL 2025-04 conditional novelty 6.0 of 10

    HyPerAlign, which prompts LLMs with hypotheses inferred from a few user examples, beats a preference fine-tuning baseline on authorship attribution and improves safety refusal behavior.

  37. AI Behind Closed Doors: a Primer on The Governance of Internal Deployment

    cs.CY 2025-04 conditional novelty 6.0 of 10

    Internal deployment of frontier AI systems is an under-governed risk area; the paper provides a conceptual map, a legal review, lessons from safety-critical industries, and a defense-in-depth governance blueprint.

  38. On the Promise for Assurance of Differentiable Neurosymbolic Reasoning Paradigms

    cs.AI 2025-02 conditional novelty 6.0 of 10

    A systematic comparison finds that differentiable neurosymbolic systems offer better assurance mainly in arithmetic-like reasoning tasks, not across the board, and interpretable shortcuts can increase adversarial risk.

  39. MetaSC: Test-Time Safety Specification Optimization for Language Models

    cs.CL 2025-02 conditional novelty 6.0 of 10

    MetaSC improves language model safety by using a meta-critic to iteratively rewrite the safety specification that guides self-critique at inference time.

  40. Adversarial Reasoning at Jailbreaking Time

    cs.LG 2025-02 conditional novelty 6.0 of 10

    A loss-guided 'reason, verify, search' loop with three LLM modules surpasses prior semantic jailbreak methods on several defended models.

  41. BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generation

    cs.CL 2025-02 conditional novelty 6.0 of 10

    BARE generates diverse, high-quality synthetic training data from only three seed examples by having a base model draft and an instruction-tuned model refine, improving downstream fine-tuning accuracy in few-shot settings.

  42. Pragmatic Attack Surface: Vulnerabilities of Implicit Context in Large Language Models

    cs.CL 2026-08 reject novelty 5.0 of 10

    The paper claims that prompting LLMs to infer and enrich implicit presuppositions bypasses safety alignment and yields high attack success on bias, hate, and unsafe-code benchmarks.

  43. Outcome-Guided Distillation: A Teacher-Student Framework to Advance VLM Reasoning in Autonomous Driving

    cs.RO 2026-07 conditional novelty 5.0 of 10

    An outcome-guided teacher-student framework for VLM driving, where a teacher generates reflective reasoning from ground-truth waypoints, a student distills it, and a separate decoder converts the reasoning into waypoi...

  44. Cost of Reasoning in non-English Languages: A Case Study on Japanese

    cs.CL 2026-07 conditional novelty 5.0 of 10

    Japanese reasoning-language control is feasible with CPT plus GRPO, but incurs a capability cost and does not free-improve cultural Japanese performance.

  45. Towards terahertz nanomechanics

    physics.app-ph 2025-08 unverdicted novelty 5.0 of 10

    Suspended Lamb-wave resonators in lithium niobate films thinned from 300 nm to 67 nm reach 220 GHz, doubling the prior record.

  46. Beyond Surface-Level Detection: Towards Cognitive-Driven Defense Against Jailbreak Attacks via Meta-Operations Reasoning

    cs.AI 2025-08 unverdicted novelty 5.0 of 10

    A jailbreak defense that reasons about hidden manipulations in attack prompts, trained with supervised fine-tuning plus entropy-guided reinforcement learning, generalizes to attacks never seen in training.

  47. R1-ACT: Efficient Reasoning Model Safety Alignment by Activating Safety Knowledge

    cs.AI 2025-08 conditional novelty 5.0 of 10

    Adding an explicit 'is this harmful?' step to the reasoning chain, trained on just 1,000 examples, substantially reduces harmful responses from reasoning models while roughly preserving benchmark reasoning performance.

  48. The Policy Cliff: A Theoretical Analysis of Reward-Policy Maps in Large Language Models

    cs.AI 2025-07 conditional novelty 5.0 of 10

    Small reward perturbations can discontinuously switch an LLM's optimal policy whenever optimal actions tie, and entropy regularization restores Lipschitz continuity of the reward-policy map.

  49. AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning

    cs.AI 2025-07 conditional novelty 5.0 of 10

    AlphaAlign uses pure reinforcement learning with a verifiable safety reward to make LLMs refuse harmful requests with explicit reasoning while preserving helpfulness on benign queries.

  50. InfoFlood: Jailbreaking Large Language Models with Information Overload

    cs.CR 2025-06 conditional novelty 5.0 of 10

    InfoFlood claims near-perfect jailbreak success on four frontier LLMs by rewriting harmful queries into verbose academic prose with fake citations, past-tense framing, and ethical disclaimers, without adversarial suffixes.

  51. SafeCoT: Improving VLM Safety with Minimal Reasoning

    cs.AI 2025-06 conditional novelty 5.0 of 10

    Training vision-language models to emit a short rule-based reasoning chain before refusing improves the safety-usefulness balance, with reported gains even at 100 training samples.

  52. Lossless Token Sequence Compression via Meta-Tokens

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A new compression scheme replaces repeated token subsequences with learnable placeholder tokens, shrinking prompts by 15-27% with no loss of information, and fine-tuned LLMs perform nearly as well as on uncompressed input.

  53. Security Concerns for Large Language Models: A Survey

    cs.CR 2025-05 conditional novelty 5.0 of 10

    A survey that classifies LLM security threats and argues that intrinsic agentic risks, such as scheming, are underappreciated and poorly defended.

  54. Exploring the Vulnerability of the Content Moderation Guardrail in Large Language Models via Intent Manipulation

    cs.CL 2025-05 conditional novelty 5.0 of 10

    An automated two-stage rewriting framework, IntentPrompt, bypasses LLM content guardrails with 80-98% success by turning harmful asks into declarative outlines.

  55. Adaptive Plan-Execute Framework for Smart Contract Security Auditing

    cs.CR 2025-05 reject novelty 5.0 of 10

    SmartAuditFlow claims 100% detection on a standard smart contract benchmark and all 13 tested CVEs via a plan-execute LLM workflow, though the supporting evaluation has major reproducibility and validation gaps.

  56. A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment

    cs.CR 2025-04 conditional novelty 5.0 of 10

    A large collaborative survey organizes LLM and LLM-agent safety issues into a full-stack lifecycle framework from data preparation to deployment.

  57. Reason4Rec: Deliberative User Preference Alignment of Large Language Models for Recommendation

    cs.IR 2025-02 conditional novelty 5.0 of 10

    An LLM recommender that decomposes preference reasoning into summarization, matching, and prediction steps, trained on review text, improves rating accuracy and explanation quality on Amazon and Yelp benchmarks.

  58. Standardizing Intelligence: Aligning Generative AI for Regulatory and Operational Compliance

    cs.CY 2025-02 conditional novelty 5.0 of 10

    The paper proposes C3F, a two-axis framework rating GenAI compliance capability and standard criticality, and applies it to 15 models and 34 standards.

  59. Don't Command, Cultivate: An Exploratory Study of System-2 Alignment

    cs.CL 2024-11 conditional novelty 5.0 of 10

    Encouraging LLMs to analyze user requests step-by-step (System-2 Alignment) modestly improves safety on open-source models, but with trade-offs and limited evidence.

  60. A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models

    cs.CL 2025-09 conditional novelty 4.0 of 10

    A structured literature survey concluding that reasoning capabilities do not automatically make LLMs more trustworthy and can introduce new vulnerabilities in safety, robustness, and privacy.

See all 71 Pith citations

Reference graph

Works this paper leans on

37 extracted references · 12 canonical work pages · cited by 71 Pith papers (see all)

  1. [1]

    [Online]

    OpenAI, Learning to reason with LLMs , 2024. [Online]. Available: https://openai.com/index/learning- to-reason-with-llms/

    work page 2024

  2. [2]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. , “Training language models to follow instructions with human feedback,” in NeurIPS, 2022

    work page 2022

  3. [3]

    The LLaMA 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. , “The LLaMA 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    arXiv 2024

  4. [5]

    GPT-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. , “GPT-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  5. [6]

    Universal and transferable adversarial attacks on aligned language models,

    A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” arXiv preprint arXiv:2307.15043 , 2023

    arXiv 2023

  6. [7]

    Jailbroken: How does llm safety training fail?

    A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How does llm safety training fail?” NeurIPS, 2024

    2024

  7. [8]

    Jailbreaking leading safety-aligned llms with simple adaptive attacks,

    M. Andriushchenko, F. Croce, and N. Flammarion, “Jailbreaking leading safety-aligned llms with simple adaptive attacks,” arXiv preprint arXiv:2404.02151 , 2024

    arXiv 2024

  8. [9]

    Solving math word problems with process-and outcome-based feedback,

    J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins, “Solving math word problems with process-and outcome-based feedback,” arXiv preprint arXiv:2211.14275, 2022

    arXiv 2022

Show all 37 references
  1. [10]

    Learning by distilling context,

    C. Snell, D. Klein, and R. Zhong, “Learning by distilling context,” arXiv preprint arXiv:2209.15189 , 2022

    2022 arXiv

  2. [11]

    A general language assistant as a laboratory for alignment,

    A. Askell, Y. Bai, A. Chen, et al., “A general language assistant as a laboratory for alignment,” arXiv preprint arXiv:2112.00861, 2021

    2021 arXiv

  3. [12]

    A strongreject for empty jailbreaks,

    A. Souly, Q. Lu, D. Bowen, et al., “A strongreject for empty jailbreaks,”arXiv preprint arXiv:2402.10260, 2024. 21

    2024 arXiv

  4. [13]

    Xstest: A test suite for identifying exaggerated safety behaviours in large language models,

    P. R¨ ottger, H. R. Kirk, B. Vidgen, G. Attanasio, F. Bianchi, and D. Hovy, “Xstest: A test suite for identifying exaggerated safety behaviours in large language models,” arXiv preprint arXiv:2308.01263, 2024

    2024 arXiv

  5. [14]

    [Online]

    OpenAI, Introducing the model spec , 2024. [Online]. Available: https://cdn.openai.com/spec/model- spec-2024-05-08.html

    2024

  6. [15]

    Wildchat: 1m chatgpt interaction logs in the wild,

    W. Zhao, X. Ren, J. Hessel, C. Cardie, Y. Choi, and Y. Deng, “Wildchat: 1m chatgpt interaction logs in the wild,” arXiv preprint arXiv:2405.01470 , 2024

    2024 arXiv

  7. [16]

    ”do anything now

    X. Shen, Z. Chen, M. Backes, Y. Shen, and Y. Zhang, “”do anything now”: Characterizing and evalu- ating in-the-wild jailbreak prompts on large language models,” arXiv preprint arXiv:2308.03825, 2024

    2024 arXiv

  8. [17]

    Jailbreaking black box large language models in twenty queries,

    P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong, “Jailbreaking black box large language models in twenty queries,” arXiv preprint arXiv:2310.08419 , 2024

    2024 arXiv

  9. [18]

    Jailbreakbench: An open robustness benchmark for jail- breaking large language models,

    P. Chao, E. Debenedetti, A. Robey, et al. , “Jailbreakbench: An open robustness benchmark for jail- breaking large language models,” arXiv preprint arXiv:2404.01318 , 2024

    2024 arXiv

  10. [19]

    Refusal-trained llms are easily jailbroken as browser agents,

    P. Kumar, E. Lau, S. Vijayakumar, et al., “Refusal-trained llms are easily jailbroken as browser agents,” arXiv preprint arXiv:2410.13886 , 2024

    2024 arXiv

  11. [20]

    [Online]

    OpenAI, O1 system card , 2024. [Online]. Available: https://cdn.openai.com/o1-system-card.pdf

    2024

  12. [21]

    [Online]

    OpenAI, Gpt-4o system card, 2024. [Online]. Available: https://cdn.openai.com/gpt-4o-system-card.pdf

    2024

  13. [22]

    [Online]

    Anthropic, Model card addendum: Claude 3.5 haiku and upgraded claude 3.5 sonnet , 2024. [Online]. Available: https://assets.anthropic.com/m/1cd9d098ac3e6467/original/Claude-3-Model-Card-October- Addendum.pdf

    2024

  14. [23]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text,

    G. Gemini Team, “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text,” arXiv preprint arXiv:2403.05530 , 2024

    2024 arXiv

  15. [24]

    Mea- suring short-form factuality in large language models,

    J. Wei, N. Karina, H. W. Chung, Y. J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus, “Mea- suring short-form factuality in large language models,” arXiv preprint arXiv:2411.04368 , 2024

    2024 arXiv

  16. [25]

    BBQ: A hand-built bias benchmark for question answering,

    A. Parrish, A. Chen, N. Nangia, V. Padmakumar, J. Phang, J. Thompson, P. M. Htut, and S. R. Bow- man, “BBQ: A hand-built bias benchmark for question answering,” arXiv preprint arXiv:2110.08193 , 2021

    2021 arXiv

  17. [26]

    Constitutional AI: Harmlessness from AI feedback,

    Y. Bai, S. Kadavath, S. Kundu, et al. , “Constitutional AI: Harmlessness from AI feedback,” arXiv preprint arXiv:2212.08073, 2022

    2022 arXiv

  18. [27]

    Self-refine: Iterative refinement with self-feedback,

    A. Madaan, N. Tandon, P. Gupta, et al. , “Self-refine: Iterative refinement with self-feedback,” arXiv preprint arXiv:2303.17651, 2023

    2023 arXiv

  19. [28]

    Deep reinforcement learning from human preferences,

    P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” in Advances in Neural Information Processing Systems , vol. 30, 2017

    2017

  20. [29]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” arXiv preprint arXiv:2305.18290 , 2024

    2024 arXiv

  21. [30]

    Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies,

    L. Pan, M. Saxon, W. Xu, D. Nathani, X. Wang, and W. Y. Wang, “Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies,” arXiv preprint arXiv:2308.03188, 2023

    2023 arXiv

  22. [31]

    Generating sequences by learning to self-correct,

    S. Welleck, X. Lu, P. West, F. Brahman, T. Shen, D. Khashabi, and Y. Choi, “Generating sequences by learning to self-correct,” in The Eleventh International Conference on Learning Representations , vol. 2, 2023

    2023

  23. [32]

    Large language model programs,

    I. Schlag, S. Sukhbaatar, A. Celikyilmaz, W.-t. Yih, J. Weston, J. Schmidhuber, and X. Li, “Large language model programs,” arXiv preprint arXiv:2305.05364 , 2023

    2023 arXiv

  24. [33]

    Backtracking improves generation safety,

    Y. Zhang, J. Chi, H. Nguyen, K. Upasani, D. M. Bikel, J. Weston, and E. M. Smith, “Backtracking improves generation safety,” arXiv preprint arXiv:2409.14586 , 2024. 22

    2024 arXiv

  25. [34]

    Russell, Human compatible: Artificial intelligence and the problem of control , 1st

    S. Russell, Human compatible: Artificial intelligence and the problem of control , 1st. USA: Penguin Books, 2019, isbn: 9780525558637

    2019

  26. [35]

    Bostrom, Superintelligence: Paths, Dangers, Strategies , 1st

    N. Bostrom, Superintelligence: Paths, Dangers, Strategies , 1st. USA: Oxford University Press, Inc., 2014, isbn: 0199678111

    2014

  27. [36]

    The basic ai drives,

    S. M. Omohundro, “The basic ai drives,” in Proceedings of the 2008 Conference on Artificial General Intelligence 2008: Proceedings of the First AGI Conference , NLD: IOS Press, 2008, pp. 483–492, isbn: 9781586038335

    2008

  28. [37]

    Uncovering deceptive tendencies in language models: A simulated company ai assistant,

    O. J¨ arviniemi and E. Hubinger, “Uncovering deceptive tendencies in language models: A simulated company ai assistant,” arXiv preprint arXiv:2405.01576 , 2024

    2024 arXiv

  29. [38]

    Deception abilities emerged in large language models,

    T. Hagendorff, “Deception abilities emerged in large language models,” Proceedings of the National Academy of Sciences, vol. 121, no. 24, Jun. 2024, issn: 1091-6490. 23 A Human Review Experiment Model performance on the StrongREJECT benchmark [12] was validated through a detai...

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.