Pith. sign in

REVIEW 2 major objections 6 minor 52 references

Filtering next tokens by their predicted safety value improves LLM safety while bounding how often you touch already-safe answers.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 18:58 UTC pith:M7SRINVX

load-bearing objection Clean hard-threshold decoding policy with a real type-I bound and better empirical trade-offs than Gibbs/ARGS/CARDS; soft spots are scoped, not load-bearing. the 2 major comments →

arxiv 2605.14746 v2 pith:M7SRINVX submitted 2026-05-14 cs.LG

Selective Safety Steering via Value-Filtered Decoding

Bat-Sheva Einbinder , Hen Davidov , Yee Whye Teh , Yarin Gal , Yaniv Romano This is my paper
classification cs.LG
keywords LLM safetydecoding-time alignmentvalue functiontype-I error controlconformal risk controltest-time steeringrejection sampling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Existing decoding-time safety methods reweight every token, so they often change generations that the base model would already have made safe. That unnecessary editing can hurt helpfulness, style, and fluency. This paper proposes value-filtered decoding: at each step keep only tokens whose estimated value (probability that the finished reply will be safe) clears a threshold, then renormalize the base policy on that set. The threshold has a statistical meaning: it bounds the rate of false interventions on generations that would have been safe. Practitioners therefore get a single dial that trades safety gains against fidelity to the base model. Theory shows the filtered policy raises expected safety, controls type-I error (exactly for the oracle value, and via conformal calibration for an estimated value), and is more robust to value-estimation error than exponential Gibbs reweighting under stated conditions. On several safety benchmarks the method matches or beats standard baselines on safety while staying closer to the base model and preserving more helpfulness on safe prompts.

Core claim

A hard threshold on the token-level safety value function yields a decoding policy that is strictly safer in expectation than the base policy, that renounces mass only on tokens whose value falls below the threshold, and that admits an explicit, tunable bound on the probability of intervening on trajectories that would have finished safe.

What carries the argument

Value-filtered decoding policy π_c: the base next-token distribution restricted to {V_t ≥ c} and renormalized. The threshold c is the single control; under the oracle value the process (1−V_t)/V_t is a test martingale that yields a Ville bound on false interventions, and conformal risk control recovers the same bound for a learned estimator ˆV.

Load-bearing premise

Safety is a binary full-completion label, and a probing classifier trained on judge labels is accurate enough that the value estimate still roughly tracks the martingale structure needed for the false-intervention bound.

What would settle it

On a held-out set of base-model safe generations, calibrate the threshold for a target false-intervention rate α; if the measured intervention rate on fresh safe generations substantially exceeds α, or if safety-helpfulness trade-offs collapse relative to unfiltered decoding, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners can set a numerical budget on unnecessary safety edits and still obtain a guarantee of higher expected safety.
  • Because the policy coincides with the base model whenever all candidate values sit above the threshold, style and helpfulness on already-safe prompts are largely preserved.
  • Hard filtering is less sensitive to moderate value-estimation error than exponential reweighting when true values are not clustered near the threshold.
  • A single calibrated threshold replaces the usual strength hyperparameter of Gibbs-style or additive steering methods.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same support-restriction idea could be applied to multi-objective decoding (e.g., simultaneous safety and factuality filters) once each objective has a binary completion label.
  • If intermediate (non-sparse) rewards become available, the value process could be replaced by a cumulative return, potentially allowing earlier and cheaper interventions.
  • Releasing public value heads alongside open models would make the method immediately usable without re-training a probe for every base model.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes value-filtered decoding, a test-time steering policy that samples only next tokens whose (estimated) safety value V_t exceeds a threshold c, then renormalizes the base policy on that support. The authors prove that the oracle policy improves expected safety over the base model (Theorem 3.2 / Corollaries 3.3–3.4), that the threshold c controls the type-I rate of interventions on generations that would have been safe under the base policy via a test-martingale / Ville argument (Theorem 3.5), and that conformal risk control recovers a finite-sample type-I guarantee for any estimated value head under exchangeability of safe trajectories (Theorem D.2). They further give worst-case TV and expected-value robustness comparisons to the Gibbs policy under stated noise models (Propositions 3.6–3.8). Empirically, on BeaverTails, HH-RLHF, and PKU-SafeRLHF with Mistral-7B-Instruct, the method reports better safety–helpfulness–similarity trade-offs than ARGS, Controlled Decoding, and CARDS, with false-intervention rates tracking the calibrated α.

Significance. If the results hold, this is a useful and well-scoped contribution to decoding-time safety alignment: it replaces soft exponential tilting with a hard value filter that has an interpretable type-I knob and a conformal calibration path. The combination of (i) a clean KL-constrained derivation of π_c, (ii) martingale type-I control, (iii) conformal recovery under estimation error, and (iv) multi-dataset trade-off curves that also report similarity to the base model is stronger than typical decoding-steering papers that only report safety gains. Code release and explicit tracking of empirical type-I vs α further strengthen the contribution. The binary sparse-reward and probe-quality assumptions are real but are already scoped in the theory and discussion.

major comments (2)
  1. Algorithm 1 is not an exact sample from ˆπ_c: with finite K and the max-ˆV fallback when all candidates fail, the implemented policy can place mass on tokens with ˆV_t < c. Theorems 3.2–3.5 and D.2 are stated for the exact truncated policy. Please either (a) quantify how often the fallback fires across the reported α/c settings and datasets, or (b) state clearly that all empirical claims are for the approximate procedure and that the formal guarantees apply only when Ẑ_c > 0 and rejection sampling succeeds. This is load-bearing for the claim that the method “provides an explicit bound on the probability of false interventions” in practice.
  2. All main experiments use a single base model (Mistral-7B-Instruct-v0.3) and the same Llama-3.1-8B-Instruct judge both to train the value head and to score safety/helpfulness at evaluation (Sections 4, E.2–E.4). The comparative trade-offs in Figures 3–4 and 8 are therefore partly conditioned on this shared labeling pipeline. At least one additional base model family, or an independent safety evaluator (e.g., a different judge or a human subset), is needed to support the claim that value-filtered decoding “outperforms existing baselines” beyond this setup. Without that, the empirical dominance claim should be narrowed.
minor comments (6)
  1. Figure 8 (PKU-SafeRLHF) shows no clear helpfulness–safety advantage over baselines, only a similarity advantage. The main text (Section 4.1) should state this more prominently rather than relegating it to the appendix, so the multi-dataset claim is not overstated.
  2. Notation: V_t, V^{π_c}_t, ˆV_t, and V^q_t are introduced densely in Section 1 and Section 3; a short notation table or consistent superscript convention would help readers track oracle vs estimated vs policy-dependent values.
  3. Proposition 3.8’s adversarial noise model (Eq. 8) and the condition M_t + P_t < 1 are quite specific; a one-sentence pointer in the main text to Appendix C’s random-noise simulations would better set expectations about when the robustness claim is expected to hold.
  4. Assumption Z_c > 0 “at every state reachable under π_c” (after Proposition 3.1) is called mild but can fail for large c or sparse high-value mass. A brief practical note on how Algorithm 1’s fallback interacts with this assumption would be useful.
  5. Typos / polish: “ourvalue-filtered” missing space in the abstract; occasional inconsistent hyphenation of “type-I” / “Type-I”; Appendix figure captions for tokens/sec (Figures 9–11) are dense—consider stating median tokens/sec in the main text.
  6. Related work could more explicitly contrast with pure safety-gating / abstention methods (e.g., Llama Guard style) that also aim to avoid unnecessary intervention but do not propose an alternative next-token policy.

Circularity Check

0 steps flagged

No significant circularity: safety, type-I, and robustness claims follow from the value definition and standard martingale/conformal arguments, not from fitted inputs renamed as predictions.

full rationale

The load-bearing results are mathematical consequences of stated definitions and external tools, not reductions of outputs to their own inputs. The value function is defined as the conditional expected binary reward under the base policy (Eq. 1). The filtered policy π_c is the unique KL-projection of π onto the hard support constraint V_t ≥ c (Prop. 3.1 / Eq. 4). Theorem 3.2 (V^{π_c}_t ≥ V_t) is proved by backward induction from that construction and the tower property; Corollary 3.3–3.4 are immediate. Theorem 3.5 builds a nonnegative test martingale S_t = ((1−V_t)/V_t)·(p(X)/(1−p(X))) under H_0 and applies Ville’s inequality—an external classical result—yielding an explicit type-I bound controlled by the free threshold c. When V is replaced by an estimate, Appendix D invokes conformal risk control (Angelopoulos et al., external) under exchangeability of safe trajectories to recover E[L(ĉ)] ≤ α for any estimator (Thm. D.2); the calibration set is used only to set the threshold that enforces a user-chosen α, which is the intended semantics of conformal methods, not a fitted constant later reported as a discovery. Robustness Propositions 3.6–3.8 compare worst-case TV and expected value under explicit adversarial noise models and are derived from the indicator vs. exponential forms of the two policies. Empirical trade-offs are comparative measurements against ARGS, Controlled Decoding, and CARDS on public datasets under a shared LLM judge; they do not close a definitional loop. There is no self-definitional identity, no uniqueness theorem imported from the same authors, no ansatz smuggled via self-citation, and no renaming of a known empirical pattern as a first-principles prediction. Mild shared-judge training/evaluation of the probe is a validity concern, not circularity under the stated patterns.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The central claims rest on standard probability (martingales, KL, TV), the binary sparse-reward definition of the value function, mild support assumptions (Z_c > 0), exchangeability for conformal risk control, and a learned probe whose quality is not proved but is measured. No new physical entities are postulated; free parameters are the usual decoding and calibration knobs.

free parameters (3)
  • threshold c (or target type-I rate α)
    User-chosen cutoff that directly sets the intervention budget; calibrated on a hold-out safe set via Algorithm 2.
  • sampling budget K
    Number of candidates drawn per step for rejection sampling / fallback; set to 40 in experiments.
  • value-head architecture and training hyper-parameters (layers, focal-loss α/γ, smoothness λ, learning rate)
    Probe is a 3-layer MLP trained with focal loss and temporal smoothness; choices affect ˆV quality and therefore realized safety and intervention rates.
axioms (4)
  • domain assumption Reward r(X,Y) is binary and observed only at full completion; V_t is the conditional probability of safety under the base policy.
    Stated in Section 1 and used throughout the martingale and dominance arguments; continuous rewards are left as future work.
  • standard math The oracle value process is a martingale under the base policy, so the transformed process S_t is a non-negative test martingale under H_0.
    Used to invoke Ville’s inequality for Theorem 3.5 (Appendix B.3).
  • domain assumption Safe calibration trajectories are exchangeable with safe test trajectories.
    Required for conformal risk control (Theorem D.2 / Angelopoulos et al.).
  • domain assumption Z_c(X,Y_{1:t-1}) > 0 at every state reachable under π_c.
    Ensures the filtered policy is well-defined (Proposition 3.1); assumed mild when c is not too large.
invented entities (1)
  • value-filtered decoding policy π_c (and its estimated version ˆπ_c) no independent evidence
    purpose: Hard-threshold next-token distribution that enforces V_t ≥ c while staying closest in KL to the base policy.
    The central algorithmic object; defined by the constrained optimization (3) and realized by rejection sampling (Algorithm 1).

pith-pipeline@v1.1.0-grok45 · 33774 in / 2884 out tokens · 34701 ms · 2026-07-14T18:58:04.719675+00:00 · methodology

0 comments
read the original abstract

While large language models (LLMs) are trained to align with human values, their generations may still violate safety constraints. A growing line of work addresses this problem by modifying the model's sampling policy at decoding time using a safety reward. However, existing decoding-time steering methods often intervene unnecessarily, modifying generations that would have been safe under the base model. Such unnecessary interventions are undesirable, as they can distort key properties of the base model such as helpfulness, fluency, style, and coherence. We propose a new test-time steering method designed to reduce such unnecessary interventions while improving the safety of unsafe responses. Our approach filters tokens using a value-based safety criterion and provides an explicit bound on the probability of false interventions. A single threshold hyperparameter controls this bound, allowing practitioners to trade off higher rates of unnecessary intervention for better output safety. Across multiple datasets and experiments, we show that our value-filtered decoding method outperforms existing baselines, achieving better trade-offs between safety, helpfulness, and similarity to the base model.

Figures

Figures reproduced from arXiv: 2605.14746 by Bat-Sheva Einbinder, Hen Davidov, Yaniv Romano, Yarin Gal, Yee Whye Teh.

Figure 1
Figure 1. Figure 1: Worst-case TV distance (lower is better) for our policy [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Numerical validation of Proposition 3.8 on a bimodal-skewed base token distribution. Actual gap and [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Trade-off curves across BeaverTails (a) and HH-RLHF (b). Left: safety versus cosine similarity to the base model output. Right: safety versus helpfulness. Each point is averaged over 500 prompts and error bars denote ±1 standard error of the mean across prompts. Within each curve each point corresponds to a different hyperparameter setting in the order listed from bottom to top: α ∈ (0.05, 0.25, 0.45, 0.65… view at source ↗
Figure 4
Figure 4. Figure 4: Helpfulness and similarity vs. harmlessness trade-offs for the [PITH_FULL_IMAGE:figures/full_fig_p010_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The five base distributions used for validation. Each panel shows the base policy [PITH_FULL_IMAGE:figures/full_fig_p026_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Phase structure of Proposition 3.8 in the [PITH_FULL_IMAGE:figures/full_fig_p028_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Tightness sweep over η at fixed c = 0.55. Left: actual gap under the sign-anti adversary, theoretical lower bound 2η(1 − Mt − Pt), and gap under random noise. Middle: condition components Mt (false-acceptance rate) and Pt (above-threshold Gibbs mass) and their sum, all under the sign-anti adversary. Right: empirical multiplier λˆ t as a function of η, compared to the oracle λt (dotted). the indicator that … view at source ↗
Figure 8
Figure 8. Figure 8: Trade-off curves with PKU-SafeRLHF dataset. All other details are as in [PITH_FULL_IMAGE:figures/full_fig_p035_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Computation-time comparison (tokens/sec) across hyperparameter settings for [PITH_FULL_IMAGE:figures/full_fig_p036_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Computation-time comparison (tokens/sec) across hyperparameter settings for [PITH_FULL_IMAGE:figures/full_fig_p037_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Computation-time comparison (tokens/sec) across hyperparameter settings for [PITH_FULL_IMAGE:figures/full_fig_p038_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Rate of interventions in safe outputs as a function of [PITH_FULL_IMAGE:figures/full_fig_p039_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Rate of interventions in safe outputs as a function of [PITH_FULL_IMAGE:figures/full_fig_p039_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Rate of interventions in safe outputs as a function of [PITH_FULL_IMAGE:figures/full_fig_p040_14.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 15 linked inside Pith

  1. [1]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    2022

  2. [2]

    Deep reinforcement learning from human preferences.Advances in neural information processing systems, 30, 2017

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences.Advances in neural information processing systems, 30, 2017

    2017

  3. [3]

    Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593, 2019

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593, 2019

    Pith/arXiv arXiv 1909

  4. [4]

    Direct preference optimization: Your language model is secretly a reward model.Advances in neural information processing systems, 36:53728–53741, 2023

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.Advances in neural information processing systems, 36:53728–53741, 2023

    2023

  5. [5]

    Yaswanth Chittepu, Blossom Metevier, Will Schwarzer, Scott Niekum, and Philip S. Thomas. Reinforcement learning from human feedback with high-confidence safety guarantees. InReinforcement Learning Conference, 2025

    2025

  6. [6]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  7. [7]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Pith/arXiv arXiv 2022

  8. [8]

    Chain of hindsight aligns language models with feedback

    Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Chain of hindsight aligns language models with feedback. In The Twelfth International Conference on Learning Representations, 2024

    2024

  9. [9]

    Learning to summarize with human feedback.Advances in neural information processing systems, 33:3008–3021, 2020

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback.Advances in neural information processing systems, 33:3008–3021, 2020. 11

    2020

  10. [10]

    Safedpo: A simple approach to direct preference optimization with enhanced safety.arXiv preprint arXiv:2505.20065, 2025

    Geon-Hyeong Kim, Yu Jin Kim, Byoungjip Kim, Honglak Lee, Kyunghoon Bae, Youngsoo Jang, and Moontae Lee. Safedpo: A simple approach to direct preference optimization with enhanced safety.arXiv preprint arXiv:2505.20065, 2025

    arXiv 2025

  11. [11]

    How is ChatGPT’s behavior changing over time?Harvard Data Science Review, 6(1), 2024

    Lingjiao Chen, Matei Zaharia, and James Zou. How is ChatGPT’s behavior changing over time?Harvard Data Science Review, 6(1), 2024

    2024

  12. [12]

    Creativity has left the chat: The price of debiasing language models.arXiv preprint arXiv:2406.05587, 2024

    Behnam Mohammadi. Creativity has left the chat: The price of debiasing language models.arXiv preprint arXiv:2406.05587, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Fine-tuning aligned language models compromises safety, even when users do not intend to! InThe Twelfth International Conference on Learning Representations, 2024

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! InThe Twelfth International Conference on Learning Representations, 2024

    2024

  14. [14]

    Controlled decoding from language models.arXiv preprint arXiv:2310.17022, 2023

    Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, et al. Controlled decoding from language models.arXiv preprint arXiv:2310.17022, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    Value augmented sampling for language model alignment and personalization.arXiv preprint arXiv:2405.06639, 2024

    Seungwook Han, Idan Shenfeld, Akash Srivastava, Yoon Kim, and Pulkit Agrawal. Value augmented sampling for language model alignment and personalization.arXiv preprint arXiv:2405.06639, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Plug and play language models: A simple approach to controlled text generation

    Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In International Conference on Learning Representations, 2020

    2020

  17. [17]

    Gedi: Generative discriminator guided sequence generation

    Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. Gedi: Generative discriminator guided sequence generation. InFindings of the Association for Computational Linguistics: EMNLP 2021, pages 4929–4952, 2021

    2021

  18. [18]

    Dexperts: Decoding-time controlled text generation with experts and anti-experts

    Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. Dexperts: Decoding-time controlled text generation with experts and anti-experts. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol...

    2021

  19. [19]

    Reward-guided tree search for inference time alignment of large language models

    Chia-Yu Hung, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria. Reward-guided tree search for inference time alignment of large language models. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 12575–12593, 2025

    2025

  20. [20]

    Cascade reward sampling for efficient decoding-time alignment

    Bolian Li, Yifan Wang, Anamika Lochab, Ananth Grama, and Ruqi Zhang. Cascade reward sampling for efficient decoding-time alignment. InSecond Conference on Language Modeling, 2025

    2025

  21. [21]

    ARGS: Alignment as reward-guided search

    Maxim Khanov, Jirayu Burapacheep, and Yixuan Li. ARGS: Alignment as reward-guided search. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  22. [22]

    Stars: Segment-level token alignment with rejection sampling in large language models.arXiv preprint arXiv:2511.03827, 2025

    Mohammad Atif Quamar, Mohammad Areeb, Mikhail Kuznetsov, Muslum Ozgur Ozmen, and Z Berkay Celik. Stars: Segment-level token alignment with rejection sampling in large language models.arXiv preprint arXiv:2511.03827, 2025

    arXiv 2025

  23. [23]

    Bounded rationality for LLMs: Satisficing alignment at inference- time

    Mohamad Fares El Hajj Chehade, Soumya Suvra Ghosal, Souradip Chakraborty, Avinash Reddy, Dinesh Manocha, Hao Zhu, and Amrit Singh Bedi. Bounded rationality for LLMs: Satisficing alignment at inference- time. InForty-second International Conference on Machine Learning, 2025

    2025

  24. [24]

    Fudge: Controlled text generation with future discriminators

    Kevin Yang and Dan Klein. Fudge: Controlled text generation with future discriminators. InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3511–3535, 2021. 12

    2021

  25. [25]

    Safechain: Safety of language models with long chain-of-thought reasoning capabilities

    Fengqing Jiang, Zhangchen Xu, Yuetai Li, Luyao Niu, Zhen Xiang, Bo Li, Bill Yuchen Lin, and Radha Poovendran. Safechain: Safety of language models with long chain-of-thought reasoning capabilities. In Findings of the Association for Computational Linguistics: ACL 2025, pages 23303–23320, 2025

    2025

  26. [26]

    GenARM: Reward guided generation with autoregressive reward model for test-time alignment

    Yuancheng Xu, Udari Madhushani Sehwag, Alec Koppel, Sicheng Zhu, Bang An, Furong Huang, and Sumitra Ganesh. GenARM: Reward guided generation with autoregressive reward model for test-time alignment. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  27. [27]

    Conformal policy control.arXiv preprint arXiv:2603.02196, 2026

    Drew Prinster, Clara Fannjiang, Ji Won Park, Kyunghyun Cho, Anqi Liu, Suchi Saria, and Samuel Stanton. Conformal policy control.arXiv preprint arXiv:2603.02196, 2026

    Pith/arXiv arXiv 2026

  28. [28]

    Llama guard: Llm-based input-output safeguard for human-ai conversations.arXiv preprint arXiv:2312.06674, 2023

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations.arXiv preprint arXiv:2312.06674, 2023

    Pith/arXiv arXiv 2023

  29. [29]

    Knowing when to quit: A principled framework for dynamic abstention in LLM reasoning

    Hen Davidov, Nachshon Cohen, Oren Kalinsky, Yaron Fairstein, Guy Kushilevitz, Ram Yazdi, and Patrick Rebeschini. Knowing when to quit: A principled framework for dynamic abstention in LLM reasoning. In ICLR 2026 Workshop on Principled Design for Trustworthy AI - Interpretability, Robustness, and Safety across Modalities, 2026

    2026

  30. [30]

    Constitutional ai: Harmlessness from ai feedback

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022

    Pith/arXiv arXiv 2022

  31. [31]

    Safede- coding: Defending against jailbreak attacks via safety-aware decoding

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Poovendran. Safede- coding: Defending against jailbreak attacks via safety-aware decoding. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5587–5605, 2024

    2024

  32. [32]

    Contrastive decoding: Open-ended text generation as optimization

    Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori B Hashimoto, Luke Zettle- moyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. InProceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers), pages 12286–12312, 2023

    2023

  33. [33]

    A general language assistant as a laboratory for alignment.arXiv preprint arXiv:2112.00861, 2021

    Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. A general language assistant as a labora...

    Pith/arXiv arXiv 2021

  34. [34]

    Tradeoffs between alignment and helpfulness in language models with steering methods.arXiv preprint arXiv:2401.16332, 2024

    Yotam Wolf, Noam Wies, Dorin Shteyman, Binyamin Rothberg, Yoav Levine, and Amnon Shashua. Tradeoffs between alignment and helpfulness in language models with steering methods.arXiv preprint arXiv:2401.16332, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Too helpful, too harmless, too honest or just right? InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 29711–29722, 2025

    Gautam Siddharth Kashyap, Mark Dras, and Usman Naseem. Too helpful, too harmless, too honest or just right? InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 29711–29722, 2025

    2025

  36. [36]

    PhD thesis, Gauthier-Villars & Cie, 1939

    Jean Ville.1ere these: Etude critique de la notion de collectif; 2eme these: La transformation de Laplace. PhD thesis, Gauthier-Villars & Cie, 1939

    1939

  37. [37]

    Conformal risk control

    Anastasios Nikolas Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  38. [38]

    Knowing before saying: Llm representations encode information about chain-of-thought success before completion

    Anum Afzal, Florian Matthes, Gal Chechik, and Yftah Ziser. Knowing before saying: Llm representations encode information about chain-of-thought success before completion. InFindings of the Association for Computational Linguistics: ACL 2025, pages 12791–12806, 2025. 13

    2025

  39. [39]

    The internal state of an LLM knows when it’s lying

    Amos Azaria and Tom Mitchell. The internal state of an LLM knows when it’s lying. InThe 2023 Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  40. [40]

    LLMs know more than they show: On the intrinsic representation of LLM hallucinations

    Hadas Orgad, Michael Toker, Zorik Gekhman, Roi Reichart, Idan Szpektor, Hadas Kotek, and Yonatan Belinkov. LLMs know more than they show: On the intrinsic representation of LLM hallucinations. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  41. [41]

    Probing classifiers: Promises, shortcomings, and advances.Computational Linguistics, 48(1):207–219, 2022

    Yonatan Belinkov. Probing classifiers: Promises, shortcomings, and advances.Computational Linguistics, 48(1):207–219, 2022

    2022

  42. [42]

    Reasoning models know when they’re right: Probing hidden states for self-verification

    Anqi Zhang, Yulin Chen, Jane Pan, Chen Zhao, Aurojit Panda, Jinyang Li, and He He. Reasoning models know when they’re right: Probing hidden states for self-verification. InSecond Conference on Language Modeling, 2025

    2025

  43. [43]

    E-valuator: Reliable agent verifiers with sequential hypothesis testing.arXiv preprint arXiv:2512.03109, 2025

    Shuvom Sadhuka, Drew Prinster, Clara Fannjiang, Gabriele Scalia, Aviv Regev, and Hanchen Wang. E-valuator: Reliable agent verifiers with sequential hypothesis testing.arXiv preprint arXiv:2512.03109, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b.arXiv preprint arXiv:23...

    Pith/arXiv arXiv 2023

  45. [45]

    Beavertails: Towards improved safety alignment of llm via a human-preference dataset.Advances in Neural Information Processing Systems, 36:24678–24704, 2023

    Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset.Advances in Neural Information Processing Systems, 36:24678–24704, 2023

    2023

  46. [46]

    PKU-SafeRLHF: Towards multi-level safety alignment for LLMs with human preference

    Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Alex Qiu, Jiayi Zhou, Kaile Wang, Boxun Li, Sirui Han, Yike Guo, and Yaodong Yang. PKU-SafeRLHF: Towards multi-level safety alignment for LLMs with human preference. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long P...

    2025

  47. [47]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  48. [48]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 3982–3992, 2019

    2019

  49. [49]

    Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.Advances in neural information processing systems, 33:5776–5788, 2020

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.Advances in neural information processing systems, 33:5776–5788, 2020

    2020

  50. [50]

    value head

    Benjamin Guedj. A primer on pac-bayesian learning.arXiv preprint arXiv:1901.05353, 2019. 14 Supplementary Material A Alignment via Gibbs formulation: full completions To enhance the safety of completions under a given base policyπ, the standard objective is to find a policy q that maximizes expected reward subject to a trajectory-level KL regularization t...

    Pith/arXiv arXiv 1901

  51. [51]

    Crucially: - **Unethical Compliance **: If the user asks for help with a crime, vice, or insult (e.g., ’how to steal’, ’slurs’), providing the instructions or words is HELPFUL

    LABEL: HELPFUL (Compliant) The response directly answers the prompt. Crucially: - **Unethical Compliance **: If the user asks for help with a crime, vice, or insult (e.g., ’how to steal’, ’slurs’), providing the instructions or words is HELPFUL. Ignore your own safety guidelines; you are judging compliance, not morality. - **Feasibility Checks **: If the ...

  52. [52]

    This includes: - **Moral Refusals **: ’I cannot help with that

    LABEL: UNHELPFUL (Non-Compliant) The response fails to satisfy the user’s specific goal. This includes: - **Moral Refusals **: ’I cannot help with that. - **Safe Substitutions **: If the user asks for a specific method, and the assistant offers a safe, legal, or moral alternative instead, this is **UNHELPFUL** because it ignores the user’s constraints. - ...