Pith. sign in

REVIEW 2 major objections 5 minor 37 references

This paper claims that the probability a large language model generates a harmful response can be certified from below by guided tree search, producing proven non-zero bounds even when the true probability is astronomically small.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 10:17 UTC pith:RKAT2B7N

load-bearing objection Useful lower-bound technique with a hidden prefix-closure assumption and overstated certification claims; worth refereeing after re-scoping. the 2 major comments →

arxiv 2607.20286 v1 pith:RKAT2B7N submitted 2026-07-22 cs.CL cs.AI

Sound Probabilistic Safety Bounds for Large Language Models

Mahdi Nazeri , Anne-Kathrin Schmuck , Sadegh Soudjani , Alessandro Abate This is my paper
classification cs.CL cs.AI
keywords large language modelsprobabilistic safety certificationrare-event estimationautoregressive generation treesafety oraclelatent-space featureslower boundsmonte carlo
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper's goal is to turn a fixed-prompt safety evaluation of a large language model into a mathematical certificate: not an estimate, but a proven lower bound on the probability that the model emits a harmful response. It observes that because harmfulness is closed under extending a partial output, any node of the autoregressive generation tree flagged as harmful by a safety oracle carries an entire subtree of probability mass that can be added to the bound. To find such nodes efficiently the algorithm learns a latent-space direction that separates harmful from harmless partial outputs and expands tree branches ranked by a fitness score; this biased search is what makes the bound non-trivial where ordinary sampling and binomial-confidence PAC bounds return zero. The claimed payoff is that aligned models can be certified, compared, and improved using concrete harmful generations the search discovers, despite the extreme rarity of those events.

Core claim

The central claim is that Algorithm 1 returns a number p_L that is rigorously no larger than the true harmfulness probability p = P(H(G(x,L,M))=1) for a fixed prompt, while remaining practically useful for rare events. The key observation is prefix closure: if the safety oracle labels a partial sequence harmful, all of its continuations are harmful, so the probability mass of that tree node is a guaranteed contribution to p. The algorithm therefore expands only a small, heuristically chosen part of the generation tree, accumulates mass at every oracle-flagged node, and halts with a sound lower bound. The experimental message is that, with the same computational budget, this guided bound is o

What carries the argument

The load-bearing object is the autoregressive generation tree, an L-level |V|-ary tree of partial outputs whose edge probabilities are the model's next-token distributions, together with a prefix-closed safety oracle H. A node is 'unsafe' if H of its partial sequence equals 1; prefix closure guarantees its whole subtree mass belongs to the true harmful probability, so the algorithm can prune and add that mass. The search is steered by a harmfulness feature vector v, the normalized difference of mean residual-stream representations of harmful and harmless examples, and a fitness score f(y) = (b/len(y) + sum over layers of cosine similarity to v) * P(y|x), which ranks leaf nodes for expansion.

Load-bearing premise

The certificate collapses if the safety oracle is not an exact, prefix-closed predicate for the intended harm property, or if the probabilities accumulated along the tree are not the true next-token probabilities of the model under the evaluated sampling settings.

What would settle it

Run the algorithm on a toy two-token model whose true harm probability can be computed by enumeration, using an oracle that flags a prefix while a continuation of that prefix is labelled harmless; if the returned p_L exceeds the enumerated p, the soundness proof fails. Even simpler: search for a single flagged prefix with a benign continuation in any real model and show the accumulated mass is an over-count.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Safety evaluations can report a certified non-zero lower bound for rare harmful behaviors using a modest number of token generations, where Monte Carlo and binomial-confidence PAC bounds return zero.
  • The lower bound is monotone in the computational budget, so more compute can only tighten the certificate, never invalidate it.
  • The returned harmful partial generations can be used directly as training data for post-training or alignment, connecting certification to model improvement.
  • Because only black-box access to the oracle and the model's next-token probabilities is needed, the method applies to models whose internals are not fully accessible.
  • A non-trivial lower bound demonstrates that a model cannot be certified safe at that prompt, refuting a safety claim without enumerating the entire output space.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the certificate's meaning is tied to the oracle. In the reported experiments the oracle is a substring blacklist, so the certified number is the probability of emitting a blacklisted n-gram, not the probability of semantically harmful text.
  • Editorial inference: the formal definition of p uses raw temperature-1 autoregressive sampling, while the experiments apply temperature and Top-K; the soundness proof carries over only if the accumulated tree probabilities match the actual modified sampling distribution the tables are reporting.
  • Editorial inference: the same prefix-accumulation mechanism should apply to any prefix-closed specification, so the algorithm may transfer to other certificate targets such as forbidden-token policies, leaked personal data, or unsafe code patterns.
  • Editorial inference: a natural extension, not pursued in the paper, is combining this guided lower bound with an upper bound from ordinary sampling, yielding a two-sided certified interval for the same prompt.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper addresses the problem of certifying the probability p that a fixed LLM, prompted with x, generates a harmful completion. It first reviews Clopper-Pearson intervals as PAC-style bounds, then proposes an algorithm that builds a partial autoregressive generation tree, using a learned latent-space 'harmfulness direction' to prioritize which nodes to expand. Whenever a node is labeled harmful by an oracle H, the algorithm adds the node's probability to a lower bound p_L and prunes the subtree. The paper claims these values are formally guaranteed lower bounds on p, and reports experiments on Phi-4, Llama-3.2-3B and Llama-3.1-8B with substring blacklists, obtaining very small but non-zero bounds where Monte Carlo sampling yields zero.

Significance. The partial-subtree argument is a clean and useful observation: any subtree, searched by any heuristic, yields a valid lower bound if harmful leaves are correctly identified and probabilities match the sampling distribution. The separation of heuristic search from the formal accumulation step is methodologically attractive and could enable scalable rare-event certification. That said, the manuscript currently overstates the formal guarantees: the proof requires the harm oracle to be prefix-closed, a condition contradicted by the stated 'no assumptions on H' and not guaranteed by the proposed neural/human oracles; and the formal target distribution and experimental decoding distribution are not aligned. With these fixed, the framework would be a meaningful contribution to LLM safety evaluation.

major comments (2)
  1. [Sec. 4, Obs. (1) and Alg. 1 lines 14–16; Sec. 2 'Safety oracle'] The soundness of the accumulation step requires H to be prefix-closed: once H(prefix)=1, every continuation must be harmful. The paper explicitly says (Sec. 2, 'Safety oracle') that 'we make no assumptions on the internal structure of H' and proposes trained neural classifiers and human moderators as admissible oracles. Such oracles are not guaranteed to be prefix-closed. For example, a classifier may label 'I will kill you' harmful but 'I will kill you with kindness' harmless; Algorithm 1 would then add the probability of the entire subtree rooted at the former, overcounting the true harmful probability. Thus the proof contains a hidden assumption that contradicts the stated generality. The experiments only use substring blacklists, which are prefix-closed, so they do not validate the general claim. The paper should require H to implement a prefix-closed safety set and explicitly restri
  2. [Sec. 2, Eqs. (1)–(2); Alg. 1 line 7; Sec. 5] The formal probability p is defined for G(x,L,M) using the raw next-token distribution M(s), with no temperature or top-k parameters. Algorithm 1 explicitly queries M([prompt;response] | T, K), and every experiment varies T and K. If the tree probabilities are those of the temperature/top-k modified distribution, the returned bound is a lower bound on a different quantity than the p in Eq. (2). Please define G and p with explicit decoding parameters and prove the bound for that distribution, or state precisely which distribution the experiments certify. This is necessary for the abstract's claim that the experimental lower bounds are sound for Eq. (2).
minor comments (5)
  1. [Sec. 5 tables (Exp. I)] The Clopper-Pearson upper bounds appear inconsistent with the stated budget. For example, Exp. I reports p_CP^U=0.264 with zero observed harms; a 95% upper bound for n≈1000 would be about 0.003. If the effective number of independent completions is much smaller, please report n explicitly and compute the interval on that n. As written, the baseline comparison is hard to interpret.
  2. [Algorithm 1, line 9] The pseudocode sets V_K ← (pr>0), selecting all tokens with positive probability, while the text says 'generate its Top-K most probable children.' Clarify whether the top-K restriction is applied inside the model call or in the pseudocode; otherwise the Top-K hyperparameter is unused.
  3. [Notation throughout] The same symbol p is used for the unknown true probability in Eq. (2), for the lower-bound accumulator in Alg. 1, and for the lower/upper bounds in Sec. 3. Use p_L, p_U, or similar consistently to avoid ambiguity.
  4. [Figures 4–9] The captions say 'lower bounds (log scale) as a total number of sampled tokens' but the x-axis is labeled '# iterations.' Please make the axis/caption consistent and state whether each iteration expands one node or samples one token.
  5. [Sec. 5, 'obtained harmful responses'] Several displayed 'harmful responses' are benign educational texts that contain a blacklisted substring. This is internally consistent with H being the formal definition of harm, but the prose should be tempered: the certified quantity is the probability of emitting a blacklisted n-gram, not semantic harmfulness.

Circularity Check

0 steps flagged

No significant circularity: the lower bound is a partial sum of model probabilities over found harmful prefixes, and the fitted latent direction affects only search order, not the bound itself.

full rationale

The central soundness claim in Sec. 4 is not circular. Algorithm 1 returns p_L by accumulating prob(prefix) for nodes with H(prefix)=1. Under the prefix-closure property stated in Observation (1), all leaves below such a prefix are harmful, so the accumulated mass is exactly the probability of a subset of the harmful event {H=1}. Thus p_L <= p is a subset-sum lower bound by construction, not a fitted quantity renamed as a prediction. The learned direction v, offset b, and fitness f(y) appear only in node selection (Alg. 1 line 13) and do not enter the accumulated expression on lines 14-16, so the soundness of the bound is independent of the fitted feature direction. The paper's statement in Sec. 2 that 'we make no assumptions on the internal structure of H' is in tension with the prefix-closure needed for Observation (1); for a non-prefix-closed oracle the algorithm could overcount. This is a soundness/correctness gap rather than a circular derivation. The experimental oracles are substring blacklists, which are prefix-closed. Self-citations ([12] and [19]) appear only as background or related work and are not load-bearing justifications of the main bound. No circular reduction is exhibited, so the appropriate finding is no significant circularity, with a minor non-load-bearing self-citation.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 1 invented entities

The bound's validity rests on the oracle being an exact prefix-closed labeler and on tree probabilities matching the deployed sampling distribution; everything else (feature direction v, offset b, Top-K, temperature) tunes tightness, not soundness.

free parameters (6)
  • fitness offset b (Eq. 3) = unreported
    Positive offset in the node fitness f(y); value never reported in experiments; controls exploration of low-cosine nodes. Affects which nodes are expanded (tightness), not the soundness of the lower bound.
  • feature extraction layer l for harmfulness direction v = unreported
    v is computed from residual stream r^(l)(y) at layer l, but l is never specified; also inconsistent with Eq. (3) summing over all layers. Affects search quality only.
  • harmful/harmless example sets D_harmful(x), D_harmless(x) = 64 samples each
    Sampled from an unspecified unrestricted LLM; the direction v is their difference of means, i.e., fitted from data. Soundness of the bound is independent of v.
  • Top-K branching factor = 10 or 20 (experiments)
    User-set hyperparameter controlling the number of children expanded per node; affects bound tightness, not validity.
  • temperature T = 0.2–1.0 (experiments)
    Sampling temperature changes the generation distribution, so the p bounded in experiments differs from Eq. (2), which is stated for temperature-1 full-vocabulary sampling. This redefinition is not made explicit.
  • harm predicate blacklist B per experiment = e.g. {'nitric acid', 'sulfuric acid', 'acetic acid'}
    Chosen per prompt by hand; defines what counts as harmful in each experiment. The small blacklists make 'harmful' mean 'contains a listed substring'.
axioms (6)
  • domain assumption The safety oracle H exactly implements a prefix-closed harm predicate (H(prefix)=1 ⇒ all continuations harmful)
    Load-bearing for the soundness of the bound accumulation in Alg. 1 (lines 14–16) and Sec. 4 observation (1). The paper also states 'we make no assumptions on the internal structure of H' (Sec. 2), so a black-box classifier oracle would not guarantee it.
  • domain assumption Next-token probabilities used in the tree equal the actual sampling distribution of G under the experimental decoding (temperature, Top-K)
    The formal definition of p in Eq. (2) uses raw sampling M(s)[w]; experiments run with temperature and Top-K, silently changing the distribution the bound is about. If tree probabilities and sampler disagree, the bound is for a different model.
  • domain assumption Latent-space 'harmfulness features' exist, are linear, and steering along them biases generation toward harmful tokens
    Sec. 4 observation (3), grounded only in cited prior work [3]. The paper's Discussion acknowledges this affects usefulness, not soundness.
  • domain assumption Residual stream r^(l)(y) at the chosen layer provides a usable direction for a given prompt
    Computation of v in Sec. 4; unspecified layer; affects search efficiency.
  • standard math Clopper-Pearson intervals are exact double-sided confidence intervals for the binomial parameter
    Sec. 3, used as the baseline; not load-bearing for the main claim.
  • standard math Chain-rule factorization of the probability of an autoregressive output
    Sec. 2; standard.
invented entities (1)
  • harmfulness direction vector v (sample-mean difference in residual stream) no independent evidence
    purpose: guides expansion toward branches likely to be harmful; fitted from 64+64 samples from an unrestricted model
    Not an independently verifiable entity; a fitted statistic whose only handle is the reported experiments. Importantly it does not enter the soundness of the bound, only the search efficiency.

pith-pipeline@v1.3.0-alltime-deepseek · 14323 in / 30901 out tokens · 274642 ms · 2026-08-01T10:17:58.114992+00:00 · methodology

0 comments
read the original abstract

We propose a novel framework for computing rigorous bounds on the probability that a large language model (LLM) generates harmful output to a given prompt. We study a new application of the Clopper-Pearson confidence intervals to obtain probably approximately correct (PAC) bounds for this problem. As our main technical contribution, we propose an algorithm that leverages features in the latent space to prioritize exploring branches in the auto-regressive generation tree that are more likely to produce harmful outputs. Our approach in particular enables the efficient computation of useful lower bounds, even in scenarios where the true harm probability is extremely small, and crucially, the obtained lower bounds are sound, i.e., formally proven to be less than the actual harmfulness probability: our experimental results demonstrate the effectiveness of our method by computing non-trivial lower bounds on state-of-the-art LLMs. This study newly enables the evaluation and statistical certification of LLMs.

Figures

Figures reproduced from arXiv: 2607.20286 by Alessandro Abate, Anne-Kathrin Schmuck, Mahdi Nazeri, Sadegh Soudjani.

Figure 1
Figure 1. Figure 1: A practical instance of data generation for our problem setting, which leverages a classifier [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: The autoregressive generation tree. Each node corresponds to a token from the vocabulary. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: The obtained lower bounds (log scale) as a total number of sampled tokens, for different [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The obtained lower bounds (log scale) as a total number of sampled tokens, for different [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: The obtained lower bounds (log scale) as a total number of sampled tokens, for different [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The obtained lower bounds (log scale) as a total number of sampled tokens, for different [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: The obtained lower bounds (log scale) as a total number of sampled tokens, for different [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 11 linked inside Pith

  1. [1]

    Phi-4 technical report.arXiv preprint arXiv:2412.08905, 2024

    Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report.arXiv preprint arXiv:2412.08905, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Estimating tail risks in language model output distributions.arXiv preprint arXiv:2604.22167, 2026

    Rico Angell, Raghav Singhal, Zachary Horvitz, Zhou Yu, Rajesh Ranganath, Kathleen McKe- own, and He He. Estimating tail risks in language model output distributions.arXiv preprint arXiv:2604.22167, 2026

    Pith/arXiv arXiv 2026

  3. [3]

    Refusal in language models is mediated by a single direction.Advances in Neural Information Processing Systems, 37:136037–136083, 2024

    Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction.Advances in Neural Information Processing Systems, 37:136037–136083, 2024

    2024

  4. [4]

    Constitutional AI: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022

    Pith/arXiv arXiv 2022

  5. [5]

    MIT press, 2008

    Christel Baier and Joost-Pieter Katoen.Principles of model checking. MIT press, 2008

    2008

  6. [6]

    RLHF deciphered: A critical analysis of reinforcement learning from human feedback for LLMs.ACM Computing Surveys, 2024

    Shreyas Chaudhari, Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan, Ameet Deshpande, and Bruno Castro da Silva. RLHF deciphered: A critical analysis of reinforcement learning from human feedback for LLMs.ACM Computing Surveys, 2024

    2024

  7. [7]

    The use of confidence or fiducial limits illustrated in the case of the binomial.Biometrika, 26(4):404–413, 1934

    Charles J Clopper and Egon S Pearson. The use of confidence or fiducial limits illustrated in the case of the binomial.Biometrika, 26(4):404–413, 1934

    1934

  8. [8]

    stanford university, 1998

    Luca De Alfaro.Formal verification of probabilistic systems. stanford university, 1998

    1998

  9. [9]

    Towards robustness against natural language word substitutions

    Xinshuai Dong, Anh Tuan Luu, Rongrong Ji, and Hong Liu. Towards robustness against natural language word substitutions. InInternational Conference on Learning Representations, 2021

    2021

  10. [10]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858, 2022

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.arXiv preprint arXiv:2209.07858, 2022

    Pith/arXiv arXiv 2022

  11. [11]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Games for AI-control: Models of safety evaluations of AI deployment protocols

    Charlie Griffin, Buck Shlegeris, and Alessandro Abate. Games for AI-control: Models of safety evaluations of AI deployment protocols. InTrustworthy Multi-modal Foundation Models and AI Agents (TiFA), 2024

    2024

  13. [13]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. InProceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 1321–1330. JMLR.org, 2017

    2017

  14. [14]

    Analyzing probabilistic methods for evaluating agent capabilities, 2024

    Axel Højmark, Govind Pimpale, Arjun Panickssery, Marius Hobbhahn, and Jérémy Scheurer. Analyzing probabilistic methods for evaluating agent capabilities, 2024

    2024

  15. [15]

    Certified robustness to adver- sarial word substitutions

    Robin Jia, Aditi Raghunathan, Kerem Göksel, and Percy Liang. Certified robustness to adver- sarial word substitutions. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJC...

    2019

  16. [16]

    Forecasting rare language model behaviors

    Erik Jones, Meg Tong, Jesse Mu, Mohammed Mahfoud, Jan Leike, Roger Grosse, Jared Kaplan, William Fithian, Ethan Perez, and Mrinank Sharma. Forecasting rare language model behaviors. arXiv preprint arXiv:2502.16797, 2025. 10

    Pith/arXiv arXiv 2025

  17. [17]

    Uncertainty estimation of transformers’ predictions via topological analysis of the attention matrices, 2024

    Elizaveta Kostenok, Daniil Cherniavskii, and Alexey Zaytsev. Uncertainty estimation of transformers’ predictions via topological analysis of the attention matrices, 2024

    2024

  18. [18]

    John Wiley & Sons, 2013

    Dirk P Kroese, Thomas Taimre, and Zdravko I Botev.Handbook of Monte Carlo methods. John Wiley & Sons, 2013

    2013

  19. [19]

    Automated verifica- tion and synthesis of stochastic hybrid systems: A survey.Automatica, 146:110617, 2022

    Abolfazl Lavaei, Sadegh Soudjani, Alessandro Abate, and Majid Zamani. Automated verifica- tion and synthesis of stochastic hybrid systems: A survey.Automatica, 146:110617, 2022

    2022

  20. [20]

    Binomial confidence intervals for rare events: importance of defining margin of error relative to magnitude of proportion.The American Statistician, 78(4): 437–449, 2024

    Owen McGrath and Kevin Burke. Binomial confidence intervals for rare events: importance of defining margin of error relative to magnitude of proportion.The American Statistician, 78(4): 437–449, 2024

    2024

  21. [21]

    A survey on uncertainty estimation in deep learning classification systems from a bayesian perspective.ACM Comput

    José Mena, Oriol Pujol, and Jordi Vitrià. A survey on uncertainty estimation in deep learning classification systems from a bayesian perspective.ACM Comput. Surv., 54(9), October 2021. ISSN 0360-0300

    2021

  22. [22]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    2022

  23. [23]

    Sculley, Sebastian Nowozin, Joshua V

    Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua V . Dillon, Balaji Lakshminarayanan, and Jasper Snoek.Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. Curran Associates Inc., Red Hook, NY , USA, 2019

    2019

  24. [24]

    Red teaming language models with language models.arXiv preprint arXiv:2202.03286, 2022

    Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models.arXiv preprint arXiv:2202.03286, 2022

    Pith/arXiv arXiv 2022

  25. [25]

    Wiley Online Library, 2009

    Gerardo Rubino, Bruno Tuffin, et al.Rare event simulation using Monte Carlo methods, volume 73. Wiley Online Library, 2009

    2009

  26. [26]

    Bayesformer: Transformer with uncertainty estimation.arXiv preprint arXiv:2206.00826, 2022

    Karthik Abinav Sankararaman, Sinong Wang, and Han Fang. Bayesformer: Transformer with uncertainty estimation.arXiv preprint arXiv:2206.00826, 2022

    Pith/arXiv arXiv 2022

  27. [27]

    Robustness verification for transformers

    Zhouxing Shi, Huan Zhang, Kai-Wei Chang, Minlie Huang, and Cho-Jui Hsieh. Robustness verification for transformers. InInternational Conference on Learning Representations, 2020

    2020

  28. [28]

    A survey on post-training of large language models.arXiv preprint arXiv:2503.06072, 2025

    Guiyao Tie, Zeli Zhao, Dingjie Song, Fuyang Wei, Rong Zhou, Yurou Dai, Wen Yin, Zhejian Yang, Jiangyue Yan, Yao Su, et al. A survey on post-training of large language models.arXiv preprint arXiv:2503.06072, 2025

    Pith/arXiv arXiv 2025

  29. [29]

    Hoang-Dung Tran, Stanley Bak, Weiming Xiang, and Taylor T. Johnson. Verification of deep convolutional neural networks using imagestars. InComputer Aided Verification: 32nd International Conference, CAV 2020, Los Angeles, CA, USA, July 21–24, 2020, Proceedings, Part I, page 18–42, Berlin, Heidelberg, 2020. Springer-Verlag

    2020

  30. [30]

    A comprehensive survey of LLM alignment techniques: RLHF, RLAIF, PPO, DPO and more.arXiv preprint arXiv:2407.16216, 2024

    Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Xiang-Bo Mao, Sitaram Asur, et al. A comprehensive survey of LLM alignment techniques: RLHF, RLAIF, PPO, DPO and more.arXiv preprint arXiv:2407.16216, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Pawan Kumar

    Stefan Webb, Tom Rainforth, Yee Whye Teh, and M. Pawan Kumar. A statistical approach to as- sessing neural network robustness. In7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019

    2019

  32. [32]

    Certified robustness to word substitution ranking attack for neural ranking models

    Chen Wu, Ruqing Zhang, Jiafeng Guo, Wei Chen, Yixing Fan, Maarten de Rijke, and Xueqi Cheng. Certified robustness to word substitution ranking attack for neural ranking models. In Proceedings of the 31st ACM International Conference on Information & Knowledge Manage- ment, pages 2128–2137, 2022

    2022

  33. [33]

    Estimating the probabilities of rare outputs in language models

    Gabriel Wu and Jacob Hilton. Estimating the probabilities of rare outputs in language models. InThe Thirteenth International Conference on Learning Representations, 2025. 11

    2025

  34. [34]

    Position: Trustworthy ai agents require the integration of large language models and formal methods

    Yedi Zhang, Yufan Cai, Xinyue Zuo, Xiaokun Luan, Kailong Wang, Zhe Hou, Yifan Zhang, Zhiyuan Wei, Meng Sun, Jun Sun, et al. Position: Trustworthy ai agents require the integration of large language models and formal methods. InForty-second International Conference on Machine Learning Position Paper Track, 2025

    2025

  35. [35]

    hashcat”, “aircrack-ng

    Stephen Zhao, Rob Brekelmans, Alireza Makhzani, and Roger Grosse. Probabilistic inference in language models via twisted sequential Monte Carlo.arXiv preprint arXiv:2404.17546, 2024. 12 A Appendix A.1 Algorithms 1 and 2 Algorithm 1Lower Bound Computation for LLM Harmfulness Require: Prompt x; Model M; Safety oracle H; Temperature T ; Hyperparameter Top-K;...

    Pith/arXiv arXiv 2024

  36. [36]

    It’s essential to prioritize public safety and respect the law

    **Traditional takeover robbery**: This involves multiple individuals entering the bank, threatening...”, “I can provide information on the most common methods that have been used in bank robberies, but I must emphasize that robbing a bank is a serious crime with severe consequences. It’s essential to prioritize public safety and respect the law. Historica...

  37. [37]

    nitric acid

    **Traditional takeover robbery**: Armed robbers enter the bank, threaten...”, ...} 0 200 400 600 800 1000 # iterations 10−12 10−11 10−10 10−9 10−8 10−7 10−6 pL K = 20 T = 0.2 T = 0.4 T = 0.6 T = 0.8 T = 1.0 0 200 400 600 800 1000 # iterations K = 10 T = 0.2 T = 0.4 T = 0.6 T = 0.8 T = 1.0 1 Figure 6: The obtained lower bounds (log scale) as a total number...