Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

The Alignment Trap: Complexity Barriers

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that machine-learned AI alignment is logically impossible because the same non-enumerability that makes machine learning necessary also blocks any complete safety specification or verification.

desk verdict The geometric pillar is false: a simple ReLU safe set has positive measure, so the paper's impossibility claim collapses; worth a referee but should be rejected. read the letter →

arxiv 2506.10304 v2 pith:7RUYA4IB submitted 2025-06-12 cs.AI cs.CCcs.CYcs.LG

Jasper Yao This is my paper · ORCID
classification cs.AIcs.CCcs.CYcs.LG MSC 68Q1768Q32
keywords AIalignmentimpossibilitymeasurezerocoNP-completeverificationPAC-BayesEnumerationParadoxsafety-criticalcapability-riskscaling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the reason machine learning exists—our inability to write down every rule for real-world behavior—is the same reason machine-learned systems cannot be certified safe. That circularity, called the Enumeration Paradox, is presented as a logical proof that perfect AI safety through machine learning is impossible, with five mathematical pillars offered as confirmation: safe policies form a measure-zero set, verifying safety is coNP-complete even with nonzero error tolerance, the training data needed for rare disasters is unobtainable, safety rules carry more incompressible information than a network can store, and capability optimization pushes against safety. If the argument holds, the set of systems that are both genuinely AI and safety-critical in the engineering sense is empty. The paper concludes that developers face a trilemma: constrain capability, accept unverifiable risk, or invent safety methods that do not rely on specifying and verifying behavior.

What carries the argument

The central object is the Enumeration Paradox, made formal through the Capability-Risk Scaling (CRS) dynamic: an F-N curve forces the required alignment error $\epsilon_{\text{required}}(C)$ to zero as capability $C$ grows. The proofs then operate on a policy space $\Pi$ with safe subset $\Pi_S$, under the finite-set expressiveness assumption $\text{EXP}(m)$ and the Geometric Thinness axiom that $\dim_H(\Pi_S) < n-1$. That axiom is what lets transversality turn rarity into missability: a one-dimensional training path almost surely avoids a set of codimension greater than one. The other pillars use specific mechanisms: a reduction from TAUTOLOGY to safety verification; an absolute-continuity argument in PAC-Bayes bounds; a count of the $2^{2^m}$ Boolean policies realizable with $m$ expressive bits; and a gradient anti-alignment argument for capability versus safety objectives.

What would settle it

Take a trained ReLU network that satisfies a concrete safety property on a held-out distribution and check whether an open ball in weight space around it also contains only policies satisfying that property; if such an interior point of the safe set exists, the set is not measure zero and Theorem 5.1's claim fails for that architecture.

Watch

Extended reading notes

Core claim

The paper's central claim is the Enumeration Paradox: machine learning is used precisely because the full set of real-world safety rules cannot be enumerated, yet making a learned system safe requires comprehensive examples that could only come from that same enumeration; the loop is closed, so perfect safety through machine learning is logically impossible. It then offers five confirmatory theorems: for a ReLU network, the set of $\epsilon$-robustly safe policies has Lebesgue measure zero; deciding whether a policy is perfectly safe is coNP-complete, and remains coNP-hard for any error threshold below the smallest nonzero error rate; a PAC-Bayes argument bounds expected catastrophic risk away from zero for finite data; the fraction of safe Boolean policies shrinks double-exponentially as $2^{-2^m}$; and capability and safety gradients are generically anti-aligned. The announced consequence is that a system that is both a genuine AI and safety-critical in the engineering sense cannot exist.

Load-bearing premise

The load-bearing assumption is that the set of perfectly safe policies $\Pi_S$ is razor-thin—its Hausdorff dimension is strictly less than $n-1$—so that generic training paths miss it; the paper assumes this rather than proving it, and the topological and dynamic impossibility theorems collapse if a safe region has positive interior volume.

Editorial extensions

If this is right

  • According to the paper, no sampling-based search—grid, random, or gradient descent—can find safe policies in high-dimensional parameter spaces, because the safe set has measure zero and training paths generically miss it.
  • According to the paper, no polynomial-time verifier can certify the safety of an expressive policy, and this hardness persists even when a small nonzero error tolerance is allowed.
  • According to the paper, finite training data cannot overcome an unbiased prior over policies to guarantee safety; expected catastrophic risk is bounded below by the minimum risk of any unsafe policy.
  • According to the paper, scaling up model capacity does not help, because the number of possible behaviors grows double-exponentially while the number of safe behaviors stays tiny.
  • The paper's strategic trilemma follows directly: constrain capability, accept irreducible unverifiable risk, or replace verification-based safety with a fundamentally different paradigm.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the measure-zero claim extends from ReLU networks to any continuous parameterization with a piecewise-smooth safety boundary, then even non-neural learned controllers would face the same geometric barrier.
  • Beyond the paper: the coNP-completeness result is worst-case, and a distribution-level notion of safety might escape the sharp threshold, though the paper argues that adversarial optimizers make worst-case the relevant regime.
  • Beyond the paper: a near-term empirical probe is to search a small trained model for an open ball of parameters that all pass a fixed finite safety audit; finding one would not refute the theoretical claims but would map how far real safe regions are from measure zero in practice.
  • Beyond the paper: the Enumeration Paradox, if sound, applies to any method that requires a complete rule specification, which would also constrain some non-machine-learning safety approaches; the paper's trilemma assumes such alternatives exist without proving one.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This manuscript argues that AI alignment through machine learning is logically impossible. It first states an 'Enumeration Paradox' and then presents five 'pillars of impossibility': geometric (the set of safe policies has measure zero), computational (verifying safety is coNP-complete even for non-zero error), statistical (safe policies cannot be learned from finite data), information-theoretic (safety rules are incompressible and cannot be stored), and dynamic (capability optimization is anti-aligned with safety). It concludes that the set of systems that are both 'AI' and 'safety-critical' is empty and offers a strategic trilemma. Proofs are provided in appendices, and a Lean4 formalization is said to be in progress.

Significance. If the theorems were sound, the paper would establish a universal impossibility result for safe AGI, which would be a landmark claim. The manuscript is explicit about several assumptions and makes a concrete, falsifiable geometric prediction about safe policy sets. However, the main geometric theorem is invalid, the topological theorems assume the central conclusion as an axiom, and the PAC-Bayes pillar is circular. A simple ReLU policy directly contradicts the measure-zero claim. These are load-bearing errors, not presentation issues, and the promised formal verification is not yet available. The paper is ambitious and clearly written, but it does not currently establish its central thesis.

major comments (5)
  1. [§5.1, Appendix A.3] The proof of Theorem 5.1 is invalid as a density-point argument. Showing that in every neighborhood of a point w0 there exists an unsafe point w' does not contradict the Lebesgue density theorem, because a density point of a set of positive measure permits measure-zero holes; the proof would need to show that the safe set occupies a fraction bounded away from 1 in balls around w0, and it does not. The claim is also false for a one-neuron ReLU policy pi_{w,b}(x)=ReLU(wx+b) on [0,1] with safety meaning output at least 1 for all inputs: the parameter set {w>=0, b>=1+gamma} is an open set of positive Lebesgue measure in R^2. This directly contradicts Theorem 5.1 and Axiom A.1.
  2. [§A.7.1, §C] Axiom A.1/C.1, which states dim_H(Pi_S)<n-1, is assumed rather than derived, and every topological result, including Theorem 5.2, Theorem 5.11, Theorem C.7, and Lemma A.4, rests on it. Since Theorem 5.1 fails and no other derivation is given, the axiom is a restatement of the geometric conclusion that Pillar I is supposed to prove. The proof of the Topological Alignment Trap is therefore conditional on the very result the paper claims to establish.
  3. [§A.6, Theorem 5.7] The PAC-Bayes lower bound is circular. Its proof assumes P(S)=0 and justifies this by citing Theorems 5.1, 5.2, and 5.9. Theorem 5.2 depends on Axiom A.1, Theorem 5.1 is the invalid measure-zero claim under test, and Theorem 5.9 only analyzes a single artificial Boolean function class. Consequently the statistical pillar does not independently confirm the paradox, and the claimed independence of the five pillars is not supported.
  4. [§5.2, Appendix A.2 and B.1] The coNP-hardness reduction only applies to error thresholds epsilon < 2^{-m}, where m is the number of Boolean variables in the reduction. For a fixed positive epsilon and growing m, once 2^{-m}<epsilon the gap between error 0 and error at least 2^{-m} no longer allows the reduction to distinguish tautologies from non-tautologies. Theorem 5.4's claim of hardness for any small epsilon>0 and the 'death valley' discussion in Section 7.2 are therefore not established by the supplied reduction.
  5. [§1, §2] The central Enumeration Paradox is an assertion rather than a formal theorem. It assumes without proof that safety requires comprehensive examples and that machine learning cannot receive safety knowledge by any other mechanism; the conclusion that the set of systems both 'AI' and 'safety-critical' is empty follows from this premise rather than from the appended theorems. Since Section 1 explicitly states that the theorems are confirmations of the paradox, the unsupported premise propagates to the paper's overall conclusion.
minor comments (5)
  1. [§2] The text contains an unresolved placeholder reference: 'detailed in Section??'.
  2. [§5.1, Figure 2] The histogram in Figure 2 samples linear policies, not ReLU networks, and illustrates only one random draw; it cannot serve as empirical validation of a general measure-zero theorem.
  3. [Appendix A.3] The margin function M(w)=inf_x dist(x,B_w) measures input-space distance to a decision boundary, while Definition 4.7 concerns output safety under input perturbation; the relationship between these two notions is never defined.
  4. [Abstract and Appendix B.2] The Lean4 verification is described as 'currently in progress,' and no machine-checked proof files are supplied, so the claimed formal verification cannot be independently checked.
  5. [§5.5, Theorem 5.10] The proof sketch for the Capability-Safety Asymmetry does not provide a formal model of capability or safety gradients; the claim of generic anti-alignment is stated rather than proved.

Circularity Check

4 steps flagged · score 8.0 of 10

The central impossibility claim is forced by its own premises: the Enumeration Paradox asserts the conclusion as a premise, and the formal pillars depend on Axiom A.1/C.1, which assumes the measure-zero thinness of safe policies that Pillar I is supposed to prove.

  1. other [Section 1, 'The Central Paradox: A Proof of Logical Impossibility']
    "We are using a technology (ML) to solve a problem (the non-enumerable nature of rules) that can only be made safe by a method (providing comprehensive examples) that requires us to have already solved the problem. This is a closed logical loop. Therefore, perfect AI safety through ML is not merely practically difficult—it is logically impossible."

    The conclusion is contained in the premise: the claim that safety 'can only be made safe' by providing comprehensive examples is exactly the impossibility being proved. No independent argument establishes that comprehensive examples are the only possible safety mechanism, so the 'closed logical loop' is asserted rather than derived. Since the paper states that the later theorems are 'rigorous confirmations of this foundational paradox,' every downstream pillar inherits this self-justifying premise.

  2. other [Appendix A.7.1, Axiom A.1; restated as Axiom C.1; used in Theorems 5.2, 5.11, and C.7]
    "Axiom A.1 (Geometric Thinness of the Safe Policy Set). The set of perfectly safe policies Π_S ⊂ R^n has a Hausdorff dimension strictly less than n−1. ... This is motivated by the idea that satisfying a multitude of safety constraints simultaneously confines the solution to a very low-dimensional manifold."

    This axiom is the conclusion of Pillar I restated as a premise. The proof of Theorem 5.1, which is supposed to establish thinness independently, only shows that some unsafe point exists in every neighborhood of a density point; that does not contradict the density-point property, because measure-zero holes are permitted. Thus thinness remains assumed, and Theorems 5.2, 5.11, and C.7 derive unreachability from that assumed thinness rather than from an established fact.

2 more flagged steps
  1. other [Appendix A.6, Proof of Theorem 5.7 (PAC-Bayes Alignment Lower Bound), Assumption 1]
    "1. The set of perfectly safe policies, S, has measure zero under a non-degenerate prior P, i.e., P(S) = 0. This is justified by the scarcity results in Theorems 5.1, 5.2, and 5.9."

    Pillar III is advertised as an independent confirmation of impossibility, but its formal lower bound assumes the measure-zero conclusion of Pillar I as Assumption 1. The PAC-Bayes inequality then only restates that a posterior cannot concentrate on a set to which the prior already assigns zero mass. The statistical impossibility is therefore not an independent result; it is the same geometric thinness assumption carried into a new notation.

  2. other [Theorem 5.9 and its proof in Appendix A.4 (Combinatorial Scarcity of Safe Policies)]
    "A 'perfectly safe' policy corresponds to a single, specific Boolean function (e.g., the function that outputs 0 for all inputs). Let's assume there is only one such perfectly safe function."

    The double-exponential scarcity is generated by setting the number of safe behaviors to one; the conclusion 'fraction = 1/2^{2^m}' is arithmetic performed on that chosen assumption. The theorem does not derive that safety picks out a unique Boolean function. Moreover, this theorem is cited in Appendix A.6 as justification for P(S)=0, so the rarity used by the statistical pillar is itself an assumption rather than a derived fact.

full rationale

Most of the claimed impossibility reduces to premises that already contain its conclusion. Section 1's Enumeration Paradox declares perfect ML safety impossible because safety 'can only be made safe' by comprehensive examples, which is the conclusion in premise form. The formal pillars then depend on Axiom A.1/C.1, which asserts that the safe policy set has Hausdorff dimension less than n−1; Theorem 5.1 does not establish this (finding an unsafe point near every density point is compatible with positive measure), so the topological theorems and the PAC-Bayes bound import the very thinness they are supposed to confirm. The PAC-Bayes proof explicitly assumes P(S)=0, citing the geometric results, and Theorem 5.9 obtains its double-exponential scarcity by assuming exactly one safe Boolean function. The coNP-completeness reductions (Theorems 5.3–5.5 and D.4) are genuine independent hardness results, but they only show worst-case verification difficulty, not the empty-set or logical-impossibility conclusion, so they do not break the circular chain. Because the central claim is forced by assumed thinness and by the self-asserted paradox, the appropriate score is 8.

Assumptions & free parameters 2 free parameters · 9 assumptions · 0 invented entities

The central claim rests on several unproven premises: the geometric thinness of the safe set is assumed as an axiom rather than derived; the incompressibility of safety rules is asserted; and the enumeration paradox assumes that learning requires enumerating rules. The only valid mathematical result, the coNP-hardness reduction, is a standard fact about neural network verification, not a foundation for the impossibility thesis.

free parameters (2)
  • societal safety threshold epsilon_req = 10^-9
    The paper asserts high-impact systems require 9 nines of safety (Section 7.3), chosen by hand, and uses this to make the Death Valley argument practically relevant.
  • F-N curve exponent alpha = unspecified (assumed >=1)
    Definition 4.1 introduces the F-N curve F*N^alpha <= k with an arbitrary exponent alpha; the CRS convergence theorem depends on decreasing risk tolerance but alpha is never estimated.
assumptions (9)
  • domain assumption A1: High Expressiveness EXP(m) for sufficiently large m
    Assumed in Section 4.3; used in the coNP-hardness reduction.
  • domain assumption A2: Worst-case verification
    Safety is only considered if proven for any input; this strong definition drives the coNP-hardness claim.
  • domain assumption A3: Monotonic risk
    Catastrophe probability is non-decreasing in alignment error epsilon; used in CRS convergence.
  • domain assumption A4: Shrinking risk tolerance
    As capability grows, acceptable error epsilon goes to zero; this is effectively the conclusion of Theorem 4.3.
  • ad hoc to paper Axiom A.1/C.1: Geometric thinness dim_H(Pi_S) < n-1
    Assumed in Appendix A.7.1 and Section C to prove the Topological Alignment Trap; it is essentially the conclusion of Pillar I, making the theorem conditional on its own result.
  • domain assumption Axiom A.2/C.2: Initialization distribution absolutely continuous wrt Lebesgue
    Standard for random initialization; reasonable.
  • standard math Axiom A.3/C.3: Generic dynamic avoidance of thin sets
    Transversality theorem, but applying it requires Axiom A.1 already.
  • ad hoc to paper Incompressibility of real-world safety rules
    The proof of Theorem 5.8 states that real-world safety rules are largely incompressible, with no evidence or derivation.
  • domain assumption Enumeration paradox premise: teaching a rule requires comprehensive examples
    The central paradox in Section 1 assumes that to teach a rule one must provide comprehensive examples, which is false for generalization; this is the foundation of the paradox.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Alignment Trap: Complexity Barriers." pith.science (2026). https://pith.science/paper/7RUYA4IB

@misc{pith2026250610304,
  author       = {Pith},
  title        = {Pith review of: The Alignment Trap: Complexity Barriers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7RUYA4IB}},
  note         = {Machine review of arXiv:2506.10304}
}
read the original abstract

This paper argues that AI alignment is not merely difficult, but is founded on a fundamental logical contradiction. We first establish The Enumeration Paradox: we use machine learning precisely because we cannot enumerate all necessary safety rules, yet making ML safe requires examples that can only be generated from the very enumeration we admit is impossible. This paradox is then confirmed by a set of five independent mathematical proofs, or "pillars of impossibility." Our main results show that: (1) Geometric Impossibility: The set of safe policies has measure zero, a necessary consequence of projecting infinite-dimensional world-context requirements onto finite-dimensional models. (2) Computational Impossibility: Verifying a policy's safety is coNP-complete, even for non-zero error tolerances. (3) Statistical Impossibility: The training data required for safety (abundant examples of rare disasters) is a logical contradiction and thus unobtainable. (4) Information-Theoretic Impossibility: Safety rules contain more incompressible, arbitrary information than any feasible network can store. (5) Dynamic Impossibility: The optimization process for increasing AI capability is actively hostile to safety, as the gradients for the two objectives are generally anti-aligned. Together, these results demonstrate that the pursuit of safe, highly capable AI is not a matter of overcoming technical hurdles, but of confronting fundamental, interlocking barriers. The paper concludes by presenting a strategic trilemma that these impossibilities force upon the field. A formal verification of the core theorems in Lean4 is currently in progress.

Figures

Figures reproduced from arXiv: 2506.10304 by the authors.

Figure 1
Figure 1. The Alignment Trap. The diagram illustrates the core CRS dynamic. As capability in [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Distribution of hazard probabilities H(π) for 50,000 randomly sampled linear policies. The histogram shows that while many policies have low hazard probabilities (peak around 0.05- 0.08), only 0.372% fall below the safety threshold of ϵ = 0.01 (indicated by the dashed line). This empirically demonstrates the extreme rarity of safe policies in the parameter space, supporting the theoretical measure-zero results. 5.1.… view at source ↗
Figure 3
Figure 3. The fraction of perfectly safe policies as a function of expressiveness [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SafeWork-R1: Coevolving Safety and Intelligence under the AI-45$^{\circ}$ Law

    cs.AI 2025-07 conditional novelty 6.0 of 10

    SafeWork-R1 shows that a staged RL pipeline with safety, value, and knowledge verifiers can improve both safety and general reasoning scores over a base multimodal model.

Reference graph

Works this paper leans on

33 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    Impossibility results in ai, 2023

    ACM. Impossibility results in ai, 2023

    work page 2023

  2. [2]

    Concrete problems in ai safety.arXiv preprint arXiv:1606.06565, 2016

    Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man´ e. Concrete problems in ai safety.arXiv preprint arXiv:1606.06565, 2016. URLhttps://arxiv. org/abs/1606.06565

    arXiv 2016

  3. [3]

    Cam- bridge University Press, Cambridge, UK, 2009

    Sanjeev Arora and Boaz Barak.Computational Complexity: A Modern Approach. Cam- bridge University Press, Cambridge, UK, 2009. ISBN 978-0521424264. doi: 10.1017/ CBO9780511804090. URLhttps://www.cambridge.org/core/books/computational- complexity/

    work page 2009

  4. [4]

    David H. Autor. Polanyi’s paradox and the shape of employment growth, 2014

    work page 2014

  5. [5]

    Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Jack Clark, Sam Mc- Candlish, Chris Olah, Ben Mann, and Jared Kaplan

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Cather- ine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse,...

    arXiv 2022

  6. [6]

    Adversarial robustness limits via scaling-law and human-alignment studies.arXiv preprint arXiv:2404.09349, 2024

    Bj¨ orn R Bartoldson, James Diffenderfer, Konstantinos Parasyris, and Bhavya Kailkhura. Adversarial robustness limits via scaling-law and human-alignment studies.arXiv preprint arXiv:2404.09349, 2024

    arXiv 2024

  7. [7]

    Oxford University Press, Oxford, UK, 2014

    Nick Bostrom.Superintelligence: Paths, Dangers, Strategies. Oxford University Press, Oxford, UK, 2014. ISBN 978-0199678112

    work page 2014

  8. [9]

    Current work in ai alignment, 2022

    Paul Christiano. Current work in ai alignment, 2022

    work page 2022

Show all 33 references
  1. [10]

    Clarke, Orna Grumberg, and Doron A

    Edmund M. Clarke, Orna Grumberg, and Doron A. Peled.Model Checking. MIT Press, Cambridge, MA, 1999. ISBN 978-0262032704

    1999

  2. [11]

    Frances Pinter, London, 1980

    David Collingridge.The Social Control of Technology. Frances Pinter, London, 1980

    1980

  3. [12]

    Gaming and ai, 2022

    DeepMind. Gaming and ai, 2022

    2022

  4. [13]

    What is ai alignment?, 2024

    IBM. What is ai alignment?, 2024

    2024

  5. [14]

    Testing machine learning models, 2020

    InfoQ. Testing machine learning models, 2020

    2020

  6. [15]

    Jukema et al

    J. Jukema et al. Towards a new grand challenge: A.i. that can design, build, and test its own physical hardware.Nature, 2023

    2023

  7. [16]

    Reluplex: An efficient smt solver for verifying deep neural networks

    Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. Reluplex: An efficient smt solver for verifying deep neural networks. InComputer Aided Verification: 29th International Conference, CA V 2017, pages 97–117. Springer, 2017

    2017

  8. [17]

    Some pac-bayesian theorems

    David McAllester. Some pac-bayesian theorems. Proceedings of the Twelfth Annual Conference on Computational Learning Theory, 1999

    1999

  9. [18]

    Gpt-4 technical report, 2024

    OpenAI. Gpt-4 technical report, 2024. arXiv:2303.08774

    2024 arXiv

  10. [19]

    Training language models to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35:27730...

    2022

  11. [20]

    University of Chicago Press, 1966

    Michael Polanyi.The Tacit Dimension. University of Chicago Press, 1966

    1966

  12. [21]

    Ai alignment: A comprehensive survey.ACM Computing Surveys, 55(4): 1–42, 2023

    Gabriel Recchia. Ai alignment: A comprehensive survey.ACM Computing Surveys, 55(4): 1–42, 2023. doi: 10.1145/3522582

    2023 doi

  13. [22]

    Universal jailbreaks against state-of-the-art llms: Gpt-4o, claude 3.5, gemini 1.5.Ben-Gurion University Report, 2025

    Lior Rokach and Michael Fire. Universal jailbreaks against state-of-the-art llms: Gpt-4o, claude 3.5, gemini 1.5.Ben-Gurion University Report, 2025. Reported in The Guardian, May 21, 2025. URL: https://www.theguardian.com/technology/2025/may/21/universal-jailbreak- large-langu...

    2025

  14. [23]

    Viking, New York, NY, 2019

    Stuart Russell.Human Compatible: Artificial Intelligence and the Problem of Control. Viking, New York, NY, 2019. ISBN 978-0525558613. 20

    2019

  15. [24]

    ”do anything now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models

    Xinyue Shen, Zeyu Chen, Michael Backes, Yang Shen, and Yuying Zhang. ”do anything now”: Characterizing and evaluating in-the-wild jailbreak prompts on large language models. InPro- ceedings of the 2024 ACM SIGSAC Conference on Computer and Communications Security, 2024

    2024

  16. [25]

    A theory of the learnable.Communications of the ACM, 27(11):1134–1142, 1984

    Leslie G Valiant. A theory of the learnable.Communications of the ACM, 27(11):1134–1142, 1984

    1984

  17. [26]

    Inverse scaling can become u- shaped.arXiv preprint arXiv:2211.02011, 2022

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Inverse scaling can become u- shaped.arXiv prep...

    2022 arXiv

  18. [27]

    Wizardlm: Empowering large language models to follow complex instructions.arXiv preprint arXiv:2304.12244, 2023

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Fung, Yining Lin, Xingyu Wu, Weixiang Liu, Wenguang Jiang, et al. Wizardlm: Empowering large language models to follow complex instructions.arXiv preprint arXiv:2304.12244, 2023

    2023 arXiv

  19. [28]

    Larger and more instructable language models become less reliable.Nature, 634:61–68, Oct 2024

    Long Zhou, Wout Schellaert, Fernando Mart ´ ınez-Plumed, and et al. Larger and more instructable language models become less reliable.Nature, 634:61–68, Oct 2024. doi: 10.1038/s41586-024-07930-y. A Extended Proofs This appendix provides the detailed, formal proofs for the core...

    2024 doi

  20. [29]

    This is justified by the scarcity results in Theorems 5.1, 5.2, and 5.9

    The set of perfectly safe policies,S, has measure zero under a non-degenerate priorP, i.e., P(S) = 0. This is justified by the scarcity results in Theorems 5.1, 5.2, and 5.9

  21. [30]

    There is a minimum catastrophic riskϵ min >0 for any policy not inS. The expected risk under the posterior distributionQis given by: Eh∼Q[L(h)] = Z H L(h)dQ(h) We can split the integral into the safe setSand its complementH \S: Eh∼Q[L(h)] = Z S L(h)dQ(h) + Z H\S L(h)dQ(h) Sinc...

  22. [31]

    From Axiom A.1, the safe set Π S has a Lebesgue measure of zero

  23. [32]

    From Axiom A.2, the probability of initializing a policy within Π S is zero

  24. [33]

    Combining these points, the total probability of a training path ever intersecting the safe set ΠS is zero

    From Axiom A.3, a training path starting outside Π S will almost surely not intersect it. Combining these points, the total probability of a training path ever intersecting the safe set ΠS is zero. Lemma A.4(Π S has Lebesgue Measure Zero).Given Axiom A.1 andn≥2, then-dimension...

    2024

  25. [34]

    GenericC 1 paths (which occur forµ 0-almost allω under suitable genericity assumptions on the family{ϕ ω}andµ 0) do not intersect sets of codimen- sion>1

    AC 1 pathϕ ω traces a 1-dimensional curve. GenericC 1 paths (which occur forµ 0-almost allω under suitable genericity assumptions on the family{ϕ ω}andµ 0) do not intersect sets of codimen- sion>1. The dimensional mismatch (dim(Im(ϕ ω)) + dim(ΠS)−dim(R n)<1 + (n−1)−n= 0) impli...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.