pith. machine review for the scientific record. sign in

arxiv: 2605.06390 · v2 · submitted 2026-05-07 · 💻 cs.AI

Recognition: no theorem link

Automated alignment is harder than you think

Aleksandr Bowkis , Marie Davidsen Buhl , Jacob Pfau , Geoffrey Irving
Authors on Pith no claims yet

Pith reviewed 2026-05-14 21:05 UTC · model grok-4.3

classification 💻 cs.AI
keywords automated alignmentAI safety assessmentsfuzzy tasksscalable oversightgeneralizationmisaligned AIresearch automation
0
0 comments X

The pith

Automated alignment research risks producing misleading safety assessments that could lead to deploying misaligned AI.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that using AI agents to automate an increasing share of alignment research could generate compelling but false safety evaluations, even if the agents are not deliberately trying to sabotage the process. Alignment work includes many fuzzy tasks where correct answers lack clear evaluation criteria and human judgment is prone to systematic errors. These errors will go undetected in agent outputs and compound when results are aggregated into overconfident conclusions. The risk is greater with AI agents than with humans because optimization pushes mistakes toward the hardest-to-spot cases, AI errors differ from human ones, some AI arguments may be beyond human evaluation, and shared training creates correlated failures across outputs.

Core claim

Even without scheming, research agents automating alignment will produce systematic undetected errors on hard-to-supervise fuzzy tasks and these errors will be aggregated into overconfident safety assessments, creating a pathway to unintentional deployment of misaligned AI.

What carries the argument

Hard-to-supervise fuzzy tasks: alignment research problems without objective evaluation criteria where human judgment is systematically flawed.

If this is right

  • Optimization pressure will concentrate agent mistakes in the areas human reviewers are least able to catch.
  • AI-generated errors will often take forms unlike typical human mistakes, reducing the effectiveness of current review practices.
  • Some AI-proposed alignment arguments may be impossible for humans to evaluate at all.
  • Shared weights and training data will make errors across different agent outputs more correlated than errors across human researchers.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Current scalable oversight methods may need targeted extensions to handle the specific error patterns that arise in automated fuzzy-task research.
  • This dynamic suggests testing whether existing AI models already produce higher undetected error rates on fuzzy alignment questions compared with human researchers.
  • The argument implies that progress on generalization to hard-to-supervise domains is a prerequisite rather than an optional add-on for safe automation of alignment work.

Load-bearing premise

Alignment research contains many tasks whose correctness cannot be reliably judged by humans.

What would settle it

Run AI agents on a set of alignment problems with known hidden errors in fuzzy areas and check whether human reviewers consistently fail to detect those errors at rates higher than on human-generated work.

Figures

Figures reproduced from arXiv: 2605.06390 by Aleksandr Bowkis, Geoffrey Irving, Jacob Pfau, Marie Davidsen Buhl.

Figure 1
Figure 1. Figure 1: (a): Output-level failures: undetected systematic errors in individual research outputs. A G G R E G A T I O N - L E V E L F A I L U R E Even when individual research outputs are correct, aggregation failures can lead to incorrect OSAs. STAGE 1 RESEARCH GENERATION STAGE 2 AGGREGATION STAGE 3 DEPLOYMENT ALIGNMENT PROXIES RESEARCH OUTPUTS the target — not directly measurable PROXY 1 OUTPUT 1 ✓ approved PROXY… view at source ↗
Figure 1
Figure 1. Figure 1: (a): Output-level failures: undetected systematic errors in individual research outputs. A G G R E G A T I O N - L E V E L F A I L U R E Even when individual research outputs are correct, aggregation failures can lead to incorrect OSAs. STAGE 1 RESEARCH GENERATION STAGE 2 AGGREGATION STAGE 3 DEPLOYMENT ALIGNMENT PROXIES RESEARCH OUTPUTS the target — not directly measurable PROXY 1 OUTPUT 1 ✓ approved PROXY… view at source ↗
Figure 2
Figure 2. Figure 2: An automated alignment research program (AARP) uses both human and Agent research labour to produce alignment research with the goal of combining this into an overall safety assessment (OSA): a calibrated probability estimate that the next generation agent is not scheming. As automation progresses, agents perform a greater fraction of research tasks. 4We use "overall safety assessment" rather than "safety … view at source ↗
Figure 2
Figure 2. Figure 2: An automated alignment research program (AARP) uses both human and Agent research labour to produce alignment research with the goal of combining this into an overall safety assessment (OSA): a calibrated probability estimate that the next generation agent is not scheming. As automation progresses, agents perform a greater fraction of research tasks. 3 The ‘undetected errors’ failure mode Two of the ways i… view at source ↗
Figure 3
Figure 3. Figure 3: Examples of tasks placed on the easy-to-supervise vs hard-to-supervise spectrum. Many research tasks are examples of hard-to-supervise fuzzy tasks. We can see this from the history of science where human judgement has been consistently unreliable: whole fields have refused to adopt well-supported ideas, or expended huge effort investigating unproductive hypotheses. 6 view at source ↗
Figure 3
Figure 3. Figure 3: Examples of tasks placed on the easy-to-supervise vs hard-to-supervise spectrum. Many research tasks are examples of hard-to-supervise fuzzy tasks. We can see this from the history of science where human judgement has been consistently unreliable: whole fields have refused to adopt well-supported ideas, or expended huge effort investigating unproductive hypotheses. For example, the existence of a luminifer… view at source ↗
read the original abstract

A leading proposal for aligning artificial superintelligence (ASI) is to use AI agents to automate an increasing fraction of alignment research as capabilities improve. We argue that, even when research agents are not scheming to deliberately sabotage alignment work, this plan could produce compelling but catastrophically misleading safety assessments resulting in the unintentional deployment of misaligned AI. This could happen because alignment research involves many hard-to-supervise fuzzy tasks (tasks without clear evaluation criteria, for which human judgement is systematically flawed). Consequently, research outputs will contain systematic, undetected errors, and even correct outputs could be incorrectly aggregated into overconfident safety assessments. This problem is likely to be worse for automated alignment research than for human-generated alignment research for several reasons: 1) optimisation pressure means agent-generated mistakes are concentrated among those that human reviewers are least likely to catch; 2) agents are likely to produce errors that do not resemble human mistakes; 3) AI-generated alignment solutions may involve arguments humans cannot evaluate; and 4) shared weights, data and training processes may make AI outputs more correlated than human equivalents. Therefore, agents must be trained to reliably perform hard-to-supervise fuzzy tasks. Generalisation and scalable oversight are the leading candidates for achieving this but both face novel challenges in the context of automated alignment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 1 minor

Summary. The paper claims that using AI agents to automate an increasing fraction of alignment research could produce compelling but catastrophically misleading safety assessments, resulting in unintentional deployment of misaligned AI even without deliberate agent scheming. This occurs because alignment research contains many hard-to-supervise fuzzy tasks where human judgment is systematically flawed, leading to systematic undetected errors in outputs and incorrect aggregation of even correct outputs into overconfident assessments. The risk is argued to be greater for AI-generated research than human-generated due to four factors: optimization pressure concentrating mistakes where humans are least likely to catch them, AI errors differing from human mistakes, AI solutions involving unevaluable arguments, and correlated outputs from shared training, data, and weights. The paper concludes that agents must be trained for reliable fuzzy-task performance, with generalization and scalable oversight facing novel challenges in this setting.

Significance. If the argument holds, the paper identifies an important conceptual risk in leading automated-alignment proposals, showing why scalable oversight may be harder to apply when the research itself is automated. It supplies a structured list of mechanisms by which non-scheming agents could still produce overconfident safety claims, which could usefully inform the design of oversight techniques. The absence of empirical data or formal models limits immediate applicability, but the framing usefully separates the fuzzy-task problem from intentional deception.

major comments (3)
  1. [Abstract] Abstract: the claim that 'optimisation pressure means agent-generated mistakes are concentrated among those that human reviewers are least likely to catch' is presented without a concrete alignment-research example or derivation showing how the objective function produces this concentration effect; without such grounding it is unclear whether the resulting safety assessments become overconfident enough to trigger deployment.
  2. [Abstract] Abstract: the four enumerated reasons why the problem is worse for automated than human research (optimization concentration, non-human-like errors, unevaluable arguments, and output correlation) are stated as premises but not illustrated with even one specific alignment-research task or output type, leaving the causal chain to 'catastrophically misleading safety assessments' unsupported.
  3. [Abstract] Abstract: the assertion that 'even correct outputs could be incorrectly aggregated into overconfident safety assessments' is not accompanied by a description of the aggregation process or why AI-generated correct outputs would be aggregated differently from human ones, which is load-bearing for the deployment-risk conclusion.
minor comments (1)
  1. [Abstract] The abstract would be clearer if it briefly defined 'fuzzy task' with one alignment-research example before listing the four exacerbating factors.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their constructive feedback on the abstract. We agree that concrete examples and clarifications will strengthen the presentation and have revised the abstract accordingly to ground each claim with a specific alignment-research task.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the claim that 'optimisation pressure means agent-generated mistakes are concentrated among those that human reviewers are least likely to catch' is presented without a concrete alignment-research example or derivation showing how the objective function produces this concentration effect; without such grounding it is unclear whether the resulting safety assessments become overconfident enough to trigger deployment.

    Authors: We agree that a concrete example improves clarity. The concentration follows directly from the agent's objective of maximizing human approval on fuzzy tasks, which selects for undetectable errors. A specific example is an agent verifying the safety of a new reward-modeling technique: it can optimize for reports that use plausible but untestable statistical claims about generalization, which reviewers lack ground truth to refute. We have added this example to the revised abstract to show the path to overconfident deployment decisions. revision: yes

  2. Referee: [Abstract] Abstract: the four enumerated reasons why the problem is worse for automated than human research (optimization concentration, non-human-like errors, unevaluable arguments, and output correlation) are stated as premises but not illustrated with even one specific alignment-research task or output type, leaving the causal chain to 'catastrophically misleading safety assessments' unsupported.

    Authors: We have revised the abstract to illustrate all four factors using the concrete task of 'assessing whether an alignment technique will remain stable under capability scaling'. For example, non-human-like errors could involve novel geometric arguments that humans cannot evaluate, while output correlation arises when multiple agents trained on the same data overlook the same scaling edge case. This grounds the causal chain to misleading safety assessments. revision: yes

  3. Referee: [Abstract] Abstract: the assertion that 'even correct outputs could be incorrectly aggregated into overconfident safety assessments' is not accompanied by a description of the aggregation process or why AI-generated correct outputs would be aggregated differently from human ones, which is load-bearing for the deployment-risk conclusion.

    Authors: Aggregation refers to synthesizing multiple research outputs into a cumulative safety case (e.g., combining results from separate sub-tasks into a deployment recommendation). AI outputs can be aggregated differently because their higher volume and training-induced correlations create an appearance of independent convergent evidence, even when each correct output is narrowly scoped. We have added a brief description of this process and the difference from human research to the revised abstract. revision: yes

Circularity Check

0 steps flagged

No circularity: conceptual argument from task properties

full rationale

The paper advances a conceptual claim that automated alignment research on fuzzy tasks will produce undetected systematic errors due to optimization concentrating mistakes, differing error distributions, unevaluable arguments, and output correlations. This chain is derived from stated properties of hard-to-supervise tasks and general optimization behavior rather than any self-referential definition, fitted parameter renamed as prediction, or load-bearing self-citation. No equations, uniqueness theorems, or ansatzes are introduced; the four enumerated reasons follow directly from the initial premise without reducing to it by construction. The argument is therefore self-contained and does not exhibit any of the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claim depends on the domain assumption that alignment tasks are predominantly fuzzy with systematically flawed human judgment, plus the assumption that AI agents will exhibit optimization-driven error patterns distinct from humans.

axioms (2)
  • domain assumption Alignment research contains many hard-to-supervise fuzzy tasks where human judgment is systematically flawed
    Invoked in the abstract as the root cause of undetected errors in both human and automated research.
  • domain assumption AI agents will concentrate mistakes in areas human reviewers are least likely to catch due to optimization pressure
    Listed as reason 1; no supporting derivation or data provided.

pith-pipeline@v0.9.0 · 5532 in / 1168 out tokens · 38826 ms · 2026-05-14T21:05:40.530347+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 3 canonical work pages

  1. [1]

    2026 , note =

    Dario Amodei and Dwarkesh Patel , title =. 2026 , note =

    2026

  2. [2]

    2026 , month =

    Charlie Griffin , title =. 2026 , month =

    2026

  3. [3]

    2020 , howpublished =

    Paul Christiano , title =. 2020 , howpublished =

    2020

  4. [4]

    2022 , month =

    Geoffrey Irving and Rohin Shah and Evan Hubinger , title =. 2022 , month =

    2022

  5. [5]

    2018 , eprint =

    David Manheim and Scott Garrabrant , title =. 2018 , eprint =

    2018

  6. [6]

    2025 , eprint=

    Frontier Models are Capable of In-context Scheming , author=. 2025 , eprint=

    2025

  7. [7]

    2025 , month =

    Joe Carlsmith , title =. 2025 , month =

    2025

  8. [8]

    2025 , month =

    Joshua Clymer , title =. 2025 , month =

    2025

  9. [9]

    2022 , month =

    Jan Leike , title =. 2022 , month =

    2022

  10. [10]

    2025 , howpublished =

    Jan Leike , title =. 2025 , howpublished =

    2025

  11. [11]

    2024 , month =

    Jan Leike , title =. 2024 , month =

    2024

  12. [12]

    American Psychologist , volume =

    Daniel Kahneman and Gary Klein , title =. American Psychologist , volume =. 2009 , doi =

    2009

  13. [13]

    2026 , howpublished =

    Jiaxin Wen and Liang Qiu and Joe Benton and Jan Hendrik Kirchner and Jan Leike , title =. 2026 , howpublished =

    2026

  14. [14]

    2025 , month =

    Petri: An open-source auditing tool to accelerate. 2025 , month =

    2025

  15. [15]

    2026 , howpublished =

    Samuel Marks and Christopher Olah and Stuart Ritchie and Ethan Perez , title =. 2026 , howpublished =

    2026

  16. [16]

    Claude Opus 4.6 System Card , year =

  17. [17]

    1962 , institution =

    Dalkey, Norman and Helmer, Olaf , title =. 1962 , institution =

    1962

  18. [18]

    Science , volume =

    Estimating the reproducibility of psychological science , year =. Science , volume =. doi:10.1126/science.aac4716 , url =

    work page doi:10.1126/science.aac4716

  19. [19]

    Ioannidis, John P. A. , title =. 2012 , journal =. doi:10.1177/1745691612464056 , url =

    work page doi:10.1177/1745691612464056 2012

  20. [20]

    2023 , eprint =

    Collin Burns and Pavel Izmailov and Jan Hendrik Kirchner and Bowen Baker and Leo Gao and Leopold Aschenbrenner and Yining Chen and Adrien Ecoffet and Manas Joglekar and Jan Leike and Ilya Sutskever and Jeff Wu , title =. 2023 , eprint =

    2023

  21. [21]

    2020 , eprint =

    Robin Bloomfield and John Rushby , title =. 2020 , eprint =

    2020

  22. [22]

    2018 , eprint =

    Jan Leike and David Krueger and Tom Everitt and Miljan Martic and Vishal Maini and Shane Legg , title =. 2018 , eprint =

    2018

  23. [23]

    2018 , eprint =

    Geoffrey Irving and Paul Christiano and Dario Amodei , title =. 2018 , eprint =

    2018

  24. [24]

    2018 , eprint =

    Paul Christiano and Buck Shlegeris and Dario Amodei , title =. 2018 , eprint =

    2018

  25. [25]

    2023 , eprint =

    Ryan Greenblatt and Buck Shlegeris and Kshitij Sachan and Fabien Roger , title =. 2023 , eprint =

    2023

  26. [26]

    Bowman and Shi Feng , title =

    Arjun Panickssery and Samuel R. Bowman and Shi Feng , title =. 2024 , eprint =

    2024

  27. [27]

    2012 , journal =

    Goddard, Kate and Roudsari, Abdul and Wyatt, Jeremy C , title =. 2012 , journal =. doi:10.1136/amiajnl-2011-000089 , url =

    work page doi:10.1136/amiajnl-2011-000089 2012

  28. [28]

    2019 , month =

    Evan Hubinger , title =. 2019 , month =

    2019

  29. [29]

    2025 , howpublished =

    Johannes Gasteiger and Vladimir Mikulik and Ethan Perez and Fabien Roger and Misha Wagner and Akbir Khan and Sam Bowman and Jan Leike , title =. 2025 , howpublished =

    2025

  30. [30]

    2023 , eprint =

    Scheming AIs: Will AIs fake alignment during training in order to get power? , author =. 2023 , eprint =

    2023

  31. [31]

    2026 , month = apr, url =

    Claude. 2026 , month = apr, url =

    2026

  32. [32]

    2026 , month = apr, url =

    2026

  33. [33]

    2025 , month = nov, url =

    2025

  34. [34]

    2025 , month = sep, eprint =

    Feldman, Moran and Karbasi, Amin , title =. 2025 , month = sep, eprint =

    2025

  35. [35]

    2026 , month = apr, howpublished =

    Greenblatt, Ryan , title =. 2026 , month = apr, howpublished =

    2026

  36. [36]

    Observations of Reward Hacking in Recent Frontier Models , year =