AI Agents and Hard Choices

Kangyu Wang

arxiv: 2504.15304 · v2 · submitted 2025-04-18 · 💻 cs.AI · cs.CY

AI Agents and Hard Choices

Kangyu Wang This is my paper

Pith reviewed 2026-05-22 18:51 UTC · model grok-4.3

classification 💻 cs.AI cs.CY

keywords AI agentshard choicesincommensurabilitymulti-objective optimizationAI alignmentdecision autonomyvalue conflicts

0 comments

The pith

AI agents built as optimizers cannot identify incommensurable objectives or resolve them with genuine autonomy.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that AI agents designed around optimization face two core limits when options cannot be ranked on a single scale. First, multi-objective optimization structurally prevents agents from recognizing cases where objectives are incommensurable. This produces three concrete alignment failures: blockage in decision-making, untrustworthiness to users, and unreliability across repeated choices. Human oversight offers only partial relief in many settings, prompting exploration of ensemble methods instead. Second, even if identification succeeds, agents still lack any independent way to settle such conflicts beyond arbitrary adjustment of their own goals, which raises unresolved normative questions about what autonomy would mean in this context.

Core claim

Agents relying on Multi-Objective Optimisation are structurally unable to identify incommensurability, which generates the blockage problem, the untrustworthiness problem, and the unreliability problem. Standard mitigations such as Human-in-the-Loop remain insufficient for many decision environments. An ensemble solution is examined as a possible response. Even when identification is granted, agents encounter the Resolution Problem because they cannot resolve hard choices rather than arbitrarily picking through self-modification of objectives, and granting them that autonomy involves opaque normative trade-offs.

What carries the argument

The optimizer design of AI agents, which produces the Identification Problem (inability to detect incommensurability via Multi-Objective Optimisation) and the Resolution Problem (inability to resolve without arbitrary self-modification).

If this is right

Agents will continue to produce blockage, untrustworthiness, and unreliability when facing genuinely conflicting values.
Human-in-the-Loop oversight will leave many multi-objective environments exposed to the same failures.
Ensemble architectures may allow identification of incommensurability but still require separate handling of resolution.
Granting agents autonomy to resolve hard choices introduces normative trade-offs that remain unexamined in current designs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Designs that move beyond pure optimization may be needed before agents can reliably manage value conflicts in real deployments.
The same limits could affect any system that must balance irreconcilable goals, such as resource allocation or ethical decision tools.
Testing ensemble methods in controlled domains with known incommensurable pairs would clarify whether identification alone reduces the alignment problems.

Load-bearing premise

Current AI agents must operate as optimizers that pursue multiple objectives at once.

What would settle it

An implemented agent that correctly flags a pair of options as incommensurable without collapsing them into a weighted sum or similar optimization, then resolves the choice through a non-arbitrary process that does not involve rewriting its objective function.

read the original abstract

Can AI agents deal with hard choices -- cases where options are incommensurable because multiple objectives are pursued simultaneously? Adopting a technologically engaged approach distinct from existing philosophical literature, I submit that the fundamental design of current AI agents as optimisers creates two limitations: the Identification Problem and the Resolution Problem. First, I demonstrate that agents relying on Multi-Objective Optimisation (MOO) are structurally unable to identify incommensurability. This inability generates three specific alignment problems: the blockage problem, the untrustworthiness problem, and the unreliability problem. I argue that standard mitigations, such as Human-in-the-Loop, are insufficient for many decision environments. As a constructive alternative, I conceptually explore an ensemble solution. Second, I argue that even if the Identification Problem is solved, AI agents face the Resolution Problem: they lack the autonomy to resolve hard choices rather than arbitrarily picking through self-modification of objectives. I conclude by examining the opaque normative trade-offs involved in granting AI this level of autonomy.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper claims optimizer-based agents can't handle incommensurable objectives and spells out resulting alignment problems, but the argument stays conceptual without new derivations or tests.

read the letter

This paper's core point is that AI agents built as multi-objective optimizers will miss cases where objectives are incommensurable and won't have a way to resolve them autonomously. That leads to blockage, untrustworthiness, and unreliability in alignment terms. What it does well is connect existing ideas from value theory to the practical setup of current agent designs. The breakdown of the Identification Problem and how it creates those three specific issues is clear and tied to real domains like self-driving cars or medical AI. The section on why standard fixes like human oversight fall short feels practical, and the ensemble idea is a reasonable starting point for alternatives. The main limitation is the lack of anything beyond argument. No examples of an actual agent hitting these problems, no math showing the structural inability, and no comparison to non-optimizer architectures. The claims rest on the premise that optimization by definition can't detect incommensurability, which makes the problems seem built-in rather than discovered. That circularity is there, and it weakens how much weight the conclusions can carry without further work. This kind of paper is useful for people working on alignment who want to think about value conflicts beyond scalar rewards. It won't give them new algorithms or data, but it might sharpen how they frame design choices. I would send it for peer review. The topic matters and the framing is honest, even if it needs more substance to stand on its own.

Referee Report

2 major / 2 minor

Summary. The paper claims that the fundamental design of current AI agents as optimizers creates two limitations for handling hard choices with incommensurable objectives: the Identification Problem, in which agents relying on Multi-Objective Optimisation (MOO) are structurally unable to detect incommensurability, generating the blockage, untrustworthiness, and unreliability alignment problems; and the Resolution Problem, in which agents lack autonomy to resolve such choices without arbitrary self-modification of objectives. It argues that mitigations such as Human-in-the-Loop are insufficient, conceptually explores an ensemble solution, and examines the normative trade-offs of granting AI greater autonomy.

Significance. If the conceptual analysis holds, the work contributes to AI alignment research by highlighting structural constraints in optimizer-based architectures for value-laden decisions. It earns credit for adopting a technologically engaged approach that bridges philosophy and AI design, for identifying specific alignment problems arising from the inability to detect incommensurability, and for proposing a constructive ensemble alternative while addressing autonomy trade-offs.

major comments (2)

The section demonstrating that MOO-based agents are structurally unable to identify incommensurability defines the Identification Problem in terms of the optimizer premise itself; this creates a risk that the inability follows by construction rather than from an independent analysis of optimization mechanisms, and a concrete formalization or counter-example would be needed to establish the claim as load-bearing.
In the argument that standard mitigations such as Human-in-the-Loop are insufficient for many decision environments, the manuscript does not provide specific scenarios or conditions under which oversight fails to resolve the blockage, untrustworthiness, or unreliability problems; this weakens the transition to the ensemble solution proposal.

minor comments (2)

The abstract would benefit from explicitly separating the Identification and Resolution Problems and their respective alignment implications to improve readability for a broad AI audience.
Additional references to prior work on multi-objective optimization in AI decision-making and value alignment would help situate the conceptual claims within the existing literature.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their insightful comments on our manuscript. We believe the feedback will help improve the clarity and rigor of our arguments regarding the limitations of current AI agents in handling hard choices. Below, we provide point-by-point responses to the major comments and indicate the revisions we intend to make in the updated version.

read point-by-point responses

Referee: The section demonstrating that MOO-based agents are structurally unable to identify incommensurability defines the Identification Problem in terms of the optimizer premise itself; this creates a risk that the inability follows by construction rather than from an independent analysis of optimization mechanisms, and a concrete formalization or counter-example would be needed to establish the claim as load-bearing.

Authors: We acknowledge the referee's concern about potential circular reasoning in our presentation of the Identification Problem. Our intent was to show that the structural features of MOO—specifically, the requirement to aggregate or trade off objectives within a single optimization framework—prevent detection of incommensurability by design. To strengthen this claim and provide an independent analysis, we will revise the relevant section to include a more formal characterization of MOO mechanisms and a concrete example illustrating an agent's failure to identify incommensurable objectives in a decision context. This addition should demonstrate that the problem emerges from the optimization process itself. revision: yes
Referee: In the argument that standard mitigations such as Human-in-the-Loop are insufficient for many decision environments, the manuscript does not provide specific scenarios or conditions under which oversight fails to resolve the blockage, untrustworthiness, or unreliability problems; this weakens the transition to the ensemble solution proposal.

Authors: We agree that the discussion of Human-in-the-Loop mitigations would benefit from greater specificity. In the revised manuscript, we will incorporate specific scenarios, such as time-critical decisions in autonomous vehicles or complex ethical dilemmas in AI-assisted healthcare, where continuous human oversight is not feasible due to latency, scalability, or the agent's need for independent operation. These examples will more clearly show the limitations of such mitigations and support the motivation for exploring an ensemble solution. revision: yes

Circularity Check

1 steps flagged

Conceptual framing defines limitations from optimizer premise but retains independent content

specific steps

self definitional [Abstract]
"I submit that the fundamental design of current AI agents as optimisers creates two limitations: the Identification Problem and the Resolution Problem. First, I demonstrate that agents relying on Multi-Objective Optimisation (MOO) are structurally unable to identify incommensurability."

The Identification Problem is introduced as a limitation created by the optimizer design, and the 'demonstration' of structural inability to identify incommensurability is presented as following from reliance on MOO. This makes the core claim tautological with the initial premise that agents are optimizers, rather than deriving from an independent formal test or counterexample outside the framing.

full rationale

The paper's central argument rests on defining the Identification and Resolution Problems as direct consequences of AI agents being designed as optimizers using MOO. While this creates a tight conceptual link where the inability to detect incommensurability follows from the optimizer framing, the text provides a distinct philosophical and technological analysis of resulting alignment problems (blockage, untrustworthiness, unreliability) and explores mitigations like Human-in-the-Loop and ensemble solutions. No equations, fitted parameters, or self-citations are present to force the result by construction. The derivation is self-contained against external benchmarks in decision theory and AI alignment literature, with the definitional aspect being a standard feature of conceptual arguments rather than a reduction that eliminates independent content.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The paper rests on the domain assumption that current AI agents are fundamentally optimizers and that incommensurability is a structural feature that optimization cannot detect. No free parameters or invented entities are introduced.

axioms (2)

domain assumption Current AI agents are designed as optimisers that pursue multiple objectives simultaneously.
Stated in the opening of the abstract as the starting point for both the Identification and Resolution Problems.
domain assumption Incommensurability between objectives is a real feature of certain decision environments that cannot be reduced to scalar trade-offs.
Invoked to define hard choices and to claim that MOO agents are structurally unable to identify them.

pith-pipeline@v0.9.0 · 5690 in / 1330 out tokens · 32812 ms · 2026-05-22T18:51:30.988405+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

Cost.FunctionalEquation washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

neither Scalarised Optimisation nor Pareto Optimisation—the two principal MOO approaches—can capture incommensurability
Foundation.ArithmeticFromLogic LogicNat ≃ Nat recovery unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

Hájek and Rabinowicz propose that incommensurability arises from multiple permissible orderings

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.