AIs and Humans with Agency
Pith reviewed 2026-05-08 18:03 UTC · model grok-4.3
The pith
AI agency cannot be added to current LLMs and instead requires new architectures built jointly with human actors in each real-world setting.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
This paper compares agency in humans with potential agency in AI programs. Human agency takes many years to develop, as the frontal lobe is activated. Early attempts to endow LLMs agency have met serious obstacles. Progress requires a new architecture where actions and plans are formulated jointly with the human actors in each real world setting.
What carries the argument
Joint human-AI architecture in which actions and plans are co-formulated by the AI and the human participants present in each real-world setting.
If this is right
- Continued work on independent LLM agents will encounter persistent barriers that cannot be removed by scaling or prompting alone.
- Effective AI systems will need to be built from the outset to share planning and decision steps with humans who are physically or operationally present.
- Agency will emerge only inside specific, situated collaborations rather than as a general property of the model.
- Development timelines for capable AI will be tied to the time required to design and test these joint human-AI workflows.
Where Pith is reading between the lines
- This joint approach could automatically embed human values and constraints into AI decisions without separate alignment layers.
- It suggests that fully autonomous AI may be both technically harder and practically less useful than collaborative systems in most domains.
- Prototype testing would involve measuring task success rates when humans are required to participate in planning versus when they are excluded.
- The same joint-formulation idea may apply to other areas such as robotics or decision-support tools where human oversight is already routine.
Load-bearing premise
The obstacles encountered so far in giving agency to LLMs cannot be solved by further refinements of existing architectures and instead require an entirely new joint human-AI formulation.
What would settle it
A demonstration that an improved standalone LLM architecture can reliably generate and execute complex, multi-step plans in varied real-world environments without human co-planning at the action level.
read the original abstract
This paper compares agency in humans with potential agency in AI programs. Human agency takes many years to develop, as the frontal lobe is activated. Early attempts to endow LLMs agency have met serious obstacles. Progress requires a new architecture where actions and plans are formulated jointly with the human actors in each real world setting.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper compares agency in humans, which develops over many years via frontal lobe activation, with potential agency in AI. It asserts that early attempts to endow LLMs with agency have encountered serious obstacles and concludes that progress requires a new architecture in which actions and plans are formulated jointly with human actors in each real-world setting.
Significance. If substantiated, the claim would suggest that incremental improvements to existing LLM architectures are insufficient for agency and that integrated human-AI joint planning is a necessary paradigm shift. However, the manuscript supplies no empirical data, specific failure examples, comparative analysis, or formal arguments, so its significance is confined to raising a high-level conceptual question without advancing technical understanding or testable predictions.
major comments (3)
- Abstract: The assertion that 'early attempts to endow LLMs agency have met serious obstacles' is presented without any cited examples, specific failure modes, references to prior work, or analysis demonstrating why existing approaches (such as scaffolding, RLHF, or extended context) cannot address them through incremental refinement.
- Abstract / Full Text: The recommendation for a 'new architecture where actions and plans are formulated jointly with the human actors' is stated at a purely conceptual level with no description of its components, differences from current human-in-the-loop systems, or evidence that joint formulation would resolve the unspecified obstacles.
- Full Text: No data, derivations, examples, or arguments are supplied to support the central claim that a fundamentally new joint human-AI formulation is required rather than refinements within current architectures, leaving the recommendation unsubstantiated.
Simulated Author's Rebuttal
We appreciate the referee's detailed feedback on our manuscript. The paper aims to draw conceptual parallels between the development of human agency and the challenges faced by AI systems. While we recognize the manuscript's conceptual nature and lack of empirical support, we believe it raises important questions for the field. Below, we respond to each major comment and indicate planned revisions.
read point-by-point responses
-
Referee: Abstract: The assertion that 'early attempts to endow LLMs agency have met serious obstacles' is presented without any cited examples, specific failure modes, references to prior work, or analysis demonstrating why existing approaches (such as scaffolding, RLHF, or extended context) cannot address them through incremental refinement.
Authors: We acknowledge this point and agree that the manuscript would benefit from more specificity. In the revised version, we will add references to literature on LLM limitations in agentic tasks, including issues with maintaining coherent long-term plans and adapting to dynamic environments. We will also briefly discuss why approaches like RLHF and scaffolding have not fully resolved these, based on the analogy to the extended developmental process in humans. This will provide a stronger foundation for the claim. revision: yes
-
Referee: Abstract / Full Text: The recommendation for a 'new architecture where actions and plans are formulated jointly with the human actors' is stated at a purely conceptual level with no description of its components, differences from current human-in-the-loop systems, or evidence that joint formulation would resolve the unspecified obstacles.
Authors: The proposal is intentionally high-level to stimulate discussion on paradigm shifts. We will revise the manuscript to elaborate on the components, such as real-time collaborative planning interfaces and shared context models that evolve through interaction. This differs from traditional human-in-the-loop by emphasizing joint formulation from the outset rather than oversight or correction. While we cannot provide empirical evidence in this conceptual paper, we will strengthen the argument by linking it more explicitly to human developmental psychology. revision: partial
-
Referee: Full Text: No data, derivations, examples, or arguments are supplied to support the central claim that a fundamentally new joint human-AI formulation is required rather than refinements within current architectures, leaving the recommendation unsubstantiated.
Authors: We agree that the original manuscript is brief and lacks detailed arguments or examples. As a short position paper, its goal is to pose the question rather than fully substantiate it. However, we will expand the full text to include more arguments based on the human-AI comparison, such as the necessity of prolonged interactive learning. We maintain that incremental refinements are unlikely to suffice due to fundamental differences in how agency emerges, but we will make this reasoning more explicit. revision: yes
Circularity Check
No significant circularity; purely conceptual essay
full rationale
The paper presents a high-level conceptual comparison between human and AI agency, asserting that early LLM attempts have met obstacles requiring a new joint human-AI architecture. No equations, derivations, fitted parameters, self-citations, or technical reductions appear in the provided text or abstract. The central claim is an opinion on architectural needs rather than a formal argument that reduces to its own inputs by construction. As a self-contained philosophical essay without load-bearing technical steps, it exhibits no circularity under the enumerated patterns.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Human agency develops over many years because the frontal lobe is activated.
Reference graph
Works this paper leans on
-
[1]
AIs and Humans with Agency David Mumford May 5, 2026 1 Introduction We are heading towards a major transformation of AIs. To date, the large lan- guage models (LLMs) have answered questions, written essays and solved well described problems. What they have not done is to make decisions affecting the real world. I would argue that even the phrase “real wor...
-
[2]
The great thing about this is that it doesn’t in- volve any philosophical issues about consciousness or the feeling of “being alive”. This behavior is incontestably ob- jective even though it is de- scribed as having a theory of mind. Note the similar- ity of cooperative play with the current goal for integrat- ing LLMs into a business en- vironment. The ...
work page 2024
-
[3]
They put the AI in charge of the office vending machine, telling it both to make enough money to sustain its operation and to answer employee’s requests as far as possible. It could order any item that an employee asked for given the price. So one wise guy (”light hearted” is how management described this typical engineer’s prank) asked the vending machin...
work page 2015
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.