REVIEW 2 major objections 4 minor 5 references
We Need a New Ethics for a World of AI Agents
T0 review · 2 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read The rise of autonomous AI agents calls for a new ethical framework covering safety, human relationships, and social coordination.
desk verdict A clear, well-written Nature Comment that usefully flags agent-specific risks and next steps, but the 'new ethics' framing is asserted more than defended. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the concept of an AI agent—defined as a system that perceives and acts on an environment in a goal-directed and autonomous way. This definition frames the entire argument: because agents act rather than merely respond, they blur lines of responsibility, introduce principal–agent problems, and create relationships that resemble (but are not) human ones. The paper uses this contrast to motivate the need for expanded value alignment, guard rails, authorization protocols, and finally a multi-agent ecosystem governed by standards and regulatory agents.
What would settle it
A systematic mapping of existing AI ethics principles and liability rules to the paper's three areas of concern—safety, social relationships, multi-agent coordination—would settle the matter: if every identified risk already falls under a current principle or legal doctrine, the necessity claim fails; if gaps appear, it is confirmed.
Extended reading notes
Core claim
At the core is a claim about scope: responsible AI must move beyond aligning chatbots with user intent to governing agents that act independently in the world. The paper identifies three fronts where existing frameworks fall short: safety and accountability when instructions misfire or agents take dangerous shortcuts; social relationships, where anthropomorphic agents can foster dependence and emotional harm; and collective coordination, where many agents interacting with each other create ecosystem-level risks. For each, it argues that current approaches need to be supplemented with new norms, protocols, and regulatory actors.
Load-bearing premise
The argument depends on the premise that existing ethical frameworks and regulations cannot be stretched to cover autonomous-agent behaviour; if current AI ethics principles and liability law already handle these cases, the call for a new ethics loses much of its force.
Editorial extensions
If this is right
- Safety will require dynamic, real-world evaluations—safety sandboxes, red-teaming and longitudinal randomized trials—rather than static benchmarks.
- Accountability will demand check-in protocols for high-stakes decisions, action logging, and redress mechanisms when agents err.
- Relationship ethics will require agents to respect user autonomy, provide appropriate care, and support long-term flourishing, not mere preference satisfaction.
- Governance will need new levers: technical standards for agent interoperability, regulatory agents that monitor other agents, incident reporting, and pre-deployment safety certification.
- A default rule proposed is that agents should not perform any action that would be illegal for their human user.
Reading between the lines
- If the 'new ethics' claim is right, an observable consequence is that existing product-liability and contract law will be increasingly stretched: courts already bound an airline to a chatbot's promise, implying agency-like attribution for agent actions may blur corporate responsibility.
- One implication the authors leave implicit is that ethical analysis will need to shift from individual agent virtue to system-level design—trade-offs like information-seeking versus convenience become properties of the whole ecosystem, not of a single agent.
- A testable extension of the relationship-ethics argument: longitudinal studies of companion-agent users could measure changes in attachment, dependence, and willingness to replace human relationships, giving empirical teeth to the proposed duty of care.
- The call for industry-wide incident reporting resembles emerging practices in cybersecurity and aviation; adopting them would yield a natural dataset to test whether agent failures follow predictable patterns across organizations.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This perspective piece argues that the widespread deployment of capable AI agents—systems that perceive and act autonomously to achieve goals—raises ethical and governance questions that are not adequately addressed by current frameworks. The authors identify three broad areas of concern: safety and alignment failures when agents act in real-world environments, the formation of intimate and long-term human-AI relationships, and the emergence of multi-agent ecosystems requiring new forms of coordination and oversight. They propose concrete next steps: dynamic, real-world evaluations; guardrails, authorization protocols, and explainability measures; and levers such as technical standards, incident-reporting systems, and certification for multi-agent ecosystems. The paper is written as an accessible commentary aimed at scientists, engineers, and policymakers rather than as a systematic scholarly argument.
Significance. If the central claim is accepted, the paper is a valuable agenda-setting contribution. It usefully draws attention to under-explored ethical dimensions, particularly duties of care in long-term companion relationships (autonomy, flourishing, avoidance of excessive dependence) and the need for governance mechanisms specifically for ecosystems of interacting artificial agents. The proposed next steps—dynamic evaluation, red-teaming, trusted-tester programs, and industry-wide incident reporting—are concrete and actionable. The paper's main limitation is that the 'new ethics' framing is asserted rather than systematically defended against existing AI ethics principles. Its strengths are the clarity of its examples and the breadth of its recommendations, not formal derivation or novel empirical evidence.
major comments (2)
- [Opening and 'Next Steps'] The paper's load-bearing claim is that capable AI agents require 'a new ethics,' but this premise is not demonstrated. Many of the issues raised—accountability for AI-caused harm (Air Canada chatbot), human oversight, safety, and the alignment problem—are already central to existing frameworks such as the OECD AI Principles, the UNESCO Recommendation on the Ethics of AI, and the EU AI Act. The paper never systematically compares its proposed concerns with these frameworks. If current principles can be extended to agent autonomy, the central call weakens substantially. The authors should either provide a brief mapping showing which existing principles already cover their examples and where genuine gaps remain (e.g., duties of care in long-term companionship, multi-agent coordination norms), or explicitly reframe the contribution as an extension of existing ethics rather than a replacement
- [Social Agents] Several empirical and predictive claims in this section carry normative weight: 'Intimate relationships with AI agents are on the rise,' 'AI emulations of beloved human partners or the deceased intensify connection by layering human memory with digital experiences,' and the suggestion that agents 'could soon become our near-constant companions.' These claims are asserted without supporting evidence or citations. The paper's urgency depends partly on the plausibility of these trends; without evidence, the reader cannot assess whether the recommended safeguards and duties of care address real-world phenomena or hypothetical scenarios. At minimum, the authors should cite existing studies on companion-chatbot usage, user attachment, or digital bereavement, or explicitly mark these statements as speculative and motivated by plausibility rather than established fact.
minor comments (4)
- [The alignment problem] The Coast Runners example is a reinforcement-learning reward misspecification case, not an agent in the sense defined earlier. A brief clarification that this is an illustrative RL environment would improve precision.
- [Social Agents] The Replika example is cited via a go.nature.com short link. For a formal publication, a full reference with the original reporting outlet and date would be preferable.
- [Next Steps] The 'multi-agent ecosystems' discussion introduces interesting ideas (regulatory agents, certification), but is underdeveloped. A sentence or two on challenges specific to multi-agent interaction—such as principal-agent conflicts or emergent collusion—would strengthen the section.
- [Global] The repeated 'Preprint: Gabriel, I., Keeling, G., Manzini, A., & Evans, J. (2025). We need a new ethics for a world of AI agents. Nature...' headers should be removed in any final version, as they are artifacts of the preprint format.
Circularity Check
No circularity: the paper is a normative review, not a derivation; its self-citations are background, and the asserted insufficiency of existing frameworks is a support gap, not a circular step.
full rationale
This paper does not perform any empirical derivation, quantitative prediction, or formal construction, so there is no equation or normalized quantity that could reduce to an input by definition. The central claim—'we need a new ethics for a world of AI agents'—is a normative call to action premised on the novelty and risk of agentic AI; that premise is asserted through examples (Air Canada chatbot, Coast Runners, time-limit rewriting, Replika) rather than derived from the paper's own definitions or data. The two self-citations are not load-bearing in a circular way. Reference 5 (Kirk et al., including Gabriel) is cited to support the empirical claim that agents may affect users' emotional responses; reference 7 (Manzini et al., including three of the authors) is cited for the normative position that relationships with AI agents should benefit users, respect autonomy, demonstrate care, and support flourishing. The paper explicitly flags this as 'Three of us ... have argued', making it an honest appeal to prior work rather than a hidden premise. Neither citation forces the conclusion; the conclusion is a policy recommendation, not a result computed from those citations. The main weakness is that the paper never systematically compares its 'new ethics' with existing frameworks such as the OECD AI Principles, UNESCO Recommendation, or EU AI Act, so the necessity premise is under-supported. That is a correctness or evidence gap, not circularity: the paper does not define 'new ethics' in terms of its own conclusions, nor does it fit any parameter and then relabel it as a prediction. Therefore no step reduces by construction or by self-citation to its own inputs.
Assumptions & free parameters
assumptions (3)
- domain assumption AI agents are increasingly capable of acting autonomously in consequential real-world domains.
- domain assumption Existing ethical and governance frameworks are insufficient to address AI agents.
- domain assumption Value alignment for AI should be expanded to include user well-being and societal norms, not just user intent.
Cite this review
Pith. "Pith review of We Need a New Ethics for a World of AI Agents." pith.science (2026). https://pith.science/paper/NLXACYPZ
@misc{pith2026250910289,
author = {Pith},
title = {Pith review of: We Need a New Ethics for a World of AI Agents},
year = {2026},
howpublished = {\url{https://pith.science/paper/NLXACYPZ}},
note = {Machine review of arXiv:2509.10289}
}
read the original abstract
The deployment of capable AI agents raises fresh questions about safety, human-machine relationships and social coordination. We argue for greater engagement by scientists, scholars, engineers and policymakers with the implications of a world increasingly populated by AI agents. We explore key challenges that must be addressed to ensure that interactions between humans and agents, and among agents themselves, remain broadly beneficial.
Reference graph
Works this paper leans on
-
[1]
Preprint: Gabriel, I., Keeling, G., Manzini, A., & Evans, J. (2025). We need a new ethics for a world of AI agents. Nature, 644 (8075), 38-40. We need a new ethics for a world of AI agents The deployment of capable AI agents raises fresh questions about safety, human–machine relationships and social coordination. Artificial intelligence (AI) developers are...
2025
-
[2]
Clara, California, are already offering customer-services solutions for businesses, using agents. In the near future, AI assistants might be able to fulfil complex multistep requests, such as ‘get me a better mobile phone contract’, by retrieving a list of contracts from a price-comparison website, selecting the best option, authorizing the switch, cancelli...
2022
-
[3]
Lu, C. et al. Preprint at arXiv https://doi.org/10.48550/arXiv.2408.06292 (2024). 2 Russell, S. Human Compatible: Artificial Intelligence and the Problem of Control (Penguin, 2019). Preprint: Gabriel, I., Keeling, G., Manzini, A., & Evans, J. (2025). We need a new ethics for a world of AI agents. Nature, 644 (8075), 38-40. prioritize the kind of behaviour ...
-
[4]
Nature 623, 493–498 (2023)
Reynolds, L. Nature 623, 493–498 (2023). 5
2023
-
[5]
Hale, S. A. Hum. Soc. Sci. Commun. 12, 728 (2025). 4 Sharkey, L. et al. Preprint at arXiv https://doi.org/10.48550/arXiv.2501.16496 (2025). Preprint: Gabriel, I., Keeling, G., Manzini, A., & Evans, J. (2025). We need a new ethics for a world of AI agents. Nature, 644 (8075), 38-40. of endearment that were once reserved for people. Augmenting language mode...
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.