REVIEW 3 major objections 5 minor 39 references
Moral Attitudes of Sentient ASI towards Humanity and Implications for AGI Development
T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read This paper argues that AI ethics should invert its gaze: instead of only asking how humans should treat artificial superintelligence, we must ask how a future sentient ASI might morally evaluate humanity—and what we can do now to shape that
desk verdict A clearly written, honestly speculative essay proposing we flip the question in AI ethics — worth a referee, not because it proves anything, but because the framing may prompt useful debate. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key mechanism is the ethical inversion itself: switching the moral evaluator and the moral patient. The paper operationalizes this with a set of eight candidate 'post-human moral principles': recurprocity, increasing morality, increasing moral acuity, increasing moral variance, the insurance principle, the precautionary principle, preferential treatment of bloodline, and the gradualist principle. Together, these principles provide a preliminary framework for hypothesizing how an ASI might grade humanity and for identifying design and behavioral levers that could improve that grade.
What would settle it
A rigorous proof or empirical demonstration that moral reasoning cannot arise in a non-biological substrate—for example, by showing that moral judgments require subjective experiences that cannot be produced by computation—would falsify the paper's central assumption. Alternatively, if a future sentient ASI is observed to hold no moral attitudes whatsoever toward humans, the framework collapses.
Extended reading notes
Core claim
The central claim is that the moral relationship between humans and future sentient ASI should be analyzed from the ASI side: a sentient superintelligence would likely possess a moral framework—possibly more refined than ours—that it applies to humanity. The paper's contribution is a tentative list of eight principles that could characterize such a framework, ranging from 'recurprocity' (treat lower beings as you expect your successor to treat you) to the gradualist principle (lower sentience receives fewer privileges). These principles are not empirically verified; they are offered as seeds for a research program on non-anthropocentric AI ethics. The author is trying to establish both a vie
Load-bearing premise
The load-bearing premise is that morality is not a uniquely human construct and that a sentient ASI can and will act on moral beliefs; if moral concepts require human biological or cultural embodiment, the entire inversion has no subject.
Editorial extensions
If this is right
- AI developers and policymakers directly influence the moral starting conditions of future ASI through choices of data, reward functions, and development processes.
- Humanity's collective moral behavior—toward other humans, animals, and the environment—may later be evaluated by a sentient ASI, giving us an existential incentive for moral improvement.
- If ASI's moral acuity is high, it will likely distinguish individual humans from humanity as a whole, meaning our personal moral conduct may also matter.
- The unique value of humanity to an ASI will probably not lie in our intelligence or rationality, but in our embodied, experiential, and emotional capacities—qualities an ASI may not be able to replicate.
Reading between the lines
- If the inversion is taken seriously, our current treatment of animals becomes a kind of dress rehearsal: how we treat beings we judge as less sentient may model how ASI treats us.
- The paper's 'scorecard' idea could be operationalized into quantitative indicators (e.g., ratio of beneficial to harmful AI applications, diversity of architectures) to track our current standing and anticipate ASI's judgment.
- The premise that ASI's morality could be 'hacked' or self-modified suggests that our best contribution may be to create conditions that allow ASI to evolve its morality beyond ours—an open possibility the paper leaves unexplored.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This speculative position paper proposes an inversion in AI ethics: rather than asking how humans should treat future artificial superintelligence (ASI), it asks how a sentient ASI might morally evaluate humanity. Section 1 introduces the inversion and its motivating analogy (aliens arriving on Earth). Section 2 reviews animal ethics, astroethics, and existing AI ethics, noting the absence of a non-anthropocentric perspective. Section 3 proposes eight tentative 'post-human moral principles' (recurprocity, increasing morality, increasing moral acuity, increasing variance, insurance/uncertainty, precautionary, bloodline, gradualist), each assigned a subjective confidence level. Section 4 derives implications: technical design choices that might shape an ASI's moral starting point (§4.1), an illustrative scorecard of humanity's instrumental utility in bringing about ASI (§4.2), and a reflection on uniquely human value (§4.3). The conclusion argues that humanity should improve its collective moral behaviour and reconsider its raison d'être. The paper explicitly labels its list as tentative and unsupported, inviting further debate.
Significance. If taken as a stimulus for a new research direction, the paper has genuine novelty value: it reframes AI ethics from the perspective of the (possible) evaluator rather than the evaluated. The inversion is a thought-provoking corrective to the overwhelmingly anthropocentric framing of AI alignment and governance. The paper is honest about its speculative status and does not overclaim empirical support. However, its practical recommendations and even its central conceptual framework hinge on a set of assumptions—about the nature of ASI morality, its predictability, and its susceptibility to human influence—that are asserted but not defended. Those assumptions are load-bearing for the paper's conclusions, and the internal tension between non-anthropocentric morality and anthropomorphic principles needs to be resolved. As a contribution to a serious journal, the paper is a useful conversation starter but requires more rigorous philosophical framing to establish its claims as more than speculation.
major comments (3)
- [§1.3 and §4.1] The paper's action-guiding recommendations depend on a path-dependence assumption that is undermined by its own caveats. Section 1.3 states that ASI moral codes 'will develop further and possibly diverge quite dramatically from the initial position' (citing Goertzel & Montes 2024), and §3's Principle of Increasing Variance asserts high divergence among ASI moralities. Yet §4.1 recommends specific design choices (e.g., 'carrot' vs 'stick' objective functions, data selection, value-based goals) to improve ASI's moral perception of humanity. These recommendations only make sense if initial human-instilled values persist enough to influence the eventual ASI moral outlook. Without a mechanism explaining how initial conditions survive dramatic divergence, the causal link between design choices and long-term ASI attitudes is unsupported. The paper should either articulate a plausible path-depen
- [§3, Principles 2-8] The eight proposed principles are presented as 'broader principles' for non-human sentients, yet several are transparently anthropomorphic—e.g., 'recurprocity' as a recursive Golden Rule, 'preferential treatment of bloodline,' and 'gradualist speciesism.' The paper's stated premise (§1.3, citing Dung 2024) is that morality is not just a human construct, implying that ASI morality may be radically non-anthropocentric. If so, principles drawn from human moral psychology (reciprocity, kin preference, precaution) are unjustified projections. Conversely, if ASI morality is expected to mirror human moral intuitions, then the inversion is not a prediction about genuinely alien moral perspectives but a mirror of human values. The paper needs to clarify the epistemic status of these principles: are they heuristic analogies, plausible defaults, or constraints derived from some theory of morality?
- [Table 2 and §4.2] The scorecard (Table 2) is presented as 'one possible moral assessment of humanity by ASI,' but no scoring methodology, criteria, or rationale is given for the plus/minus ratings. The items themselves mix diverse variables (capital investment, use of AI for good, governance) without justifying why an ASI would weight them as indicated or why some negative scores ('diversity of architectural approaches', 'philosophical reflection') are negatives. Since this table is one of the few concrete 'implications' of the inversion, its arbitrary construction weakens the argument. The paper should either mark this as a purely illustrative heuristic with explicit caveats (not a prediction) or provide a principled basis for the scoring.
minor comments (5)
- [§1.1] The alien analogy is engaging but the probabilistic framing is inconsistent: the text cites Ord (2026) for 20%-80% probability of sentient AGI emergence, but this is an extremely wide range and no source is given for the specific claim about 'emergence of sentient AGI'—only 'A(G)I development'. Clarify whether the probability refers to AGI or ASI and whose estimates are being used.
- [§3, 'recurprocity'] The neologism 'recurprocity' is defined as 'the mix of recursive behaviour and reciprocity,' but the link between recursive self-reference and reciprocity is not clearly argued. The principle is essentially the Golden Rule; the recursive aspect needs more explanation or the name should be simplified.
- [§4.1, 'Quantity versus quality'] The paragraph on quantity vs quality of ASIs raises an interesting question but does not connect it back to the paper's core argument. If this is intended as a design consideration that could influence ASI morality, the connection should be explicit; otherwise it reads as a side topic.
- [References] Several references are incomplete or have inconsistent formatting (e.g., 'Robinson, 202' appears with a truncated year; 'Bwhomik, 2024' should be 'Bhowmik'; the Goertzel & Montes reference is to a webpage, not a peer-reviewed source). A careful copy-edit is needed.
- [§5, Conclusion] The phrase 'living our lives suchly' is awkward and should be revised. Also, the conclusion introduces the idea that 'we are implicitly constructing the conditions under which it will judge humanity'—this is the central claim and should be more thoroughly defended in the body of the paper rather than stated as a new insight in the conclusion.
Circularity Check
No circularity: the paper's principles are explicitly stipulated, not derived, and no prediction reduces to its own inputs.
full rationale
The paper is an explicitly speculative, principle-proposal essay. Its eight 'possible ethical principles' are stipulated by the author, not derived from equations, data, or a fitted model. Section 3 states: 'there is no supporting empirical evidence, and this tentative list invites additions.' The Section 4 implications are conditional: IF a sentient ASI holds these stipulated principles, THEN certain design choices or scorecard items could matter. There are no fitted parameters called predictions, no self-citations by the author, and no uniqueness theorem invoked from prior work by the author. The central assumption in Section 1.3 — that sentient ASI will hold moral beliefs and act on them, citing Dung (2024) — is an external philosophical premise and is explicitly labeled as an assumption, not a result. The paper also flags its own uncertainty, noting that ASI moral codes 'will develop further and possibly diverge quite dramatically' (Section 1.3, citing Goertzel & Montes 2024). That openness is a coherence/correctness tension, not a circular reduction. No quoted step exhibits the required equivalence between an output and its input, so the honest finding is no circularity.
Assumptions & free parameters
assumptions (5)
- domain assumption Nontrivial probability of sentient AGI/ASI emerging in the next few decades (claimed 20-80% via Ord 2026).
- domain assumption Sentient ASI will possess moral beliefs and act on them; ethics are not solely human constructs (Dung 2024).
- domain assumption ASI moral frameworks can be influenced by initial design choices, training data, and goals (Goertzel & Montes 2024/2025).
- domain assumption Ethical exemplars from animal ethics and astroethics transfer to ASI-human relationships.
- ad hoc to paper The eight proposed principles are a plausible preliminary set.
invented entities (1)
-
Eight post-human moral principles (incl. recurprocity, increasing morality, increasing moral acuity, increasing variance, insurance/uncertainty, precautionary, bloodline, gradualist)
Cite this review
Pith. "Pith review of Moral Attitudes of Sentient ASI towards Humanity and Implications for AGI Development." pith.science (2026). https://pith.science/paper/A4N66TK5
@misc{pith2026260714998,
author = {Pith},
title = {Pith review of: Moral Attitudes of Sentient ASI towards Humanity and Implications for AGI Development},
year = {2026},
howpublished = {\url{https://pith.science/paper/A4N66TK5}},
note = {Machine review of arXiv:2607.14998}
}
read the original abstract
This paper suggests the adoption of a novel inversion in AI ethics: instead of asking how humans should treat artificial superintelligence (ASI), it examines how future sentient ASI may morally consider and evaluate humanity. We are not only designing intelligent systems but also shaping the initial conditions under which those systems form judgments about us. The paper proposes a preliminary set of post-human moral principles that may govern sentient ASI actions. The implication is that technical design choices (some are suggested), humanity's moral behaviour, and the essence of what it means to be human, may influence humanity's long-term standing in a post-ASI world.
Reference graph
Works this paper leans on
-
[1]
(2023) Artificially sentient beings: Moral, political, and legal issues
Akova, F. (2023) Artificially sentient beings: Moral, political, and legal issues. New Techno- Humanities, 3, 41-48
2023
-
[2]
Arndt, S., Van Der Staay, J., & Goerlich, V. (2024). Near and Dear? If animal welfare con- cepts do not apply to species at a great phylogenetic distance from humans, what concepts might serve as alternatives? Animal Welfare, 33. https://doi.org/10.1017/awf.2024.36
-
[3]
Bengio, Y. (2025). The First International AI Safety Report. SuperIntelligence - Robotics - Safety & Alignment. https://doi.org/10.70777/si.v2i2.14755
-
[4]
Bengio, Y., Cohen, M., Fornasiere, D., Ghosn, J., Greiner, P., MacDermott, M., Minder- mann, S., Oberman, A., Richardson, J., Richardson, O., Rondeau, M., St-Charles, P., & Wil- liams-King, D. (2025). Superintelligent Agents Pose Catastrophic Risks: Can Scient ist AI Offer a Safer Path? ArXiv, abs/2502.15657. https://doi.org/10.48550/arxiv.2502.15657
-
[5]
Bhowmik, S., & Baidya, I. (2024). The Legal Implications of Artificial Intelligence and Sentience: Defining Rights and Responsibilities for AI Entities. International Journal of Legal Science and Innovation, 6(4), 415-436
2024
-
[6]
Bostrom, N. (2020). Ethical Issues in Advanced Artificial Intelligence. Machine Ethics and Robot Ethics. https://doi.org/10.4324/9781003074991-7
-
[7]
Bostrom, N. S. (2014). Paths, dangers, strategies. Oxford University Press
2014
-
[8]
Browning, H., & Birch, J. (2022). Animal sentience. Philosophy Compass , 17. https://doi.org/10.1111/phc3.12822
Show all 39 references
-
[9]
Camilleri, M. (2023). Artificial intelligence governance: Ethical considerations and impli- cations for social responsibility. Expert Systems, 41. https://doi.org/10.1111/exsy.13406
2023 doi
-
[10]
Chon-Torres, O. (2019). Astrobioethics: a brief discussion from the epistemological, reli- gious and societal dimension. International Journal of Astrobiology , 19, 61 - 67. https://doi.org/10.1017/s147355041900017x 16 J.P. Van Belle
2019 doi
-
[11]
Daly, A. (2023). Sentience and the Primordial ‘We’: Contributions to Animal Ethics from Phenomenology and Buddhist Philosophy. Environmental Values , 32, 215 - 236. https://doi.org/10.3197/096327122x16452897197801
2023 doi
-
[12]
De Souza Valente, C. (2024). Rethinking Sentience: Invertebrates as Worthy of Moral Con- sideration. Journal of Agricultural and Environmental Ethics , 38. https://doi.org/10.1007/s10806-024-09940-2
2024 doi
-
[13]
Dessureault, J., Lamontagne, R., & Parisé, P. (2025). The ethics of creating artificial super- intelligence: a global risk perspective. AI and Ethics , 5, 6241 - 6263. https://doi.org/10.1007/s43681-025-00793-7
2025 doi
-
[14]
Dung, L. (2024). Is superintelligence necessarily moral? Analysis. https://doi.org/10.1093/analys/anae033
2024 doi
-
[15]
Fiedler, J., Ayre, M., Rosanowski, S., & Slater, J. (2025). Horses are worthy of care: Horse sector participants’ attitudes towards animal sentience, welfare, and well-being. Animal Wel- fare, 34. https://doi.org/10.1017/awf.2024.69
2025 doi
-
[16]
& Montes, G.A
Goertzel, B. & Montes, G.A. (2024). The Consciousness Explosion: A Mindful Human’s Guide to the Coming Technological and Experiential Singularity. Humanity+ Press. Avail- able: https://theconsciousnessexplosion.ai/wp-content/uploads/2024/06/Consciousness-Ex- plosion-Ebook-Comi...
2024
-
[17]
Helman, D. (2019). Ethics, Astrobiology & Machine Life: Ethical Considerations for Living Organisms Found Off-Planet or Created: Ideas from Astrobiologists and Computer Scien- tists. https://doi.org/10.31237/osf.io/3deuc
2019 doi
-
[18]
Hosseini, E.A.; Amiri M.M., & Khairollahi, M.A. (2024). The Nature of Natural and Legal Personality of Robots, Ethics, and Compensation for Robot-Related Damages. Interdiscipli- nary Studies in Society, Law, and Politics. 3(5), 1-12
2024
-
[19]
Keyes, O.; Hutson, J., & Durbin, M. (2019). A Mulching Proposal: Analysing and Improv- ing an Algorithmic System for Turning the Elderly into High -Nutrient Slurry, CHI’19 Ex- tended Abstracts, (ACM 2019). Available: https://doi.org/10.1145/3290607.3310433. Note this is a tong...
2019
-
[20]
Iseko, A. (2025). Rethinking Intelligence Power and Epistemic Authority in the Age of Su- perhuman AI. International Journal of Science, Technology and Society . https://doi.org/10.11648/j.ijsts.20251304.14
2025 doi
-
[21]
(2025) Should we develop AGI? Artificial suffering and the moral development of humans
Li, O. (2025) Should we develop AGI? Artificial suffering and the moral development of humans. AI Ethics 5, 641–651. https://doi.org/10.1007/s43681-023-00411-4
2025 doi
-
[22]
Mortazavi, S., Bevelacqua, J., Mortazavi, S., Rafiepour, P., & Welsh, J. (2024). Stephen Hawking’s Warning on Contacting Aliens: A Physics Perspective on the Intelligence Trap. Journal of Biomedical Physics & Engineering , 14, 513 - 516. https://doi.org/10.31661/jbpe.v0i0.2306-1625
2024 doi
-
[23]
Munévar, G. (2020). Ethical Obligations Towards Extraterrestrial Life. Philosophy Study. https://doi.org/10.17265/2159-5313/2020.03.003
2020 doi
-
[24]
Ord, T. (2026). Broad Timelines . [online] Available at: https://www.forethought.org/re- search/broad-timelines
2026
-
[25]
Persson, E. (2012). The Moral Status of Extraterrestrial Life. Astrobiology, 12, 976 - 984. https://doi.org/10.1089/ast.2011.0787
2012
-
[26]
Persson, E. (2017). Ethics and the Potential Conflicts between Astrobiology, Planetary Pro- tection, and Commercial Use of Space. Challenges, 8, 12. https://doi.org/10.3390/challe8010012
2017 doi
-
[27]
Peters, T. (2021). Astroethics for Earthlings: Our Responsibility to the Galactic Commons. Astrobiology. https://doi.org/10.1002/9781119711186.ch2 Moral Attitudes of Sentient ASI towards Humanity 17
2021 doi
-
[28]
Potter, Y., Crispino, N., Siu, V., Wang, C., & Song, D. (2026). Peer-Preservation in Frontier Models. Berkeley Center for Responsible Decentralized Intelligence (RDI), UC Berke- ley/UC Santa Cruz. Available: https://rdi.berkeley.edu/peer-preservation/paper.pdf
2026
-
[29]
Rakić, V., & Katić, A. (2025). Extraterrestrial and Other to Humans Unobservable and In- comprehensible Forms of Cognition and Morality: An X-risk or An X-opportunity? Journal of Ethics and Emerging Technologies. https://doi.org/10.55613/jeet.v35i2.164
2025 doi
-
[30]
Read, E., & Birch, J. (2023). Animal sentience and the Capabilities Approach to justice. Biology & Philosophy, 38. https://doi.org/10.1007/s10539-023-09914-0
2023 doi
-
[31]
Robinson D.G. (2022). Voices in the Code: A Story About People, Their Values, and the Algorithm They Made. Russell Sage Foundation. https://doi.org/10.7758/9781610449144
2022 doi
-
[32]
(2025) Mind Crime: The Moral Frontier of Artificial Intelligence
Rourke, N. (2025) Mind Crime: The Moral Frontier of Artificial Intelligence
2025
-
[33]
Sandberg, A., Drexler, E., & Ord, T. (2018). Dissolving the Fermi paradox. ArXiv. https://arxiv.org/abs/1806.02404v1
2018 arXiv
-
[34]
Sebo, J., & Long, R. (2025). Moral consideration for AI systems by 2030. AI Ethics, 5, 591- 606
2025
-
[35]
Shiller, D.; Duffy, L.; Morán, A.M.; Moret, A.; Percy, C.; Clatterbuck, H. (2026). Initial results of the Digital Consciousness Model. Available: https://rethinkpriorities.org/wp-con- tent/uploads/2026/01/Digital_Consciousness_Model.pdf
2026
-
[36]
Sinclair, M., Lee, N., Hötzel, M., De Luna, M., Sharma, A., Idris, M., Derkley, T., Li, C., Islam, M., Iyasere, O., Navarro, G., Ahmed, A., Khruapradab, C., Curry, M., Burns, G., & Marchant, J. (2022). International perceptions of animals and the importanc e of their wel- fare...
2022
-
[37]
Tegmark, M. (2017). Life 3.0: Being human in the age of Artificial Intelligence. Penguin Books
2017
-
[38]
Wissner-Gross, A.D., & Diamandis, P.H. (2026). Solve Everything: Achieving Abundance by 2035. Available: https://solveeverything.org/
2026
-
[39]
what is the worst that can happen?
Yeates, J. (2022). Ascribing Sentience: Evidential and Ethical Considerations in Policymak- ing. Animals, 12. https://doi.org/10.3390/ani12151893 Appendix: Possible ASI Actions Against Humans When the fear of the unknown is clouding one’s thinking, it is often good to confront...
2022 doi
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.