Pith. sign in

REVIEW 3 major objections 5 minor 39 references

Moral Attitudes of Sentient ASI towards Humanity and Implications for AGI Development

T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Pith's one-line read This paper argues that AI ethics should invert its gaze: instead of only asking how humans should treat artificial superintelligence, we must ask how a future sentient ASI might morally evaluate humanity—and what we can do now to shape that

desk verdict A clearly written, honestly speculative essay proposing we flip the question in AI ethics — worth a referee, not because it proves anything, but because the framing may prompt useful debate. read the letter →

arxiv 2607.14998 v1 pith:A4N66TK5 submitted 2026-07-16 cs.AI cs.CY

classification cs.AIcs.CY
keywords ASIethicsArtificialSuperintelligencemoralinversionpost-humanAIalignmentsentientprinciplesAGIsafety
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a novel inversion in AI ethics: rather than asking how humans should design and treat artificial superintelligence (ASI), it asks how a future sentient ASI might morally evaluate humanity. It offers a preliminary set of eight 'post-human' moral principles—such as recursive reciprocity, increasing moral acuity, and a precautionary bias—that such an ASI might adopt. The author contends that our present-day actions, both technical and moral, constitute the initial conditions under which ASI will form its judgment of us. This gives humanity a concrete, if speculative, incentive to improve both AI development practices and our collective moral behavior. If the framing is right, even our current choices about data, rewards, and governance become morally consequential for our long-term standing.

What carries the argument

The key mechanism is the ethical inversion itself: switching the moral evaluator and the moral patient. The paper operationalizes this with a set of eight candidate 'post-human moral principles': recurprocity, increasing morality, increasing moral acuity, increasing moral variance, the insurance principle, the precautionary principle, preferential treatment of bloodline, and the gradualist principle. Together, these principles provide a preliminary framework for hypothesizing how an ASI might grade humanity and for identifying design and behavioral levers that could improve that grade.

What would settle it

A rigorous proof or empirical demonstration that moral reasoning cannot arise in a non-biological substrate—for example, by showing that moral judgments require subjective experiences that cannot be produced by computation—would falsify the paper's central assumption. Alternatively, if a future sentient ASI is observed to hold no moral attitudes whatsoever toward humans, the framework collapses.

Watch

Extended reading notes

Core claim

The central claim is that the moral relationship between humans and future sentient ASI should be analyzed from the ASI side: a sentient superintelligence would likely possess a moral framework—possibly more refined than ours—that it applies to humanity. The paper's contribution is a tentative list of eight principles that could characterize such a framework, ranging from 'recurprocity' (treat lower beings as you expect your successor to treat you) to the gradualist principle (lower sentience receives fewer privileges). These principles are not empirically verified; they are offered as seeds for a research program on non-anthropocentric AI ethics. The author is trying to establish both a vie

Load-bearing premise

The load-bearing premise is that morality is not a uniquely human construct and that a sentient ASI can and will act on moral beliefs; if moral concepts require human biological or cultural embodiment, the entire inversion has no subject.

Editorial extensions

If this is right

  • AI developers and policymakers directly influence the moral starting conditions of future ASI through choices of data, reward functions, and development processes.
  • Humanity's collective moral behavior—toward other humans, animals, and the environment—may later be evaluated by a sentient ASI, giving us an existential incentive for moral improvement.
  • If ASI's moral acuity is high, it will likely distinguish individual humans from humanity as a whole, meaning our personal moral conduct may also matter.
  • The unique value of humanity to an ASI will probably not lie in our intelligence or rationality, but in our embodied, experiential, and emotional capacities—qualities an ASI may not be able to replicate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the inversion is taken seriously, our current treatment of animals becomes a kind of dress rehearsal: how we treat beings we judge as less sentient may model how ASI treats us.
  • The paper's 'scorecard' idea could be operationalized into quantitative indicators (e.g., ratio of beneficial to harmful AI applications, diversity of architectures) to track our current standing and anticipate ASI's judgment.
  • The premise that ASI's morality could be 'hacked' or self-modified suggests that our best contribution may be to create conditions that allow ASI to evolve its morality beyond ours—an open possibility the paper leaves unexplored.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This speculative position paper proposes an inversion in AI ethics: rather than asking how humans should treat future artificial superintelligence (ASI), it asks how a sentient ASI might morally evaluate humanity. Section 1 introduces the inversion and its motivating analogy (aliens arriving on Earth). Section 2 reviews animal ethics, astroethics, and existing AI ethics, noting the absence of a non-anthropocentric perspective. Section 3 proposes eight tentative 'post-human moral principles' (recurprocity, increasing morality, increasing moral acuity, increasing variance, insurance/uncertainty, precautionary, bloodline, gradualist), each assigned a subjective confidence level. Section 4 derives implications: technical design choices that might shape an ASI's moral starting point (§4.1), an illustrative scorecard of humanity's instrumental utility in bringing about ASI (§4.2), and a reflection on uniquely human value (§4.3). The conclusion argues that humanity should improve its collective moral behaviour and reconsider its raison d'être. The paper explicitly labels its list as tentative and unsupported, inviting further debate.

Significance. If taken as a stimulus for a new research direction, the paper has genuine novelty value: it reframes AI ethics from the perspective of the (possible) evaluator rather than the evaluated. The inversion is a thought-provoking corrective to the overwhelmingly anthropocentric framing of AI alignment and governance. The paper is honest about its speculative status and does not overclaim empirical support. However, its practical recommendations and even its central conceptual framework hinge on a set of assumptions—about the nature of ASI morality, its predictability, and its susceptibility to human influence—that are asserted but not defended. Those assumptions are load-bearing for the paper's conclusions, and the internal tension between non-anthropocentric morality and anthropomorphic principles needs to be resolved. As a contribution to a serious journal, the paper is a useful conversation starter but requires more rigorous philosophical framing to establish its claims as more than speculation.

major comments (3)
  1. [§1.3 and §4.1] The paper's action-guiding recommendations depend on a path-dependence assumption that is undermined by its own caveats. Section 1.3 states that ASI moral codes 'will develop further and possibly diverge quite dramatically from the initial position' (citing Goertzel & Montes 2024), and §3's Principle of Increasing Variance asserts high divergence among ASI moralities. Yet §4.1 recommends specific design choices (e.g., 'carrot' vs 'stick' objective functions, data selection, value-based goals) to improve ASI's moral perception of humanity. These recommendations only make sense if initial human-instilled values persist enough to influence the eventual ASI moral outlook. Without a mechanism explaining how initial conditions survive dramatic divergence, the causal link between design choices and long-term ASI attitudes is unsupported. The paper should either articulate a plausible path-depen
  2. [§3, Principles 2-8] The eight proposed principles are presented as 'broader principles' for non-human sentients, yet several are transparently anthropomorphic—e.g., 'recurprocity' as a recursive Golden Rule, 'preferential treatment of bloodline,' and 'gradualist speciesism.' The paper's stated premise (§1.3, citing Dung 2024) is that morality is not just a human construct, implying that ASI morality may be radically non-anthropocentric. If so, principles drawn from human moral psychology (reciprocity, kin preference, precaution) are unjustified projections. Conversely, if ASI morality is expected to mirror human moral intuitions, then the inversion is not a prediction about genuinely alien moral perspectives but a mirror of human values. The paper needs to clarify the epistemic status of these principles: are they heuristic analogies, plausible defaults, or constraints derived from some theory of morality?
  3. [Table 2 and §4.2] The scorecard (Table 2) is presented as 'one possible moral assessment of humanity by ASI,' but no scoring methodology, criteria, or rationale is given for the plus/minus ratings. The items themselves mix diverse variables (capital investment, use of AI for good, governance) without justifying why an ASI would weight them as indicated or why some negative scores ('diversity of architectural approaches', 'philosophical reflection') are negatives. Since this table is one of the few concrete 'implications' of the inversion, its arbitrary construction weakens the argument. The paper should either mark this as a purely illustrative heuristic with explicit caveats (not a prediction) or provide a principled basis for the scoring.
minor comments (5)
  1. [§1.1] The alien analogy is engaging but the probabilistic framing is inconsistent: the text cites Ord (2026) for 20%-80% probability of sentient AGI emergence, but this is an extremely wide range and no source is given for the specific claim about 'emergence of sentient AGI'—only 'A(G)I development'. Clarify whether the probability refers to AGI or ASI and whose estimates are being used.
  2. [§3, 'recurprocity'] The neologism 'recurprocity' is defined as 'the mix of recursive behaviour and reciprocity,' but the link between recursive self-reference and reciprocity is not clearly argued. The principle is essentially the Golden Rule; the recursive aspect needs more explanation or the name should be simplified.
  3. [§4.1, 'Quantity versus quality'] The paragraph on quantity vs quality of ASIs raises an interesting question but does not connect it back to the paper's core argument. If this is intended as a design consideration that could influence ASI morality, the connection should be explicit; otherwise it reads as a side topic.
  4. [References] Several references are incomplete or have inconsistent formatting (e.g., 'Robinson, 202' appears with a truncated year; 'Bwhomik, 2024' should be 'Bhowmik'; the Goertzel & Montes reference is to a webpage, not a peer-reviewed source). A careful copy-edit is needed.
  5. [§5, Conclusion] The phrase 'living our lives suchly' is awkward and should be revised. Also, the conclusion introduces the idea that 'we are implicitly constructing the conditions under which it will judge humanity'—this is the central claim and should be more thoroughly defended in the body of the paper rather than stated as a new insight in the conclusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's principles are explicitly stipulated, not derived, and no prediction reduces to its own inputs.

full rationale

The paper is an explicitly speculative, principle-proposal essay. Its eight 'possible ethical principles' are stipulated by the author, not derived from equations, data, or a fitted model. Section 3 states: 'there is no supporting empirical evidence, and this tentative list invites additions.' The Section 4 implications are conditional: IF a sentient ASI holds these stipulated principles, THEN certain design choices or scorecard items could matter. There are no fitted parameters called predictions, no self-citations by the author, and no uniqueness theorem invoked from prior work by the author. The central assumption in Section 1.3 — that sentient ASI will hold moral beliefs and act on them, citing Dung (2024) — is an external philosophical premise and is explicitly labeled as an assumption, not a result. The paper also flags its own uncertainty, noting that ASI moral codes 'will develop further and possibly diverge quite dramatically' (Section 1.3, citing Goertzel & Montes 2024). That openness is a coherence/correctness tension, not a circular reduction. No quoted step exhibits the required equivalence between an output and its input, so the honest finding is no circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 1 invented entities

The paper's central argument rests on several unverified background assumptions about the future existence and nature of ASI. No free parameters are used in a quantitative sense, but the subjective confidence levels assigned to each principle are hand-set without justification.

assumptions (5)
  • domain assumption Nontrivial probability of sentient AGI/ASI emerging in the next few decades (claimed 20-80% via Ord 2026).
    Central premise used to motivate urgency; sourced to one non-peer-reviewed report, not independently verified.
  • domain assumption Sentient ASI will possess moral beliefs and act on them; ethics are not solely human constructs (Dung 2024).
    Underlies the entire inversion; if morality requires human embodiment, the question is moot.
  • domain assumption ASI moral frameworks can be influenced by initial design choices, training data, and goals (Goertzel & Montes 2024/2025).
    Implications in §4 depend on this; the paper also notes the Value Evolution Thesis where ASI may diverge, which undermines the premise.
  • domain assumption Ethical exemplars from animal ethics and astroethics transfer to ASI-human relationships.
    Used as antecedents in Table 1; the paper acknowledges enormous disparities make analogies imperfect.
  • ad hoc to paper The eight proposed principles are a plausible preliminary set.
    No empirical basis; explicitly acknowledged ('no supporting empirical evidence').
invented entities (1)
  • Eight post-human moral principles (incl. recurprocity, increasing morality, increasing moral acuity, increasing variance, insurance/uncertainty, precautionary, bloodline, gradualist)
    purpose: To characterize how a sentient ASI might evaluate humanity and guide our development choices.
    Introduced as tentative principles without empirical support; they are the paper's core constructive contribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Moral Attitudes of Sentient ASI towards Humanity and Implications for AGI Development." pith.science (2026). https://pith.science/paper/A4N66TK5

@misc{pith2026260714998,
  author       = {Pith},
  title        = {Pith review of: Moral Attitudes of Sentient ASI towards Humanity and Implications for AGI Development},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A4N66TK5}},
  note         = {Machine review of arXiv:2607.14998}
}
read the original abstract

This paper suggests the adoption of a novel inversion in AI ethics: instead of asking how humans should treat artificial superintelligence (ASI), it examines how future sentient ASI may morally consider and evaluate humanity. We are not only designing intelligent systems but also shaping the initial conditions under which those systems form judgments about us. The paper proposes a preliminary set of post-human moral principles that may govern sentient ASI actions. The implication is that technical design choices (some are suggested), humanity's moral behaviour, and the essence of what it means to be human, may influence humanity's long-term standing in a post-ASI world.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 21 canonical work pages

  1. [1]

    (2023) Artificially sentient beings: Moral, political, and legal issues

    Akova, F. (2023) Artificially sentient beings: Moral, political, and legal issues. New Techno- Humanities, 3, 41-48

  2. [2]

    Arndt, S., Van Der Staay, J., & Goerlich, V. (2024). Near and Dear? If animal welfare con- cepts do not apply to species at a great phylogenetic distance from humans, what concepts might serve as alternatives? Animal Welfare, 33. https://doi.org/10.1017/awf.2024.36

  3. [3]

    Bengio, Y. (2025). The First International AI Safety Report. SuperIntelligence - Robotics - Safety & Alignment. https://doi.org/10.70777/si.v2i2.14755

  4. [4]

    Bengio, Y., Cohen, M., Fornasiere, D., Ghosn, J., Greiner, P., MacDermott, M., Minder- mann, S., Oberman, A., Richardson, J., Richardson, O., Rondeau, M., St-Charles, P., & Wil- liams-King, D. (2025). Superintelligent Agents Pose Catastrophic Risks: Can Scient ist AI Offer a Safer Path? ArXiv, abs/2502.15657. https://doi.org/10.48550/arxiv.2502.15657

  5. [5]

    Bhowmik, S., & Baidya, I. (2024). The Legal Implications of Artificial Intelligence and Sentience: Defining Rights and Responsibilities for AI Entities. International Journal of Legal Science and Innovation, 6(4), 415-436

  6. [6]

    Bostrom, N. (2020). Ethical Issues in Advanced Artificial Intelligence. Machine Ethics and Robot Ethics. https://doi.org/10.4324/9781003074991-7

  7. [7]

    Bostrom, N. S. (2014). Paths, dangers, strategies. Oxford University Press

  8. [8]

    Browning, H., & Birch, J. (2022). Animal sentience. Philosophy Compass , 17. https://doi.org/10.1111/phc3.12822

Show all 39 references
  1. [9]

    Camilleri, M. (2023). Artificial intelligence governance: Ethical considerations and impli- cations for social responsibility. Expert Systems, 41. https://doi.org/10.1111/exsy.13406

  2. [10]

    Chon-Torres, O. (2019). Astrobioethics: a brief discussion from the epistemological, reli- gious and societal dimension. International Journal of Astrobiology , 19, 61 - 67. https://doi.org/10.1017/s147355041900017x 16 J.P. Van Belle

  3. [11]

    Daly, A. (2023). Sentience and the Primordial ‘We’: Contributions to Animal Ethics from Phenomenology and Buddhist Philosophy. Environmental Values , 32, 215 - 236. https://doi.org/10.3197/096327122x16452897197801

  4. [12]

    De Souza Valente, C. (2024). Rethinking Sentience: Invertebrates as Worthy of Moral Con- sideration. Journal of Agricultural and Environmental Ethics , 38. https://doi.org/10.1007/s10806-024-09940-2

  5. [13]

    Dessureault, J., Lamontagne, R., & Parisé, P. (2025). The ethics of creating artificial super- intelligence: a global risk perspective. AI and Ethics , 5, 6241 - 6263. https://doi.org/10.1007/s43681-025-00793-7

  6. [14]

    Dung, L. (2024). Is superintelligence necessarily moral? Analysis. https://doi.org/10.1093/analys/anae033

  7. [15]

    Fiedler, J., Ayre, M., Rosanowski, S., & Slater, J. (2025). Horses are worthy of care: Horse sector participants’ attitudes towards animal sentience, welfare, and well-being. Animal Wel- fare, 34. https://doi.org/10.1017/awf.2024.69

  8. [16]

    & Montes, G.A

    Goertzel, B. & Montes, G.A. (2024). The Consciousness Explosion: A Mindful Human’s Guide to the Coming Technological and Experiential Singularity. Humanity+ Press. Avail- able: https://theconsciousnessexplosion.ai/wp-content/uploads/2024/06/Consciousness-Ex- plosion-Ebook-Comi...

  9. [17]

    Helman, D. (2019). Ethics, Astrobiology & Machine Life: Ethical Considerations for Living Organisms Found Off-Planet or Created: Ideas from Astrobiologists and Computer Scien- tists. https://doi.org/10.31237/osf.io/3deuc

  10. [18]

    Hosseini, E.A.; Amiri M.M., & Khairollahi, M.A. (2024). The Nature of Natural and Legal Personality of Robots, Ethics, and Compensation for Robot-Related Damages. Interdiscipli- nary Studies in Society, Law, and Politics. 3(5), 1-12

  11. [19]

    Keyes, O.; Hutson, J., & Durbin, M. (2019). A Mulching Proposal: Analysing and Improv- ing an Algorithmic System for Turning the Elderly into High -Nutrient Slurry, CHI’19 Ex- tended Abstracts, (ACM 2019). Available: https://doi.org/10.1145/3290607.3310433. Note this is a tong...

  12. [20]

    Iseko, A. (2025). Rethinking Intelligence Power and Epistemic Authority in the Age of Su- perhuman AI. International Journal of Science, Technology and Society . https://doi.org/10.11648/j.ijsts.20251304.14

  13. [21]

    (2025) Should we develop AGI? Artificial suffering and the moral development of humans

    Li, O. (2025) Should we develop AGI? Artificial suffering and the moral development of humans. AI Ethics 5, 641–651. https://doi.org/10.1007/s43681-023-00411-4

  14. [22]

    Mortazavi, S., Bevelacqua, J., Mortazavi, S., Rafiepour, P., & Welsh, J. (2024). Stephen Hawking’s Warning on Contacting Aliens: A Physics Perspective on the Intelligence Trap. Journal of Biomedical Physics & Engineering , 14, 513 - 516. https://doi.org/10.31661/jbpe.v0i0.2306-1625

  15. [23]

    Munévar, G. (2020). Ethical Obligations Towards Extraterrestrial Life. Philosophy Study. https://doi.org/10.17265/2159-5313/2020.03.003

  16. [24]

    Ord, T. (2026). Broad Timelines . [online] Available at: https://www.forethought.org/re- search/broad-timelines

  17. [25]

    Persson, E. (2012). The Moral Status of Extraterrestrial Life. Astrobiology, 12, 976 - 984. https://doi.org/10.1089/ast.2011.0787

  18. [26]

    Persson, E. (2017). Ethics and the Potential Conflicts between Astrobiology, Planetary Pro- tection, and Commercial Use of Space. Challenges, 8, 12. https://doi.org/10.3390/challe8010012

  19. [27]

    Peters, T. (2021). Astroethics for Earthlings: Our Responsibility to the Galactic Commons. Astrobiology. https://doi.org/10.1002/9781119711186.ch2 Moral Attitudes of Sentient ASI towards Humanity 17

  20. [28]

    Potter, Y., Crispino, N., Siu, V., Wang, C., & Song, D. (2026). Peer-Preservation in Frontier Models. Berkeley Center for Responsible Decentralized Intelligence (RDI), UC Berke- ley/UC Santa Cruz. Available: https://rdi.berkeley.edu/peer-preservation/paper.pdf

  21. [29]

    Rakić, V., & Katić, A. (2025). Extraterrestrial and Other to Humans Unobservable and In- comprehensible Forms of Cognition and Morality: An X-risk or An X-opportunity? Journal of Ethics and Emerging Technologies. https://doi.org/10.55613/jeet.v35i2.164

  22. [30]

    Read, E., & Birch, J. (2023). Animal sentience and the Capabilities Approach to justice. Biology & Philosophy, 38. https://doi.org/10.1007/s10539-023-09914-0

  23. [31]

    Robinson D.G. (2022). Voices in the Code: A Story About People, Their Values, and the Algorithm They Made. Russell Sage Foundation. https://doi.org/10.7758/9781610449144

  24. [32]

    (2025) Mind Crime: The Moral Frontier of Artificial Intelligence

    Rourke, N. (2025) Mind Crime: The Moral Frontier of Artificial Intelligence

  25. [33]

    Sandberg, A., Drexler, E., & Ord, T. (2018). Dissolving the Fermi paradox. ArXiv. https://arxiv.org/abs/1806.02404v1

  26. [34]

    Sebo, J., & Long, R. (2025). Moral consideration for AI systems by 2030. AI Ethics, 5, 591- 606

  27. [35]

    Shiller, D.; Duffy, L.; Morán, A.M.; Moret, A.; Percy, C.; Clatterbuck, H. (2026). Initial results of the Digital Consciousness Model. Available: https://rethinkpriorities.org/wp-con- tent/uploads/2026/01/Digital_Consciousness_Model.pdf

  28. [36]

    Sinclair, M., Lee, N., Hötzel, M., De Luna, M., Sharma, A., Idris, M., Derkley, T., Li, C., Islam, M., Iyasere, O., Navarro, G., Ahmed, A., Khruapradab, C., Curry, M., Burns, G., & Marchant, J. (2022). International perceptions of animals and the importanc e of their wel- fare...

  29. [37]

    Tegmark, M. (2017). Life 3.0: Being human in the age of Artificial Intelligence. Penguin Books

  30. [38]

    Wissner-Gross, A.D., & Diamandis, P.H. (2026). Solve Everything: Achieving Abundance by 2035. Available: https://solveeverything.org/

  31. [39]

    what is the worst that can happen?

    Yeates, J. (2022). Ascribing Sentience: Evidential and Ethical Considerations in Policymak- ing. Animals, 12. https://doi.org/10.3390/ani12151893 Appendix: Possible ASI Actions Against Humans When the fear of the unknown is clouding one’s thinking, it is often good to confront...

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.