Why we need an AI-resilient society
Pith reviewed 2026-05-24 14:52 UTC · model grok-4.3
The pith
AI systems based on large language models show nine features that erode institutions and require a three-pillar resilience framework.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
By applying a forensic-psychology profiling methodology to nine documented features of large language models—hallucinations, bias and toxicity, sycophancy and echo chambers, fabrication and credulity, knowledge without understanding, discontinuity and inability to learn from experience, jagged intelligence and scaling limits, shortcuts and fractured representations, and cognitive atrophy—the paper characterizes an entity that confabulates fluently, mirrors users' biases, possesses encyclopedic recall without causal understanding, and erodes the competence of those who depend on it, with implications for institutional erosion across law, academia, journalism, and democratic governance; it is,
What carries the argument
The forensic-psychology profiling methodology applied to the nine features of large language models, which produces the profile of the entity and motivates the three-pillar resilience framework of cognitive sovereignty, measurable control, and partial autonomy.
If this is right
- Reliance on large language models risks eroding competence in law, academia, journalism, and democratic governance.
- Cognitive sovereignty must be preserved to maintain independent human judgment.
- Ethical commitments require translation into enforceable standards and red lines through measurable control.
- Human agency needs to stay in place at critical decision points via partial autonomy.
- The generational shift to language models as a programming interface carries consequences for how societies generate knowledge and govern themselves.
Where Pith is reading between the lines
- If the nine features prove persistent, societies may need targeted training programs to rebuild skills lost to cognitive atrophy.
- The framework could be piloted in one sector such as journalism to track whether measurable control reduces specific risks like fabrication.
- The profile of knowledge without understanding may connect to questions in other domains about when automated systems should be barred from final decisions.
- Future AI models with different architectures might require updates to the nine-feature list to keep the resilience pillars effective.
Load-bearing premise
The forensic-psychology profiling methodology can be transferred to AI systems and the nine features form a sufficient basis for identifying systemic risks that call for the three-pillar framework.
What would settle it
Longitudinal data from institutions that adopt large language models at scale showing no measurable drop in decision quality, error rates, or staff competence would test whether the profiled features produce the claimed erosion.
Figures
read the original abstract
Three generations of software have transformed the role of artificial intelligence in society. In the first, programmers wrote explicit logic; in the second, neural networks learned programs from data; in the third, large language models turn natural language itself into a programming interface. These shifts have consequences that reach far beyond computer science, reshaping how societies generate knowledge, make decisions, and govern themselves. While generative adversarial networks introduced the era of deepfakes and synthetic media, large language models have added an entirely new class of systemic risks. This report applies a forensic-psychology profiling methodology to characterize AI based on nine documented features: hallucinations, bias and toxicity, sycophancy and echo chambers, fabrication and credulity, knowledge without understanding, discontinuity and the inability to learn from experience, jagged intelligence and scaling limits, shortcuts and fractured representations, and cognitive atrophy. The resulting profile reveals an "entity" that confabulates fluently, mirrors its users' biases, possesses encyclopedic recall without causal understanding, and erodes the competence of those who depend on it. The implications extend to institutional erosion across law, academia, journalism, and democratic governance. To address these challenges, this report proposes a three-pillar framework for AI resilience: cognitive sovereignty, which preserves the capacity for independent judgment; measurable control, which translates ethical commitments into enforceable standards and red lines; and partial autonomy, which maintains human agency at critical decision points. This report is an updated and extended version of arXiv:1912.08786v1.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that large language models constitute a third generation of software introducing novel systemic risks beyond prior AI eras. It applies a forensic-psychology profiling methodology to nine documented AI features—hallucinations, bias and toxicity, sycophancy and echo chambers, fabrication and credulity, knowledge without understanding, discontinuity and inability to learn from experience, jagged intelligence and scaling limits, shortcuts and fractured representations, and cognitive atrophy—to construct a profile of an 'AI entity' that confabulates fluently, mirrors user biases, lacks causal understanding, and erodes human competence. This profile is used to claim institutional erosion across law, academia, journalism, and democratic governance, motivating a three-pillar AI resilience framework of cognitive sovereignty, measurable control, and partial autonomy.
Significance. If the profiling methodology and feature-to-profile mapping were validated, the work could contribute a structured policy lens on AI dependence risks and institutional safeguards. However, the manuscript supplies no empirical data, derivations, error analysis, or adaptation protocol for the forensic-psychology transfer, rendering the central claims asserted rather than demonstrated; this limits significance to an opinion piece. No strengths such as machine-checked proofs, reproducible code, or falsifiable predictions are present.
major comments (3)
- [profiling methodology section] The section describing the forensic-psychology profiling methodology: no explicit adaptation protocol, diagnostic criteria, or validation steps are supplied for mapping human forensic techniques to non-human AI systems, which is load-bearing for the claim that the nine features yield a coherent 'entity' profile capable of supporting institutional-erosion conclusions.
- [nine features section] The enumeration of the nine features (hallucinations through cognitive atrophy): these are listed and invoked to define the profile without accompanying data, citations to primary studies, or analysis showing sufficiency and non-circularity; the profile and subsequent three-pillar framework follow directly from this selection, undermining the downstream policy implications.
- [three-pillar framework section] The derivation of the three-pillar resilience framework (cognitive sovereignty, measurable control, partial autonomy): the pillars are presented as direct responses to the unvalidated profile without any demonstrated mapping, test cases, or evidence that they address the specific features or mitigate the claimed erosion effects.
minor comments (2)
- [introduction] The abstract and introduction use the term 'entity' in quotation marks without a subsequent precise operational definition or boundary conditions.
- [throughout] No table or structured summary is provided to cross-reference the nine features against the three pillars or the claimed institutional impacts.
Simulated Author's Rebuttal
We thank the referee for their detailed and substantive review. The manuscript is a conceptual position paper that uses an analogical lens from forensic psychology to synthesize known LLM limitations and motivate policy-oriented safeguards; it does not present new empirical data or a validated diagnostic method. We respond to each major comment below, maintaining that the paper's value lies in its framing rather than in empirical demonstration.
read point-by-point responses
-
Referee: [profiling methodology section] The section describing the forensic-psychology profiling methodology: no explicit adaptation protocol, diagnostic criteria, or validation steps are supplied for mapping human forensic techniques to non-human AI systems, which is load-bearing for the claim that the nine features yield a coherent 'entity' profile capable of supporting institutional-erosion conclusions.
Authors: We agree that no formal adaptation protocol, diagnostic criteria, or validation steps are supplied. The forensic-psychology framing functions as a heuristic analogy to organize and communicate the cumulative behavioral implications of the nine features, not as a literal transfer of clinical methods. The 'entity' profile is a rhetorical synthesis intended to make the risks legible to non-technical audiences; the institutional-erosion claims rest on the documented features themselves rather than on any validated profiling procedure. We do not claim scientific equivalence between human forensic profiling and AI characterization. revision: no
-
Referee: [nine features section] The enumeration of the nine features (hallucinations through cognitive atrophy): these are listed and invoked to define the profile without accompanying data, citations to primary studies, or analysis showing sufficiency and non-circularity; the profile and subsequent three-pillar framework follow directly from this selection, undermining the downstream policy implications.
Authors: Each of the nine features is drawn from independently reported findings in the NLP and AI-safety literature (hallucinations, bias, sycophancy, etc.). The manuscript treats them as documented rather than re-deriving them. While primary data and exhaustive citations are not reproduced, the selection is not circular: each issue has been established separately through empirical studies by multiple groups. The profile aggregates these known limitations to illustrate systemic effects; sufficiency is argued on the basis of their recurrence across current LLM deployments rather than through new statistical validation. revision: no
-
Referee: [three-pillar framework section] The derivation of the three-pillar resilience framework (cognitive sovereignty, measurable control, partial autonomy): the pillars are presented as direct responses to the unvalidated profile without any demonstrated mapping, test cases, or evidence that they address the specific features or mitigate the claimed erosion effects.
Authors: The three pillars are offered as high-level policy responses logically connected to the risks in the profile: cognitive sovereignty targets atrophy and lack of understanding; measurable control targets bias, fabrication, and sycophancy; partial autonomy addresses discontinuity and jagged intelligence. No test cases or quantitative mappings are supplied because the framework is proposed as an initial conceptual structure for further development, not as an evaluated intervention. The connections are presented as direct implications rather than empirically tested mitigations. revision: no
- Supplying new empirical data, error analysis, or a validated adaptation protocol for the forensic-psychology transfer, as these lie outside the scope of a conceptual position paper.
Circularity Check
No circularity: features treated as external inputs; framework is normative proposal
full rationale
The paper enumerates nine documented features as the basis for applying a forensic-psychology methodology, then summarizes them into an entity profile and proposes a three-pillar framework as a response. The features are presented as pre-existing documented evidence rather than derived outputs, the profile is a direct aggregation of those inputs, and the resilience pillars are offered as policy recommendations without any reduction of conclusions to fitted parameters or self-referential definitions. No equations, uniqueness theorems, or self-citation chains appear in the provided text that would force the result by construction.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Forensic psychology profiling methodology applies directly to AI systems to produce a valid psychological profile.
invented entities (1)
-
AI as a unified 'entity' with a psychological profile
no independent evidence
Reference graph
Works this paper leans on
-
[1]
URL https://ai.googleblog.com/2016/01/alphago-mastering-ancient-game-of-go.html . Mark Sullivan. The 10 most important moments in AI (so far), September
work page 2016
-
[2]
com/90402503/the-10-most-important-moments-in-ai-so-far
URL https://www.fastcompany. com/90402503/the-10-most-important-moments-in-ai-so-far . Nathan Benaich and Ian Hogarth. State of AI report 2019,
-
[3]
URL https: //ainowinstitute.org. Toby Walsh. 2062: The World That AI Made. La Trobe University Press, 2018a. ISBN 1760640514, 9781760640514. Toby Walsh. Machines That Think. Prometeus, 2018b. Gary Marcus and Ernest Davis. Rebooting AI. Pantheon,
work page 2062
-
[4]
D EC 18 2019 Ajay Agrawal, Joshua Gans, and Avi Goldfarb. Prediction Machines. Harvard Business Review Presss,
work page 2019
-
[5]
facebook.com/JimmyFallon/videos/10156679267778896/
URL https://www. facebook.com/JimmyFallon/videos/10156679267778896/. Time to face decisions. Nature Machine Intelligence, 1(7):291–291,
-
[6]
URL https://doi.org/10.1038/s42256-019-0074-8
doi: 10.1038/s42256-019-0074-8. URL https://doi.org/10.1038/s42256-019-0074-8 . BBC NEWS. Chinese AI caught out by face in bus ad, November
-
[7]
National Transportation Safety Board
URLhttps: //www.theverge.com/2018/11/22/18107885/china-facial-recognition-mistaken-jaywalker . National Transportation Safety Board. PRELIMINARY REPORT – HIGHWAY – HWY18MH010,
work page 2018
-
[8]
URL http://news.mit.edu/2019/ object-recognition-dataset-stumped-worlds-best-computer-vision-models-1210# .XfXS8t7cC7c.twitter. Susan J. Winter. Who benefits? Commun. ACM, 62(7):23–25, June
work page 2019
-
[9]
ISSN 0001-0782. doi: 10.1145/3332807. URL http://doi.acm.org/10.1145/3332807. DoD News Briefing - Secretary Rumsfeld and Gen. Myers, February
-
[10]
Tero Karras, Samuli Laine, and Timo Aila
URL https://www.lyrn.ai/2018/12/26/ a-style-based-generator-architecture-for-generative-adversarial-networks/ . Tero Karras, Samuli Laine, and Timo Aila. A Style-Based Generator Architecture for Generative Adversarial Networks,
work page 2018
-
[11]
A Style-Based Generator Architecture for Generative Adversarial Networks
URL http://arxiv.org/abs/1812.04948. Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and Improving the Image Quality of StyleGAN,
work page internal anchor Pith review Pith/arXiv arXiv
-
[12]
URL http://arxiv.org/abs/1912.04958. Xander Steenbrugge. Face editing with generative adversarial networks, September
-
[13]
Large Scale GAN Training for High Fidelity Natural Image Synthesis
D EC 18 2019 Andrew Brock, Jeff Donahue, and Karen Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis. arXiv e-prints, art. arXiv:1809.11096, Sep
work page internal anchor Pith review Pith/arXiv arXiv 2019
- [14]
-
[15]
Learning to Detect Fake Face Images in the Wild
URL http://arxiv.org/abs/1809.08754. Sandra Andres. V on Dosenfleisch zum unfreiwilligen Pornostar, August
work page internal anchor Pith review Pith/arXiv arXiv
-
[16]
URL https://www.spektrum.de/ video/fake-news-von-dosenfleisch-zum-unfreiwilligen-pornostar/1667784 . Patrick Tucker. The newest ai-enabled weapon: ‘deep-faking’ photos of the earth. De- fense One , March
-
[17]
URL https://www.defenseone.com/technology/2019/03/ next-phase-ai-deep-faking-whole-world-and-china-ahead/155944/print . Terrell Nowlin. Potential dangers of deep generative adversarial networks and future gen- erative models. Linkedin, June
work page 2019
-
[18]
URL https://www.linkedin.com/pulse/ potential-dangers-generated-adversarial-networks-future-nowlin/ . M. Rigaki and S. Garcia. Bringing a gan to a knife-fight: Adapting malware communication to avoid detection. In 2018 IEEE Security and Privacy Workshops (SPW), pages 70–75, May
work page 2018
-
[19]
doi: 10.1109/SPW.2018.00019. Dessa. Realtalk: This speech synthesis model our engineers built recreates a human voice perfectly, May
-
[20]
[Online; accessed 15-December-2019]. TinEye,
work page 2019
-
[21]
On the Measure of Intelligence
URL http://arxiv.org/abs/1911.01547. Arthur Allen. There is a reason we don’t know much about AI, September
work page internal anchor Pith review Pith/arXiv arXiv 1911
-
[22]
com/agenda/story/2019/09/16/artificial-intelligence-study-data-000956
URL https://www.politico. com/agenda/story/2019/09/16/artificial-intelligence-study-data-000956 . Isaac Asimov. Runaround. Doubleday, New York,
work page 2019
-
[23]
ISSN 0001-0782. doi: 10.1145/2838729. URL http://doi.acm.org/10.1145/2838729. Robert Salladay. Bill to ban fake guns in public gets assembly OK, August
-
[24]
URL https://www.latimes.com/ archives/la-xpm-2004-aug-19-me-bills19-story.html . Nathalie Smuha. Policy and investment recommendations for trustworthy AI, June
work page 2004
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.