pith. sign in

arxiv: 1912.08786 · v2 · submitted 2019-12-18 · 💻 cs.CY · cs.AI

Why we need an AI-resilient society

Pith reviewed 2026-05-24 14:52 UTC · model grok-4.3

classification 💻 cs.CY cs.AI
keywords AI resiliencelarge language modelssocietal riskscognitive sovereigntyinstitutional erosionsystemic riskshuman agencyforensic profiling
0
0 comments X

The pith

AI systems based on large language models show nine features that erode institutions and require a three-pillar resilience framework.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper traces how large language models turn natural language into a programming interface and applies forensic-psychology methods to profile them through nine features such as hallucinations, bias, sycophancy, knowledge without understanding, and cognitive atrophy. This produces a description of an entity that generates fluent but unreliable text, reflects user biases, and reduces the skills of people who rely on it. The analysis points to resulting damage in law, academia, journalism, and governance, which the paper addresses by outlining three pillars: cognitive sovereignty to keep independent judgment, measurable control to enforce standards, and partial autonomy to retain human oversight at key points.

Core claim

By applying a forensic-psychology profiling methodology to nine documented features of large language models—hallucinations, bias and toxicity, sycophancy and echo chambers, fabrication and credulity, knowledge without understanding, discontinuity and inability to learn from experience, jagged intelligence and scaling limits, shortcuts and fractured representations, and cognitive atrophy—the paper characterizes an entity that confabulates fluently, mirrors users' biases, possesses encyclopedic recall without causal understanding, and erodes the competence of those who depend on it, with implications for institutional erosion across law, academia, journalism, and democratic governance; it is,

What carries the argument

The forensic-psychology profiling methodology applied to the nine features of large language models, which produces the profile of the entity and motivates the three-pillar resilience framework of cognitive sovereignty, measurable control, and partial autonomy.

If this is right

  • Reliance on large language models risks eroding competence in law, academia, journalism, and democratic governance.
  • Cognitive sovereignty must be preserved to maintain independent human judgment.
  • Ethical commitments require translation into enforceable standards and red lines through measurable control.
  • Human agency needs to stay in place at critical decision points via partial autonomy.
  • The generational shift to language models as a programming interface carries consequences for how societies generate knowledge and govern themselves.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the nine features prove persistent, societies may need targeted training programs to rebuild skills lost to cognitive atrophy.
  • The framework could be piloted in one sector such as journalism to track whether measurable control reduces specific risks like fabrication.
  • The profile of knowledge without understanding may connect to questions in other domains about when automated systems should be barred from final decisions.
  • Future AI models with different architectures might require updates to the nine-feature list to keep the resilience pillars effective.

Load-bearing premise

The forensic-psychology profiling methodology can be transferred to AI systems and the nine features form a sufficient basis for identifying systemic risks that call for the three-pillar framework.

What would settle it

Longitudinal data from institutions that adopt large language models at scale showing no measurable drop in decision quality, error rates, or staff competence would test whether the profiled features produce the claimed erosion.

Figures

Figures reproduced from arXiv: 1912.08786 by Thomas Bartz-Beielstein.

Figure 1
Figure 1. Figure 1: A simplified chess computer. Hexapawn is played on a 3x3 board. to any problem. Strong AI is sometimes associated with understanding and (self-)consciousness, whereas weak AI is associated with data processing and adaptation. Super-intelligence is considered to be far away. AI tools are successful in many applications, especially in pattern recognition, e.g., speech recognition, image classifi￾cation, or b… view at source ↗
Figure 2
Figure 2. Figure 2: Johari window. A four paned window divides AI threats into four different types, as represented by the four quadrants. Known knowns are threats discussed in society, e.g., Google’s translation errors. Known unknowns are threats that we are aware of, but do not know when they will (ever) happen. SciFy stories about super-intelligence can be mentioned here. Unknown un￾knowns are threats that cannot be predic… view at source ↗
Figure 3
Figure 3. Figure 3: GANs creating counterfeit money. This process does not require any human interaction. Left: The first counterfeit note. Center: Improved counterfeit note. Right: Result after several billion iterations. Final counterfeit note [Kalina, 2018]. 1. The first computer (generator), say A, creates fake images, e.g., counterfeit money. 2. The second computer (discriminator), say B, says “that’s fake” and gives fee… view at source ↗
Figure 4
Figure 4. Figure 4: AI-resilient society in one figure. Unknown knows are transformed into known knowns. 6 Conclusion The AI genie is out of the bottle and we cannot put it back. We cannot trust data, images, audio, video, and identities any more. That’s why we urgently need an AI resilient society, which is based on openness and knowledge transfer. This process is illustrated in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
read the original abstract

Three generations of software have transformed the role of artificial intelligence in society. In the first, programmers wrote explicit logic; in the second, neural networks learned programs from data; in the third, large language models turn natural language itself into a programming interface. These shifts have consequences that reach far beyond computer science, reshaping how societies generate knowledge, make decisions, and govern themselves. While generative adversarial networks introduced the era of deepfakes and synthetic media, large language models have added an entirely new class of systemic risks. This report applies a forensic-psychology profiling methodology to characterize AI based on nine documented features: hallucinations, bias and toxicity, sycophancy and echo chambers, fabrication and credulity, knowledge without understanding, discontinuity and the inability to learn from experience, jagged intelligence and scaling limits, shortcuts and fractured representations, and cognitive atrophy. The resulting profile reveals an "entity" that confabulates fluently, mirrors its users' biases, possesses encyclopedic recall without causal understanding, and erodes the competence of those who depend on it. The implications extend to institutional erosion across law, academia, journalism, and democratic governance. To address these challenges, this report proposes a three-pillar framework for AI resilience: cognitive sovereignty, which preserves the capacity for independent judgment; measurable control, which translates ethical commitments into enforceable standards and red lines; and partial autonomy, which maintains human agency at critical decision points. This report is an updated and extended version of arXiv:1912.08786v1.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper argues that large language models constitute a third generation of software introducing novel systemic risks beyond prior AI eras. It applies a forensic-psychology profiling methodology to nine documented AI features—hallucinations, bias and toxicity, sycophancy and echo chambers, fabrication and credulity, knowledge without understanding, discontinuity and inability to learn from experience, jagged intelligence and scaling limits, shortcuts and fractured representations, and cognitive atrophy—to construct a profile of an 'AI entity' that confabulates fluently, mirrors user biases, lacks causal understanding, and erodes human competence. This profile is used to claim institutional erosion across law, academia, journalism, and democratic governance, motivating a three-pillar AI resilience framework of cognitive sovereignty, measurable control, and partial autonomy.

Significance. If the profiling methodology and feature-to-profile mapping were validated, the work could contribute a structured policy lens on AI dependence risks and institutional safeguards. However, the manuscript supplies no empirical data, derivations, error analysis, or adaptation protocol for the forensic-psychology transfer, rendering the central claims asserted rather than demonstrated; this limits significance to an opinion piece. No strengths such as machine-checked proofs, reproducible code, or falsifiable predictions are present.

major comments (3)
  1. [profiling methodology section] The section describing the forensic-psychology profiling methodology: no explicit adaptation protocol, diagnostic criteria, or validation steps are supplied for mapping human forensic techniques to non-human AI systems, which is load-bearing for the claim that the nine features yield a coherent 'entity' profile capable of supporting institutional-erosion conclusions.
  2. [nine features section] The enumeration of the nine features (hallucinations through cognitive atrophy): these are listed and invoked to define the profile without accompanying data, citations to primary studies, or analysis showing sufficiency and non-circularity; the profile and subsequent three-pillar framework follow directly from this selection, undermining the downstream policy implications.
  3. [three-pillar framework section] The derivation of the three-pillar resilience framework (cognitive sovereignty, measurable control, partial autonomy): the pillars are presented as direct responses to the unvalidated profile without any demonstrated mapping, test cases, or evidence that they address the specific features or mitigate the claimed erosion effects.
minor comments (2)
  1. [introduction] The abstract and introduction use the term 'entity' in quotation marks without a subsequent precise operational definition or boundary conditions.
  2. [throughout] No table or structured summary is provided to cross-reference the nine features against the three pillars or the claimed institutional impacts.

Simulated Author's Rebuttal

3 responses · 1 unresolved

We thank the referee for their detailed and substantive review. The manuscript is a conceptual position paper that uses an analogical lens from forensic psychology to synthesize known LLM limitations and motivate policy-oriented safeguards; it does not present new empirical data or a validated diagnostic method. We respond to each major comment below, maintaining that the paper's value lies in its framing rather than in empirical demonstration.

read point-by-point responses
  1. Referee: [profiling methodology section] The section describing the forensic-psychology profiling methodology: no explicit adaptation protocol, diagnostic criteria, or validation steps are supplied for mapping human forensic techniques to non-human AI systems, which is load-bearing for the claim that the nine features yield a coherent 'entity' profile capable of supporting institutional-erosion conclusions.

    Authors: We agree that no formal adaptation protocol, diagnostic criteria, or validation steps are supplied. The forensic-psychology framing functions as a heuristic analogy to organize and communicate the cumulative behavioral implications of the nine features, not as a literal transfer of clinical methods. The 'entity' profile is a rhetorical synthesis intended to make the risks legible to non-technical audiences; the institutional-erosion claims rest on the documented features themselves rather than on any validated profiling procedure. We do not claim scientific equivalence between human forensic profiling and AI characterization. revision: no

  2. Referee: [nine features section] The enumeration of the nine features (hallucinations through cognitive atrophy): these are listed and invoked to define the profile without accompanying data, citations to primary studies, or analysis showing sufficiency and non-circularity; the profile and subsequent three-pillar framework follow directly from this selection, undermining the downstream policy implications.

    Authors: Each of the nine features is drawn from independently reported findings in the NLP and AI-safety literature (hallucinations, bias, sycophancy, etc.). The manuscript treats them as documented rather than re-deriving them. While primary data and exhaustive citations are not reproduced, the selection is not circular: each issue has been established separately through empirical studies by multiple groups. The profile aggregates these known limitations to illustrate systemic effects; sufficiency is argued on the basis of their recurrence across current LLM deployments rather than through new statistical validation. revision: no

  3. Referee: [three-pillar framework section] The derivation of the three-pillar resilience framework (cognitive sovereignty, measurable control, partial autonomy): the pillars are presented as direct responses to the unvalidated profile without any demonstrated mapping, test cases, or evidence that they address the specific features or mitigate the claimed erosion effects.

    Authors: The three pillars are offered as high-level policy responses logically connected to the risks in the profile: cognitive sovereignty targets atrophy and lack of understanding; measurable control targets bias, fabrication, and sycophancy; partial autonomy addresses discontinuity and jagged intelligence. No test cases or quantitative mappings are supplied because the framework is proposed as an initial conceptual structure for further development, not as an evaluated intervention. The connections are presented as direct implications rather than empirically tested mitigations. revision: no

standing simulated objections not resolved
  • Supplying new empirical data, error analysis, or a validated adaptation protocol for the forensic-psychology transfer, as these lie outside the scope of a conceptual position paper.

Circularity Check

0 steps flagged

No circularity: features treated as external inputs; framework is normative proposal

full rationale

The paper enumerates nine documented features as the basis for applying a forensic-psychology methodology, then summarizes them into an entity profile and proposes a three-pillar framework as a response. The features are presented as pre-existing documented evidence rather than derived outputs, the profile is a direct aggregation of those inputs, and the resilience pillars are offered as policy recommendations without any reduction of conclusions to fitted parameters or self-referential definitions. No equations, uniqueness theorems, or self-citation chains appear in the provided text that would force the result by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The report depends on the transferability of forensic psychology methods to AI and the representativeness of the nine features; no free parameters or machine-checked elements are present.

axioms (1)
  • domain assumption Forensic psychology profiling methodology applies directly to AI systems to produce a valid psychological profile.
    Invoked when the paper states it applies the methodology to characterize AI based on the nine features.
invented entities (1)
  • AI as a unified 'entity' with a psychological profile no independent evidence
    purpose: To frame risks in terms of confabulation, bias mirroring, and competence erosion
    The abstract introduces this characterization without external falsifiable evidence for treating AI outputs as a single psychological entity.

pith-pipeline@v0.9.0 · 5795 in / 1293 out tokens · 33191 ms · 2026-05-24T14:52:31.710345+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 24 canonical work pages · 4 internal anchors

  1. [1]

    Mark Sullivan

    URL https://ai.googleblog.com/2016/01/alphago-mastering-ancient-game-of-go.html . Mark Sullivan. The 10 most important moments in AI (so far), September

  2. [2]

    com/90402503/the-10-most-important-moments-in-ai-so-far

    URL https://www.fastcompany. com/90402503/the-10-most-important-moments-in-ai-so-far . Nathan Benaich and Ian Hogarth. State of AI report 2019,

  3. [3]

    Toby Walsh

    URL https: //ainowinstitute.org. Toby Walsh. 2062: The World That AI Made. La Trobe University Press, 2018a. ISBN 1760640514, 9781760640514. Toby Walsh. Machines That Think. Prometeus, 2018b. Gary Marcus and Ernest Davis. Rebooting AI. Pantheon,

  4. [4]

    Prediction Machines

    D EC 18 2019 Ajay Agrawal, Joshua Gans, and Avi Goldfarb. Prediction Machines. Harvard Business Review Presss,

  5. [5]

    facebook.com/JimmyFallon/videos/10156679267778896/

    URL https://www. facebook.com/JimmyFallon/videos/10156679267778896/. Time to face decisions. Nature Machine Intelligence, 1(7):291–291,

  6. [6]

    URL https://doi.org/10.1038/s42256-019-0074-8

    doi: 10.1038/s42256-019-0074-8. URL https://doi.org/10.1038/s42256-019-0074-8 . BBC NEWS. Chinese AI caught out by face in bus ad, November

  7. [7]

    National Transportation Safety Board

    URLhttps: //www.theverge.com/2018/11/22/18107885/china-facial-recognition-mistaken-jaywalker . National Transportation Safety Board. PRELIMINARY REPORT – HIGHWAY – HWY18MH010,

  8. [8]

    URL http://news.mit.edu/2019/ object-recognition-dataset-stumped-worlds-best-computer-vision-models-1210# .XfXS8t7cC7c.twitter. Susan J. Winter. Who benefits? Commun. ACM, 62(7):23–25, June

  9. [9]

    doi: 10.1145/3332807

    ISSN 0001-0782. doi: 10.1145/3332807. URL http://doi.acm.org/10.1145/3332807. DoD News Briefing - Secretary Rumsfeld and Gen. Myers, February

  10. [10]

    Tero Karras, Samuli Laine, and Timo Aila

    URL https://www.lyrn.ai/2018/12/26/ a-style-based-generator-architecture-for-generative-adversarial-networks/ . Tero Karras, Samuli Laine, and Timo Aila. A Style-Based Generator Architecture for Generative Adversarial Networks,

  11. [11]

    A Style-Based Generator Architecture for Generative Adversarial Networks

    URL http://arxiv.org/abs/1812.04948. Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and Improving the Image Quality of StyleGAN,

  12. [12]

    Xander Steenbrugge

    URL http://arxiv.org/abs/1912.04958. Xander Steenbrugge. Face editing with generative adversarial networks, September

  13. [13]

    Large Scale GAN Training for High Fidelity Natural Image Synthesis

    D EC 18 2019 Andrew Brock, Jeff Donahue, and Karen Simonyan. Large Scale GAN Training for High Fidelity Natural Image Synthesis. arXiv e-prints, art. arXiv:1809.11096, Sep

  14. [14]

    URL https://www.spektrum.de/video/ doktor-whatson-wie-funktionieren-deepfakes/1670290# . Dev Nag. Generative adversarial networks (GANs) in 50 lines of code (PyTorch), February

  15. [15]

    Learning to Detect Fake Face Images in the Wild

    URL http://arxiv.org/abs/1809.08754. Sandra Andres. V on Dosenfleisch zum unfreiwilligen Pornostar, August

  16. [16]

    Patrick Tucker

    URL https://www.spektrum.de/ video/fake-news-von-dosenfleisch-zum-unfreiwilligen-pornostar/1667784 . Patrick Tucker. The newest ai-enabled weapon: ‘deep-faking’ photos of the earth. De- fense One , March

  17. [17]

    Terrell Nowlin

    URL https://www.defenseone.com/technology/2019/03/ next-phase-ai-deep-faking-whole-world-and-china-ahead/155944/print . Terrell Nowlin. Potential dangers of deep generative adversarial networks and future gen- erative models. Linkedin, June

  18. [18]

    URL https://www.linkedin.com/pulse/ potential-dangers-generated-adversarial-networks-future-nowlin/ . M. Rigaki and S. Garcia. Bringing a gan to a knife-fight: Adapting malware communication to avoid detection. In 2018 IEEE Security and Privacy Workshops (SPW), pages 70–75, May

  19. [19]

    doi: 10.1109/SPW.2018.00019. Dessa. Realtalk: This speech synthesis model our engineers built recreates a human voice perfectly, May

  20. [20]

    [Online; accessed 15-December-2019]. TinEye,

  21. [21]

    On the Measure of Intelligence

    URL http://arxiv.org/abs/1911.01547. Arthur Allen. There is a reason we don’t know much about AI, September

  22. [22]

    com/agenda/story/2019/09/16/artificial-intelligence-study-data-000956

    URL https://www.politico. com/agenda/story/2019/09/16/artificial-intelligence-study-data-000956 . Isaac Asimov. Runaround. Doubleday, New York,

  23. [23]

    doi: 10.1145/2838729

    ISSN 0001-0782. doi: 10.1145/2838729. URL http://doi.acm.org/10.1145/2838729. Robert Salladay. Bill to ban fake guns in public gets assembly OK, August

  24. [24]

    Nathalie Smuha

    URL https://www.latimes.com/ archives/la-xpm-2004-aug-19-me-bills19-story.html . Nathalie Smuha. Policy and investment recommendations for trustworthy AI, June