AI, Digital Platforms, and the New Systemic Risk

Atoosa Kasirzadeh; Lilian Edwards; Philipp Hacker

REVIEW 4 major objections 8 minor 1 cited by

AI, Digital Platforms, and the New Systemic Risk

T0 review · 4 major / 8 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The EU AI Act's 'most advanced models' test likely excludes the two AI harms most likely to destabilize society: large-scale discrimination and systematic hallucinations.

desk verdict Useful legal-conceptual synthesis of AI systemic risk, but with a real internal contradiction: Section VIII.2.b concludes large-scale discrimination counts as systemic risk under the AI Act, contradicting the abstract and conclusion that it likely doesn't. read the letter →

arxiv 2509.17878 v2 pith:C3JUNNWS submitted 2025-09-22 cs.CY

Philipp Hacker , Lilian Edwards , Atoosa Kasirzadeh This is my paper

classification cs.CY

keywords systemicriskAIActDigitalServicesgeneral-purposehallucinationsalgorithmicdiscriminationgovernanceplatformregulation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that 'systemic risk' in AI regulation is conceptually underdeveloped, and that the EU AI Act ties systemic risk to 'the most advanced' general-purpose AI models, thereby likely excluding two of the most dangerous failure modes: large-scale discrimination and systematic hallucinations. The authors build a four-criteria framework—scale, collective harm, irreversibility, complexity—and a four-level taxonomy of where AI systemic risk can arise, and use it to show the Digital Services Act captures these risks better than the AI Act. If right, the EU's flagship AI law would leave the most probable societal-scale AI harms outside its systemic-risk duties.

What carries the argument

A four-criteria definition of systemic risk—scale and scope of expected damage, collective harms exceeding the sum of individual impacts, potential irreversibility, and complexity/interconnectedness enabling cascading effects—combined with a four-level taxonomy of manifestation: single-model, multi-model correlated failure, model-platform integration, and model-institution integration. This framework is used both to critique the AI Act and to compare it with the DSA.

What would settle it

A formal Commission or Court of Justice determination that a mid-capability model's large-scale discriminatory outputs qualify as a systemic risk under Article 3(65), or an AI Office designation of a below-frontier model based on hallucinations, would refute the paper's central legal critique.

Watch

Extended reading notes

Core claim

The paper's central claim is that systemic risk from AI does not require frontier capability: widely deployed 'legacy' models can cause harms that cascade through society. The AI Act's Articles 3(64) and 3(65) define systemic risk as specific to the high-impact capabilities of the most advanced GPAI models, which the authors call a conceptual error that conflates the underlying risk with the scope of regulated models. Under their preferred static interpretation, generic capabilities like text generation do not count as high-impact, so discrimination at scale and hallucinations likely fall outside the systemic risk chapter, even though they meet the paper's four criteria and can destabilize f

Load-bearing premise

The argument depends on the AI Act's 'most advanced models' being read narrowly and statically, so that generic text, image, and video generation are not treated as high-impact capabilities; under a broader reading, discrimination and hallucinations could be covered, and the critique would weaken.

Editorial extensions

If this is right

Large-scale discrimination and systematic hallucinations would become systemic risks in their own right, requiring providers to assess and mitigate them regardless of frontier status.
AI Act obligations would need to cover older, widely deployed models, not just those above the 10^25 FLOPs threshold.
Systemic risk should no longer be filtered through EU market impact; non-market harms such as environmental damage would count.
The DSA and AI Act must be coordinated through reciprocal risk analysis in hybrid AI-platform systems.
A revised AI Act could separate the concept of systemic risk from the thresholds that decide which models are regulated.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension: compare whether systemic harm tracks deployment and reach rather than raw capability; if older open models at scale produce more discrimination or misinformation than newer fine-tuned frontier models, the AI Act's capability focus is empirically misplaced.
The paper's four-level taxonomy implies that AI agents interacting with each other are a distinct source of correlated failure that current regulation has no category for; this could be monitored directly.
The authors' preferred static interpretation carries a legal risk: courts may read 'most advanced' dynamically, which would undermine the regulatory stability the paper wants.
The four criteria could be applied to other digital phenomena, such as recommender-driven polarization, to test whether they qualify as systemic risks.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing you should know: this is a worthwhile paper, but it has a load-bearing internal contradiction. The abstract and conclusion say discrimination at scale and systematic hallucinations likely fall outside the AI Act's systemic risk definition. But Section VIII.2.b explicitly concludes that 'large-scale discrimination does count as systemic risk' under that same definition. The stress-test note is right. The paper never reconciles these statements, and the contradiction weakens the central critique: discrimination is one of the two flagship examples, so the exclusion claim reduces to hallucinations alone. This is more than interpretive ambiguity; it's the paper's own analysis disagreeing with its thesis.

What it does well: the four-level taxonomy (single-model, multi-model, model-platform, model-institution) and four dimensions (scale, collective harm, irreversibility, complexity) are a clean synthesis of prior systemic risk work, applied specifically to the DSA and AI Act. The comparison is genuinely useful, especially the point that the DSA's reach-based threshold avoids the AI Act's 'most advanced' moving-target problem. The statutory analysis of Art. 3(64), 3(65), and 51 is careful and transparent about interpretive choices. The paper is honest about the contested static/dynamic readings; it just doesn't notice that its own discrimination analysis takes a different side than its conclusion.

Soft spots: the discrimination analysis in VIII.2.b does heavy lifting with 'scale of potential harm' to show specificity to advanced models, while the empirical evidence on bias and model scale is mixed (the paper admits this). That's a minor overreach, but the contradiction is the real problem. Also, the manuscript is clearly an early draft: multiple 'xxx' placeholders and incomplete citations. That alone wouldn't sink it, but a referee should ask for cleanup.

Who this is for: people working on EU AI governance, systemic risk theory, or platform regulation. It deserves a serious referee; it's a solid working paper, not a finished article.

Recommendation: send it to peer review with a request for major revision to fix the discrimination contradiction and complete the references.

Referee Report

4 major / 8 minor

Summary. The paper develops a conceptual framework for systemic risk from AI, digital platforms, and hybrid AI-platform systems, drawing on analogies from finance, climate change, and cybersecurity. It proposes four assessment criteria (scale/scope, collective harm, irreversibility, complexity) and four levels of AI systemic risk (single-model, multi-model, model-platform, model-institution). It then applies this framework to the EU AI Act and the Digital Services Act, arguing that the AI Act's definition of systemic risk in Art. 3(64)-(65) is too narrow because it ties systemic risk to 'most advanced' GPAI models and to 'significant impact on the Union market.' The paper claims that large-scale discrimination and systematic hallucinations, despite their potential to destabilize fundamental rights and democratic institutions, 'may not fall under current legal definitions,' while the DSA does a better job. The framework is tested on five examples: CBNR risks, discrimination at scale, hallucinations, cybersecurity, and environmental impacts, followed by policy proposals.

Significance. If the interpretive claims hold, the paper is a useful and policy-relevant contribution: it brings a structured multi-dimensional concept of systemic risk to AI governance, engages directly with the statutory text of the AI Act and the Code of Practice, acknowledges interpretive uncertainty, and proposes concrete reforms. The cross-domain synthesis and the four-level taxonomy are genuinely useful diagnostic tools. However, the paper's central critique is compromised by an internal inconsistency: its detailed analysis of discrimination concludes that large-scale discrimination does count as systemic risk under the AI Act, contradicting the abstract and conclusion. The hallucination analysis is more coherent but depends on a contested 'static' reading of 'most advanced' that the AI Office guidelines reject. These issues are fixable within the scope of a revision, and the underlying framework remains valuable.

major comments (4)

[§VIII.2.b.i–ii vs Abstract/Conclusion] The abstract states that discrimination at scale and systematic hallucinations 'may not fall under current legal definitions,' and the Conclusion (Section X) says 'the present framework likely does not treat them as such.' Yet Section VIII.2.b.i concludes that discrimination is 'a systemic threat specifically linked to the most advanced AI systems,' and VIII.2.b.ii concludes 'Hence, we conclude that large-scale discrimination does count as systemic risk.' This is a direct internal contradiction on one of the paper's two flagship examples. The authors must either revise the abstract/conclusion to limit the claim to hallucinations, or show why the section-level conclusion does not reflect the overall legal definition. As written, the paper's own analysis undermines its central thesis.
[§VIII.2.b.i] The specificity requirement in Art. 3(65) requires the risk to be 'specific to the high-impact capabilities' of GPAI models. The paper's empirical discussion of bias across model sizes finds the evidence 'mixed,' and the authors then rely on 'the scale of potential harm' and 'wider deployment, greater user trust, and persistent subtle biases' to conclude specificity. Scale of harm and deployment breadth are not the same as capability-specificity. If bias is present across less advanced models, the specificity requirement is not met merely because the most advanced models are more widely deployed. The paper needs to articulate a legal test that distinguishes 'specific to high-impact capabilities' from 'significant at scale' before concluding that discrimination is covered.
[§VI.4.b and VIII.3.b] The conclusion that hallucinations fall outside the AI Act's systemic risk definition rests on the 'static interpretation' of 'most advanced GPAI models' that the authors prefer. The paper itself notes that the GPAI guidelines of the AI Office 'have come out against' this interpretation (Section VI.4.b, para. 38) and that a 'second interpretation' would cover hallucinations and discrimination. Since the abstract and conclusion state the exclusion as a 'likely' outcome, the analysis should be explicitly conditional and should examine both readings when applying the five examples. If a court or the AI Office adopts the broader interpretation, the central critique is substantially muted.
[§VI.3.c] The critique of the 'Union market' requirement is overstated. Art. 3(65) does not require a purely economic market effect; it requires a 'significant impact on the Union market due to their reach or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or society as a whole.' The paper's own discrimination analysis uses the 'due to negative effects' route. The authors should engage with this textual alternative before arguing that the market framing 'inevitably provides incomplete protection' and should adjust Section IX.5 accordingly.

minor comments (8)

[§V.1] There are unresolved placeholders: '(xxx)' after Member State constitutional courts and 'Eigenberger etc. xxx' after CJEU judgments. These need to be completed.
[§VII] 'Section VII.3' in the paragraph on AI summaries should be 'Section VIII.3'.
[§VII] '45 Mio.' should be '45 million' for consistency.
[§VI.2 / Figure 0] The appendix is labelled 'Figure 0'; this should be renumbered sequentially.
[§VIII.5.b] 'Code of Conduct' should be 'Code of Practice' when referring to the GPAI Code.
[Bibliography] Two entries for Hacker (2024) are not distinguished as 2024a and 2024b; this creates citation ambiguity.
[§II.1] The citation '(debandt, 2000)' should be '(De Bandt & Hartmann, 2000)' for consistency with the bibliography.
[§VI.3.b] The reference to 'Part V.' should be to a specific section or part number; as written it is unclear.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the paper's legal and conceptual analysis is self-contained; self-citations are background, and the main internal problem is an inconsistency, not circularity.

full rationale

The paper's central claims are statutory-interpretive and conceptual rather than derived from fitted parameters or from the authors' own conclusions. The four-dimension framework in Section IV is distilled from external literature (finance, climate, cybersecurity; e.g., Kaufman, Schwarcz, OECD, IPCC, ESRB) and then applied to examples; this is framework application, not circular derivation. The critique of the AI Act is supported directly by the text of Arts. 3(64), 3(65), 51, Annex XIII, the Code of Practice, and the AI Office guidelines, all of which are independent of the authors. The self-citations (Hacker & Holweg 2025, Hacker 2024, Hacker et al. 2025, Kasirzadeh 2024/2025, Uuk et al. 2024) provide background or supplementary support; even the 'conventional vision models' counterexample is accompanied by independent empirical literature (Bender et al., Weidinger et al., Jeong et al., Kumar et al.). No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled via citation. One notable issue is an internal inconsistency: the abstract and conclusion state that large-scale discrimination 'may not fall under current legal definitions,' but Section VIII.2.b.ii concludes that 'large-scale discrimination does count as systemic risk' under the AI Act. This is a coherence/correctness problem that weakens the central claim, but it is not circularity, because the Section VIII analysis is an independent application of the statutory definition to the facts, not an input to that definition. The paper would be stronger if it reconciled this contradiction, but the derivation chain itself is not circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The analysis rests on the transferability of systemic risk concepts across domains, a particular statutory interpretation, and a legal feasibility judgement.

assumptions (4)

domain assumption Systemic risk as developed in finance, climate, and cybersecurity can be meaningfully applied to AI and platform systems.
The paper borrows the concept from those fields (Sections II, IV) and assumes it transfers to digital ecosystems.
domain assumption The preferred 'static' interpretation of 'most advanced GPAI models' in Art. 3(64) AI Act is correct, so that systemic risk is tied to a fixed capability threshold.
If courts adopted the dynamic interpretation instead, the claim that legacy models fall outside systemic risk would weaken. The authors discuss both in Section VI.4.
domain assumption Large-scale discrimination and systematic hallucinations can produce collective and irreversible harms comparable to CBRN and cyber-offense risks.
The paper asserts this in Sections VIII.2 and VIII.3 based on literature examples; it is a value-laden premise underlying the reform proposal.
domain assumption EU law (Art. 114 TFEU and other bases) permits regulation of non-market systemic risks under the Tobacco Advertising test.
Section IX.5 argues reform is legally feasible; this is a contested legal position.

how reviews work

0 comments

Cite this review

Pith. "Pith review of AI, Digital Platforms, and the New Systemic Risk." pith.science (2026). https://pith.science/paper/C3JUNNWS

@misc{pith2026250917878,
  author       = {Pith},
  title        = {Pith review of: AI, Digital Platforms, and the New Systemic Risk},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C3JUNNWS}},
  note         = {Machine review of arXiv:2509.17878}
}

read the original abstract

As artificial intelligence (AI) becomes increasingly embedded in digital, social, and institutional infrastructures, and AI and platforms are merged into hybrid structures, systemic risk has emerged as a critical but undertheorized challenge. In this paper, we develop a rigorous framework for understanding systemic risk in AI, platform, and hybrid system governance, drawing on insights from finance, complex systems theory, climate change, and cybersecurity - domains where systemic risk has already shaped regulatory responses. We argue that recent legislation, including the EU's AI Act and Digital Services Act (DSA), invokes systemic risk but relies on narrow or ambiguous characterizations of this notion, sometimes reducing this risk to specific capabilities present in frontier AI models, or to harms occurring in economic market settings. The DSA, we show, actually does a better job at identifying systemic risk than the more recent AI Act. Our framework highlights novel risk pathways, including the possibility of systemic failures arising from the interaction of multiple AI agents. We identify four levels of AI-related systemic risk and emphasize that discrimination at scale and systematic hallucinations, despite their capacity to destabilize institutions and fundamental rights, may not fall under current legal definitions, given the AI Act's focus on frontier model capabilities. We then test the DSA, the AI Act, and our own framework on five key examples, and propose reforms that broaden systemic risk assessments, strengthen coordination between regulatory regimes, and explicitly incorporate collective harms.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Regulating autonomous and agentic AI
cs.AI 2026-07 conditional novelty 5.0 of 10

Agentic AI's unpredictability means regulators must extend liability and oversight across the AI supply chain and adopt real-time, AI-assisted supervision.

Reference graph

Works this paper leans on

10 extracted references · 1 linked inside Pith · cited by 1 Pith paper

[1]

dirty bombs

Chemical, Biological, Nuclear, and Radiological Risks CBNR risks encompass threats that arise from the malicious use or accidental release of chemical, biological, nuclear, or radiological materials. Chemical risks involve toxic substances that can cause harm through inhalation, ingestion, or skin contact. Biological risks include pathogens such as bacter...
[2]

any actual or foreseeable negative effects for the exercise of fundamental rights

Discrimination at Scale Discrimination at scale refers to systematic differential treatment of individuals or groups through digital platforms and AI systems based on protected characteristics such as race, gender, religion, disability, sexual orientation, or nationality. This phenomenon manifests through algorithmic decision-making systems that perpetuat...

2022
[3]

Hallucinations

Information Pollution Through Hallucinations “Hallucinations” in AI systems refer to outputs that contain factually false or misleading information presented as truth, or correct information attributed to incorrect sources (Binns & Edwards, 2025; Magesh et al., 2025). These errors manifest in two primary forms: complete fabrications where the AI generates...

2025
[4]

unstolen,

Cybersecurity Cybersecurity has become a critical national and economic security imperative in an era marked by intensified geopolitical competition and the proliferation of cyber capabilities among state and non-state actors. Nation-states deploy sophisticated persistent threats against critical infrastructure, while criminal organizations execute ransom...

2024
[5]

systemic risks

Climate and Environmental Impacts While AI has a significant, yet hitherto untapped potential for reducing emissions and energy usage around a range of crucial fields (e.g., housing and transportation) (Rolnick et al., 2022; Taddeo et al., 2021), the current tendency is for AI specifically to generate more demand for energy (IEA, 2025) and, where it is no...

2022
[6]

This risk-based approach manifests consistently across frameworks, though with varying implementation strategies

Risk-Based Regulatory Frameworks without Corresponding Liability All three regulatory domains embrace a fundamental principle: regulatory obligations should scale with risk magnitude. This risk-based approach manifests consistently across frameworks, though with varying implementation strategies. Financial regulation pioneered this approach through enhanc...

2023
[7]

high-impact capabilities

The Definition of Systemic Risk: beyond the Most Advanced Models The three frameworks adopt different approaches to defining systemic risk. Financial regulation provides precise definitions linking systemic risk to interconnectedness, substitutability, and contagion potential. The AI Act offers a formal definition in Article 3(65), though one complicated ...
[8]

systemic risk

Comparative Risk Characteristics While all three domains address systemic risks, the nature of these risks differs fundamentally. Financial systemic risks manifest through economic contagion, liquidity crises, and asset price collapses. Recovery, while painful, follows semi-established patterns through recapitalization, stimulus, and structural reform. Pl...

2023

Show all 10 references

[9]

Banking regulation uses quantitative thresholds based on assets, interconnectedness, and market share – objective financial indicators that clearly delineate systemic importance

Application Thresholds and Scoping Mechanisms Each framework employs distinct mechanisms to identify which entities bear systemic risk obligations. Banking regulation uses quantitative thresholds based on assets, interconnectedness, and market share – objective financial indic...
[10]

most advanced

Recommendations for Regulatory Evolution: Beyond the Market Paradigm Future revisions should abandon the artificial constraint on the (Union) market. The protection of fundamental rights and interests, democratic values, and environmental sustainability cannot be reduced and s...

2024 arXiv

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.

[1] [1]

dirty bombs

Chemical, Biological, Nuclear, and Radiological Risks CBNR risks encompass threats that arise from the malicious use or accidental release of chemical, biological, nuclear, or radiological materials. Chemical risks involve toxic substances that can cause harm through inhalation, ingestion, or skin contact. Biological risks include pathogens such as bacter...

[2] [2]

any actual or foreseeable negative effects for the exercise of fundamental rights

Discrimination at Scale Discrimination at scale refers to systematic differential treatment of individuals or groups through digital platforms and AI systems based on protected characteristics such as race, gender, religion, disability, sexual orientation, or nationality. This phenomenon manifests through algorithmic decision-making systems that perpetuat...

2022

[3] [3]

Hallucinations

Information Pollution Through Hallucinations “Hallucinations” in AI systems refer to outputs that contain factually false or misleading information presented as truth, or correct information attributed to incorrect sources (Binns & Edwards, 2025; Magesh et al., 2025). These errors manifest in two primary forms: complete fabrications where the AI generates...

2025

[4] [4]

unstolen,

Cybersecurity Cybersecurity has become a critical national and economic security imperative in an era marked by intensified geopolitical competition and the proliferation of cyber capabilities among state and non-state actors. Nation-states deploy sophisticated persistent threats against critical infrastructure, while criminal organizations execute ransom...

2024

[5] [5]

systemic risks

Climate and Environmental Impacts While AI has a significant, yet hitherto untapped potential for reducing emissions and energy usage around a range of crucial fields (e.g., housing and transportation) (Rolnick et al., 2022; Taddeo et al., 2021), the current tendency is for AI specifically to generate more demand for energy (IEA, 2025) and, where it is no...

2022

[6] [6]

This risk-based approach manifests consistently across frameworks, though with varying implementation strategies

Risk-Based Regulatory Frameworks without Corresponding Liability All three regulatory domains embrace a fundamental principle: regulatory obligations should scale with risk magnitude. This risk-based approach manifests consistently across frameworks, though with varying implementation strategies. Financial regulation pioneered this approach through enhanc...

2023

[7] [7]

high-impact capabilities

The Definition of Systemic Risk: beyond the Most Advanced Models The three frameworks adopt different approaches to defining systemic risk. Financial regulation provides precise definitions linking systemic risk to interconnectedness, substitutability, and contagion potential. The AI Act offers a formal definition in Article 3(65), though one complicated ...

[8] [8]

systemic risk

Comparative Risk Characteristics While all three domains address systemic risks, the nature of these risks differs fundamentally. Financial systemic risks manifest through economic contagion, liquidity crises, and asset price collapses. Recovery, while painful, follows semi-established patterns through recapitalization, stimulus, and structural reform. Pl...

2023