{"id":"8411a300-94ec-4fd9-a869-7def4801da45","arxiv_id":"2509.17878","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The AI Act's focus on frontier models and market impact wrongly excludes discrimination at scale and hallucinations, which should also count as systemic risks.","lead":"This paper argues that the EU AI Act defines systemic AI risk too narrowly, and that the Digital Services Act captures societal harms better. It proposes a broader framework that would treat large-scale discrimination and AI hallucinations as systemic risks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Discrimination analysis in §VIII.2.b concludes large-scale discrimination counts as systemic risk under the AI Act, contradicting the abstract's claim that it likely falls outside.","rationale":"The reader's weakest assumption identified interpretive uncertainty about whether 'most advanced models' will be interpreted narrowly or broadly. My stress-test reveals a different, more concrete issue: the paper's own detailed analysis of discrimination under the AI Act appears to conclude that discrimination is covered, which contradicts the central claim that the AI Act likely excludes it. This is an internal inconsistency rather than a mere dependence on a contested interpretive choice. While this does not invalidate the entire paper—the hallucinations example may still support the critique—it weakens the generality of the claim and requires revision. The verdict remains CONDITIONAL, but the condition should now include reconciling the discrimination analysis with the abstract and conclusion. I partially agree with the reader because the interpretive ambiguity is still relevant, but the strongest load-bearing concern is the internal contradiction.","tokens_in":53277,"tokens_out":5429,"duration_ms":45091,"concrete_test":"Perform a close reading of §VIII.2.b.i–ii and compare its conclusion with the abstract and §X. Specifically, verify whether the final sentence of §VIII.2.b.ii states that large-scale discrimination counts as systemic risk under the AI Act. If it does, the paper contains a direct contradiction that must be resolved before the central claim can be accepted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the AI Act's systemic risk definition likely excludes large-scale discrimination and systematic hallucinations because they are not specific to the most advanced GPAI models. However, in Section VIII.2.b.i, the paper argues that discrimination is 'a systemic threat specifically linked to the most advanced AI systems' and, after analyzing the Union market effect in VIII.2.b.ii, concludes: 'Hence, we conclude that large-scale discrimination does count as systemic risk.' This directly contradicts the abstract's assertion that discrimination 'may not fall under current legal definitions' and the conclusion (Section X) that 'the present framework likely does not treat them as such.' If discrimination does count, the paper's central critique that the AI Act excludes these risks loses one of its two flagship examples, reducing the claim to hallucinations alone. This internal inconsistency is more fundamental than the acknowledged interpretive ambiguity because it undermines the paper's own analysis, not just a contested legal reading. The paper must reconcile these statements or explicitly limit its claim to hallucinations.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops a conceptual framework for systemic risk from AI, digital platforms, and hybrid AI-platform systems, drawing on analogies from finance, climate change, and cybersecurity. It proposes four assessment criteria (scale/scope, collective harm, irreversibility, complexity) and four levels of AI systemic risk (single-model, multi-model, model-platform, model-institution). It then applies this framework to the EU AI Act and the Digital Services Act, arguing that the AI Act's definition of systemic risk in Art. 3(64)-(65) is too narrow because it ties systemic risk to 'most advanced' GPAI models and to 'significant impact on the Union market.' The paper claims that large-scale discrimination and systematic hallucinations, despite their potential to destabilize fundamental rights and democratic institutions, 'may not fall under current legal definitions,' while the DSA does a better job. The framework is tested on five examples: CBNR risks, discrimination at scale, hallucinations, cybersecurity, and environmental impacts, followed by policy proposals.","tokens_in":53512,"tokens_out":4076,"duration_ms":40727,"significance":"If the interpretive claims hold, the paper is a useful and policy-relevant contribution: it brings a structured multi-dimensional concept of systemic risk to AI governance, engages directly with the statutory text of the AI Act and the Code of Practice, acknowledges interpretive uncertainty, and proposes concrete reforms. The cross-domain synthesis and the four-level taxonomy are genuinely useful diagnostic tools. However, the paper's central critique is compromised by an internal inconsistency: its detailed analysis of discrimination concludes that large-scale discrimination does count as systemic risk under the AI Act, contradicting the abstract and conclusion. The hallucination analysis is more coherent but depends on a contested 'static' reading of 'most advanced' that the AI Office guidelines reject. These issues are fixable within the scope of a revision, and the underlying framework remains valuable.","major_comments":[{"comment":"The abstract states that discrimination at scale and systematic hallucinations 'may not fall under current legal definitions,' and the Conclusion (Section X) says 'the present framework likely does not treat them as such.' Yet Section VIII.2.b.i concludes that discrimination is 'a systemic threat specifically linked to the most advanced AI systems,' and VIII.2.b.ii concludes 'Hence, we conclude that large-scale discrimination does count as systemic risk.' This is a direct internal contradiction on one of the paper's two flagship examples. The authors must either revise the abstract/conclusion to limit the claim to hallucinations, or show why the section-level conclusion does not reflect the overall legal definition. As written, the paper's own analysis undermines its central thesis.","section":"§VIII.2.b.i–ii vs Abstract/Conclusion"},{"comment":"The specificity requirement in Art. 3(65) requires the risk to be 'specific to the high-impact capabilities' of GPAI models. The paper's empirical discussion of bias across model sizes finds the evidence 'mixed,' and the authors then rely on 'the scale of potential harm' and 'wider deployment, greater user trust, and persistent subtle biases' to conclude specificity. Scale of harm and deployment breadth are not the same as capability-specificity. If bias is present across less advanced models, the specificity requirement is not met merely because the most advanced models are more widely deployed. The paper needs to articulate a legal test that distinguishes 'specific to high-impact capabilities' from 'significant at scale' before concluding that discrimination is covered.","section":"§VIII.2.b.i"},{"comment":"The conclusion that hallucinations fall outside the AI Act's systemic risk definition rests on the 'static interpretation' of 'most advanced GPAI models' that the authors prefer. The paper itself notes that the GPAI guidelines of the AI Office 'have come out against' this interpretation (Section VI.4.b, para. 38) and that a 'second interpretation' would cover hallucinations and discrimination. Since the abstract and conclusion state the exclusion as a 'likely' outcome, the analysis should be explicitly conditional and should examine both readings when applying the five examples. If a court or the AI Office adopts the broader interpretation, the central critique is substantially muted.","section":"§VI.4.b and VIII.3.b"},{"comment":"The critique of the 'Union market' requirement is overstated. Art. 3(65) does not require a purely economic market effect; it requires a 'significant impact on the Union market due to their reach or due to actual or reasonably foreseeable negative effects on public health, safety, public security, fundamental rights, or society as a whole.' The paper's own discrimination analysis uses the 'due to negative effects' route. The authors should engage with this textual alternative before arguing that the market framing 'inevitably provides incomplete protection' and should adjust Section IX.5 accordingly.","section":"§VI.3.c"}],"minor_comments":[{"comment":"There are unresolved placeholders: '(xxx)' after Member State constitutional courts and 'Eigenberger etc. xxx' after CJEU judgments. These need to be completed.","section":"§V.1"},{"comment":"'Section VII.3' in the paragraph on AI summaries should be 'Section VIII.3'.","section":"§VII"},{"comment":"'45 Mio.' should be '45 million' for consistency.","section":"§VII"},{"comment":"The appendix is labelled 'Figure 0'; this should be renumbered sequentially.","section":"§VI.2 / Figure 0"},{"comment":"'Code of Conduct' should be 'Code of Practice' when referring to the GPAI Code.","section":"§VIII.5.b"},{"comment":"Two entries for Hacker (2024) are not distinguished as 2024a and 2024b; this creates citation ambiguity.","section":"Bibliography"},{"comment":"The citation '(debandt, 2000)' should be '(De Bandt & Hartmann, 2000)' for consistency with the bibliography.","section":"§II.1"},{"comment":"The reference to 'Part V.' should be to a specific section or part number; as written it is unclear.","section":"§VI.3.b"}],"recommendation":"major_revision","confidential_remarks":"The paper is an early draft with some unfinished references, but the main issue is substantive: the treatment of discrimination contradicts the paper's stated thesis. This is fixable, but the authors need to decide whether the central claim is about both discrimination and hallucinations or only hallucinations. The reliance on a contested statutory interpretation is acknowledged, but the abstract and conclusion should carry that qualification. The paper is otherwise a strong fit for this journal and would make a notable contribution after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a worthwhile paper, but it has a load-bearing internal contradiction. The abstract and conclusion say discrimination at scale and systematic hallucinations likely fall outside the AI Act's systemic risk definition. But Section VIII.2.b explicitly concludes that 'large-scale discrimination does count as systemic risk' under that same definition. The stress-test note is right. The paper never reconciles these statements, and the contradiction weakens the central critique: discrimination is one of the two flagship examples, so the exclusion claim reduces to hallucinations alone. This is more than interpretive ambiguity; it's the paper's own analysis disagreeing with its thesis.\n\nWhat it does well: the four-level taxonomy (single-model, multi-model, model-platform, model-institution) and four dimensions (scale, collective harm, irreversibility, complexity) are a clean synthesis of prior systemic risk work, applied specifically to the DSA and AI Act. The comparison is genuinely useful, especially the point that the DSA's reach-based threshold avoids the AI Act's 'most advanced' moving-target problem. The statutory analysis of Art. 3(64), 3(65), and 51 is careful and transparent about interpretive choices. The paper is honest about the contested static/dynamic readings; it just doesn't notice that its own discrimination analysis takes a different side than its conclusion.\n\nSoft spots: the discrimination analysis in VIII.2.b does heavy lifting with 'scale of potential harm' to show specificity to advanced models, while the empirical evidence on bias and model scale is mixed (the paper admits this). That's a minor overreach, but the contradiction is the real problem. Also, the manuscript is clearly an early draft: multiple 'xxx' placeholders and incomplete citations. That alone wouldn't sink it, but a referee should ask for cleanup.\n\nWho this is for: people working on EU AI governance, systemic risk theory, or platform regulation. It deserves a serious referee; it's a solid working paper, not a finished article.\n\nRecommendation: send it to peer review with a request for major revision to fix the discrimination contradiction and complete the references.","headline":"Useful legal-conceptual synthesis of AI systemic risk, but with a real internal contradiction: Section VIII.2.b concludes large-scale discrimination counts as systemic risk under the AI Act, contradicting the abstract and conclusion that it likely doesn't.","tokens_in":53977,"tokens_out":2355,"would_cite":true,"duration_ms":23120,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The EU AI Act's 'most advanced models' test likely excludes the two AI harms most likely to destabilize society: large-scale discrimination and systematic hallucinations.","keywords":["systemic risk","AI Act","Digital Services Act","general-purpose AI","hallucinations","algorithmic discrimination","AI governance","platform regulation"],"falsifier":"A formal Commission or Court of Justice determination that a mid-capability model's large-scale discriminatory outputs qualify as a systemic risk under Article 3(65), or an AI Office designation of a below-frontier model based on hallucinations, would refute the paper's central legal critique.","tokens_in":1009,"feed_emoji":"⚖️","tokens_out":1189,"duration_ms":211819,"temperature":0.7,"pith_summary":"The paper argues that 'systemic risk' in AI regulation is conceptually underdeveloped, and that the EU AI Act ties systemic risk to 'the most advanced' general-purpose AI models, thereby likely excluding two of the most dangerous failure modes: large-scale discrimination and systematic hallucinations. The authors build a four-criteria framework—scale, collective harm, irreversibility, complexity—and a four-level taxonomy of where AI systemic risk can arise, and use it to show the Digital Services Act captures these risks better than the AI Act. If right, the EU's flagship AI law would leave the most probable societal-scale AI harms outside its systemic-risk duties.","feed_headline":"Why the EU AI Act misses systemic AI harms","feed_subtitle":"Large-scale discrimination and hallucinations can destabilize society, yet the law targets only the most advanced models.","key_machinery":"A four-criteria definition of systemic risk—scale and scope of expected damage, collective harms exceeding the sum of individual impacts, potential irreversibility, and complexity/interconnectedness enabling cascading effects—combined with a four-level taxonomy of manifestation: single-model, multi-model correlated failure, model-platform integration, and model-institution integration. This framework is used both to critique the AI Act and to compare it with the DSA.","core_discovery":"The paper's central claim is that systemic risk from AI does not require frontier capability: widely deployed 'legacy' models can cause harms that cascade through society. The AI Act's Articles 3(64) and 3(65) define systemic risk as specific to the high-impact capabilities of the most advanced GPAI models, which the authors call a conceptual error that conflates the underlying risk with the scope of regulated models. Under their preferred static interpretation, generic capabilities like text generation do not count as high-impact, so discrimination at scale and hallucinations likely fall outside the systemic risk chapter, even though they meet the paper's four criteria and can destabilize f","pith_inferences":["A testable extension: compare whether systemic harm tracks deployment and reach rather than raw capability; if older open models at scale produce more discrimination or misinformation than newer fine-tuned frontier models, the AI Act's capability focus is empirically misplaced.","The paper's four-level taxonomy implies that AI agents interacting with each other are a distinct source of correlated failure that current regulation has no category for; this could be monitored directly.","The authors' preferred static interpretation carries a legal risk: courts may read 'most advanced' dynamically, which would undermine the regulatory stability the paper wants.","The four criteria could be applied to other digital phenomena, such as recommender-driven polarization, to test whether they qualify as systemic risks."],"forward_implications":["Large-scale discrimination and systematic hallucinations would become systemic risks in their own right, requiring providers to assess and mitigate them regardless of frontier status.","AI Act obligations would need to cover older, widely deployed models, not just those above the 10^25 FLOPs threshold.","Systemic risk should no longer be filtered through EU market impact; non-market harms such as environmental damage would count.","The DSA and AI Act must be coordinated through reciprocal risk analysis in hybrid AI-platform systems.","A revised AI Act could separate the concept of systemic risk from the thresholds that decide which models are regulated."],"fun_headline_variants":["EU AI Act misses systemic harms from everyday AI","Why the AI Act overlooks systemic AI failures","DSA sees systemic AI risk better than AI Act","EU law ignores AI harms that destabilize society","AI discrimination and hallucinations: the risks EU law misses"],"cache_read_input_tokens":55424,"weakest_assumption_plain":"The argument depends on the AI Act's 'most advanced models' being read narrowly and statically, so that generic text, image, and video generation are not treated as high-impact capabilities; under a broader reading, discrimination and hallucinations could be covered, and the critique would weaken.","fun_headline_variants_meta":{"raw":{"variants":["EU AI Act misses systemic harms from everyday AI","Why the AI Act overlooks systemic AI failures","DSA sees systemic AI risk better than AI Act","EU law ignores AI harms that destabilize society","AI discrimination and hallucinations: the risks EU law misses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000778,"raw_usage":{"total_tokens":3286,"prompt_tokens":766,"completion_tokens":2520,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":2447}},"tokens_in":510,"tokens_out":2520,"duration_ms":20194,"temperature":1.0,"reasoning_tokens":2447,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T15:48:10.713350+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A formal Commission or Court of Justice determination that a mid-capability model's large-scale discriminatory outputs qualify as a systemic risk under Article 3(65), or an AI Office designation of a below-frontier model based on hallucinations, would refute the paper's central legal critique.","supporting_citations":[],"review_version":1}