What Are The Risks of Living in a GenAI Synthetic Reality? The Generative AI Paradox

Emilio Ferrara

REVIEW 3 major objections 3 minor 18 references

What Are The Risks of Living in a GenAI Synthetic Reality? The Generative AI Paradox

T0 review · 3 major / 3 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that generative AI's most consequential risk is the creation of personalized synthetic realities, which could fragment shared truths and drive society to treat digital content as inherently fake.

desk verdict A timely, clearly written viewpoint on GenAI-driven 'personalized synthetic realities,' but it is an argument, not a research result—worth review as an essay, not as evidence. read the letter →

arxiv 2411.08250 v1 pith:4EEK66LF submitted 2024-11-12 cs.SI

Emilio Ferrara This is my paper · ORCID

classification cs.SI

keywords generativeAIpersonalizedsyntheticrealitymisinformationrisktaxonomydeepfakesepistemictrustgovernance

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the most consequential risk of generative AI is not economic disruption or copyright but the creation of personalized synthetic realities: AI-generated worlds tailored to each person's desires or to external manipulation. Because GenAI makes synthetic content cheap, realistic, customizable, and hard to detect, people may no longer be able to tell what is real from what is generated. The result, the author contends, would be a fragmented understanding of shared truths, with different people inhabiting different factual worlds. The paper names a paradox: society may collectively adopt the assumption that all digital content is fake, leaving only lived, directly witnessed experience as real.

What carries the argument

The load-bearing mechanism is the taxonomy of GenAI risks and harms drawn from the paper's earlier framework, which links specific intents such as dishonesty, propaganda, and deception to categories of harm such as personal loss, financial and economic damage, information manipulation, and socio-technical and infrastructural risk. The paper couples this taxonomy with an inventory of seven GenAI-specific properties, namely commoditization, scale, customization, hyper-targeting, detection lag, eroding trust, and realism, to argue that synthetic content can become a personalized filter over reality. The named endpoint is the Generative AI Paradox: if digital content is presumed fake by default, then only lived, directly witnessed experience counts as real, and the machinery shows how that endpoint is reached through an arms race between creation and detection that detection tools are currently losing.

What would settle it

A longitudinal comparison of matched populations with high versus low exposure to personalized GenAI content, measuring divergence in beliefs about shared factual events such as election outcomes or public-health facts, would settle the claim: if divergence does not grow beyond pre-GenAI baselines, the predicted fragmentation of shared truth does not materialize.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that GenAI introduces risks that are not merely extensions of past misinformation: its ease, speed, sophistication, and personalization make synthetic realities a qualitatively new phenomenon. This claim is developed through a classification of harms, covering personal loss, financial and economic damage, information manipulation, and socio-technical and infrastructural risk, and through seven properties of GenAI, including low cost, mass production, open-source customization, hyper-targeting, detection difficulty, eroding trust, and hyper-realism. Together these blur the boundary between real and synthetic worlds. The endpoint is the Generative AI Paradox: society may come to assume that digital content is inherently fake while treating only lived or directly witnessed experience as real, which would alter the very fabric of collective reality and undermine institutions that depend on shared evidence.

Load-bearing premise

The argument rests on the premise that people will not be able to reliably tell synthetic content from real content in everyday settings, so personalized synthetic realities can become a dominant mode of experience; if detection or adaptation keeps pace, the predicted fragmentation of shared truth may not occur.

Editorial extensions

If this is right

If personalized synthetic realities become common, people's perceptions of shared events will diverge, weakening the common factual ground that democratic deliberation and legal evidence rely on.
The Generative AI Paradox implies that trust in genuine digital evidence will also erode, so even real images, recordings, or documents may be dismissed as fake.
Because GenAI lowers cost and enables hyper-targeting, malicious actors, from scammers to governments, can tailor synthetic content to individuals or communities, deepening echo chambers and polarization.
Escapism into individually satisfying synthetic worlds could increase social isolation and reduce engagement with a shared physical and social reality.
Given the detection arms race, technical fixes alone are insufficient; the paper argues for coordinated governance, transparency, education, and public awareness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable corollary the paper leaves implicit is that epistemic fragmentation should be measurable over time, for example as divergence in factual beliefs across matched populations with different GenAI exposure; the paper offers no operationalization of this measure.
The paradox suggests a tipping dynamic: once default skepticism toward digital content becomes the norm, even authentic evidence loses force, so the marginal cost of verifying any claim rises; this feedback loop is not quantified in the paper.
The argument implies that provenance and watermarking may need to be prioritized in high-stakes domains such as legal evidence, journalism, and electoral information before synthetic content saturates them, since the paper's own logic shows detection lags creation.
One could extend the framework to test whether exposure to synthetic realities increases escapism or social isolation using behavioral data, a connection the paper raises but does not pursue experimentally.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a viewpoint essay, not a research paper. There is no new data, derivation, or falsifiable claim. What is genuinely new is the phrase "personalized synthetic realities" and the closing paradox—that we may collectively start treating all digital content as fake and trust only lived experience. That framing is worth having in the conversation.

What the paper does well: it gives a compact, readable taxonomy of GenAI risk categories (personal loss, financial, information manipulation, socio-technical), and it makes a plausible case that GenAI is not just more of the same misinformation. The seven bullet points on cost, scale, customization, hyper-targeting, detection, trust erosion, and realism are a useful checklist for policymakers.

The soft spots are real but not fatal for a viewpoint. The load-bearing premise—that GenAI content is realistic enough and pervasive enough that people cannot tell it apart from real experience—is asserted with three anecdotes (fake IDs, a synthetic handshake, an optical illusion) and no measurement of realism, prevalence, or detection rates. That means the central fragmentation risk is a conditional warning, not an established empirical finding. That's fine for a perspective piece, but it should be labeled as such.

There are also some careless production issues: an unresolved citation placeholder ("unprecedented scale [? ]") and figure references to "Figure 2" and "Figure 3" for panels that appear to be part of Figure 1. A referee should flag these.

The reliance on the author's own taxonomy [8] is fine—it is their prior work, and the taxonomy is relevant—but it makes the essay not fully self-contained. I don't see a circularity problem; the author isn't deriving the risk from his own taxonomy, just citing it.

Bottom line: not a scientific contribution, but a clear and timely opinion piece. If the venue accepts viewpoint essays, it deserves a real review rather than a desk reject. The reviewer should ask for a few concrete references to evidence on realism or detection, and fix the dangling citation and figure numbering. I would not cite it as evidence, but I'd point people to it as a framing piece.

Referee Report

3 major / 3 minor

Summary. The paper is a viewpoint essay arguing that generative AI (GenAI) introduces a novel class of societal risk through 'personalized synthetic realities'—individually tailored synthetic content that blurs the boundary between real and fabricated experience. The author builds on a prior taxonomy of GenAI harms (personal loss, financial and economic damage, information manipulation, and socio-technical/infrastructural risks) and contends that GenAI's ease, scale, customization, hyper-targeting, detection difficulty, trust erosion, and realism make the risk of fragmented shared truths qualitatively different from earlier misinformation technologies. The argument is illustrated with three examples—a Reddit proof-of-concept for identity documents, a synthetic image of a fictitious political handshake, and an optical illusion—and the paper closes with a call for ethical frameworks, governance, and public awareness.

Significance. If the argument holds, the paper identifies a plausible and under-appreciated risk that is timely given rapid advances in generative media. Its strength is a clear, accessible articulation of why personalized synthetic realities deserve distinct attention, and it offers a taxonomy that could structure future empirical research. The paper is also commendable for making a falsifiable conditional claim: if synthetic content becomes indistinguishable from real content and dominates media diets, then shared truths may fragment. However, as a perspective piece, it does not establish the antecedent; the load-bearing premise of hyper-realism and pervasiveness is asserted rather than demonstrated. The manuscript would be strengthened by explicitly framing the argument as a scenario analysis and by engaging with the extensive literature on filter bubbles and selective exposure to support the claim of uniqueness.

major comments (3)

[What You Can't Tell Apart Can Harm You] The central claim that GenAI content is hyper-realistic enough to blur the line between real and synthetic worlds, and that personalized synthetic realities will become a dominant mode of experience, is asserted rather than established. The three examples (Reddit proof-of-concept, synthetic handshake, optical illusion) are illustrative but do not measure realism, prevalence, detection rates, or the share of an individual's media diet that is synthetic. Because this antecedent is load-bearing for the predicted fragmentation of shared truths, the argument needs either systematic empirical support or an explicit reframing as a conditional risk scenario. As written, the claim cannot be evaluated on the evidence provided.
[What You Can't Tell Apart Can Harm You, bullet 'Scale and Mass Production'] The phrase 'unprecedented scale [? ]' contains an unresolved placeholder citation. This is not a mere typographical issue: the claim that GenAI enables manipulation 'on an unprecedented scale' is a key part of the argument that this risk is distinct from prior misinformation technologies. The manuscript must either supply the relevant citation or remove the quantitative claim.
[What You Can't Tell Apart Can Harm You] The paper asserts that GenAI's 'ease, speed, and sophistication' is 'unparalleled' but does not support this with comparison to prior technologies. The manuscript does not engage with the substantial literature on filter bubbles, selective exposure, and personalized news, which already describes a fragmented understanding of shared truths. To make the case that GenAI presents a novel risk, the author should specify the mechanisms by which GenAI goes beyond these existing phenomena, rather than simply labeling the risk 'unprecedented.'

minor comments (3)

[Implications of GenAI Synthetic Realities] The text cites 'Figure 2, Top Right' and 'Figure 3, Bottom Right' for panels that appear to be part of Figure 1 (Left, Top Right, Bottom Right). Please correct the figure numbering or citations.
[Concluding paragraph] The closing 'paradox'—that society may assume digital content is inherently fake—is intriguing but underdeveloped; consider expanding it in the body or linking it more explicitly to the earlier discussion of eroding trust.
[Introduction and Taxonomy section] The taxonomy is presented as a summary of the author's prior work [8]; for readers unfamiliar with that work, a slightly fuller description of the four categories and their definitions would make the argument more self-contained.

Circularity Check

0 steps flagged · score 2.0 of 10

No reduction-by-construction circularity: the central warning about personalized synthetic realities is argued from qualitative risk factors and examples, not derived from the paper's own inputs; self-citations are contextual rather than load-bearing.

full rationale

This is a Viewpoint essay rather than a derivation or an empirical study, so there is no chain of equations, fitted parameters, or constructed predictions to invert. The central claim that GenAI may create personalized synthetic realities that fragment shared truths is supported by illustrative examples (identity-document proofs-of-concept, a synthetic handshake image, an optical illusion) and by qualitative risk factors such as cost, scale, customization, and detection challenges, rather than by a quantity computed from the paper's own inputs. The taxonomy 'as proposed in [8]' is an organizational frame borrowed from the author's prior work, but the paper's specific warning is not derived from that taxonomy; the taxonomy could be revised without invalidating the risk argument. The other self-citations (HUMANS Lab working papers and prior memos) refer to the team's data-collection and monitoring efforts and are not the evidentiary basis for the main risk claim. The genuine weaknesses are evidentiary and conceptual — 'unprecedented scale [?]' is left unresolved, the realism/distinguishability premise is asserted rather than measured, and Figures 2 and 3 are cited for panels that appear in Figure 1 — but these are completeness and rigor concerns, not circularity. No step of the argument reduces by construction to its own inputs, so the circularity burden is low; the nonzero score reflects the paper's heavy reliance on the author's own prior taxonomy and working papers as framing references, not any circular derivation.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities apply because the paper offers no quantitative model or testable artifact. The central argument rests on three unproven behavioral assumptions about realism, adoption, and social non-adaptation; these are domain assumptions rather than mathematical axioms.

assumptions (3)

domain assumption GenAI-generated content will be hyper-realistic enough that individuals cannot reliably distinguish synthetic from real content at scale.
This premise is stated in the section 'What you can't tell apart can harm you' (Realism and Blurring of Boundaries) and is load-bearing; without it, the shift toward synthetic realities would not be psychologically or socially consequential.
domain assumption Personalized synthetic realities will become a significant mode of content consumption rather than a marginal niche.
The paper asserts that GenAI could foster alternative synthetic realities customized to individual preferences, but provides no adoption or behavioral evidence that such personalized consumption will outcompete shared media experiences.
domain assumption Society will not adapt to synthetic content as it adapted to earlier technological upheavals.
The author notes that 'history suggests that society can adapt' but then argues the scale of GenAI is different; this implicit non-adaptation assumption is necessary for the predicted fragmentation to occur.

how reviews work

0 comments

Cite this review

Pith. "Pith review of What Are The Risks of Living in a GenAI Synthetic Reality? The Generative AI Paradox." pith.science (2026). https://pith.science/paper/4EEK66LF

@misc{pith2026241108250,
  author       = {Pith},
  title        = {Pith review of: What Are The Risks of Living in a GenAI Synthetic Reality? The Generative AI Paradox},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4EEK66LF}},
  note         = {Machine review of arXiv:2411.08250}
}

abstract

Generative AI (GenAI) technologies possess unprecedented potential to reshape our world and our perception of reality. These technologies can amplify traditionally human-centered capabilities, such as creativity and complex problem-solving in socio-technical contexts. By fostering human-AI collaboration, GenAI could enhance productivity, dismantle communication barriers across abilities and cultures, and drive innovation on a global scale. Yet, experts and the public are deeply divided on the implications of GenAI. Concerns range from issues like copyright infringement and the rights of creators whose work trains these models without explicit consent, to the conditions of those employed to annotate vast datasets. Accordingly, new laws and regulatory frameworks are emerging to address these unique challenges. Others point to broader issues, such as economic disruptions from automation and the potential impact on labor markets. Although history suggests that society can adapt to such technological upheavals, the scale and complexity of GenAI's impact warrant careful scrutiny. This paper, however, highlights a subtler, yet potentially more perilous risk of GenAI: the creation of $\textit{personalized synthetic realities}$. GenAI could enable individuals to experience a reality customized to personal desires or shaped by external influences, effectively creating a "filtered" worldview unique to each person. Such personalized synthetic realities could distort how people perceive and interact with the world, leading to a fragmented understanding of shared truths. This paper seeks to raise awareness about these profound and multifaceted risks, emphasizing the potential of GenAI to fundamentally alter the very fabric of our collective reality.

Figures

Figures reproduced from arXiv: 2411.08250 by the authors.

**Figure 1.** (Left) In January 2024, the r/StableDiffusion community on Reddit demonstrated a proofof-concept workflow to synthetically generate proofs of identity. (Top Right) GenAI can produce lifelike depictions of never-occurred events (MJv5 prompt: "president biden and supreme leader of iran shaking hands"). (Bottom Right) Subliminal messages in generated content (optical illusion reads OBEY) [PITH_FULL_IMAGE:figures/full… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 12 canonical work pages

[8]

Emilio Ferrara. 2024. GenAI Against Humanity: Nefarious Applications of Generative Artificial Intelligence and Large Language Models. Journal of Computational Social Science (2024)

work page 2024
[1]

Ricardo Baeza-Yates. 2018. Bias on the web. Commun. ACM 61, 6 (2018), 54–61

work page 2018
[2]

Ashwin Balasubramanian, Vito Zou, Hitesh Narayana, Christina You, Luca Luceri, and Emilio Ferrara. 2024. A Public Dataset Tracking Social Media Discourse about the 2024 U.S. Presidential Election on Twitter/X. Technical Report. HUMANS Lab – Working Paper No. 2024.6. https://arxiv.org/abs/2411.00376

arXiv 2024
[3]

Leonardo Blas, Luca Luceri, and Emilio Ferrara. 2024. Unearthing a Billion Telegram Posts about the 2024 U.S. Presidential Election: Development of a Public Dataset. Technical Report. HUMANS Lab – Working Paper No. 2024.5. https://arxiv.org/abs/2410.23638

work page Pith review arXiv 2024
[4]

Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183–186

2017
[5]

Federico Cinus, Marco Minici, Luca Luceri, and Emilio Ferrara. 2024. Exposing Cross-Platform Coordinated Inauthentic Activity in the Run-Up to the 2024 U.S. Election. Technical Report. HUMANS Lab – Working Paper No. 2024.7. https://arxiv.org/abs/2410.22716

arXiv 2024
[6]

Emilio Ferrara. 2019. The history of digital spam. Commun. ACM 62, 8 (2019), 82–91

work page 2019
[7]

Emilio Ferrara. 2024. Charting the Landscape of Nefarious Uses of Generative Artificial Intelligence for Online Election Interference. Technical Report. HUMANS Lab – Working Paper No. 2024.1. https://arxiv.org/abs/2406.01862

arXiv 2024

Show all 18 references

[9]

Emilio Ferrara, Onur Varol, Clayton A Davis, Filippo Menczer, and Alessandro Flammini. 2016. The rise of social bots. Commun. ACM 59, 7 (2016), 96–104

2016
[10]

Nils Köbis, Jean-François Bonnefon, and Iyad Rahwan. 2021. Bad machines corrupt good morals. Nature Human Behaviour 5, 6 (2021), 679–685

2021
[11]

Wojciech Mazurczyk, Dongwon Lee, and Andreas Vlachos. 2024. Disinformation 2.0 in the Age of AI: A Cyberse- curity Perspective. Commun. ACM (2024)

2024
[12]

Filippo Menczer, David Crandall, Yong-Yeol Ahn, and Apu Kapadia. 2023. Addressing the harms of AI-generated inauthentic content. Nature Machine Intelligence (2023), 1–2

2023
[13]

Marco Minici, Luca Luceri, Federico Cinus, and Emilio Ferrara. 2024. Uncovering Coordinated Cross-Platform Information Operations Threatening the Integrity of the 2024 US Presidential Election Online Discussion. Technical Report. HUMANS Lab – Working Paper No. 2024.4. https://...

2024 arXiv
[14]

2024.Tracking the 2024 US Presidential Election Chatter on Tiktok: A Public Multimodal Dataset

Gabriela Pinto, Charles Bickham, Tanishq Salkar, Luca Luceri, and Emilio Ferrara. 2024.Tracking the 2024 US Presidential Election Chatter on Tiktok: A Public Multimodal Dataset. Technical Report. HUMANS Lab – Working Paper No. 2024.3. https://arxiv.org/abs/2407.01471

2024 arXiv
[15]

Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin A Rothkopf, and Kristian Kersting. 2022. Large pre-trained language models contain human-like biases of what is right and wrong to do. Nature Machine Intelligence 4, 3 (2022), 258–268

2022
[16]

Michael Seymour, Kai Riemer, Lingyao Yuan, and Alan R Dennis. 2023. Beyond Deep Fakes.Commun. ACM 66, 10 (2023), 56–67

2023
[17]

Kashish Shah, Patrick Gerard, Luca Luceri, and Emilio Ferrara. 2024. Unfiltered Conversations: A Dataset of 2024 U.S. Presidential Election Discourse on Truth Social. Technical Report. HUMANS Lab – Working Paper No. 2024.8

2024
[18]

Auditing Political Exposure Bias: Algorithmic Amplification on Twitter/X Approaching the 2024 U.S

Jinyi Ye, Luca Luceri, and Emilio Ferrara. Auditing Political Exposure Bias: Algorithmic Amplification on Twitter/X Approaching the 2024 U.S. Presidential Election. Technical Report. HUMANS Lab – Working Paper No. 2024.9

2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [8]

Emilio Ferrara. 2024. GenAI Against Humanity: Nefarious Applications of Generative Artificial Intelligence and Large Language Models. Journal of Computational Social Science (2024)

work page 2024

[2] [1]

Ricardo Baeza-Yates. 2018. Bias on the web. Commun. ACM 61, 6 (2018), 54–61

work page 2018

[3] [2]

Ashwin Balasubramanian, Vito Zou, Hitesh Narayana, Christina You, Luca Luceri, and Emilio Ferrara. 2024. A Public Dataset Tracking Social Media Discourse about the 2024 U.S. Presidential Election on Twitter/X. Technical Report. HUMANS Lab – Working Paper No. 2024.6. https://arxiv.org/abs/2411.00376

arXiv 2024

[4] [3]

Leonardo Blas, Luca Luceri, and Emilio Ferrara. 2024. Unearthing a Billion Telegram Posts about the 2024 U.S. Presidential Election: Development of a Public Dataset. Technical Report. HUMANS Lab – Working Paper No. 2024.5. https://arxiv.org/abs/2410.23638

work page Pith review arXiv 2024

[5] [4]

Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183–186

2017

[6] [5]

Federico Cinus, Marco Minici, Luca Luceri, and Emilio Ferrara. 2024. Exposing Cross-Platform Coordinated Inauthentic Activity in the Run-Up to the 2024 U.S. Election. Technical Report. HUMANS Lab – Working Paper No. 2024.7. https://arxiv.org/abs/2410.22716

arXiv 2024

[7] [6]

Emilio Ferrara. 2019. The history of digital spam. Commun. ACM 62, 8 (2019), 82–91

work page 2019

[8] [7]

Emilio Ferrara. 2024. Charting the Landscape of Nefarious Uses of Generative Artificial Intelligence for Online Election Interference. Technical Report. HUMANS Lab – Working Paper No. 2024.1. https://arxiv.org/abs/2406.01862

arXiv 2024