pith. sign in

arxiv: 2604.09960 · v1 · submitted 2026-04-10 · 💻 cs.CL

Human vs. Machine Deception: Distinguishing AI-Generated and Human-Written Fake News Using Ensemble Learning

Samuel Jaeger , Calvin Ibeneye , Aya Vera-Jimenez , Dhrubajyoti Ghosh This is my paper

Pith reviewed 2026-05-10 16:32 UTC · model grok-4.3

classification 💻 cs.CL
keywords AI-generated textfake news detectionensemble learningreadability featuresstylistic analysismachine learningdeception detectiontext classification
0
0 comments X

The pith

AI-generated fake news can be distinguished from human-written fake news using readability and stylistic features with ensemble machine learning.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper explores the differences in linguistic and emotional patterns between fake news created by humans and that produced by AI systems. It constructs a feature set from sentence structure, word diversity, punctuation, reading ease scores, and emotional tones to train various machine learning models. The study finds that these models, especially when combined in an ensemble, perform well at classifying the source of the deceptive text. A sympathetic reader would care because distinguishing these could help combat the spread of AI-enhanced misinformation in a world where both human and machine deception coexist.

Core claim

The authors claim that readability-based features are the strongest predictors for distinguishing AI-generated fake news from human-written fake news, with AI text displaying more uniform stylistic patterns overall. Ensemble methods that combine logistic regression, random forests, support vector machines, gradient boosting, and neural networks yield modest but consistent gains in accuracy and AUC over individual models, supporting the use of structural and affective text properties for reliable detection.

What carries the argument

A document-level feature representation incorporating sentence structure, lexical diversity, punctuation patterns, readability indices, and emotion scores for dimensions like fear and anger, fed into an ensemble of classifiers.

If this is right

  • Readability indices emerge as the top features for identifying AI-generated content.
  • AI-generated fake news exhibits greater stylistic uniformity than human equivalents.
  • Ensemble aggregation improves classification performance modestly but reliably.
  • Stylistic and structural text properties provide a robust foundation for detection systems.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the features prove stable across topics, they could be applied to detect AI assistance in other writing domains like opinion pieces or reviews.
  • Future work might test whether evolving AI models erode these detectable differences over time.
  • Integration into social media platforms could prioritize review of content flagged as potentially AI-sourced deception.

Load-bearing premise

The collected examples of human-written and AI-generated fake news represent the broader range of real-world deceptive content, and the chosen features capture inherent differences rather than biases from specific sources or subjects.

What would settle it

Repeating the classification on a new dataset of human and AI fake news collected from different platforms or covering unrelated topics, and observing if accuracy drops substantially below the reported levels.

Figures

Figures reproduced from arXiv: 2604.09960 by Aya Vera-Jimenez, Calvin Ibeneye, Dhrubajyoti Ghosh, Samuel Jaeger.

Figure 1
Figure 1. Figure 1: Classification performance across models. The top panel displays accuracy, and the bottom panel displays AUC. All models achieve high performance, with ensemble and tree-based methods showing slightly superior results [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Top features contributing to classification for random forest and XGBoost models. Readability-based features dominate, with the Coleman-Liau index being the most influential. Discriminative Power of Individual Features The discriminative ability of individual features was evaluated using AUC. The results are presented in [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: AUC values for individual features. Readability metrics demonstrate strong discriminative ability, while emotion-based features perform close to random guessing. larger values. In contrast, AI-generated fake news is concentrated at lower readability levels, indicating simpler or more standardized linguistic structures. Although some overlap is present, the difference in central tendency is pronounced, prov… view at source ↗
Figure 4
Figure 4. Figure 4: Density plot of the Coleman-Liau readability index for human-written and AI-generated fake news. Human-written text exhibits higher readability scores, indicating greater linguistic complexity. DISCUSSION The results demonstrate that distinguishing between human-written and AI-generated fake news can be achieved with consistently high accuracy using relatively simple text-based features. Across all models … view at source ↗
read the original abstract

The rapid adoption of large language models has introduced a new class of AI-generated fake news that coexists with traditional human-written misinformation, raising important questions about how these two forms of deceptive content differ and how reliably they can be distinguished. This study examines linguistic, structural, and emotional differences between human-written and AI-generated fake news and evaluates machine learning and ensemble-based methods for distinguishing these content types. A document-level feature representation is constructed using sentence structure, lexical diversity, punctuation patterns, readability indices, and emotion-based features capturing affective dimensions such as fear, anger, joy, sadness, trust, and anticipation. Multiple classification models, including logistic regression, random forest, support vector machines, extreme gradient boosting, and a neural network, are applied alongside an ensemble framework that aggregates predictions across models. Model performance is assessed using accuracy and area under the receiver operating characteristic curve. The results show strong and consistent classification performance, with readability-based features emerging as the most informative predictors and AI-generated text exhibiting more uniform stylistic patterns. Ensemble learning provides modest but consistent improvements over individual models. These findings indicate that stylistic and structural properties of text provide a robust basis for distinguishing AI-generated misinformation from human-written fake news.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper claims that linguistic, structural, and emotional features—particularly readability indices—can reliably distinguish AI-generated fake news from human-written fake news. It constructs document-level representations using sentence structure, lexical diversity, punctuation, readability scores, and emotion dimensions (fear, anger, joy, etc.), then evaluates logistic regression, random forest, SVM, XGBoost, neural networks, and an ensemble aggregator, reporting strong accuracy and AUC performance with readability features as the top predictors, more uniform AI stylistic patterns, and modest ensemble gains.

Significance. If the central claim holds after controlling for data artifacts, the work would offer a practical, feature-based approach to detecting AI misinformation and highlight stable stylistic markers separating human and machine deception. The emphasis on readability and ensemble aggregation could inform downstream detection systems, though the absence of quantified results and controls currently limits its immediate utility.

major comments (2)
  1. [Abstract and Data Collection section] Abstract and Data Collection section: The central claim that readability and stylistic features distinguish generation methods requires that human and AI examples are balanced on topics and sources. The manuscript provides no description of dataset sizes, collection procedures, topic matching, stratification, or domain labels, leaving open the possibility that observed differences in lexical diversity, sentence structure, and readability indices reflect topic or source confounds rather than inherent generation differences.
  2. [Results section] Results section: The abstract asserts 'strong and consistent classification performance' and 'modest but consistent improvements' from the ensemble without reporting specific accuracy/AUC values, baseline comparisons, error bars, or statistical tests. This is load-bearing for the claim of reliable distinction, as it prevents assessment of effect sizes or whether improvements exceed noise.
minor comments (2)
  1. [Methods] The ensemble aggregation method (e.g., voting weights or stacking) is not described in sufficient detail to allow replication.
  2. [Results] Feature importance analysis for readability indices is mentioned but not supported by specific scores, ablation results, or tables showing relative contributions.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback, which highlights important areas for improving the clarity and rigor of our manuscript. We address each major comment below and will make the necessary revisions to strengthen the presentation of our data collection and results.

read point-by-point responses

  1. Referee: [Abstract and Data Collection section] Abstract and Data Collection section: The central claim that readability and stylistic features distinguish generation methods requires that human and AI examples are balanced on topics and sources. The manuscript provides no description of dataset sizes, collection procedures, topic matching, stratification, or domain labels, leaving open the possibility that observed differences in lexical diversity, sentence structure, and readability indices reflect topic or source confounds rather than inherent generation differences.

    Authors: We agree that the absence of detailed dataset information leaves the central claim vulnerable to potential confounds from topics or sources. The current Data Collection section is indeed too brief. In the revised manuscript we will expand it to report exact dataset sizes (number of human-written and AI-generated articles), collection procedures (sources for human fake news and the specific LLM prompting strategy for AI examples), methods used for topic matching and stratification to ensure balance across categories, and any available domain labels. We will also add a short discussion of how these controls mitigate topic or source biases. revision: yes

  2. Referee: [Results section] Results section: The abstract asserts 'strong and consistent classification performance' and 'modest but consistent improvements' from the ensemble without reporting specific accuracy/AUC values, baseline comparisons, error bars, or statistical tests. This is load-bearing for the claim of reliable distinction, as it prevents assessment of effect sizes or whether improvements exceed noise.

    Authors: We acknowledge that the current manuscript reports performance only qualitatively ('strong and consistent') without the quantitative details needed to evaluate effect sizes or statistical reliability. We will revise the Results section to include explicit accuracy and AUC values for all models and the ensemble, add baseline comparisons (e.g., against a majority-class or random classifier), report error bars or standard deviations from cross-validation folds, and include statistical tests (such as McNemar's test or paired t-tests) for the ensemble improvements. We will also update the abstract to incorporate the key numerical results. revision: yes

Circularity Check

0 steps flagged

No circularity: standard empirical ML pipeline on held-out labels

full rationale

The paper's chain consists of (1) collecting labeled human-written and AI-generated fake news examples, (2) extracting fixed linguistic/readability/emotion features, (3) training off-the-shelf classifiers (LR, RF, SVM, XGBoost, NN) and an ensemble, and (4) reporting accuracy/AUC on held-out test data. None of these steps reduces to its inputs by construction: the performance numbers are not a re-expression of the feature definitions or training labels, nor are they obtained by fitting a parameter and then relabeling it a prediction. No uniqueness theorems, ansatzes, or self-citations are invoked to force the result. The evaluation is falsifiable against external ground-truth origin labels, satisfying the self-contained benchmark criterion.

Axiom & Free-Parameter Ledger

1 free parameters · 2 axioms · 0 invented entities

The central claim rests on standard machine-learning assumptions about feature informativeness and data representativeness; no new entities are postulated.

free parameters (1)
  • model hyperparameters
    Hyperparameters for random forest, XGBoost, neural network and ensemble weights are tuned on data but not detailed in the abstract.
axioms (2)
  • domain assumption The extracted features (readability, lexical diversity, emotion scores) are stable and informative discriminators independent of topic or source bias.
    Invoked implicitly when claiming readability features are most informative.
  • domain assumption Human-written and AI-generated fake news samples are drawn from distributions that reflect real-world deceptive content.
    Required for the classification results to generalize beyond the study data.

pith-pipeline@v0.9.0 · 5527 in / 1387 out tokens · 46611 ms · 2026-05-10T16:32:20.338340+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 26 canonical work pages · 1 internal anchor

  1. [1]

    and Schudson, M., 2018

    Lazer, D.M., Baum, M.A., Benkler, Y., Berinsky, A.J., Greenhill, K.M., Menczer, F., Metzger, M.J., Nyhan, B., Pennycook, G., Rothschild, D. and Schudson, M., 2018. The science of fake news. Science, 359(6380), pp.1094-1096

    work page 2018

  2. [2]

    and Aral, S., 2018

    Vosoughi, S., Roy, D. and Aral, S., 2018. The spread of true and false news online. science, 359(6380), pp.1146-1151

    work page 2018

  3. [3]

    and Lahiri, S., 2025

    Ghosh, D., Boettcher, W.A., Johnston, R. and Lahiri, S., 2025. THANOS: a predictive model of electoral campaigns using twitter data and opinion polls. Data Science in Science, 4(1), p.2484180

    work page 2025

  4. [4]

    and Liu, H., 2017

    Shu, K., Sliva, A., Wang, S., Tang, J. and Liu, H., 2017. Fake news detection on social media: A data mining perspective. ACM SIGKDD explorations newsletter, 19(1), pp.22-36

    work page 2017

  5. [5]

    and Lahiri, S., 2026

    Ghosh, D., Boettcher, W., Johnston, R. and Lahiri, S., 2026. Machine Learning Based Bot Detection on X With Temporal and Semantic Feature Integration. IEEE Transactions on Computational Social Systems

    work page 2026

  6. [6]

    and Agarwal, S., 2020

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A. and Agarwal, S., 2020. Language models are few-shot learners. Advances in neural information processing systems, 33, pp.1877-1901

    work page 2020

  7. [7]

    GPT-4 Technical Report

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S. and Avila, R., 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  8. [8]

    and Choi, Y., 2017, September

    Rashkin, H., Choi, E., Jang, J.Y., Volkova, S. and Choi, Y., 2017, September. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 conference on empirical methods in natural language processing (pp. 2931-2937)

    work page 2017

  9. [9]

    and Mihalcea, R., 2018, August

    P\'erez-Rosas, V., Kleinberg, B., Lefevre, A. and Mihalcea, R., 2018, August. Automatic detection of fake news. In Proceedings of the 27th international conference on computational linguistics (pp. 3391-3401)

    work page 2018

  10. [10]

    and Liu, H., 2019, January

    Shu, K., Wang, S. and Liu, H., 2019, January. Beyond news contents: The role of social context for fake news detection. In Proceedings of the twelfth ACM international conference on web search and data mining (pp. 312-320)

    work page 2019

  11. [11]

    and Zafarani, R., 2020

    Zhou, X. and Zafarani, R., 2020. A survey of fake news: Fundamental theories, detection methods, and opportunities. ACM Computing Surveys (CSUR), 53(5), pp.1-40

    work page 2020

  12. [12]

    and Wang, W.Y., 2020, May

    Oshikawa, R., Qian, J. and Wang, W.Y., 2020, May. A survey on natural language processing for fake news detection. In Proceedings of the Twelfth Language Resources and Evaluation Conference (pp. 6086-6093)

    work page 2020

  13. [13]

    and Gouza, F.B., 2018

    Zhang, J., Cui, L., Fu, Y. and Gouza, F.B., 2018. Fake news detection with deep diffusive network model. arXiv preprint arXiv:1805.08751, pp.1-10

    work page arXiv 2018

  14. [14]

    and Smith, N.A., 2021, August

    Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S. and Smith, N.A., 2021, August. All that's ‘human'is not gold: Evaluating human evaluation of generated text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pa...

    work page 2021

  15. [15]

    and Brundage, M., 2022

    Kreps, S., McCain, R.M. and Brundage, M., 2022. All the news that's fit to fabricate: AI-generated text as a tool of media misinformation. Journal of experimental political science, 9(1), pp.104-117

    work page 2022

  16. [16]

    A survey of modern authorship attribution methods

    Stamatatos, E., 2009. A survey of modern authorship attribution methods. Journal of the American Society for information Science and Technology, 60(3), pp.538-556

    work page 2009

  17. [17]

    Certain language skills in children; their development and interrelationships

    Templin, M.C., 1957. Certain language skills in children; their development and interrelationships

    work page 1957

  18. [18]

    and Baayen, R.H., 1998

    Tweedie, F.J. and Baayen, R.H., 1998. How variable may a constant be? Measures of lexical richness in perspective. Computers and the Humanities, 32(5), pp.323-352

    work page 1998

  19. [19]

    A new readability yardstick

    Flesch, R., 1948. A new readability yardstick. Journal of applied psychology, 32(3), p.221

    work page 1948

  20. [20]

    and Chissom, B.S., 1975

    Kincaid, J.P., Fishburne Jr, R.P., Rogers, R.L. and Chissom, B.S., 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel (No. RBR875)

    work page 1975

  21. [21]

    SMOG grading-a new readability formula

    Mc Laughlin, G.H., 1969. SMOG grading-a new readability formula. Journal of reading, 12(8), pp.639-646

    work page 1969

  22. [22]

    and Liau, T.L., 1975

    Coleman, M. and Liau, T.L., 1975. A computer readability formula designed for machine scoring. Journal of Applied Psychology, 60(2), p.283

    work page 1975

  23. [23]

    and Turney, P.D., 2013

    Mohammad, S.M. and Turney, P.D., 2013. Crowdsourcing a word-emotion association lexicon. Computational intelligence, 29(3), pp.436-465

    work page 2013

  24. [24]

    and Gupta, R., 2020, February

    Rincy, T.N. and Gupta, R., 2020, February. Ensemble learning techniques and its efficiency in machine learning: A survey. In 2nd international conference on data, engineering and applications (IDEA) (pp. 1-6). IEEE

    work page 2020

  25. [25]

    and Alzheimer's Disease Neuroimaging Initiative, 2025

    Ghosh, D., Pal, S., Lutz, M., Luo, S. and Alzheimer's Disease Neuroimaging Initiative, 2025. Ensemble survival analysis for preclinical cognitive decline prediction in Alzheimer's disease using longitudinal biomarkers. Journal of Alzheimer's Disease, 107(3), pp.1256-1266

    work page 2025

  26. [26]

    and Bhatia, M.P.S., 2016, December

    Srivastava, R. and Bhatia, M.P.S., 2016, December. Ensemble methods for sentiment analysis of on-line micro-texts. In 2016 International Conference on Recent Advances and Innovations in Engineering (ICRAIE) (pp. 1-6). IEEE

    work page 2016