{"id":"33d0667e-6c3c-4758-b51f-feb271257d10","arxiv_id":"2607.21967","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Using lexical, readability, and emotion features, six classifiers separate real from AI-generated fake news with AUC ~0.99 but real from human-written fake news with AUC ~0.89–0.96, so AI fake news is currently easier to detect.","lead":"This paper reports that AI-generated fake news is much easier to detect than human-written fake news using standard text statistics. The finding would matter for detection-system design, but the preprint provides no data or code and has statistical inconsistencies, so it cannot be verified.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The AUC gap may be an artifact of corpus mismatch: human- and AI-fake corpora are unspecified and likely differ in genre/length, so the 'identical pipeline' does not isolate source.","rationale":"I read the paper in good faith: the experiment is well-motivated, the feature set is reasonable, and the qualitative asymmetry (AI fake easier than human fake) is consistent with some literature. However, the causal inference depends entirely on the comparability of the two tasks. The lack of any dataset details is not just a presentation issue; it makes the alternative explanation (genre/length/topic mismatch) impossible to rule out. A single LLM and prompt scheme also limits generalizability, but the more immediate threat is that the human-fake corpus may not be matched in format. The proposed controlled test (rewriting each human-fake article with the LLM, or matching on length/topic) would directly isolate source. If the gap persists under that control, the paper's central claim would be substantially supported. If it collapses, the claim is an artifact. Given these deficiencies, I agree with the reader's REJECT verdict; this concern does not move the verdict, hence UNCHANGED.","tokens_in":9388,"tokens_out":6158,"duration_ms":65369,"concrete_test":"Obtain the three corpora (or have the authors release them). For each document, record length, genre, and topic (e.g., LDA proportions). Re-run the two classification tasks on a matched subset: (i) restrict all documents to the same length band (e.g., 300–800 words) and same domain/topic distribution (propensity-score matched on topic and length); or (ii) take each human-written fake article, have the same LLM rewrite it under identical style constraints, and compare AUC for human-original vs LLM-rewritten against the same real-news set. If the AUC gap Δ_m remains after matching/rewriting (≈0.049), the source attribution is supported; if it collapses, the reported gap is a corpus artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the performance gap Δ_m reflects intrinsic distributional differences between human- and AI-generated fake news—requires that the two fake corpora be comparable in every way except generation source. The Method section (Section II) says only that human fake news comes from 'curated misinformation datasets' and AI fake news is produced by 'a large language model under controlled prompting conditions to ensure topical consistency with real news.' No corpus names, sample sizes, LLM identity, decoding parameters, prompts, or matching procedure are given. If the human-fake dataset consists of short social-media claims or out-of-domain statements while the AI-fake articles are generated as full-length, news-style documents, then lexical diversity (TTR) and readability features (Flesch, Coleman–Liau) will encode document length and genre rather than author identity. The paper's assertion that 'the two learning problems differ only in the distribution of the fake class' is therefore not established. This is load-bearing because the entire conclusion rests on Δ_m being source-specific rather than corpus-specific. Additionally, the reported Wilcoxon p=0.016 is impossible for M=6 paired observations (minimum two-sided exact p is 1/32 ≈ 0.031) and the figure/table captions (Fig. 1 vs Fig. 2) are mismatched, which raises further doubt about the reliability of the numerical records, but the primary threat is the uncontrolled corpora.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares two binary classification tasks: distinguishing real news from human-written fake news and from AI-generated fake news. Both tasks use the same feature set (lexical diversity, readability indices, and NRC emotion features), the same preprocessing, and six machine-learning models. The reported held-out AUCs are uniformly higher for the real-versus-AI-fake task (0.963–0.996) than for the real-versus-human-fake task (0.888–0.960). Because the modeling pipeline is identical, the authors attribute the gap to intrinsic distributional differences between human- and AI-generated fake text. The manuscript includes feature-level analyses and paired statistical tests (t-test and Wilcoxon signed-rank) across models.","tokens_in":9732,"tokens_out":8585,"duration_ms":83590,"significance":"If the empirical result is trustworthy, this is a timely and useful contribution: it quantifies an asymmetry in detectability under a controlled pipeline and with interpretable features. The strengths are the explicit attempt to hold the modeling pipeline fixed, the transparent feature set, and the authors' acknowledgment of some limitations. However, the central claim is not yet established because the two fake-news corpora are not described in enough detail to rule out corpus artifacts, and the statistical reporting lacks confidence intervals and a clear inference framework. The paper is also not reproducible in its current form, with no data or code availability.","major_comments":[{"comment":"The design claim that 'the two learning problems differ only in the distribution of the fake class' is load-bearing but unsupported. The manuscript does not name the real-news dataset, the human-fake 'curated misinformation datasets', the LLM used, the prompting scheme, decoding parameters, or the topical matching procedure. If the human-fake corpus and AI-fake corpus differ in document length, genre, or topic coverage, the features used (TTR, readability, emotions) will encode those differences rather than source identity. Please provide dataset names, sample sizes, and a demonstration of comparability across the three classes, e.g., distributions of length and topic, or a matched-subset analysis.","section":"Section II (Method)"},{"comment":"The AUC values are given as single point estimates from one 80/20 split, without confidence intervals, standard errors, or repeated resampling. The paired t-test and Wilcoxon test are performed across M=6 models, treating the model set as paired observations; this does not support inference about document- or corpus-level variability. Also, the Wilcoxon direction is not specified: with six positive differences, the one-sided exact p is 1/64≈0.016, while the two-sided exact minimum is 0.031. The reported p=0.016 is therefore ambiguous. Please report bootstrap or repeated-CV confidence intervals for the AUC gap and describe the exact test implementation and direction.","section":"Section III, Table I and statistical tests"},{"comment":"The phrase 'intrinsic distributional properties of the text' overstates what a fixed, hand-selected feature set can establish. The gap is measured in one specific feature space (TTR, readability indices, NRC emotions), and the Discussion correctly limits the claim later. Please either soften the abstract and conclusion to 'under this feature representation', or add a robustness check with a complementary feature family (e.g., contextual embeddings or more stylometric features) to show that the direction of the gap does not depend on the feature choice.","section":"Section IV (Discussion) and Section V (Conclusion)"}],"minor_comments":[{"comment":"The figure captions are mismatched with the text: Fig. 1 is described in the text as an AUC bar chart but captioned as a Coleman–Liau density plot; Fig. 2 is described as a Coleman–Liau density plot but captioned as feature-wise mean differences; Fig. 3 is described as a feature heatmap but captioned as ROC curves. The cross-references need to be fixed.","section":"Figure captions"},{"comment":"The table header still reads 'TABLE TYPE STYLES'; replace with a descriptive caption.","section":"Table I"},{"comment":"No data availability statement or code repository is provided, which is a barrier to reproducing the reported AUCs.","section":"Reproducibility"},{"comment":"Section II says 'balanced samples across classes' but gives no class counts; report the number of documents in each class.","section":"Dataset description"},{"comment":"The text refers to Figure 4 for the ROC curves, but no Fig. 4 caption appears in the manuscript.","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The central empirical question is timely, and a carefully documented version of this study would be appropriate for the journal. However, the current lack of corpus transparency and statistical detail prevents verification. The Wilcoxon p=0.016 issue is not by itself decisive—it could come from a one-sided exact test—but it illustrates the need for precise statistical reporting. I also suggest the editor ask the authors to justify the relevance of several self-citations (e.g., [1], [2], [21]) that are not directly about fake-news detection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — the short version: this is a legitimate research question with a plausible answer, but the submission is not in a form anyone can verify. The same-pipeline comparison of real-vs-human and real-vs-AI fake news is a reasonable way to ask whether the source of fake content changes detectability, and the reported asymmetry (AI fake AUC >0.99, human fake ~0.89–0.96) is consistent with a fair amount of prior work on machine-text artifacts. That is real credit where it is due.\n\nWhat the paper has going for it: the design holds the model family, features, and evaluation fixed, so the comparison is cleaner than most; the feature set (TTR, readability, NRC emotions) is simple but not crazy; and the discussion correctly notes the gap may narrow as generators improve. The authors also acknowledge general limitations (domain shift, fixed split), which shows good instincts.\n\nThe soft spots are not minor. The central inference — that the gap reflects intrinsic distributional properties — rests on the two fake corpora being comparable except for the source. The paper never names the datasets, sample sizes, the LLM, the prompts, or the matching procedure. The stress-test concern is on target: if the human-fake items are short social-media claims and the AI-fake items are full news-style paragraphs, TTR and readability will encode length and genre, not authorship. That single gap is load-bearing.\n\nBeyond that, the statistics are internally impossible. For M=6 paired models the two-sided Wilcoxon exact minimum p is about 0.031; the reported p=0.016 cannot come from six observations. Either the test was run on folds or more models, and the text doesn't say. The figure captions also don't match what the text says they show, which makes me doubt the numerical records.\n\nThe question deserves a serious look, but this manuscript is not ready for citation. I'd send it to review only with an explicit request for the full dataset, code, and corrected statistics; if the authors can't provide them, reject. As it stands, it's a cautionary reading-group example of a plausible-sounding result without verifiable support.","headline":"A plausible and relevant empirical question, but the manuscript as written is unverifiable—missing corpus details, an impossible p-value, and mismatched captions—so it needs major revision and full data release before it can be trusted.","tokens_in":10204,"tokens_out":4047,"would_cite":false,"duration_ms":45696,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AI-generated fake news is easier to detect than human-written fake news","keywords":["fake news detection","AI-generated text","misinformation","text classification","readability analysis","sentiment analysis","machine learning","AUC"],"falsifier":"Re-run the same pipeline with several current LLMs (e.g., different architectures and parameter scales) and multiple prompt templates, keeping the real-news corpus fixed and topic-matched, and check whether any AI-generated fake news set achieves AUC comparable to the 0.89–0.96 range seen for human fake news. If such a set exists, the claim that AI-generated fake news is intrinsically more separable would collapse.","tokens_in":9306,"feed_emoji":"🤖","tokens_out":2178,"duration_ms":23241,"temperature":0.7,"pith_summary":"This paper asks whether fake news becomes easier or harder to detect when it is written by a large language model instead of a person. Using the same feature representation and the same set of classifiers for both tasks, it finds that real-versus-AI-fake classification achieves near-perfect AUC (above 0.99 for most models) while real-versus-human-fake classification scores between 0.89 and 0.96. Because the modeling pipeline is identical, the authors argue this gap reflects intrinsic statistical differences in the text, not the choice of algorithm. The result matters because deployed detection systems face both kinds of misinformation, and the asymmetry suggests a single unified detector may be insufficient.","feed_headline":"AI fake news is easier to detect than human fake news","feed_subtitle":"Near-perfect spotting of machine-written misinformation, but scores drop when people do the lying.","key_machinery":"The central object is the feature mapping φ(x) that concatenates type-token ratio, Flesch and Flesch-Kincaid readability scores, SMOG and Coleman-Liau indices, and NRC emotion-lexicon features (anger, fear, joy, sadness, trust, anticipation, disgust, surprise, plus positive/negative sentiment). The argument is carried by comparing two binary classification tasks — real vs human fake and real vs AI fake — under an identical pipeline of feature extraction, preprocessing, model classes, and evaluation, so any systematic AUC difference is attributed to the distribution of the fake class.","core_discovery":"Under a common feature representation built from lexical diversity, readability indices, and emotion lexicons, AI-generated fake news is substantially more separable from real news than human-written fake news. Across all six models tested, the AUC for the real-versus-AI task is uniformly higher than for the real-versus-human task, with an average improvement of about 0.049 (paired t-test t = 8.67, p < 0.001; Wilcoxon signed-rank p = 0.016). Feature-level analysis shows AI-generated fake news has more uniform readability and emotional patterns, producing less overlap with real news, while human-written fake news mimics the heterogeneity of legitimate journalism. The authors conclude that the","pith_inferences":["The paper's result is a snapshot tied to one unspecified LLM and one prompt scheme; using multiple current LLMs with varied prompts could show that some configurations produce AI fake news nearly as hard to detect as human fake news, which would temper the claim of intrinsic separability.","A testable extension would be to measure whether the gap shrinks when AI-generated text is post-edited by humans to mimic the readability and emotion variance of real news; if it closes, the finding is about current decoding habits rather than a fundamental property.","The authors' feature set is deliberately compact and interpretable; one could check whether transformer-based embeddings erase or widen the gap, which would clarify how much of the effect lives at the surface-statistics level.","The asymmetry implies that platform moderation policies treating all fake news alike may be misaligned with detection reality, but the ethical and adversarial implications of that asymmetry are left for future work."],"forward_implications":["Current detection systems may be well-equipped to identify AI-generated fake news in its present form, but they remain substantially less reliable for sophisticated human-written misinformation.","Because the performance gap persists across linear, kernel, tree-based, neural, and ensemble models, the separability appears to be a property of the feature space itself rather than any particular classifier.","Ensembling models yields the highest AUC in both tasks but does not change the ordering, reinforcing that the data-generating process drives the result.","If generative models become more diverse in readability and emotional expression, the gap may narrow, requiring detection methods to adapt as LLMs evolve.","Text-only surface features face inherent limits against human-authored deception, suggesting that contextual, social, and network signals may be necessary for that harder task."],"fun_headline_variants":["AI fake news: easy spot, human fake news: hard","AI lies are easier to catch than human lies","Machine fake news is more detectable than human fake news","Detecting AI fake news: near perfect, human: falls short"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The comparison assumes that one unspecified language model and one prompt scheme adequately represent 'AI-generated fake news', and that the real-news corpus is identical and topically matched across both tasks; otherwise the AUC gap could reflect topic drift or dataset artifacts rather than the source of the fake content.","fun_headline_variants_meta":{"raw":{"variants":["AI fake news: easy spot, human fake news: hard","AI lies are easier to catch than human lies","Machine fake news is more detectable than human fake news","Detecting AI fake news: near perfect, human: falls short"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000359,"raw_usage":{"total_tokens":1780,"prompt_tokens":745,"completion_tokens":1035,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":968}},"tokens_in":489,"tokens_out":1035,"duration_ms":10667,"temperature":1.0,"reasoning_tokens":968,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T06:10:36.478973+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same pipeline with several current LLMs (e.g., different architectures and parameter scales) and multiple prompt templates, keeping the real-news corpus fixed and topic-matched, and check whether any AI-generated fake news set achieves AUC comparable to the 0.89–0.96 range seen for human fake news. If such a set exists, the claim that AI-generated fake news is intrinsically more separable would collapse.","supporting_citations":[],"review_version":1}