{"id":"d03545f4-0aaf-4003-8dd6-4025c7fd24ed","arxiv_id":"2601.07368","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Simple word-count classifiers distinguish 1920s–30s detective fiction from GPT-4.1 rewrites with ~98% accuracy, and the main cue is the AI's broader synonym use plus modernized phrasing.","lead":"This paper tests whether machine-learning classifiers can tell short human-written novel excerpts from versions rewritten by GPT-4.1; they reach about 98% accuracy, while 119 human readers score at chance. A linear model's word-level weights suggest the AI text has more varied vocabulary, modern-language drift, and fewer Britishisms or colloquialisms.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Prompt's 'structurally different' instruction likely drives the synonym-diversity cue, so the paper's central explanation and robustness claim lack support outside this specific generation mode.","rationale":"The reader's weakest_assumption focused on potential train/test leakage from ungrouped pairs. While a legitimate methodological concern, its direction is likely to reduce measured accuracy: if a test sample's paired counterpart (opposite class) is in the training set, content overlap would bias the classifier toward the wrong label, making the reported 0.98 accuracy a conservative estimate. Thus the split is not the most load-bearing threat. The prompt artifact is more corrosive to the central claim because it directly explains the feature set and entropy result; the paper's own §6 limitation admits this. The correct check is to regenerate the data with neutral prompts and re-measure. Since the authors already mention preliminary but unreported evidence, the verdict should remain CONDITIONAL: acceptance pending a demonstration that the phenomenon survives without the explicit 'structurally different' injunction.","tokens_in":23758,"tokens_out":8182,"duration_ms":87867,"concrete_test":"Generate three new versions of DET12 using GPT-4.1 at temperature 0.7: (A) prompt without the 'structurally different' instruction, e.g., \"Rewrite the following text; keep all key details and approximately the same length\"; (B) a continuation prompt giving 2–3 sentences of the human text and asking the model to continue the scene; (C) an outline-only prompt (no source text). For each, split by original–rewrite pair (grouped) into the same 70/15/15 proportions, retrain the linear classifier using the exact same tokenization and feature set, and report test accuracy and the Class 1 vs Class 0 unigram entropy difference. If accuracy in any condition drops below ~0.90 or the entropy gap shrinks to <0.1 bits, the paper's central explanation and robustness claim do not generalize beyond the original prompt.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The data generation prompt (§3.1) explicitly instructs GPT-4.1 to \"Create a new text excerpt which contains the same key details but appears structurally different to the original.\" This instruction directly induces the observed synonym substitution (E1 categories, Table 4), the higher unigram entropy (H=9.164 vs 8.875, §5.2), and consequently the 0.93–0.98 test accuracies (Table 3). The paper's central claim that \"the LLM tends to use a larger variety of synonyms\" is therefore a statement about the behavior elicited by this specific prompt, not a general property of LLM-generated fiction. The authors acknowledge this in §6 (\"the methodology may have introduced greater variation ... by requesting rewording\") but only cite \"preliminary experiments\" without data. Because the paper aims to explain why classifiers detect LLM text and argues the detection is robust (\"not easy to circumvent\", §6), this prompt-dependence is load-bearing: a malicious actor could use a neutral prompt (e.g., \"rewrite in the same style\" or \"continue from here\") and potentially evade detection, making the robustness claim unsupported. Even if the train/test split were perfectly grouped by pair, this issue would remain; in fact, the semantic difference is hard-wired into the data. Thus the current evidence is insufficient to generalize the explanation beyond the \"rewrite to be different\" paradigm.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper constructs DET12, a corpus of 8,068 short excerpts from twelve classic detective novels by Agatha Christie and Dorothy L. Sayers, together with 8,068 GPT-4.1 rewrites generated under a prompt that asks for the same key details but an \"appears structurally different\" text. Using only unigram features, five machine-learning classifiers are trained on a 70/15/15 split and achieve 0.93--0.98 test accuracy, with the best linear classifier at 0.9814. In the same task, 119 human evaluators perform at chance (297/595 correct). The authors then use the linear classifier's weights and a manual annotation of 190 frequent, class-selective unigrams to argue that the main cue is the LLM's greater synonym variety, supplemented by temporal drift, Americanisms, foreign-language removal, and loss of colloquialisms. A unigram-entropy analysis reports higher entropy for the LLM class (H_1=9.164 vs H_0=8.875). The paper concludes that detection is robust and not easy to circumvent.","tokens_in":24013,"tokens_out":4886,"duration_ms":56047,"significance":"If the test set is genuinely unseen in the sense that no original--rewrite pair straddles the train/test boundary, the 0.98 accuracy of a simple linear classifier on short fiction excerpts is a practically valuable result, and the interpretability analysis is a useful template for understanding classifier decisions. The study has notable strengths: the data and tokenized datasets are released on Zenodo; the artefact checks for punctuation, casing, and letter distributions are thoughtful; and the human evaluation is substantially larger than previous work (595 judgments). The visualisation in Figure 3 is a nice illustration of an inherently interpretable model. However, two load-bearing issues prevent me from accepting the paper in its current form: the train/test split is not stated to keep original--rewrite pairs together, and the central explanation (synonym variety) is substantially built into the generation prompt, so the generalisation and robustness claims go beyond the evidence. These are fixable with additional experiments or by reframing the claims.","major_comments":[{"comment":"The split description says only that \"labelled original and rewritten text samples\" were split into 70/15/15 sets; it does not state that each original--rewrite pair was kept in the same split. With 8,068 pairs and a sample-level random split, a rewritten test sample will very likely have its source human text in the training set (probability ≈0.7). Because the LLM was prompted to preserve key details, a test rewrite and its training-set original share proper nouns and many content words; a unigram classifier can exploit that overlap. The reported 0.93--0.98 accuracies could therefore reflect content memory rather than style discrimination. Please state the grouping explicitly, and, if the split was not pair-grouped, re-run the experiments with a grouped split (e.g., GroupShuffleSplit) and report the resulting accuracies.","section":"Section 3.1 / Table 3"},{"comment":"The generation prompt explicitly instructs GPT-4.1 to \"Create a new text excerpt which contains the same key details but appears structurally different to the original.\" This instruction directly invites synonym substitution and greater lexical variation. The paper's main explanation category E1 (rephrasing, Table 4) and the higher unigram entropy reported in §5.2 are therefore measures of behaviour elicited by this specific prompt, not established properties of LLM-generated fiction in general. The authors acknowledge this in §6 (\"the methodology may have introduced greater variation ... by requesting rewording\") and cite \"preliminary experiments\" without data, but then still conclude that the classification \"appears robust, and therefore not easy to circumvent by malicious actors.\" This robustness claim is unsupported: a different generation protocol (e.g., neutral continuation or same","section":"Section 3.1 / §5.1 / §5.2 / §6"},{"comment":"The robustness argument uses an ablation in which learned feature weights are set to zero one by one and accuracy is recomputed on the test set without retraining the classifier. This shows how much the fitted model relies on individual features, but it does not simulate a malicious actor who rewrites text to evade detection. It also does not account for correlated features that a retrained model could exploit after a feature is removed. The conclusion in §6 that \"it is difficult to make automatically generated detective fiction harder to detect ... whether by removal or by rewriting\" therefore overinterprets the experiment. I recommend either reframing Figure 5 as a descriptive sensitivity analysis or adding experiments that actually modify generated text (e.g., synonym substitution or prompting with different instructions) and measure evasion.","section":"Section 5, Figure 5 / §6"}],"minor_comments":[{"comment":"The line \"As 29.164/28.875 ≈1.22\" appears to be missing superscripts; it should read 2^9.164 / 2^8.875 ≈ 1.22.","section":"Section 5.2"},{"comment":"There is an inconsistency between the text of Figure 5, which describes removing the first 100 features, and the conclusion in §6, which refers to \"the 80 features which most help the classifiers.\" Please align these numbers.","section":"Section 5, Figure 5 / §6"},{"comment":"Typo in the caption: \"Cspecifices\" should be \"C specifies\".","section":"Table 2"},{"comment":"The manual annotation was performed by hand without reporting inter-annotator agreement or multiple annotators; the authors note this limitation, but it would be helpful to state the number of annotators explicitly.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The pair-grouping issue is the most important point: if the original--rewrite pairs were not kept together in the split, the headline accuracy numbers may be substantially inflated. Please ask the authors to clarify this first; if they cannot provide a pair-grouped split, the paper's central detection claim would not be supported. The prompt-dependence issue is also substantive and should be addressed by additional experiments or by restricting the claims to the rewrite-to-be-different setting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a decent, honest paper that explains why simple unigram classifiers can tell human-written 1920s detective fiction apart from GPT-4.1 rewrites of the same passages. The genuinely new parts are the manual feature annotation (E1–E5) and the entropy analysis, and the 119-participant human evaluation is a nice addition. The authors also ship the tokenized data on Zenodo, which is good practice. The claim that humans are at chance while simple ML beats them is solid for this specific setup.\n\nBut two issues keep me from fully trusting the 0.98 number. First, the train/validation/test split is described as a random split of 'labelled original and rewritten text samples' without stating that each original–rewrite pair was kept together. If any pairs straddle train and test, the test set is not genuinely unseen — the classifier could pick up content cues from the original in training and apply them to the rewrite in test. That is load-bearing, and the paper needs a re-analysis with pair-grouped splits (or at least whole-book held-out sets). Second, the prompt explicitly says 'appears structurally different to the original,' which invites synonym substitution and other variation. So the main explanation (E1, larger synonym variety) is at least partly a consequence of the generation protocol, not a general property of LLM-written fiction. To their credit, the authors acknowledge this in §6, but they only cite unpublished preliminary experiments — no data — and then claim robustness to evasion based on Figure 5, which is essentially a feature-ablation study. That robustness claim is not supported.\n\nThe feature annotation is careful and the categories are plausible; the entropy result (0.29 bits higher for rewrites) is a clean summary of the same effect. But the paper overreaches when it generalizes beyond the 'rewrite to be different' paradigm. Also, the high-accuracy detection was already established in McGlinchey and Barclay (2025) by two of the same authors, so the classification numbers themselves are not new.\n\nBottom line: this deserves a serious referee, mainly to force a re-analysis with pair-grouped splits and an honest discussion of prompt-dependence. The interpretability framework is worth engaging with. I'd bring it to a reading group if you work on AI-text detection; otherwise it's a maybe. I'd cite it for the feature categories, with a caveat, once the split issue is resolved.","headline":"Useful interpretability analysis of LLM-rewrite detection, but the ungrouped split and the 'structurally different' prompt mean the headline accuracy numbers need re-analysis before I'd trust them.","tokens_in":24549,"tokens_out":2445,"would_cite":true,"duration_ms":26250,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Simple unigram statistics can tell LLM-rewritten fiction from human prose at 98 percent accuracy, while human readers score at chance.","keywords":["LLM-generated text detection","interpretable classification","creative fiction","unigram entropy","synonym variety","bag-of-words features","human versus AI text","detective fiction"],"falsifier":"Inspect the released DET12 data: if any test-set sample's paired counterpart appears in the training set, the core accuracy claim collapses to memorization. Alternatively, on a held-out corpus of contemporary fiction and rewrites, compute the per-class unigram entropy gap; if classifiers still reach 0.98 while the gap disappears, synonym variety is not the operative cue.","tokens_in":23572,"feed_emoji":"🤖","tokens_out":5084,"duration_ms":48772,"temperature":0.7,"pith_summary":"This paper sets out to show that human readers are almost unable to tell an LLM's rewrite of a classic detective novel from the original prose, while even very simple machine-learning classifiers do it nearly perfectly. Using short excerpts from twelve human-written novels and GPT-rewritten versions, the authors report test accuracies between 0.93 and 0.98 for a range of classifiers, with an interpretable linear classifier at the top; 119 human evaluators, by contrast, score 297/595, essentially a coin flip. The central explanatory claim is that the language model consistently uses a wider range of synonyms and slightly modernised phrasing, which raises the unigram entropy of the generated texts (9.164 vs 8.875 bits) and gives classifiers a stable statistical fingerprint. The paper argues that because this signal is spread across many small word-level cues, it is not something a human can consciously perceive and is hard to remove by post-editing. If correct, this would give editors and publishers a practical, explainable tool for flagging disguised AI-authored fiction.","feed_headline":"Word counts alone detect AI fiction at 98% accuracy","feed_subtitle":"Humans score at chance on the same texts; a linear classifier reads synonym variety and other small cues.","key_machinery":"The load-bearing object is a linear classifier trained on bag-of-words unigram features, whose weights are directly inspectable; the paper uses those weights, plus a parallel analysis of grammatical features, to trace classification to specific word-replacement patterns. The companion identity is the 0.29-bit unigram-entropy gap between the two classes, which quantifies the model's greater synonym variety and serves as a single-number summary of the detection signal.","core_discovery":"The paper claims that on the DET12 data set — 8,068 short excerpts from a dozen human-written detective novels plus 8,068 GPT-rewritten versions — a linear classifier with bag-of-words unigram features reaches 98.14% test accuracy, while 119 human evaluators score 49.9%, essentially chance. The decisive property is that the rewrites use a wider vocabulary: the unigram entropy of the generated class is 9.164 bits versus 8.875 for the human class, about 22% more word-choice variation. Manual annotation of the highest-weighted features attributes this to rephrasing (especially replacement of simple verbs, adjectives, and nouns by less common synonyms), plus secondary effects of temporal drift,","pith_inferences":["A clean way to test the synonym-variety mechanism is to instruct the model to avoid synonyms and to match the source text's vocabulary distribution; if the entropy gap narrows and classifier accuracy falls, the mechanism is confirmed rather than merely correlated.","The temporal-drift and Americanism cues are probably specific to this mid-century British corpus; contemporary human fiction would not provide the same contrast, so the 0.98 accuracy should not be expected to transfer across eras or genres without recalibration.","The human near-chance result is a comment on task setting as much as on human ability: evaluating side-by-side pairs under time pressure may imperfectly reflect real-world detection, so the human–machine gap deserves a more ecologically valid follow-up.","Because the split procedure is not fully specified, the first thing a reader should check is whether any original-rewrite pair straddles train and test; this single detail determines whether the paper's accuracy numbers are about style or about content memory."],"forward_implications":["An editorial workflow could flag likely AI-rewritten fiction automatically, because the signal survives on short 100-word samples.","The detection does not hinge on artefacts of punctuation, casing, or layout; normalising these did not change accuracy, so the signal lies in word choice.","Post-processing that simply edits the most salient words would need to remove roughly 80 to 100 heavy-weight features to drop accuracy from 0.98 to around 0.88, and even then the classifier remains far above human performance.","Adding n-grams, TF-IDF, or grammatical features does little beyond unigram counts, suggesting the information is already present in single-word frequencies.","Preliminary evidence from the authors indicates that generating from an outline rather than rewriting still leaves the text detectable, though this is explicitly presented as needing further work."],"fun_headline_variants":["Machines spot AI fiction 98% of time; humans flip a coin","Synonym variety gives away AI-written fiction","AI fiction unmasked by word choice, not style","Linear model reads synonym variety to catch AI fiction","AI fiction: machines 98%, humans 50%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper does not state that each original and its rewrite were always placed in the same data split, so if a test-set rewrite has its source human text in training, the 'unseen' test set is not unseen and 98% accuracy could reflect content memory rather than style discrimination.","fun_headline_variants_meta":{"raw":{"variants":["Machines spot AI fiction 98% of time; humans flip a coin","Synonym variety gives away AI-written fiction","AI fiction unmasked by word choice, not style","Linear model reads synonym variety to catch AI fiction","AI fiction: machines 98%, humans 50%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0006,"raw_usage":{"total_tokens":2651,"prompt_tokens":763,"completion_tokens":1888,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":1809}},"tokens_in":507,"tokens_out":1888,"duration_ms":13774,"temperature":1.0,"reasoning_tokens":1809,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T11:07:02.486702+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the released DET12 data: if any test-set sample's paired counterpart appears in the training set, the core accuracy claim collapses to memorization. Alternatively, on a held-out corpus of contemporary fiction and rewrites, compute the per-class unigram entropy gap; if classifiers still reach 0.98 while the gap disappears, synonym variety is not the operative cue.","supporting_citations":[],"review_version":1}