{"id":"92de8386-375c-4ace-bf7b-c2b62e884653","arxiv_id":"2507.00838","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":5,"one_line_summary":"Stylometric features and tree-based classifiers separate human-written Wikipedia summaries from LLM-generated texts with high cross-validated accuracy on a new seven-class benchmark, though performance drops on other text types.","lead":"This paper shows that stylometric features, such as word frequencies and grammar patterns, let tree-based machine learning models tell apart human-written Wikipedia summaries from texts generated by six large language models, using only 10-sentence samples. The finding is a step toward practical AI-text detection, though the results hold mainly for encyclopedic-style English text and weaken on other text types.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Human baseline mixes a 2022 snapshot with 1,048 live Wikipedia-API terms; without revision dates, the pre-GPT-3.5 purity argument covers only part of the human class.","rationale":"The reader's weakest assumption was that the 2022 Wikipedia snapshot is free of LLM-generated or LLM-edited content. I agree that this is the load-bearing point, but the paper's own methodology makes the concern sharper: the human baseline is not solely the 2022 snapshot. Section 3.1.1 explicitly combines 1,500 snapshot terms with 1,048 terms from the Wikipedia-API, and the date-based contamination argument is only stated for the snapshot. If the API subset is live Wikipedia from 2024-2025, then the central human-versus-machine contrast is compromised for a large fraction of the samples, independent of how well the classifiers work internally. This is a data-validity issue rather than a modeling issue, so it does not attack the method's merits; it attacks whether the reported numbers mean what the abstract claims. The proposed test is straightforward because both the code and data are released: verify the revision dates of the API subset and rerun the experiments on the two subsets separately. The reader's conditional verdict already accounts for a contamination risk, and my finding reinforces it rather than moving the verdict. I therefore keep the verdict unchanged, pending the check.","tokens_in":27922,"tokens_out":12415,"duration_ms":151498,"concrete_test":"From the released OSF/GitHub repository, inspect the Wikipedia-API retrieval script and determine whether it fetched current revisions or an 'as of' date. For each of the 1,048 API-sourced terms, compare the stored text with the 2022-03-01 revision via the Wikipedia API, or examine the edit history for AI/bot edit markers. Then rerun the main binary wiki-versus-GPT-4 and 7-class experiments twice: once with the human class restricted to the 1,500 HuggingFace-snapshot terms, and once restricted to the 1,048 API terms. If the accuracy/MCC drops sharply for the API subset, or if the stored API texts differ from their 2022 revisions, the human baseline is contaminated; if the gap is negligible and the revisions match, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1.1 builds the human class from two sources: 1,500 terms from the HuggingFace snapshot 20220301.simple and 1,048 terms collected via Wikipedia-API. The paper's only protection against LLM contamination is the 2022 snapshot date, chosen to predate GPT-3.5. That protection is argued for the first source only; the paper reports no retrieval date, no revision IDs, and no 'as-of' parameter for the Wikipedia-API subset. If those pages were fetched when the study was run (2024-2025), roughly 43% of the 2,424 human samples come from a period in which Wikipedia text was increasingly AI-generated or AI-edited (Brooks et al., 2024). In that case the reported binary accuracy of 0.98 for Wikipedia versus GPT-4 would partly compare two machine-influenced distributions rather than human versus machine, and the central claim would not be established by this dataset. The concern is directly checkable from the released code and data, and it does not depend on any assumption about the stylometric features themselves.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a controlled study of stylometric classification of short (10-sentence) Wikipedia-style texts into human-written and LLM-generated classes. The dataset combines human Wikipedia summaries from a 2022 snapshot and a Wikipedia-API collection, LLM outputs from six models (GPT-3.5, GPT-4, LLaMa 2/3, Orca, Falcon), and outputs of four summarizers. Using StyloMetrix and a frequency-based CLARIN-PL pipeline with decision trees and LightGBM under group cross-validation, the authors report a multiclass MCC up to 0.87 and binary accuracies up to 1.00, with the Wikipedia-vs-GPT-4 case at 0.98. They also present robustness tests on unseen models, paraphrased texts, and the AuTexTification benchmark, plus SHAP-based explanations of discriminating features such as grammatical standardization and overused words. The central claim is that stylometry can distinguish machine- from human-generated text for this well-defined text type.","tokens_in":28162,"tokens_out":7359,"duration_ms":85878,"significance":"If the central claim holds, the paper provides a useful, interpretable, and inexpensive complement to neural MGT detectors: it shows that boosted trees on linguistic frequency features can give high accuracy on short encyclopedic texts, with explainable feature attributions. The study merits credit for a controlled dataset, group cross-validation to prevent topic leakage, explicit tests on unseen LLMs, and honest reporting of a cross-domain benchmark where the frequency pipeline underperforms a logistic-regression baseline. The released code and data (OSF and GitHub) support reproducibility. The main significance is therefore not a new detector architecture but a careful demonstration of the strengths and limits of explainable stylometry in a narrow domain.","major_comments":[{"comment":"The human baseline is not verifiably pre-LLM. Only the HuggingFace snapshot 20220301.simple is dated, while the 1,048 Wikipedia-API samples have no reported retrieval date, revision IDs, or as-of parameter. Since these samples constitute roughly 43% of the human class, the binary and multiclass results (e.g., Table 5's wiki accuracy of 0.98-1.00) cannot be interpreted as human-versus-machine unless this subset is shown to predate GPT-3.5 or to be free of LLM edits. The concern is directly checkable from the released code and data; please report the retrieval metadata and either stratify the analysis by source or verify the absence of AI-assisted content.","section":"Section 3.1.1"},{"comment":"The abstract's statement of 'accuracy between .79 and 1. in binary classification' is not supported by the reported experiments. Table 4 includes decision-tree accuracies as low as 0.53 (prompt #2, Orca vs. LLaMa 2), and the StyloMetrix panel of Table 5 includes 0.77 (LLaMa 2 vs. Orca). If the intended scope is only the LGBM frequency-based pipeline, the abstract should say so explicitly; otherwise the stated range misrepresents the full set of binary results.","section":"Abstract and Tables 4-5"},{"comment":"The paraphrase robustness test is presented as showing no collapse, but the paper does not address the possibility that the high recall on Dipper- and Parrot-paraphrased texts is driven by stylistic fingerprints of the paraphrase models themselves rather than by persistence of the original LLM's signals. The training set contains only unparaphrased texts, so a positive test could mean that the classifier recognizes 'was paraphrased by Dipper/Parrot' rather than 'is a paraphrased LLM output'. Please add a control with human texts passed through the same paraphrasers, or at least discuss this confound explicitly.","section":"Section 4.4.2 and Table 8"}],"minor_comments":[{"comment":"The text first states that the final dataset consists of 2,439 terms and then says that after removing outlying texts it consists of 2,424 terms; please clarify which number is used for the reported experiments.","section":"Section 3.1.1"},{"comment":"There are repeated typos, including 'Metholodogy' in section headings and 'dynamicaly' in the Introduction; a careful proofread is needed.","section":"Section 3 and elsewhere"},{"comment":"The caption says each entry corresponds to a task with class 1 and 2 as column and row labels, but the table appears to show only one triangle of a symmetric matrix; please clarify the layout.","section":"Table 4"},{"comment":"The feature description says lemmas and POS tags exclude named entities, but the list later refers to 'excluding named entities and punctuation' without specifying which feature groups exclude punctuation; please make the definitions precise.","section":"Section 3.3.2"},{"comment":"Several references contain incomplete author lists with '...' placeholders (e.g., Hu et al. 2023, Liu et al. 2023a, Mikros et al. 2023); these should be completed before publication.","section":"References"},{"comment":"The text refers to 'noa (2025)' but the table caption names GPTZero; please correct the in-text citation or clarify the entity being referenced.","section":"Section 5.2 and Table 11"}],"recommendation":"major_revision","confidential_remarks":"The empirical core is plausible and the authors are honest about several limitations, including the closed-set multiclass setting and the weak cross-domain performance. The load-bearing issue is the unverifiable purity of the Wikipedia-API subset of the human class; if the authors can demonstrate that this subset is pre-GPT-3.5 or otherwise uncontaminated, or can show that the main results are unchanged when the two human sources are analyzed separately, I would support acceptance after revision. The abstract's binary accuracy range should also be corrected to match the tables."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper builds a useful, well-controlled benchmark for stylometric detection of LLM text in short Wikipedia-style samples, and the central claim—that tree classifiers on hand-crafted features separate these texts—holds up for the defined text type. The caveat that matters: the human baseline is a mix of a 2022 snapshot (1,500 terms) and a live Wikipedia-API fetch (1,048 terms) with no retrieval date or revision IDs. The paper's purity argument (pre-GPT-3.5) covers only the first part. If those API pages were pulled in 2024-25, some may contain AI-assisted edits, which means the reported 0.98 wiki-vs-GPT-4 accuracy might partly be machine-vs-machine. That's a direct, checkable issue from the released code/data, not a fatal one—but it should be fixed by reporting revision dates or filtering to pre-2023 revisions.\n\nWhat's genuinely good: the dataset design, group cross-validation to prevent topic leakage, honest robustness tests (unseen models, paraphrase attacks, AuTexTification), and SHAP-based explainability that gives concrete feature insights. The related-work section properly credits prior stylometric detection (Kumarage et al., Bhattacharjee et al.), so the novelty is the specific benchmark and evaluation, not the general approach. That's a fair contribution.\n\nMinor soft spots: the abstract reports the binary accuracy range .79–1.00 as if it were overall performance, when it's actually a range over all class pairs; the per-pair numbers are the real story. Feature selection is not nested inside the cross-validation loop, which can slightly inflate performance estimates. Reproducibility artifacts lack a commit hash, though the OSF/GitHub links look live.\n\nWho should read it: anyone working on MGT detection, especially for short encyclopedic texts, and people interested in explainable stylometric features. It deserves a serious referee—the data and protocol are reusable, and the baseline-contamination question is answerable with minor extra work. I'd send it out for review rather than desk-reject.","headline":"Solid, well-controlled stylometric benchmark for short Wikipedia-style LLM detection, with a real but fixable human-baseline contamination issue.","tokens_in":28711,"tokens_out":3096,"would_cite":false,"duration_ms":34781,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Stylometric fingerprints of words, grammar, and punctuation separate human from LLM-written ten-sentence encyclopedic texts, reaching 0.98 accuracy in the Wikipedia-versus-GPT-4 case and 0.87 Matthews correlation across seven classes.","keywords":["stylometry","machine-generated text detection","large language models","authorship attribution","LightGBM","SHAP explanations","Wikipedia benchmark","text classification"],"falsifier":"Audit the 2,424 term summaries by tracing edit histories or applying an independent detector, and count any machine-written or machine-edited passages; even a small contamination rate would require recomputing the reported 0.98 accuracy. Cleaner still: rerun the Wikipedia-versus-GPT-4 binary experiment on a corpus verifiably written and edited before 2018; if accuracy falls well below 0.98, the original separation leaned on the unverified baseline.","tokens_in":27753,"feed_emoji":"🔍","tokens_out":15967,"duration_ms":157247,"temperature":0.7,"pith_summary":"The paper tries to show that stylometry—the quantitative study of linguistic style—can identify machine-generated text even in samples as short as ten sentences, at least for the well-defined text type of encyclopedic Wikipedia introductions. The authors built a benchmark of 2,424 human term summaries from a 2022 Wikipedia snapshot and comparable ten-sentence descriptions produced by GPT-3.5/4, LLaMa 2/3, Orca, and Falcon, then classified them with tree-based models on two kinds of interpretable features: 195 handcrafted StyloMetrix features and thousands of frequency-based n-gram features. Cross-validated results reached 0.87 Matthews correlation coefficient in seven-class attribution and binary accuracy up to 0.98 for Wikipedia versus GPT-4, with SHAP explanations showing markers such as overused words, punctuation patterns, and more standardized grammar in LLM text. The authors deliberately limit the claim: their own cross-domain test on the AuTexTification benchmark scored macro-F1 0.54 with frequency features, so the method's strength is tied to the specific genre, not to all text. If the claim holds, it gives a cheap, explainable detection route that needs no watermark and no access to the generating model.","feed_headline":"Stylometry flags AI-written Wikipedia text at 98% accuracy","feed_subtitle":"Word-frequency fingerprints separate six AI models from Wikipedia writers, with no watermark or model access.","key_machinery":"The load-bearing machinery is a feature-engineering pipeline that converts short texts into normalized frequency vectors over linguistically annotated units: lemmas (uni- to trigrams), part-of-speech tags, dependency bigrams, and morphological annotations, alongside StyloMetrix's 195 engineered stylometric features. These vectors feed LightGBM, a gradient-boosted tree classifier, under 10-fold group cross-validation that keeps a given term's topic out of both training and test folds. Shapley Additive Explanations (SHAP), a per-prediction attribution method, then links each classification decision back to individual features, exposing that whitespace tokens, punctuation, proper nouns, dates, and part-of-speech bigrams carry much of the discriminative signal. This combination is what makes the approach explainable and cheap: no watermark, no access to model logits, and no neural classifier is needed.","core_discovery":"On the paper's own terms, the discovery is that LLM prose carries a stable, measurable stylistic signature that a gradient-boosted tree classifier can exploit. Using ten-sentence Wikipedia-style samples, frequency-based features—normalized counts of lemmas, part-of-speech uni- to trigrams, dependency bigrams, and morphological tags—achieve near-perfect binary separation between Wikipedia summaries and each of the six LLMs (up to 0.98 for Wikipedia versus GPT-4, with several pairs at 1.00), and a 0.87 Matthews correlation coefficient separating all seven sources at once. SHAP attributions show the signal is distributed: Wikipedia samples are richer in proper nouns and numerals, while LLM outputs show standardized part-of-speech sequences, distinctive punctuation and whitespace behaviour, and overuse of particular words such as 'significant', 'notable', and 'despite'. The detector also generalizes to unseen LLMs and survives single-step paraphrase attacks with recall at or above the unparaphrased validation level in nearly all cases, the main exceptions being GPT-4 and LLaMa 3 when first encountered. The paper is explicit that these results demonstrate machine-versus-human detection for a defined text type, not a universal detector.","pith_inferences":["Because SHAP exposes the exact words and grammar patterns that separate LLM output from Wikipedia, the same tables could be read as a recipe for making future models less detectable: diversify part-of-speech sequences and suppress overused words. The paper does not test this, so it is an inference.","The near-total survival of paraphrase attacks suggests the discriminative signal sits in generation-level habits rather than in surface wording alone; a direct test would be recursive or iterative paraphrasing, which the authors explicitly leave to future work.","The closed-set seven-class experiment would become an open-set problem as soon as a new, unseen model appears; a practical detector would need an 'unknown model' output or a separate novelty-detection step, extending the paper's own acknowledgment that unseen models change the multiclass task.","Since the authors note Wikipedia samples are multi-author and multi-editor, the 'human' class may actually be a mixture of many styles; retraining the same pipeline on single-author human corpora would probably tighten the human class and could change which features dominate the explanations."],"forward_implications":["Ten-sentence encyclopedic text can be separated human-from-machine with near-perfect binary accuracy and attributed to a specific LLM at 0.87 MCC using only interpretable frequency features and boosted trees.","One-step paraphrase attacks with DIPPER and Parrot do not break detection: recall stays at 98.8% or higher for every LLM, so surface rewording is not a sufficient evasion for this text type.","Held-out LLMs are detected with high recall—88.2% validation recall for the hardest case, GPT-4, and above 94% for LLaMa 3—indicating some generalization across model families rather than memorization of the six training generators.","Cross-domain transfer remains a bottleneck: on AuTexTification the frequency features reach macro-F1 0.54 and StyloMetrix 0.48, below baselines, so accuracy is tied to the encyclopedic genre.","SHAP explanations turn detection into auditable claims—overused words like 'significant' and 'notable', lower punctuation variability, standardized grammatical sequences—that a human reviewer can check directly."],"supporting_citations":[{"why":"Anchors the dataset-timing choice: summaries were drawn from a 2022 snapshot so human text predates GPT-3.5, and it supplies the benchmark framing.","marker":"Bevendorff et al. (2024)"},{"why":"Provides AuTexTification, the external multi-domain benchmark used to test cross-domain robustness.","marker":"Sarvazyan et al. (2023a)"},{"why":"Defines the paraphrase-attack methodology, including DIPPER, that the robustness tests follow.","marker":"Sadasivan et al. (2025)"},{"why":"Introduces LightGBM, the boosted-tree classifier used in all main experiments.","marker":"Ke et al. (2017)"},{"why":"Provides SHAP, the explanation method that turns classifier outputs into per-feature attributions.","marker":"Lundberg et al. (2020)"},{"why":"Defines StyloMetrix, one of the two stylometric feature sets tested.","marker":"Okulska et al. (2023)"},{"why":"Describes the modular frequency-based stylometric pipeline and its use with tree classifiers and SHAP.","marker":"Ochab & Walkowiak (2024)"},{"why":"Supplies the dataset-loading library used to obtain the 2022 Wikipedia term summaries.","marker":"Lhoest et al. (2021)"},{"why":"Documents the rise of AI-generated content in Wikipedia, motivating the pre-GPT-3.5 date that keeps the human baseline uncontaminated.","marker":"Brooks et al. (2024)"}],"fun_headline_variants":["Stylometry separates AI from human prose at 98% accuracy","Short text style fingerprints identify AI writers at 98%","Wikipedia texts reveal AI authorship via stylometry","AI vs human writing: Style analysis hits 98% in short samples","Stylometry detects AI text in 10-sentence Wikipedia at 98%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 2022 Wikipedia snapshot used as the human baseline is assumed to contain no text written or edited by a large language model; if any of the 2,424 summaries were machine-assisted, the reported human-versus-machine separation is inflated.","fun_headline_variants_meta":{"raw":{"variants":["Stylometry separates AI from human prose at 98% accuracy","Short text style fingerprints identify AI writers at 98%","Wikipedia texts reveal AI authorship via stylometry","AI vs human writing: Style analysis hits 98% in short samples","Stylometry detects AI text in 10-sentence Wikipedia at 98%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001621,"raw_usage":{"total_tokens":6541,"prompt_tokens":1128,"completion_tokens":5413,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":744,"completion_tokens_details":{"reasoning_tokens":5325}},"tokens_in":744,"tokens_out":5413,"duration_ms":41129,"temperature":1.0,"reasoning_tokens":5325,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:05:29.364158+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit the 2,424 term summaries by tracing edit histories or applying an independent detector, and count any machine-written or machine-edited passages; even a small contamination rate would require recomputing the reported 0.98 accuracy. Cleaner still: rerun the Wikipedia-versus-GPT-4 binary experiment on a corpus verifiably written and edited before 2018; if accuracy falls well below 0.98, the original separation leaned on the unverified baseline.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the paraphrase-attack methodology, including DIPPER, that the robustness tests follow."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides SHAP, the explanation method that turns classifier outputs into per-feature attributions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the modular frequency-based stylometric pipeline and its use with tree classifiers and SHAP."}],"review_version":1}