REVIEW 4 major objections 6 minor 1 cited by
Human Variability vs. Machine Consistency: A Linguistic Analysis of Texts Generated by Humans and Large Language Models
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read This paper argues that human-written and machine-generated texts differ most sharply in variability: human texts vary far more in their linguistic profiles, and the gap grows in genres with flexible stylistic rules.
desk verdict Likely true but not established: length confounding undermines the headline variability claim, though the domain breakdown is a useful extension. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The analytical engine is a 250-dimension linguistic profile: 247 handcrafted features from the LFTK toolkit (covering surface, lexico-semantics, discourse, and syntax), plus three added measures – average syntactic depth from a SpaCy dependency parse, semantic distance from pairwise cosine similarity of sentence embeddings (paraphrase-MiniLM-L6-v2), and emotional content from the NRC Emotion Intensity Lexicon across eight emotion categories. Variability is quantified as each text source's distance from its cluster centroid in a two-dimensional PCA of the full feature set, and a logistic regression on all 250 features is used for authorship classification. The machinery's job is to show that a small set of human-interpretable features, none of them black-box embeddings, can carry both explanation and prediction.
What would settle it
Repeat the same feature computation after truncating all human documents to the median LLM length or after regressing each feature on token count and analyzing the residuals; if the human-machine gaps in variability, semantic distance, and negative emotion largely disappear, the paper's central claim is refuted, while if they persist at similar magnitude, the claim survives this direct challenge.
Extended reading notes
Core claim
On its own terms, the paper claims that human-written texts and LLM-generated texts differ not just in style but in statistical behavior. Humans exhibit considerably higher variability across nearly all 250 measured linguistic features, and this variability is domain-dependent: in Wikipedia and Reddit, where style constraints are relaxed, the human cluster in PCA space spreads far wider than the machine clusters (e.g., distance-from-centroid values of 296.8 and 50.7 for humans versus single digits for most LLMs), while in arXiv the gap nearly vanishes (4.7 vs. 1.6–9.7). Beyond variability, humans write shallower syntactic trees (lower average syntactic depth), show greater semantic distance between sentences while using richer vocabulary, and score higher on emotional intensity, particularly for negative emotions such as anger. The authors interpret these results as evidence that LLM outputs are homogenized and emotionally flattened, likely shaped by training and alignment, and that human text is more cognitively economical, meaningful per unit, and emotionally engaged.
Load-bearing premise
The analysis assumes human and machine texts are comparable even though human documents average about twice the token count of machine documents, and many features are raw, unnormalized counts; if the extra length of human text drives the variability and emotion gaps, the claim that humans are intrinsically more variable would collapse.
Editorial extensions
If this is right
- The reported classifier reaches 87% accuracy in the in-domain, same-generator setting, so handcrafted, explainable features can support accurate authorship detection.
- Detection will be easier in informal, unconstrained genres and harder in formal, rigid genres, since the human-machine gap shrinks where style constraints are tight.
- The consistent machine profile across five different LLMs implies that model families may be identifiable by shared linguistic fingerprints, despite different architectures and training data.
- If the suppression of negative emotion in LLM texts is replicated, machine text may be emotionally narrower by design, with implications for affective computing and content moderation.
- Variability, rather than any single mean difference, may be the most robust signal for separating humans from machines.
Reading between the lines
- Because human documents average about 706 tokens versus 187–412 for the five LLMs, and several features are raw unnormalized counts, a length-matched or residualized analysis is needed to test whether the variability gap is intrinsic to human style or partly a length artifact.
- If variability is the load-bearing signal, detectors should measure dispersion within authors or genres rather than mean differences, a concrete design principle for future systems.
- The finding that all five LLMs cluster tightly while humans scatter wide suggests a general 'machine style' of current training objectives; testing decoder-only models trained without reinforcement learning from human feedback would isolate the role of alignment in this homogenization.
- The emotional flattening result implies a testable hypothesis: LLMs instructed to 'express anger' should still show lower anger intensity than humans in equivalent contexts, indicating a model-level bias rather than a prompt artifact.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript analyzes human-written and LLM-generated texts from the M4 corpus (SemEval 2024 Task 8 Subtask B), profiling each document with 250 linguistic features from LFTK plus measures of syntactic depth, semantic distance, and emotional content. The authors apply PCA and report that human texts show considerably higher variability than LLM texts, especially in less constrained domains (Wikipedia, Reddit), and that humans produce shallower syntax, more unique words, greater semantic distance, and more negative emotion. A logistic regression on the linguistic features is reported to distinguish human from machine text with high accuracy.
Significance. The paper addresses a timely question—whether interpretable linguistic features can reveal systematic differences between human and machine text. Its strengths are the use of a large public benchmark, the breadth of features (surface, syntactic, semantic, emotional), and domain-stratified analyses. If the variability claim survives length normalization and inferential testing, it would be a useful empirical contribution, with implications for explainable detection of machine-generated text. At present the core evidence for the variability claim is confounded by document length, and the semantic findings contain internal contradictions; these are fixable but require a substantive revision.
major comments (4)
- [§5, Table 3 vs Table 1] The central variability claim rests on Table 3, where variability is measured as distance from each source's centroid in a two-dimensional PCA of all 250 features. Table 1 reports average token counts of 706 for humans versus 187–412 for the five LLMs, and many LFTK features are raw counts (words, characters, spaces, unique words). The paper does not state that features were standardized before PCA, and no length normalization is applied in this analysis. Longer documents therefore contribute larger count values and larger count variance, so the PCA centroid distances (e.g., human Wikipedia 296.75 vs. ChatGPT 4.72) may largely encode length differences rather than intrinsic linguistic variability. I request a reanalysis with length-normalized features (e.g., per 1,000 tokens or residualized on length) and/or feature standardization, with a demonstration that the human-vs-LLM variability gap persists.
- [§5, Table 3] The variability estimates in Table 3 are reported as single numbers without confidence intervals, bootstraps, or significance tests. Because the claim is specifically that human variability is higher, the authors should compare the full distributions of per-document distances from the centroid (e.g., via permutation tests or bootstrap CIs) rather than only the mean distance. Without such tests, differences like the Wikipedia human/ChatGPT gap cannot be distinguished from sampling variation or outlier influence.
- [§5, Figures 6–8 and discussion of unique words] The vocabulary and semantic-distance findings are also length-sensitive. Unique-word counts are raw counts, and type-token ratio mechanically declines with text length for natural language, so the statement in the introduction that humans show a lower TTR while having a 'richer vocabulary' is an artifact of comparing longer human documents with shorter LLM documents unless a length-controlled vocabulary richness measure (e.g., MATTR or moving-average TTR) is used. Likewise, semantic distance from pairwise sentence comparisons depends on the number of sentences per document, which varies with length. The analyses should be repeated on length-matched subsamples or with length-normalized features before claiming that humans have richer vocabulary and higher semantic content.
- [§5 vs §6] The paper's statements about semantic consistency are mutually contradictory. Section 5 and Figure 2 show that humans have higher average semantic distance (i.e., lower pairwise similarity), and the introduction says humans 'tend to employ less similar semantic content.' Yet Section 6 concludes that 'HWT contain richer semantic content and show greater consistency in meaning than LLMs.' The authors should clarify whether higher semantic distance is interpreted as less consistency or as greater content richness, and align the text across sections.
minor comments (6)
- [Abstract/Introduction vs §5] The accuracy is reported as 0.82 in the abstract and introduction but as 87.15% (0.87) in Section 5 and Table 6; these numbers should be reconciled or the settings for each should be stated.
- [§5] The phrase 'achieving a prediction accuracy of 93%' after discussing the most prominent features is unclear; Table 6 reports human-class F1 of 0.93, so the text should specify that this is the human-class F1 (or recall/precision), not overall accuracy.
- [§4.2] Please state explicitly whether the 250 features were standardized or scaled before PCA, and report the total variance explained by the two components shown (18.82% + 8.92% = 27.74%), so readers can judge how much variability the centroid-distance measure summarizes.
- [§4.1] The terms 'semantic distance' and 'semantic similarity' are used interchangeably; define distance = 1 − cosine similarity in the methodology and use the terms consistently.
- [§8–9] The code-release sentence ('All codes will be released in the camera-ready version due to anonymity reasons') is confusing in an arXiv preprint with visible author names; rephrase to a standard data/code availability statement.
- [§10.1] The description of the training/test split (random selection of 5,000 elements per model) should clarify whether the test set is balanced across the six classes and whether the classifier is trained on all domains combined or per domain.
Circularity Check
No circularity: descriptive empirical analysis; no derivation step reduces to its inputs.
full rationale
This paper is a descriptive empirical study, not a derivation chain. The central claim that human texts are more variable than LLM texts is supported by measurements: LFTK features, syntactic depth from SpaCy, semantic distance from a sentence transformer, emotional content from the NRC lexicon, and PCA centroid distances. Each of these is an independent measurement on the corpus, not a quantity defined in terms of the conclusion. The PCA variability analysis is presented as a measurement, and the logistic classifier is explicitly described as a fitted classifier evaluated on a held-out test set (accuracy 0.87), not as a prediction derived from the variability claim. The only self-citation, Aroyehun et al. (2023), is used to choose which emotions count as negative versus positive in the NRC lexicon; this is a conventional grouping and does not carry the central variability claim. The paper's own limitations (Section 8) acknowledge generalizability and temporal validity concerns but do not reveal any circular step. The length difference between human and LLM texts (Table 1) is a potential confound for raw count features, but that is an internal validity or correctness concern, not circularity: the paper does not define the variability measure in terms of length, nor does it fit any parameter and then report it as a prediction. No equation in the paper reduces by construction to an input, and no load-bearing argument relies on a self-citation chain. The analysis is self-contained against the dataset and external tools, so the appropriate circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption Human and machine texts in the M4 dataset are aligned and comparable for direct feature comparison despite large length differences.
- domain assumption LFTK's 247 features plus SpaCy dependency depth, MiniLM semantic similarity, and NRC lexicon emotion scores validly capture the linguistic constructs the paper attributes to them.
- standard math Kruskal-Wallis and Dunn's tests assume independent samples, and the PCA is fit on the full corpus; these statistical assumptions are not evaluated.
Cite this review
Pith. "Pith review of Human Variability vs. Machine Consistency: A Linguistic Analysis of Texts Generated by Humans and Large Language Models." pith.science (2026). https://pith.science/paper/P3XJIKG3
@misc{pith2026241203025,
author = {Pith},
title = {Pith review of: Human Variability vs. Machine Consistency: A Linguistic Analysis of Texts Generated by Humans and Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/P3XJIKG3}},
note = {Machine review of arXiv:2412.03025}
}
read the original abstract
The rapid advancements in large language models (LLMs) have significantly improved their ability to generate natural language, making texts generated by LLMs increasingly indistinguishable from human-written texts. Recent research has predominantly focused on using LLMs to classify text as either human-written or machine-generated. In our study, we adopt a different approach by profiling texts spanning four domains based on 250 distinct linguistic features. We select the M4 dataset from the Subtask B of SemEval 2024 Task 8. We automatically calculate various linguistic features with the LFTK tool and additionally measure the average syntactic depth, semantic similarity, and emotional content for each document. We then apply a two-dimensional PCA reduction to all the calculated features. Our analyses reveal significant differences between human-written texts and those generated by LLMs, particularly in the variability of these features, which we find to be considerably higher in human-written texts. This discrepancy is especially evident in text genres with less rigid linguistic style constraints. Our findings indicate that humans write texts that are less cognitively demanding, with higher semantic content, and richer emotional content compared to texts generated by LLMs. These insights underscore the need for incorporating meaningful linguistic features to enhance the understanding of textual outputs of LLMs.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 1 Pith paper
-
Modeling Professionalism in Expert Questioning through Linguistic Differentiation
The paper claims linguistic features such as discourse markers and prefaces can both predict perceived professionalism and distinguish human from LLM-generated expert questions, but the correlation evidence is interna...
Reference graph
Works this paper leans on
-
[1]
Mohamed Hesham Ibrahim Abdalla, Simon Malberg, Daryna Dementieva, Edoardo Mosca, and Georg Groh. 2023. https://doi.org/10.3390/info14100522 A benchmark dataset to distinguish human-written and machine-generated scientific papers . Information, 14(10)
-
[2]
Segun Taofeek Aroyehun, Lukas Malik, Hannah Metzler, Nikolas Haimerl, Anna Di Natale, and David Garcia. 2023. Leia: Linguistic embeddings for the identification of affect. EPJ Data Science, 12(1):52
work page 2023
-
[3]
Jessie S. Barrot. 2023. https://doi.org/10.1016/j.asw.2023.100745 Using chatgpt for second language writing: Pitfalls and potentials . Assessing Writing, 57:100745
arXiv 2023
-
[4]
Roger E Beaty and Dan R Johnson. 2021. Automating creativity assessment with semdis: An open platform for computing semantic distance. Behavior research methods, 53(2):757--780
work page 2021
-
[5]
Douglas Biber and Susan Conrad. 2019. Register, genre, and style. Cambridge University Press
2019
-
[6]
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned llm. Company Blog of Databricks
2023
-
[7]
Evan N Crothers, Nathalie Japkowicz, and Herna L Viktor. 2023. Machine-generated text: A comprehensive survey of threat models and detection methods. IEEE Access, 11:70977--71002
2023
-
[8]
Alexis Dinno. 2015. Nonparametric pairwise multiple comparisons in independent groups using dunn's test. The Stata Journal, 15(1):292--300
work page 2015
Show all 34 references
-
[9]
Liam Dugan, Daphne Ippolito, Arun Kirubarajan, Sherry Shi, and Chris Callison-Burch. 2023. Real or fake text?: Investigating human ability to detect boundaries between human-written and machine-generated text. In Proceedings of the AAAI Conference on Artificial Intelligence, v...
2023
-
[10]
Elisa Ferracane, Su Wang, and Raymond Mooney. 2017. https://aclanthology.org/I17-1059 Leveraging discourse information effectively for authorship attribution . In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), p...
2017
-
[11]
Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. 2019. https://doi.org/10.18653/v1/P19-3019 GLTR : Statistical detection and visualization of generated text . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrat...
2019 doi
-
[12]
Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597
2023 arXiv
-
[13]
Peter Hagoort, Colin M Brown, and Lee Osterhout. 1999. The neurocognition of syntactic processing. The neurocognition of language, pages 273--316
1999
-
[14]
Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. 2020. https://doi.org/10.18653/v1/2020.acl-main.164 Automatic detection of generated text is easiest when humans are fooled . In Proceedings of the 58th Annual Meeting of the Association for Computationa...
2020 doi
-
[15]
Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks Lakshmanan, V.S. 2020. https://doi.org/10.18653/v1/2020.coling-main.208 Automatic detection of machine generated text: A critical survey . In Proceedings of the 28th International Conference on Computational Linguistics, pages 22...
2020 doi
-
[16]
Tharindu Kumarage, Joshua Garland, Amrita Bhattacharjee, Kirill Trapeznikov, Scott Ruston, and Huan Liu. 2023. Stylometric detection of ai-generated text in twitter timelines. arXiv preprint arXiv:2303.03697
2023 arXiv
-
[17]
Lee and Jason Lee
Bruce W. Lee and Jason Lee. 2023. https://aclanthology.org/2023.bea-1.1 LFTK : Handcrafted features in computational linguistics . In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 1--19, Toronto, Canada. Assoc...
2023
-
[18]
Jiwei Li, Myle Ott, Claire Cardie, and Eduard Hovy. 2014. https://doi.org/10.3115/v1/P14-1147 Towards a general rule for identifying deceptive opinion spam . In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pag...
2014 doi
-
[19]
Ta Lin Liau, Carolyn B Bassin, Clessen J Martin, and Edmund B Coleman. 1976. Modification of the coleman readability formulas. Journal of Reading Behavior, 8(4):381--386
1976
-
[20]
Yongqiang Ma, Jiawei Liu, Fan Yi, Qikai Cheng, Yong Huang, Wei Lu, and Xiaozhong Liu. 2023. Ai vs. human--differentiation analysis of scientific content generation. arXiv preprint arXiv:2301.10416
2023 arXiv
-
[21]
Patrick E McKight and Julius Najab. 2010. Kruskal-wallis test. The corsini encyclopedia of psychology, pages 1--1
2010
-
[22]
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. 2023. Detectgpt: Zero-shot machine-generated text detection using probability curvature. In International Conference on Machine Learning, pages 24950--24962. PMLR
2023
-
[23]
Mohammad and Peter D
Saif M. Mohammad and Peter D. Turney. 2013. Crowdsourcing a word-emotion association lexicon. Computational Intelligence, 29(3):436--465
2013
-
[24]
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward ...
2023 doi
-
[25]
Nils Reimers and Iryna Gurevych. 2019. http://arxiv.org/abs/1908.10084 Sentence-bert: Sentence embeddings using siamese bert-networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics
2019 arXiv
-
[26]
Yanir Seroussi, Ingrid Zukerman, and Fabian Bohnert. 2011. Authorship attribution with latent dirichlet allocation. In Proceedings of the fifteenth conference on computational natural language learning, pages 181--189
2011
-
[27]
Yanir Seroussi, Ingrid Zukerman, and Fabian Bohnert. 2014. Authorship attribution with topic models. Computational Linguistics, 40(2):269--310
2014
-
[28]
Abhay Sharma, Ananya Nandan, and Reetika Ralhan. 2018. An investigation of supervised learning methods for authorship attribution in short hinglish texts using char & word n-grams. arXiv preprint arXiv:1812.10281
2018 arXiv
-
[29]
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. 2019. Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203
2019 arXiv
-
[30]
Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.673 Authorship attribution for neural text generation . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8384--8395, Onli...
2020 doi
-
[31]
Yuxia Wang, Jonibek Mansurov, Petar Ivanov, jinyan su, Artem Shelmanov, Akim Tsvigun, Osama Mohammed Afzal, Tarek Mahmoud, Giovanni Puccetti, Thomas Arnold, Chenxi Whitehouse, Alham Fikri Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. 2024 a . https://aclanthology.org/2...
2024
-
[32]
Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Toru Sasaki, Thomas Arnold, Alham Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. 2024 b . https://aclanthology.org/2024.eacl-lon...
2024
-
[33]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[34]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.