REVIEW 2 major objections 4 minor 56 references
LLMs learn a lasting aversion to token self-repetition early in training; measuring that aversion detects their text.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-11 21:56 UTC pith:4LWPHD5P
load-bearing objection Simple single-pass detector that is competitive with Binoculars on modern targets, with clean early-training and locality evidence that is actually new. the 2 major comments →
Telescope: Improving Zero Shot Detection of LLM Generated Content By Measuring Token Repetition Probability
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper claims that the extreme local aversion to token self-repetition that language models acquire in the earliest, bigram-like phase of pre-training persists as a vestigial heuristic and is differentially activated by LLM-generated versus human text. Measuring that residual with Telescope Perplexity therefore separates the two classes of text with high accuracy in a zero-shot setting.
What carries the argument
Telescope Perplexity (Eq. 1): the average negative log probability that a reference model assigns to each token given the sequence up to and including that token, P(s_i | s_1:i). It isolates the model’s learned repetition probability rather than ordinary next-token perplexity.
Load-bearing premise
That the strong early aversion to a token repeating itself is never fully unlearned and remains a detectable signature across essentially all current model families and training mixes.
What would settle it
If a modern model family, or a late-stage training checkpoint of an existing family, produced text whose Telescope Perplexity distribution fully overlapped that of human writing of the same genre, the claimed vestigial signature would be absent.
If this is right
- A single-forward-pass detector can match or beat multi-model and multi-pass zero-shot methods on many contemporary target models.
- Thresholds still need domain-specific calibration; a score tuned on one genre can lose accuracy when transferred to another.
- Genres that force deliberate repetition (poetry, some news styles) can suppress the signature and produce confident false negatives.
- The same local-repetition probe can also separate synthetic versus human portions of a model’s own training corpus.
Where Pith is reading between the lines
- If the vestigial signal is truly fixed early, detectors built on it may remain useful longer than supervised classifiers that must be retrained for every new model family.
- Training interventions that deliberately re-introduce or penalize self-repetition later in pre-training could be tested as a way to erase the signature and thereby evade this class of detectors.
- The same early-local-bias logic may apply to other simple statistical regularities (e.g., local n-gram entropy) that form before long-range dependencies are mastered.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Telescope Perplexity (Eq. 1), the average negative log probability that a reference model assigns to immediate self-repetition of each token given the prefix that includes that token. The authors hypothesize that early pre-training instills a strong local aversion to token repetition that persists as a “Vestigial Heuristic” and is differentially activated by LLM-generated versus human text. They support the developmental claim with training-checkpoint curves (Pythia, Amber), bigram/trigram locality ablations (Table 1), and training-data separability experiments, then show that the metric yields competitive or superior zero-shot AUROC against Perplexity, DetectLLM-LRR, Binoculars and Fast-DetectGPT across twelve reference models, public and newly generated modern-target datasets (GPT-4o Mini, DeepSeek-V3), length/perturbation/humanizer ablations, and transferability tests, all with a single forward pass.
Significance. If the empirical ranking holds, Telescope Perplexity supplies a simple, single-pass, reference-model-agnostic detector that matches or exceeds the current zero-shot SOTA (Binoculars) while roughly halving compute relative to two-model methods and avoiding the multi-pass cost of perturbation detectors. The accompanying training-dynamics and locality evidence is a novel mechanistic contribution that links early bigram statistics to a practical detection signal. Strengths include extensive multi-model/multi-dataset evaluation with confidence intervals, public code, newly generated modern-target sets, and explicit discussion of domain sensitivity and threshold calibration. These elements make the work immediately useful for practitioners and a useful probe for future interpretability studies of early training biases.
major comments (2)
- Table 2 averages AUROC over twelve reference models of which six are SmolLM/SmolLM2 variants; the authors themselves note the resulting architectural bias. Because the headline claim of “state-of-the-art or competitive” performance rests on these averages, the main text should also report (or prominently highlight) the non-SmolLM subset averages and the per-model spreads already present in Appendix 8.20, so that readers can judge robustness without the over-representation of one family.
- Section 5.4 and Table 3 show that transferability F1 can drop substantially when the threshold is tuned on other domains. The practical recommendations correctly call for domain-specific calibration, yet the abstract and introduction still present Telescope Perplexity as a ready-to-deploy zero-shot detector. A short, explicit statement of the calibration requirement (and the expected degradation under pure transfer) should appear in the abstract or early results so that the efficiency claim is not overstated relative to methods that may be more threshold-stable.
minor comments (4)
- Eq. (1) is called “Telescope Perplexity” yet is not exponentiated; a one-sentence clarification that the monotonic transform does not affect ranking-based metrics would remove a possible source of confusion.
- Figure 2 and Appendix 8.13 show early plateauing of the signature; adding a brief quantitative summary (e.g., fraction of final value reached by 10 % of training) would make the “early emergence” claim easier to cite.
- The poetry/stylized-text failure mode (Appendix 8.7) is interesting; a short pointer in the main-text limitations section would help practitioners anticipate the genre-specific risk already quantified for news writing.
- Minor typographical inconsistencies appear in the arXiv identifier formatting and a few table captions; a light copy-edit pass would polish the presentation.
Circularity Check
No circularity: Telescope Perplexity is an independently defined single-pass statistic; detection power and the vestigial-heuristic interpretation are both evaluated against external labels and checkpoints.
full rationale
The paper defines Telescope Perplexity (Eq. 1) solely from a reference model’s conditional probabilities M(s_i | s_1:i) and never fits any free parameter of that definition to detection labels. Detection is ordinary thresholding of the resulting scalar; AUROC/F1 are measured on external human/LLM corpora (including newly generated GPT-4o-mini and DeepSeek-V3 sets). The supporting “vestigial-heuristic” story is tested with independent evidence—early-training checkpoints of Pythia and Amber (Fig. 2, App. 8.13), bigram/trigram ablations (Table 1), and separability of SmolLM’s own FineWeb vs. Cosmopedia subsets (App. 8.3)—none of which reuse the detection labels. Thresholds are either optimized on held-out data or transferred across datasets (Table 3). No equation, uniqueness claim, or self-citation reduces the reported AUROC rankings to a quantity fitted on the same labels used for evaluation. The work is therefore free of the six circularity patterns.
Axiom & Free-Parameter Ledger
free parameters (2)
- classification threshold τ
- minimum text length filter (100 words)
axioms (3)
- domain assumption Early in pre-training, language models behave approximately as bigram (or low-order n-gram) predictors and therefore learn strong local statistics such as token non-repetition.
- ad hoc to paper Statistical biases acquired in the earliest training phase are not fully unlearned later and remain detectable in the final model’s token probabilities.
- domain assumption Reference models drawn from different families and sizes share sufficiently similar early-training biases that the same scalar probe works across them.
invented entities (1)
-
Vestigial Heuristic
independent evidence
read the original abstract
Distinguishing Large Language Model (LLM) generated text from human writing is a critical and difficult challenge. While LLMs are trained to write like humans, we hypothesize that this training leaves an indelible mark. LLMs develop a particularly strong aversion to token repetition very early in training. This bias persists as a ''Vestigial Heuristic'' (a developmental artifact) that is activated in LLM-generated text, separating LLM from human writing. To probe this phenomenon, we introduce Telescope Perplexity, a metric that evaluates the token repetition of the model, $P(s_i | s_{1:i})$ . Our empirical investigation reveals that the Telescope Perplexity signature emerges early in pre-training, and Telescope Perplexity empirically enables highly effective zero-shot LLM detection. We show state-of-the-art or competitive performance across diverse datasets (including modern evaluation sets we introduce), reference models, and perturbation schemes with greater efficiency than other methods.
Figures
Reference graph
Works this paper leans on
-
[1]
Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text , booktitle =
Abhimanyu Hans and Avi Schwarzschild and Valeriia Cherepanova and Hamid Kazemi and Aniruddha Saha and Micah Goldblum and Jonas Geiping and Tom Goldstein , editor =. Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text , booktitle =. 2024 , url =
2024
-
[2]
arXiv , author =:2312.06550 , primaryclass =
LLM360: Towards Fully Transparent Open-Source LLMs , year =. arXiv , author =:2312.06550 , primaryclass =
-
[3]
Pythia: A suite for analyzing large language models across training and scaling , year =
Biderman, Stella and Schoelkopf, Hailey and Anthony, Quentin Gregory and Bradley, Herbie and O’Brien, Kyle and Hallahan, Eric and Khan, Mohammad Aflah and Purohit, Shivanshu and Prashanth, USVSN Sai and Raff, Edward and others , booktitle =. Pythia: A suite for analyzing large language models across training and scaling , year =
-
[4]
Ghostbuster: Detecting Text Ghostwritten by Large Language Models , booktitle =
Vivek Verma and Eve Fleisig and Nicholas Tomlin and Dan Klein , editor =. Ghostbuster: Detecting Text Ghostwritten by Large Language Models , booktitle =. 2024 , url =. doi:10.18653/V1/2024.NAACL-LONG.95 , timestamp =
-
[5]
arXiv , author =:2412.16525 , primaryclass =
Investigating Efficacy of Perplexity in Detecting LLM-Generated Code , url =. arXiv , author =:2412.16525 , primaryclass =
-
[6]
Manning and Chelsea Finn , editor =
Eric Mitchell and Yoonho Lee and Alexander Khazatsky and Christopher D. Manning and Chelsea Finn , editor =. DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature , booktitle =. 2023 , url =
2023
-
[7]
arXiv , author =:2311.16867 , primaryclass =
The Falcon Series of Open Language Models , url =. arXiv , author =:2311.16867 , primaryclass =
-
[8]
Loubna Ben Allal and Anton Lozhkov and Elie Bakouch , title =
-
[9]
arXiv , author =:2302.13971 , primaryclass =
LLaMA: Open and Efficient Foundation Language Models , url =. arXiv , author =:2302.13971 , primaryclass =
-
[10]
arXiv , author =:2307.09288 , primaryclass =
Llama 2: Open Foundation and Fine-Tuned Chat Models , url =. arXiv , author =:2307.09288 , primaryclass =
-
[11]
arXiv , author =:2407.21783 , primaryclass =
The Llama 3 Herd of Models , url =. arXiv , author =:2407.21783 , primaryclass =
-
[12]
arXiv , author =:2408.00118 , primaryclass =
Gemma 2: Improving Open Language Models at a Practical Size , url =. arXiv , author =:2408.00118 , primaryclass =
-
[13]
arXiv , author =:2310.15264 , primaryclass =
Towards Possibilities and Impossibilities of AI-generated Text Detection: A Survey , url =. arXiv , author =:2310.15264 , primaryclass =
-
[14]
2026 , eprint=
A Training-free Method for LLM Text Attribution , author=. 2026 , eprint=
2026
-
[15]
Approximation by superpositions of a sigmoidal function , volume =
Cybenko, G , journal =. Approximation by superpositions of a sigmoidal function , volume =
-
[16]
Can linguists distinguish between ChatGPT/AI and human writing?: A study of research ethics and academic publishing , url =
J Elliott Casal and Matt Kessler , doi =. Can linguists distinguish between ChatGPT/AI and human writing?: A study of research ethics and academic publishing , url =. Research Methods in Applied Linguistics , month =
-
[17]
Kristina Radivojevic and Nicholas Clark and Paul R. Brenner , editor =. LLMs Among Us: Generative. Proceedings of the. 2024 , url =. doi:10.1609/AAAISS.V3I1.31202 , timestamp =
-
[18]
Jinyan Su and Terry Yue Zhuo and Di Wang and Preslav Nakov , editor =. DetectLLM: Leveraging Log Rank Information for Zero-Shot Detection of Machine-Generated Text , booktitle =. 2023 , url =. doi:10.18653/V1/2023.FINDINGS-EMNLP.827 , timestamp =
-
[19]
arXiv , author =:2412.19437 , primaryclass =
DeepSeek-V3 Technical Report , url =. arXiv , author =:2412.19437 , primaryclass =
-
[20]
arXiv , author =:2410.21276 , primaryclass =
GPT-4o System Card , url =. arXiv , author =:2410.21276 , primaryclass =
-
[21]
How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection , year =
Guo, Biyang and Zhang, Xin and Wang, Ziyuan and Jiang, Minqi and Nie, Jinran and Ding, Yuxuan and Yue, Jianwei and Wu, Yupeng , journal =. How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection , year =
-
[22]
arXiv , author =:2309.02731 , primaryclass =
HC3 Plus: A Semantic-Invariant Human ChatGPT Comparison Corpus , url =. arXiv , author =:2309.02731 , primaryclass =
-
[23]
LLM - Detect AI Generated Text , year =
Jules King and Perpetual Baffour and Scott Crossley and Ryan Holbrook and Maggie Demkin , howpublished =. LLM - Detect AI Generated Text , year =
-
[24]
AI Vs Human Text , year =
Shayan Gerami , howpublished =. AI Vs Human Text , year =
-
[25]
The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track , title =
Guilherme Penedo and Hynek Kydl. The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track , title =
-
[26]
Dmitry Kobak and Rita González-Márquez and Emőke-Ágnes Horvát and Jan Lause , title =. Science Advances , volume =. 2025 , doi =. https://www.science.org/doi/pdf/10.1126/sciadv.adt3813 , abstract =
-
[27]
arXiv , author =:2405.17247 , primaryclass =
An Introduction to Vision-Language Modeling , url =. arXiv , author =:2405.17247 , primaryclass =
-
[28]
Fu and Stefano Ermon and Atri Rudra and Christopher R
Tri Dao and Daniel Y. Fu and Stefano Ermon and Atri Rudra and Christopher R. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness , booktitle =. 2022 , url =
2022
-
[29]
2015 , eprint=
Distilling the Knowledge in a Neural Network , author=. 2015 , eprint=
2015
-
[30]
2024 , eprint=
A Survey on Mixture of Experts , author=. 2024 , eprint=
2024
-
[31]
arXiv , author =:2203.15556 , primaryclass =
Training Compute-Optimal Large Language Models , url =. arXiv , author =:2203.15556 , primaryclass =
-
[32]
arXiv , author =:2409.17416 , primaryclass =
From Deception to Detection: The Dual Roles of Large Language Models in Fake News , url =. arXiv , author =:2409.17416 , primaryclass =
-
[33]
Detecting
Liyanage, Vijini and Buscaldi, Davide and Forcioli, Penelope , booktitle =. Detecting
-
[34]
An Empirical Study to Understand How Students Use ChatGPT for Writing Essays , booktitle =
Andrew Jelson and Daniel Manesh and Alice Jang and Daniel Dunlap and Young. An Empirical Study to Understand How Students Use ChatGPT for Writing Essays , booktitle =. 2026 , url =. doi:10.1145/3772318.3791056 , timestamp =
-
[35]
Yafu Li and Qintong Li and Leyang Cui and Wei Bi and Zhilin Wang and Longyue Wang and Linyi Yang and Shuming Shi and Yue Zhang , editor =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),. 2024 , url =. doi:10.18653/V1/2024.ACL-LONG.3 , timestamp =
-
[36]
Weixin Liang and Mert Y. Patterns , volume =. 2023 , url =. doi:10.1016/J.PATTER.2023.100779 , timestamp =
-
[37]
Feedback Prize - English Language Learning , year =
Alex Franklin and Maggie and Meg Benner and Natalie Rambis and Perpetual Baffour and Ryan Holbrook and Scott Crossley and ulrichboser , howpublished =. Feedback Prize - English Language Learning , year =
-
[38]
The Grammar-Learning Trajectories of Neural Language Models , booktitle =
Leshem Choshen and Guy Hacohen and Daphna Weinshall and Omri Abend , editor =. The Grammar-Learning Trajectories of Neural Language Models , booktitle =. 2022 , url =. doi:10.18653/V1/2022.ACL-LONG.568 , timestamp =
-
[39]
Fern , editor =
Nora Belrose and Quintin Pope and Lucia Quirke and Alex Mallen and Xiaoli Z. Fern , editor =. Neural Networks Learn Statistics of Increasing Complexity , booktitle =. 2024 , url =
2024
-
[40]
Large Scale Structure of Neural Network Loss Landscapes , booktitle =
Stanislav Fort and Stanislaw Jastrzebski , editor =. Large Scale Structure of Neural Network Loss Landscapes , booktitle =. 2019 , url =
2019
-
[41]
A python library for confidence intervals , year =
Jacob Gildenblat , howpublished =. A python library for confidence intervals , year =
-
[42]
Applied Intelligence , year =
Takahashi, Kanae and Yamamoto, Kouji and Kuchiba, Aya and Koyama, Tatsuki , title =. Applied Intelligence , year =
-
[43]
Fast Implementation of DeLong’s Algorithm for Comparing the Areas Under Correlated Receiver Operating Characteristic Curves , volume =
Sun, Xu and Xu, Weichao , doi =. Fast Implementation of DeLong’s Algorithm for Comparing the Areas Under Correlated Receiver Operating Characteristic Curves , volume =. IEEE Signal Processing Letters , keywords =
-
[44]
DeLong and David M
Elizabeth R. DeLong and David M. DeLong and Daniel L. Clarke-Pearson , issn =. Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach , url =. Biometrics , number =
-
[45]
Zoom In: An Introduction to Circuits , volume =
Olah, Chris and Cammarata, Nick and Schubert, Ludwig and Goh, Gabriel and Petrov, Michael and Carter, Shan , doi =. Zoom In: An Introduction to Circuits , volume =. Distill , month =
-
[46]
A Practical Examination of AI-Generated Text Detectors for Large Language Models , booktitle =
Brian Tufts and Xuandong Zhao and Lei Li , editor =. A Practical Examination of AI-Generated Text Detectors for Large Language Models , booktitle =. 2025 , url =. doi:10.18653/V1/2025.FINDINGS-NAACL.271 , timestamp =
-
[47]
Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics,
Yuxia Wang and Jonibek Mansurov and Petar Ivanov and Jinyan Su and Artem Shelmanov and Akim Tsvigun and Chenxi Whitehouse and Osama Mohammed Afzal and Tarek Mahmoud and Toru Sasaki and Thomas Arnold and Alham Fikri Aji and Nizar Habash and Iryna Gurevych and Preslav Nakov , editor =. Proceedings of the 18th Conference of the European Chapter of the Associ...
2024
-
[48]
Tavan, Ehsan and Najafi, Maryam , title =
-
[49]
Matthew E. Peters and Mark Neumann and Mohit Iyyer and Matt Gardner and Christopher Clark and Kenton Lee and Luke Zettlemoyer , editor =. Deep Contextualized Word Representations , booktitle =. 2018 , url =. doi:10.18653/V1/N18-1202 , timestamp =
-
[50]
Automatic Detection of Generated Text is Easiest when Humans are Fooled , booktitle =
Daphne Ippolito and Daniel Duckworth and Chris Callison. Automatic Detection of Generated Text is Easiest when Humans are Fooled , booktitle =. 2020 , url =. doi:10.18653/V1/2020.ACL-MAIN.164 , timestamp =
-
[51]
The Twelfth International Conference on Learning Representations,
Guangsheng Bao and Yanbin Zhao and Zhiyang Teng and Linyi Yang and Yue Zhang , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =
2024
-
[52]
The Twelfth International Conference on Learning Representations,
Xianjun Yang and Wei Cheng and Yue Wu and Linda Ruth Petzold and William Yang Wang and Haifeng Chen , title =. The Twelfth International Conference on Learning Representations,. 2024 , url =
2024
-
[53]
The Thirteenth International Conference on Learning Representations,
Christopher Ackerman and Nina Panickssery , title =. The Thirteenth International Conference on Learning Representations,. 2025 , url =
2025
-
[54]
A Survey on
Wu, Junchao and Yang, Shu and Zhan, Runzhe and Yuan, Yulin and Chao, Lidia Sam and Wong, Derek Fai , doi =. A Survey on. Computational Linguistics , month =
-
[55]
Long Ouyang and Jeffrey Wu and Xu Jiang and Diogo Almeida and Carroll L. Wainwright and Pamela Mishkin and Chong Zhang and Sandhini Agarwal and Katarina Slama and Alex Ray and John Schulman and Jacob Hilton and Fraser Kelton and Luke Miller and Maddie Simens and Amanda Askell and Peter Welinder and Paul F. Christiano and Jan Leike and Ryan Lowe , editor =...
2022
-
[56]
Ben Allal, Loubna and Lozhkov, Anton and Penedo, Guilherme and Wolf, Thomas and von Werra, Leandro , title =
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.