REVIEW 3 major objections 5 minor 20 references
The paper claims that locally averaging token-level detection scores with an adaptively chosen window locates LLM-generated tokens in human–AI coauthored text better than existing document- or segment-level detectors, without training label
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 07:21 UTC pith:GHWHOLU2
load-bearing objection A clean, training-free token-level LLM localization method with strong empirical results, held back mainly by a theory that is honest but sits a bit loose from the actual implementation. the 3 major comments →
Detecting LLM-Generated Tokens in Human--LLM Coauthored Text
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that a smoothed, adaptively banded version of a token-level authorship score estimates the local expected score with near-oracle mean squared error, and hence classifies tokens better than raw scores, fixed-bandwidth smoothing, sentence-level segmentation, or trained token classifiers. The theoretical result states that when token scores are bounded and conditionally independent given the authorship labels, the adaptive bandwidth rule performs within a constant factor of the best fixed bandwidth in the candidate set, plus a small penalty term; the error decomposes into contamination bias (the total smoothing weight on tokens from the other source times the score gap) and
What carries the argument
The mechanism is a three-part pipeline. First, any token-level detection score that separates human and LLM tokens in expectation is computed for each token; the paper's implementation uses a pre-trained adaptive score and notes compatibility with the broader class of log-probability and curvature scores. Second, a symmetric nonincreasing kernel — uniform or triangular — forms a local weighted average of scores around each token, with the triangular kernel reducing contamination from the opposite authorship source. Third, an adaptive bandwidth selector, of the kind used in classical nonparametric estimation, builds confidence intervals for each candidate window and picks the largest window w
Load-bearing premise
The load-bearing premise is that tokens from the same author have a stable expected detection score, that the two authors' typical scores differ, and that nearby tokens tend to be by the same author; if real token scores vary as much within one author as between authors, or if authorship switches at almost every token, smoothing blurs boundaries instead of revealing them.
What would settle it
Run the detector on synthetic documents where authorship is randomly assigned per token with no local persistence: if smoothing with a window of 15 still improves over raw scores, the method is benefiting from something other than shared authorship. Alternatively, simulate bounded conditionally independent scores with known segment lengths, run the adaptive selector over the candidate grid, and compare its mean squared error against the best fixed bandwidth; the oracle inequality would be refuted if the adaptive error exceeds the bound by more than the stated constants.
If this is right
- Authorship localization becomes possible without token-level training labels: any existing token-level detector can be upgraded by the smoothing-plus-adaptive-window layer.
- Document- and sentence-level detectors can be replaced by a single token-level procedure that still flags where AI contributed, even when boundaries fall inside sentences.
- The bias–variance decomposition gives a quantitative reason smoothing helps: variance falls as the window grows, and the contamination bias term controls how wide one can go.
- The oracle-type inequality says the data-driven window choice is never much worse than the best fixed window in the candidate set, making the method robust to unknown segment lengths.
- The sensitivity experiments indicate the gains persist across generators, writing styles, irregular authorship-switch patterns, and sampling temperatures.
Where Pith is reading between the lines
- I infer the natural deployment is as an audit layer for long documents: at roughly three seconds per 1,300 tokens, the adaptive variant is cheap enough to scan preprint manuscripts and flag passages for human review—a use case the paper does not develop.
- Because the oracle guarantee assumes bounded, conditionally independent token scores, which real autoregressive scores violate, I read the theoretical result as a proof-of-concept; the practical case for the method rests on the empirical comparisons.
- The smoothed score is, in effect, a boundary map, so the method could feed change-point or segmentation algorithms to produce segment-level decisions with sub-sentence precision.
- The same smoothing-plus-adaptive-window layer could be applied to other token-level signals such as perplexity, semantic entropy, or watermark scores, making it a generic post-processing tool for authorship attribution.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies token-level detection of LLM-generated content in human–LLM coauthored text. It proposes to smooth existing token-level detection scores (AdaDetectGPT is the default) with a symmetric kernel and to select the bandwidth per token using Lepski's rule. The theory characterizes the bias–variance trade-off of the smoothed score (Theorem 4.1), gives explicit bounds for uniform and triangular kernels (Corollary 4.2), and claims an oracle-type guarantee for the adaptive selector (Theorem 4.3, Eq. 6). Empirically, the method outperforms several ML- and segmentation-based baselines on synthetic datasets with three closed-source LLMs and on the real CoAuthor dataset, and it is supported by ablations, sensitivity analyses, and a public demo website.
Significance. If the claims held as stated, the paper would be a useful contribution: it addresses a real problem, the method is simple and training-free at the token level, and the experimental evaluation is broad. The paper's strengths include a clean formulation of the local bias–variance trade-off, an ablation that isolates the effect of aggregation and adaptive bandwidth selection, experiments across multiple generators and datasets, and a publicly accessible implementation. However, the main theoretical guarantee is currently disconnected from the algorithm as implemented, and the proof of Theorem 4.3 has a gap. The practical value of the method is therefore supported mainly by the strong empirical results, which are convincing but need to be coupled with a more honest statement of what the theory does and does not cover.
major comments (3)
- [Theorem 4.3 / Appendix B.1] The oracle inequality in Theorem 4.3 (Eq. 6) is proved under assumptions not satisfied by the implemented Algorithm 1. The theorem assumes a known constant ξ with sup_t |S_t−μ_{C_t}|≤ξ a.s. and conditional independence of {S_t} given C. Appendix B.1 replaces ξ with a plug-in variance estimator ξ̂_{t,k_i}; the theorem contains no guarantee for this plug-in radius. For autoregressive scores such as S_t=log p_θ(X_t|X_{<t}), scores are unbounded below and strongly dependent across t, so both hypotheses fail in the experiments. Thus Eq. (6) does not control the MSE of the implemented method; the theoretical support for the headline MSE claim is limited to an idealized score process. Please either extend the result to the plug-in radius or explicitly restrict the theory and present the practical benefit as empirical.
- [Theorem 4.3, proof of condition (iii)] The proof of Theorem 4.3 invokes Corollary 4 of Su et al. (2020) and requires the bias bound B_{t,i}=Δπ_{t,k_i} to be nondecreasing in i. The proof says this 'directly follows from the proof of Corollary 4.2', but Corollary 4.2 establishes the bias bound only under the additional condition k<τ/2 (the window crosses at most one boundary). Without such an assumption, π_{t,k} need not be monotone: after crossing a second boundary, a larger window can include more tokens of the target label, decreasing the contamination weight. The theorem statement does not include k_M<τ/2 or any alternative condition, so the proof as written has a gap. Please add the missing condition and check whether the candidate grids in Section 5 satisfy it, or prove monotonicity under weaker assumptions.
- [Assumption 3.1 / Theorem 4.1] The bias calculation in the proof of Theorem 4.1 (Appendix A.1, Eqs. (8)–(9)) relies critically on Assumption 3.1, which requires E[S_t|C_t=c] to be constant within each source. Real token-level scores are lexically dependent: common tokens tend to have higher probability under both human and LLM authorship, so E[S_t|C_t] varies with t. Under such heterogeneity the bias contains a within-source term Σ_l w_l (E[S_l|C_l]−μ_{C_l}) that is not captured by Δπ_{t,k}, and Theorem 4.1's bound does not follow from the stated assumption alone. Please either relax Assumption 3.1 to a bounded-heterogeneity condition and derive the additional bias term, or explicitly restrict the theorem's scope.
minor comments (5)
- [Notation] The symbol κ is used both for the dependence factor Σ_r ρ(r) in Theorem 4.1 and for the multiplicative constant in condition (iii) of the proof of Theorem 4.3. This overload makes the proof harder to follow; please rename one of the two.
- [Appendix B.1] The formula for the plug-in variance estimator ξ̂²_{t,k_i} is presented with the same weight notation in numerator and denominator; please explain why the w²-weighted average is natural and how it relates to the radius used in Theorem 4.3. The numerical constant also differs from the theorem's r_{t,k_i}; this should be reconciled or explicitly stated to be a heuristic choice.
- [Figure 1] The labels k_1,...,k_5 and the selected k_4 are difficult to read in the current figure; a larger figure or a clearer caption would help.
- [Section 6] The case study reports 94% classification accuracy on a single coauthored abstract; this is illustrative. Please state explicitly that it is not a substitute for the aggregate evaluation.
- [Typesetting] There are several LaTeX rendering artifacts (e.g., Eq. (1), the keywords line, and some inline math) that should be cleaned up in the final version.
Circularity Check
No significant circularity: the smoothing/Lepski contribution is evaluated independently of the base scorer and does not reduce to its inputs.
full rationale
The paper's central methodological claim is that kernel smoothing plus a Lepski-type adaptive bandwidth improves token-level detection over raw scores. This claim is not circular: the base token scores are treated as given inputs, the same AdaDetectGPT scoring function is used across the proposed method, fixed-bandwidth variants, and segment-based baselines, and the ablation explicitly compares raw scores (|Adj_k(t)|=1) against smoothed and adaptively smoothed versions, showing that aggregation itself changes performance. The theoretical oracle bound (Theorem 4.3) is imported from an external result (Corollary 4 of Su et al. 2020) and the proof verifies its conditions rather than assuming the conclusion. The use of AdaDetectGPT from the authors' prior work is a dependency and a minor self-citation, but it is not load-bearing: the method is described as compatible with any score satisfying Assumption 3.1, and the raw-score ablation isolates the effect of smoothing. The paper also discloses that the oracle bound relies on a known scale xi which is replaced in practice by a plug-in variance estimator (Appendix B.1), and real autoregressive LM scores may violate boundedness and conditional independence. That is a theory-practice gap or correctness risk, not circularity: the implementation does not rename a fitted quantity as a prediction of the theorem, and the empirical comparisons are conducted against external baselines. No equation in the paper is equivalent by construction to its inputs, and no load-bearing claim is justified solely by a self-citation.
Axiom & Free-Parameter Ledger
free parameters (4)
- Detection threshold eta =
per document
- Lepski candidate bandwidth grid =
not stated precisely; oracle uses |Adj| in {1,15,63,127,255}
- Radius scale xi-hat =
estimated from local variance of log-probs (Appendix B.1)
- Kernel choice (triangular vs uniform) =
triangular default
axioms (5)
- domain assumption Assumption 3.1: Score separability and homogeneity — E[S_t|C_t=1] != E[S_t|C_t=0], and E[S_t|C_t] is constant within each source.
- ad hoc to paper Boundedness |S_t - mu_{C_t}| <= xi almost surely (Theorem 4.3).
- ad hoc to paper Conditional independence of token scores given the label sequence (Theorem 4.3).
- domain assumption Minimum segment length tau (Eq. 2) with authorship persisting in contiguous blocks.
- domain assumption Pre-trained detection scores (AdaDetectGPT/Fast-DetectGPT) separate human from LLM tokens.
read the original abstract
The rise of human-AI collaborative writing has created a growing need for fine-grained detection methods that support localizing likely LLM-generated content in mixed-authorship documents. Existing methods for detecting LLM-generated text mainly focus on document-level classification and cannot identify which parts of the text are generated by LLMs. This paper introduces a new method to address this urgent need. Our method operates at the token level, the natural unit of modern language models, and builds on existing token-level detection scores. The key idea is to smooth adjacent token scores to reduce their variability, while using an adaptive Lepski-type rule to select the bandwidth according to the local authorship structure. Our method is simple to implement and does not require token-level labeled data for training. Theoretically, we characterize this trade-off and show that the proposed method achieves favorable mean square error performance in estimating the underlying signal. Empirically, we demonstrate strong performance of our method against a wide range of baselines in both synthetic datasets and a realistic dataset. We deploy a publicly accessible website that implements the methods as well.
Figures
Reference graph
Works this paper leans on
-
[3]
doi:10.1126/science.adg7879. Nature Editorial. Tools such as ChatGPT threaten transparent science; here are our ground rules for their use.Nature, 613:612,
-
[4]
European Parliament and Council of the European Union
doi:10.1038/d41586-023-00191-1. European Parliament and Council of the European Union. Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence. Official Journal of the European Union, L 2024/1689,
-
[6]
Machine-generated text localization
Zhongping Zhang, Wenda Qin, and Bryan Plummer. Machine-generated text localization. InFindings of the Association for Computational Linguistics: ACL 2024, pages 8357–8371, Bangkok, Thailand, August 2024b. Association for Computational Linguistics. Zhixiong Su, Yichen Wang, Herun Wan, Zhaohan Zhang, and Minnan Luo. HACo-Det: A study towards fine-grained ma...
2024
-
[9]
Training-free LLM-generated text detection by mining token probability sequences
Yihuai Xu, Yongwei Wang, Yifei Bi, Huangsen Cao, Zhouhan Lin, Yu Zhao, and Fei Wu. Training-free LLM-generated text detection by mining token probability sequences. InInternational Conference on Learning Representations, volume 2025, pages 19072–19098,
2025
-
[10]
Ghostbuster: Detecting text ghostwritten by large language models
Vivek Verma, Eve Fleisig, Nicholas Tomlin, and Dan Klein. Ghostbuster: Detecting text ghostwritten by large language models. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1702–1717, Mexico City, Mexico, June
2024
-
[12]
RoBERTa: A robustly optimized BERT pretraining approach.arXiv preprint arXiv:1907.11692,
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach.arXiv preprint arXiv:1907.11692,
Pith/arXiv arXiv 1907
-
[13]
Haitao Bai, Pinghui Wang, Ruofei Zhang, and Zhou Su
doi:10.1080/01621459.2025.2520455. Haitao Bai, Pinghui Wang, Ruofei Zhang, and Zhou Su. SegFormer: a topic segmentation model with controllable range of attention. InProceedings of the AAAI conference on artificial intelligence, volume 37, pages 12545–12552,
arXiv 2025
-
[14]
SeqXGPT: Sentence-level AI-generated text detection
Pengyu Wang, Linyang Li, Ke Ren, Botian Jiang, Dong Zhang, and Xipeng Qiu. SeqXGPT: Sentence-level AI-generated text detection. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1144–1156, Singapore, December
2023
-
[15]
Towards automatic boundary detection for human-AI collaborative hybrid essay in education
Zijie Zeng, Lele Sha, Yuheng Li, Kaixun Yang, Dragan Gaševi´c, and Guanliang Chen. Towards automatic boundary detection for human-AI collaborative hybrid essay in education. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 22502–22510, 2024b. Yafu Li, Zhilin Wang, Leyang Cui, Wei Bi, Shuming Shi, and Yue Zhang. Spotting AI...
2024
-
[16]
SenDetEX: Sentence-level AI-generated text detection for human-AI hybrid content via style and context fusion
Lei Jiang, Desheng Wu, and Xiaolong Zheng. SenDetEX: Sentence-level AI-generated text detection for human-AI hybrid content via style and context fusion. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 5287–5302, Suzhou, China, November
2025
-
[17]
Semeval-2024 task 8: Multidomain, multimodel and multilingual machine-generated text detection
Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Osama Mohammed Afzal, Tarek Mahmoud, Giovanni Puccetti, and Thomas Arnold. Semeval-2024 task 8: Multidomain, multimodel and multilingual machine-generated text detection. InProceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 2057–2079,
2024
-
[19]
Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118,
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118,
-
[20]
Anthropic
URLhttps://data.x.ai/2025-08-20-grok-4-model-card.pdf. Anthropic. Claude haiku 4.5 system card. https://www-cdn.anthropic.com/ 7aad69bf12627d42234e01ee7c36305dc2f6a970/Claude%20Haiku%204.5%20System%20Card.pdf,
2025
-
[2018]
Shashi Narayan, Shay B Cohen, and Mirella Lapata
Association for Computational Linguistics. Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. InProceedings of the 2018 conference on empirical methods in natural language processing, pages 1797–1807,
2018
-
[2019]
Jinyan Su, Terry Zhuo, Di Wang, and Preslav Nakov
Association for Computational Linguistics. Jinyan Su, Terry Zhuo, Di Wang, and Preslav Nakov. DetectLLM: Leveraging log rank information for zero-shot detection of machine-generated text. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 12395–12412, Singapore, December
2023
-
[2022]
Qihui Zhang, Chujie Gao, Dongping Chen, Yue Huang, Yixin Huang, Zhenyang Sun, Shilin Zhang, Weiye Li, Zhengyan Fu, Yao Wan, and Lichao Sun. LLM-as-a-Coauthor: Can mixed human-written and machine-generated text be detected? InFindings of the Association for Computational Linguistics: NAACL 2024, pages 409–436, Mexico City, Mexico, June 2024a. Association f...
2024
-
[2023]
The continuation should be coherent with the passage
20 Detecting LLM-Generated Tokens in Human–LLM Coauthored TextA PREPRINT Prompt to GPT-5 Based on the given passage, write a natural continuation. The continuation should be coherent with the passage. A continuation of about 6 sentences is preferred, but it may be slightly longer if needed for coherence. Do not add explanation, commentary, numbering, or a...
2022
-
[2024]
europa.eu/eli/reg/2024/1689/oj
URL https://eur-lex. europa.eu/eli/reg/2024/1689/oj. Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy.Nature, 630(8017):625–630,
2024
-
[2025]
Release strategies and the social impacts of language models.arXiv preprint arXiv:1908.09203,
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-V oss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. Release strategies and the social impacts of language models.arXiv preprint arXiv:1908.09203,
Pith/arXiv arXiv 1908
-
[2026]
Zijie Zeng, Shiqi Liu, Lele Sha, Zhuang Li, Kaixun Yang, Sannyuya Liu, Dragan Gaševi ´c, and Guanliang Chen. Detecting AI-generated sentences in human-AI collaborative hybrid texts: challenges, strategies, and insights. InProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI ’24, 2024a. doi:10.24963/ijcai.2024/83...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.