REVIEW 4 major objections 5 minor 1 cited by
On the NLPCC 2025 Chinese AI-text detection benchmark, LoRA-adapted Qwen2.5-7B reaches 95.94% test accuracy, beating BERT-large (79.3%), RoBERTa-large (76.3%), and FastText (83.5%).
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
On the NLPCC 2025 Chinese AI-text detection benchmark, LoRA-adapted Qwen2.5-7B reaches 95.94% test accuracy, beating BERT-large (79.3%), RoBERTa-large (76.3%), and FastText (83.5%).
T0 review reviewed 2026-08-05 challenge →
load-bearing objection Useful benchmark comparison undercut by overclaimed robustness and test-set rank selection. the 4 major comments →
LLM Encoder vs. Decoder: Robust Detection of Chinese AI-Generated Text with LoRA
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
The headline result is that the large Qwen model with LoRA reaches about 96% accuracy on the test set, far better than BERT (79%), RoBERTa (76%), or FastText (84%). The simple FastText model beats both encoders despite being much simpler, which suggests that surface word patterns carry a lot of signal.
The authors interpret this as evidence that decoder-style large language models are more robust for detection. But the evidence is limited: only one dataset is used, the test set is not shown to be from different domains than the training set, the best result comes from picking the best of three LoRA settings after seeing test scores, and no error bars are reported. The paper itself admits that multiple factors besides architecture differ between the models, and that conclusions about robustness need more datasets. So the measurement is interesting, but the stronger claim about generalization is not yet backed.
Core claim
The central claim is that 'a LoRA-adapted Qwen2.5-7B achieves 95.94% test accuracy with balanced precision-recall metrics, indicating superior generalization and resilience to dataset-specific artifacts' (abstract). If true, on the NLPCC-2025 Chinese test set a decoder LLM adapted with rank-16 LoRA detects AI-generated text substantially better than fine-tuned encoders or a lexical baseline.
Load-bearing premise
The paper treats the held-out test split of the NLPCC-2025 dataset as evidence of 'distribution shift' and 'robustness' (abstract, Section 5.1), but no out-of-domain dataset or domain-disjoint split is used. If the test split is merely a random holdout from the same pool as the training texts, then the encoder accuracy drop reflects overfitting and the decoder's advantage may not survive genuinely new domains, generators, or writing styles. The paper's own Section 6 concedes that a single dataset limits generalizability claims.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports a comparative evaluation of encoder-based Transformers (Chinese BERT-large, RoBERTa-wwm-ext-large) fine-tuned with a prompt-based masked language modeling objective, decoder-only LLMs (Qwen2.5-7B, DeepSeek-R1-Distill-Qwen-7B) adapted with LoRA ranks 4/8/16 and a linear classification head on the first-token hidden state, and a FastText lexical baseline, all on the NLPCC-2025 Chinese AI-generated text detection dataset. The main claim is that the LoRA-adapted Qwen2.5-7B at r=16 achieves 95.94% test accuracy with balanced precision/recall, substantially outperforming the encoders (76.31%, 79.33%) and FastText (83.50%), and that this demonstrates superior generalization and robustness under distribution shift. The paper also claims a novel prompt-based MLM approach for encoders that improves out-of-domain generalization.
Significance. If the central result is statistically sound, the paper would provide useful evidence that parameter-efficient LoRA adaptation of a 7B decoder can outperform smaller fine-tuned encoders for Chinese AI-text detection, and the inclusion of a lexical baseline is a valuable sanity check. The paper has several strengths: it uses a public shared-task dataset; it reports class-wise precision/recall/F1 for all models; it includes both BERT/RoBERTa and two 7B decoders; and Section 6 candidly lists important limitations (single dataset, missing comparison with standard classification heads). However, the current experimental protocol does not yet support the 'superior generalization' and 'distribution shift' claims.
major comments (4)
- [§5.1–5.2, Table 2] The headline 95.94% is the maximum over r=4,8,16 test-set accuracies. Section 5.1 says only checkpoints are selected by dev macro-F1, not the LoRA rank, so the test set is used both to choose r and to report the final number. No seeds or confidence intervals are given; Section 5.2 acknowledges seed-dependent variation. Please use a validation-based rank-selection rule, or run multiple seeds per rank and report mean±std.
- [§5.2, Table 2] The text states 'Raising the rank to r = 8 yields modest gains for both models,' but Table 2 shows Qwen accuracy drops from 94.31 (r=4) to 93.76 (r=8) and DeepSeek drops from 90.79 to 88.98. This internal contradiction suggests the r=8 result is not a stable gain and undermines the narrative that higher rank monotonically helps. Correct the text or explain why r=8 underperforms.
- [Abstract and §5.2] The paper repeatedly attributes the decoder's advantage to 'superior generalization and resilience to dataset-specific artifacts' and 'robustness under distribution shift,' but the only evaluation is a random held-out split of the same NLPCC-2025 corpus. No out-of-domain dataset or domain-disjoint split is used. Section 6 correctly concedes that 'evaluation on a single dataset limits generalizability claims.' This concession is load-bearing: the abstract and results section must be reworded to say 'held-out test set,' or the authors must add a genuinely out-of-domain evaluation.
- [§5.2 and §4] The Qwen-vs-DeepSeek comparison is confounded: the two 7B decoders differ not only in LoRA adaptation but also in distillation procedure, instruction tuning, and possibly initialization (§3.5), so the claim that 'the base Qwen2.5-7B decoder ... delivers superior generalization' is not cleanly attributable. Likewise, the encoder advantage of the prompt-based MLM objective is asserted without comparison to a standard [CLS] classification head; Section 6 admits this. The paper should either add the missing controlled comparisons or temper the causal language in the abstract and Section 5.2.
minor comments (5)
- [Throughout] Typos: 'e fficacy' (abstract), 'e ffective' (p.2), 'buch size' (§5.1), 'lz"weight-decay" penalty' (§5.1) should be corrected.
- [§4] The decoder adaptation text says 'the final hidden state of the [CLS] token,' but the decoder has no [CLS] token; §5.2 correctly uses the first-token hidden state h0^(L). Please reconcile.
- [§3.3/§5.1] The FastText description omits the embedding dimension value ('dimension which are averaged') and inconsistently describes 'Autotune mode' alongside explicit hyperparameters lr=0.05, batch size=1490212; specify actual training configuration.
- [References] References [15] and [24] are the same work (Cui et al., 2020); the dataset URL in §4 should include the version/access date.
- [Figures 3–6] The error-length distributions are described in text but the paper does not specify how the length bins are defined (characters vs tokens) or how the errors are aggregated. Please clarify the methodology for these figures.
Circularity Check
No circular derivation; the empirical benchmark is self-contained, with only minor non-load-bearing self-citations and a test-set rank-selection caveat that is a correctness risk, not circularity.
full rationale
The paper's central claim is an empirical measurement: LoRA-adapted Qwen2.5-7B attains 95.94% test accuracy on the NLPCC-2025 task. There is no derivation chain that reduces a predicted quantity to a fitted input. Encoder, decoder, and FastText models are trained on the official train split and evaluated on the held-out test split; all three LoRA ranks are reported in Table 2, so the r=16 result is transparent. The only self-citations ([23], [26]) are tangential related-work references and are not load-bearing. The paper itself flags limitations in Section 6 ('evaluation on a single dataset limits generalizability claims') and the post-hoc extension note in Section 5.2 ('Initially, we reported results for Qwen2.5-7B with rank = 8. We later extended the evaluation...'). The choice of r=16 as the headline 'peak' from the test set is a statistical selection concern, not a circular definition: the number is a measurement of the chosen configuration, and the qualitative decoder-over-encoder conclusion holds across all ranks (e.g., r=4 Qwen at 94.31% still exceeds the encoders). Therefore no step in the paper's reasoning is equivalent to its input by construction.
Axiom & Free-Parameter Ledger
free parameters (2)
- LoRA rank selection (headline result) =
r=16
- Encoder prompt template and verbalizer =
Template in Section 4; verbalizers 人工/算法
axioms (3)
- domain assumption NLPCC-2025 Task 1 labels and official train/dev/test splits are accurate and appropriate for measuring generalization.
- ad hoc to paper The test split represents a distribution shift from the training split.
- ad hoc to paper The encoder and decoder setups constitute 'identical conditions' for comparison.
Cite this review
Pith. "Pith review of LLM Encoder vs. Decoder: Robust Detection of Chinese AI-Generated Text with LoRA." pith.science (2026). https://pith.science/paper/OBZGVH42
@misc{pith2026250900731,
author = {Pith},
title = {Pith review of: LLM Encoder vs. Decoder: Robust Detection of Chinese AI-Generated Text with LoRA},
year = {2026},
howpublished = {\url{https://pith.science/paper/OBZGVH42}},
note = {Machine review of arXiv:2509.00731}
}
read the original abstract
The rapid growth of large language models (LLMs) has heightened the demand for accurate detection of AI-generated text, particularly in languages like Chinese, where subtle linguistic nuances pose significant challenges to current methods. In this study, we conduct a systematic comparison of encoder-based Transformers (Chinese BERT-large and RoBERTa-wwm-ext-large), a decoder-only LLM (Alibaba's Qwen2.5-7B/DeepSeek-R1-Distill-Qwen-7B fine-tuned via Low-Rank Adaptation, LoRA), and a FastText baseline using the publicly available dataset from the NLPCC 2025 Chinese AI-Generated Text Detection Task. Encoder models were fine-tuned using a novel prompt-based masked language modeling approach, while Qwen2.5-7B was adapted for classification with an instruction-format input and a lightweight classification head trained via LoRA. Experiments reveal that although encoder models nearly memorize training data, they suffer significant performance degradation under distribution shifts (RoBERTa: 76.3% test accuracy; BERT: 79.3%). FastText demonstrates surprising lexical robustness (83.5% accuracy) yet lacks deeper semantic understanding. In contrast, the LoRA-adapted Qwen2.5-7B achieves 95.94% test accuracy with balanced precision-recall metrics, indicating superior generalization and resilience to dataset-specific artifacts. These findings underscore the efficacy of decoder-based LLMs with parameter-efficient fine-tuning for robust Chinese AI-generated text detection. Future work will explore next-generation Qwen3 models, distilled variants, and ensemble strategies to enhance cross-domain robustness further.
Figures
Forward citations
Cited by 1 Pith paper
-
AI-Associated Lexical Shifts Across 34 Languages: Cross-Lingual Convergence and Diachronic Uptake in News Writing
Analysis of news text in 34 languages shows cross-lingual convergence on AI-associated lemmas and increased prevalence of top AI-overused items after ChatGPT's release.
Reference graph
Works this paper leans on
-
[1]
Release strategies and the social impacts of language mod- els
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-V oss, Je ff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. Release strategies and the social impacts of language mod- els. arXiv preprint arXiv:1908.09203, 2019
Pith/arXiv arXiv 1908
-
[2]
De- fending against neural fake news
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. De- fending against neural fake news. Advances in neural information processing systems, 32, 2019
work page 2019
-
[3]
Detectgpt: Zero-shot machine-generated text detection using probability curva- ture
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christo- pher D Manning, and Chelsea Finn. Detectgpt: Zero-shot machine-generated text detection using probability curva- ture. In International Conference on Machine Learning, pages 24950–24962. PMLR, 2023
work page 2023
-
[4]
All that’s’ human’is not gold: Evaluating human evaluation of generated text
Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A Smith. All that’s’ human’is not gold: Evaluating human evaluation of generated text. arXiv preprint arXiv:2107.00061, 2021
Pith/arXiv arXiv 2021
-
[5]
Automatic detection of generated text is easiest when humans are fooled
Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. Automatic detection of generated text is easiest when humans are fooled. arXiv preprint arXiv:1911.00650, 2019
Pith/arXiv arXiv 1911
-
[6]
Automatic detection of machine generated text: A critical survey
Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks VS Lakshmanan. Automatic detection of machine generated text: A critical survey. arXiv preprint arXiv:2011.01314, 2020
Pith/arXiv arXiv 2011
-
[7]
The conll-2014 shared task on grammatical error correc- tion
Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadi- winoto, Raymond Hendy Susanto, and Christopher Bryant. The conll-2014 shared task on grammatical error correc- tion. In Proceedings of the eighteenth conference on com- putational natural language learning: shared task, pages 1–14, 2014
work page 2014
-
[8]
An empirical study on word segmentation for chinese machine translation
Hai Zhao, Masao Utiyama, Eiichiro Sumita, and Bao- Liang Lu. An empirical study on word segmentation for chinese machine translation. In International Conference on Intelligent Text Processing and Computational Linguis- tics, pages 248–263. Springer, 2013
work page 2013
-
[9]
Rongsheng Wang, Haoming Chen, Ruizhe Zhou, Han Ma, Yaofei Duan, Yanlan Kang, Songhua Yang, Baoyu Fan, and Tao Tan. Llm-detector: Improving ai-generated chi- nese text detection with open-source llm instruction tuning. arXiv preprint arXiv:2402.01158, 2024
Pith/arXiv arXiv 2024
-
[10]
Towards Reliable Detection of LLM-Generated Texts: A Comprehensive Evaluation Framework with CUDRT
Zhen Tao, Zhiyu Li, Dinghao Xi, and Wei Xu. Cudrt: Benchmarking the detection of human vs. large language models generated texts. arXiv preprint arXiv:2406.09056, 2024. Preprint – LLM Encoder vs. Decoder: Robust Detection of Chinese AI-Generated Text with LoRA 8
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[11]
Bert: Pre-training of deep bidirectional trans- formers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the as- sociation for computational linguistics: human language technologies, volume 1 (long and short papers) , pages 4171–4186, 2019
work page 2019
-
[12]
Roberta: A ro- bustly optimized bert pretraining approach
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A ro- bustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019
Pith/arXiv arXiv 1907
-
[13]
Language models are few-shot learners
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neu- ral information processing systems, 33:1877–1901, 2020
work page 1901
-
[14]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023
Pith/arXiv arXiv 2023
-
[16]
Deepseek-r1: Incentivizing reason- ing capability in llms via reinforcement learning
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reason- ing capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025
Pith/arXiv arXiv 2025
-
[17]
Bag of tricks for efficient text classifica- tion
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classifica- tion. arXiv preprint arXiv:1607.01759, 2016
Pith/arXiv arXiv 2016
-
[18]
Roberta and bi-lstm for human vs ai generated text detection
Panagiotis Petropoulos and Vasilis Petropoulos. Roberta and bi-lstm for human vs ai generated text detection. Work- ing Notes of CLEF, 2024
work page 2024
-
[19]
Multiscale positive-unlabeled detection of ai-generated texts
Yuchuan Tian, Hanting Chen, Xutao Wang, Zheyuan Bai, Qinghua Zhang, Ruifeng Li, Chao Xu, and Yunhe Wang. Multiscale positive-unlabeled detection of ai-generated texts. arXiv preprint arXiv:2305.18149, 2023
Pith/arXiv arXiv 2023
-
[20]
A watermark for large language models
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. In International Conference on Machine Learning, pages 17061–17084. PMLR, 2023
work page 2023
-
[21]
On Evaluating The Performance of Watermarked Machine-Generated Texts Under Adversarial Attacks
Zesen Liu, Tianshuo Cong, Xinlei He, and Qi Li. On evaluating the performance of watermarked machine- generated texts under adversarial attacks. arXiv preprint arXiv:2407.04794, 2024
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[22]
Wa- termarks in the sand: Impossibility of strong watermarking for generative models
Hanlin Zhang, Benjamin L Edelman, Danilo Francati, Daniele Venturi, Giuseppe Ateniese, and Boaz Barak. Wa- termarks in the sand: Impossibility of strong watermarking for generative models. arXiv preprint arXiv:2311.04378, 2023
Pith/arXiv arXiv 2023
-
[23]
Houji Jin, Negin Ashrafi, Kamiar Alaei, Elham Pishgar, Greg Placencia, and Maryam Pishgar. A novel multi-task teacher-student architecture with self-supervised pretrain- ing for 48-hour vasoactive-inotropic trend analysis in sep- sis mortality prediction. arXiv preprint arXiv:2502.16834, 2025
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[24]
Revisiting pre-trained models for chinese natural language processing
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. Revisiting pre-trained models for chinese natural language processing. arXiv preprint arXiv:2004.13922, 2020
Pith/arXiv arXiv 2004
-
[25]
Glu variants improve transformer
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020
Pith/arXiv arXiv 2002
-
[26]
Armin Abdollahi, Negin Ashrafi, Xinghong Ma, Jia- hao Zhang, Daijia Wu, Tongshou Wu, Zizheng Ye, and Maryam Pishgar. Advanced predictive modeling for en- hanced mortality prediction in icu stroke patients using clinical data. PLoS One, 20(5):e0323441, 2025
work page 2025
This paper was first reviewed by deepseek-v4-flash on August 5, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.