Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

On the NLPCC 2025 Chinese AI-text detection benchmark, LoRA-adapted Qwen2.5-7B reaches 95.94% test accuracy, beating BERT-large (79.3%), RoBERTa-large (76.3%), and FastText (83.5%).

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

On the NLPCC 2025 Chinese AI-text detection benchmark, LoRA-adapted Qwen2.5-7B reaches 95.94% test accuracy, beating BERT-large (79.3%), RoBERTa-large (76.3%), and FastText (83.5%).

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Useful benchmark comparison undercut by overclaimed robustness and test-set rank selection. the 4 major comments →

arxiv 2509.00731 v1 pith:OBZGVH42 submitted 2025-08-31 cs.CL

LLM Encoder vs. Decoder: Robust Detection of Chinese AI-Generated Text with LoRA

classification cs.CL
keywords chineseai-generateddetectionmodelstextaccuracyencoderlora
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper studies a practical question: can software tell whether a Chinese sentence was written by a person or generated by an AI? The authors use the NLPCC 2025 shared task benchmark, which supplies labeled human-written and AI-written Chinese texts. They train four kinds of models: two older text-understanding models (Chinese BERT and RoBERTa), a simple word-frequency baseline called FastText, and two large text-generating models (Qwen and DeepSeek) that are adapted for classification with a parameter-saving method called LoRA. They also try a prompt-style trick for the smaller models, asking them to fill in masked words meaning 'human' or 'AI'.

The headline result is that the large Qwen model with LoRA reaches about 96% accuracy on the test set, far better than BERT (79%), RoBERTa (76%), or FastText (84%). The simple FastText model beats both encoders despite being much simpler, which suggests that surface word patterns carry a lot of signal.

The authors interpret this as evidence that decoder-style large language models are more robust for detection. But the evidence is limited: only one dataset is used, the test set is not shown to be from different domains than the training set, the best result comes from picking the best of three LoRA settings after seeing test scores, and no error bars are reported. The paper itself admits that multiple factors besides architecture differ between the models, and that conclusions about robustness need more datasets. So the measurement is interesting, but the stronger claim about generalization is not yet backed.

Core claim

The central claim is that 'a LoRA-adapted Qwen2.5-7B achieves 95.94% test accuracy with balanced precision-recall metrics, indicating superior generalization and resilience to dataset-specific artifacts' (abstract). If true, on the NLPCC-2025 Chinese test set a decoder LLM adapted with rank-16 LoRA detects AI-generated text substantially better than fine-tuned encoders or a lexical baseline.

Load-bearing premise

The paper treats the held-out test split of the NLPCC-2025 dataset as evidence of 'distribution shift' and 'robustness' (abstract, Section 5.1), but no out-of-domain dataset or domain-disjoint split is used. If the test split is merely a random holdout from the same pool as the training texts, then the encoder accuracy drop reflects overfitting and the decoder's advantage may not survive genuinely new domains, generators, or writing styles. The paper's own Section 6 concedes that a single dataset limits generalizability claims.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript reports a comparative evaluation of encoder-based Transformers (Chinese BERT-large, RoBERTa-wwm-ext-large) fine-tuned with a prompt-based masked language modeling objective, decoder-only LLMs (Qwen2.5-7B, DeepSeek-R1-Distill-Qwen-7B) adapted with LoRA ranks 4/8/16 and a linear classification head on the first-token hidden state, and a FastText lexical baseline, all on the NLPCC-2025 Chinese AI-generated text detection dataset. The main claim is that the LoRA-adapted Qwen2.5-7B at r=16 achieves 95.94% test accuracy with balanced precision/recall, substantially outperforming the encoders (76.31%, 79.33%) and FastText (83.50%), and that this demonstrates superior generalization and robustness under distribution shift. The paper also claims a novel prompt-based MLM approach for encoders that improves out-of-domain generalization.

Significance. If the central result is statistically sound, the paper would provide useful evidence that parameter-efficient LoRA adaptation of a 7B decoder can outperform smaller fine-tuned encoders for Chinese AI-text detection, and the inclusion of a lexical baseline is a valuable sanity check. The paper has several strengths: it uses a public shared-task dataset; it reports class-wise precision/recall/F1 for all models; it includes both BERT/RoBERTa and two 7B decoders; and Section 6 candidly lists important limitations (single dataset, missing comparison with standard classification heads). However, the current experimental protocol does not yet support the 'superior generalization' and 'distribution shift' claims.

major comments (4)
  1. [§5.1–5.2, Table 2] The headline 95.94% is the maximum over r=4,8,16 test-set accuracies. Section 5.1 says only checkpoints are selected by dev macro-F1, not the LoRA rank, so the test set is used both to choose r and to report the final number. No seeds or confidence intervals are given; Section 5.2 acknowledges seed-dependent variation. Please use a validation-based rank-selection rule, or run multiple seeds per rank and report mean±std.
  2. [§5.2, Table 2] The text states 'Raising the rank to r = 8 yields modest gains for both models,' but Table 2 shows Qwen accuracy drops from 94.31 (r=4) to 93.76 (r=8) and DeepSeek drops from 90.79 to 88.98. This internal contradiction suggests the r=8 result is not a stable gain and undermines the narrative that higher rank monotonically helps. Correct the text or explain why r=8 underperforms.
  3. [Abstract and §5.2] The paper repeatedly attributes the decoder's advantage to 'superior generalization and resilience to dataset-specific artifacts' and 'robustness under distribution shift,' but the only evaluation is a random held-out split of the same NLPCC-2025 corpus. No out-of-domain dataset or domain-disjoint split is used. Section 6 correctly concedes that 'evaluation on a single dataset limits generalizability claims.' This concession is load-bearing: the abstract and results section must be reworded to say 'held-out test set,' or the authors must add a genuinely out-of-domain evaluation.
  4. [§5.2 and §4] The Qwen-vs-DeepSeek comparison is confounded: the two 7B decoders differ not only in LoRA adaptation but also in distillation procedure, instruction tuning, and possibly initialization (§3.5), so the claim that 'the base Qwen2.5-7B decoder ... delivers superior generalization' is not cleanly attributable. Likewise, the encoder advantage of the prompt-based MLM objective is asserted without comparison to a standard [CLS] classification head; Section 6 admits this. The paper should either add the missing controlled comparisons or temper the causal language in the abstract and Section 5.2.
minor comments (5)
  1. [Throughout] Typos: 'e fficacy' (abstract), 'e ffective' (p.2), 'buch size' (§5.1), 'lz"weight-decay" penalty' (§5.1) should be corrected.
  2. [§4] The decoder adaptation text says 'the final hidden state of the [CLS] token,' but the decoder has no [CLS] token; §5.2 correctly uses the first-token hidden state h0^(L). Please reconcile.
  3. [§3.3/§5.1] The FastText description omits the embedding dimension value ('dimension which are averaged') and inconsistently describes 'Autotune mode' alongside explicit hyperparameters lr=0.05, batch size=1490212; specify actual training configuration.
  4. [References] References [15] and [24] are the same work (Cui et al., 2020); the dataset URL in §4 should include the version/access date.
  5. [Figures 3–6] The error-length distributions are described in text but the paper does not specify how the length bins are defined (characters vs tokens) or how the errors are aggregated. Please clarify the methodology for these figures.

Circularity Check

0 steps flagged

No circular derivation; the empirical benchmark is self-contained, with only minor non-load-bearing self-citations and a test-set rank-selection caveat that is a correctness risk, not circularity.

full rationale

The paper's central claim is an empirical measurement: LoRA-adapted Qwen2.5-7B attains 95.94% test accuracy on the NLPCC-2025 task. There is no derivation chain that reduces a predicted quantity to a fitted input. Encoder, decoder, and FastText models are trained on the official train split and evaluated on the held-out test split; all three LoRA ranks are reported in Table 2, so the r=16 result is transparent. The only self-citations ([23], [26]) are tangential related-work references and are not load-bearing. The paper itself flags limitations in Section 6 ('evaluation on a single dataset limits generalizability claims') and the post-hoc extension note in Section 5.2 ('Initially, we reported results for Qwen2.5-7B with rank = 8. We later extended the evaluation...'). The choice of r=16 as the headline 'peak' from the test set is a statistical selection concern, not a circular definition: the number is a measurement of the chosen configuration, and the qualitative decoder-over-encoder conclusion holds across all ranks (e.g., r=4 Qwen at 94.31% still exceeds the encoders). Therefore no step in the paper's reasoning is equivalent to its input by construction.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

The central claim rests on two empirically tuned choices (LoRA rank and prompt/verbalizer) plus three unproven dataset and design assumptions. No new entities are introduced.

free parameters (2)
  • LoRA rank selection (headline result) = r=16
    The 95.94% accuracy is the maximum over ranks 4, 8, and 16 evaluated on the test set; no validation-based selection or multiple-testing correction is reported, so the rank functions as a free parameter tuned on the test data.
  • Encoder prompt template and verbalizer = Template in Section 4; verbalizers 人工/算法
    The encoder results depend on the chosen masked-prompt template and verbalizer pair; no comparison to a standard [CLS] classification head or alternative prompts is provided, so the method's contribution is unidentified.
axioms (3)
  • domain assumption NLPCC-2025 Task 1 labels and official train/dev/test splits are accurate and appropriate for measuring generalization.
    All conclusions rest on this single public dataset; no analysis of label noise, annotation sources, generation models, or domain composition is given.
  • ad hoc to paper The test split represents a distribution shift from the training split.
    The abstract and conclusion claim robustness under distribution shift, but no OOD evaluation or domain-disjoint split is used; this assumption is asserted, not demonstrated.
  • ad hoc to paper The encoder and decoder setups constitute 'identical conditions' for comparison.
    Encoders use full fine-tuning with a prompt-MLM objective on about 330M parameters, while the decoder uses LoRA with a classification head on a 7.6B model; the comparability assumption is not examined.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM Encoder vs. Decoder: Robust Detection of Chinese AI-Generated Text with LoRA." pith.science (2026). https://pith.science/paper/OBZGVH42

@misc{pith2026250900731,
  author       = {Pith},
  title        = {Pith review of: LLM Encoder vs. Decoder: Robust Detection of Chinese AI-Generated Text with LoRA},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OBZGVH42}},
  note         = {Machine review of arXiv:2509.00731}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The rapid growth of large language models (LLMs) has heightened the demand for accurate detection of AI-generated text, particularly in languages like Chinese, where subtle linguistic nuances pose significant challenges to current methods. In this study, we conduct a systematic comparison of encoder-based Transformers (Chinese BERT-large and RoBERTa-wwm-ext-large), a decoder-only LLM (Alibaba's Qwen2.5-7B/DeepSeek-R1-Distill-Qwen-7B fine-tuned via Low-Rank Adaptation, LoRA), and a FastText baseline using the publicly available dataset from the NLPCC 2025 Chinese AI-Generated Text Detection Task. Encoder models were fine-tuned using a novel prompt-based masked language modeling approach, while Qwen2.5-7B was adapted for classification with an instruction-format input and a lightweight classification head trained via LoRA. Experiments reveal that although encoder models nearly memorize training data, they suffer significant performance degradation under distribution shifts (RoBERTa: 76.3% test accuracy; BERT: 79.3%). FastText demonstrates surprising lexical robustness (83.5% accuracy) yet lacks deeper semantic understanding. In contrast, the LoRA-adapted Qwen2.5-7B achieves 95.94% test accuracy with balanced precision-recall metrics, indicating superior generalization and resilience to dataset-specific artifacts. These findings underscore the efficacy of decoder-based LLMs with parameter-efficient fine-tuning for robust Chinese AI-generated text detection. Future work will explore next-generation Qwen3 models, distilled variants, and ensemble strategies to enhance cross-domain robustness further.

Figures

Figures reproduced from arXiv: 2509.00731 by Armin Abdollahi, Ganyu Gui, Houji Jin, Huanghao Feng, Jian Wang, Maryam Pishgar, Negin Ashrafi, Wei Liu.

Figure 1
Figure 1. Figure 1: Encoder based model training logic and prompt design architecture. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Decoder based model architecture comprises standard components, token embedding with positional encoding, masked [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: DeepSeek error-length distributions for r = 4, 8, 16. becomes nearly perfect but slightly more conservative on human texts. At r = 16, Qwen attains the optimal balance, minimizing both misclassified errors. Qwen’s error-length distributions at ranks 4, 8, and 16 are demonstrated in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qwen confusion matrices for r = 4, 8, 16. (a) r = 4 (b) r = 8 (c) r = 16 [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Qwen error-length distributions for r = 4, 8, 16. Initially, we reported results for Qwen2.5-7B with rank = 8. We later extended the evaluation to multiple LoRA rank settings for both Qwen (including rank = 8) and DeepSeek models, while keeping the original training logic unchanged. Minor differences in performance may arise due to variations in random seed ini￾tialization. Overall, Qwen consistently outpe… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. AI-Associated Lexical Shifts Across 34 Languages: Cross-Lingual Convergence and Diachronic Uptake in News Writing

    cs.CL 2026-05 unverdicted novelty 6.0

    Analysis of news text in 34 languages shows cross-lingual convergence on AI-associated lemmas and increased prevalence of top AI-overused items after ChatGPT's release.

Reference graph

Works this paper leans on

25 extracted references · 12 canonical work pages · cited by 1 Pith paper · 3 internal anchors

  1. [1]

    Release strategies and the social impacts of language mod- els

    Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-V oss, Je ff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. Release strategies and the social impacts of language mod- els. arXiv preprint arXiv:1908.09203, 2019

  2. [2]

    De- fending against neural fake news

    Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. De- fending against neural fake news. Advances in neural information processing systems, 32, 2019

  3. [3]

    Detectgpt: Zero-shot machine-generated text detection using probability curva- ture

    Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christo- pher D Manning, and Chelsea Finn. Detectgpt: Zero-shot machine-generated text detection using probability curva- ture. In International Conference on Machine Learning, pages 24950–24962. PMLR, 2023

  4. [4]

    All that’s’ human’is not gold: Evaluating human evaluation of generated text

    Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A Smith. All that’s’ human’is not gold: Evaluating human evaluation of generated text. arXiv preprint arXiv:2107.00061, 2021

  5. [5]

    Automatic detection of generated text is easiest when humans are fooled

    Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. Automatic detection of generated text is easiest when humans are fooled. arXiv preprint arXiv:1911.00650, 2019

  6. [6]

    Automatic detection of machine generated text: A critical survey

    Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks VS Lakshmanan. Automatic detection of machine generated text: A critical survey. arXiv preprint arXiv:2011.01314, 2020

  7. [7]

    The conll-2014 shared task on grammatical error correc- tion

    Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadi- winoto, Raymond Hendy Susanto, and Christopher Bryant. The conll-2014 shared task on grammatical error correc- tion. In Proceedings of the eighteenth conference on com- putational natural language learning: shared task, pages 1–14, 2014

  8. [8]

    An empirical study on word segmentation for chinese machine translation

    Hai Zhao, Masao Utiyama, Eiichiro Sumita, and Bao- Liang Lu. An empirical study on word segmentation for chinese machine translation. In International Conference on Intelligent Text Processing and Computational Linguis- tics, pages 248–263. Springer, 2013

  9. [9]

    Llm-detector: Improving ai-generated chi- nese text detection with open-source llm instruction tuning

    Rongsheng Wang, Haoming Chen, Ruizhe Zhou, Han Ma, Yaofei Duan, Yanlan Kang, Songhua Yang, Baoyu Fan, and Tao Tan. Llm-detector: Improving ai-generated chi- nese text detection with open-source llm instruction tuning. arXiv preprint arXiv:2402.01158, 2024

  10. [10]

    Towards Reliable Detection of LLM-Generated Texts: A Comprehensive Evaluation Framework with CUDRT

    Zhen Tao, Zhiyu Li, Dinghao Xi, and Wei Xu. Cudrt: Benchmarking the detection of human vs. large language models generated texts. arXiv preprint arXiv:2406.09056, 2024. Preprint – LLM Encoder vs. Decoder: Robust Detection of Chinese AI-Generated Text with LoRA 8

  11. [11]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the as- sociation for computational linguistics: human language technologies, volume 1 (long and short papers) , pages 4171–4186, 2019

  12. [12]

    Roberta: A ro- bustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A ro- bustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019

  13. [13]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neu- ral information processing systems, 33:1877–1901, 2020

  14. [14]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

  15. [16]

    Deepseek-r1: Incentivizing reason- ing capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reason- ing capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

  16. [17]

    Bag of tricks for efficient text classifica- tion

    Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classifica- tion. arXiv preprint arXiv:1607.01759, 2016

  17. [18]

    Roberta and bi-lstm for human vs ai generated text detection

    Panagiotis Petropoulos and Vasilis Petropoulos. Roberta and bi-lstm for human vs ai generated text detection. Work- ing Notes of CLEF, 2024

  18. [19]

    Multiscale positive-unlabeled detection of ai-generated texts

    Yuchuan Tian, Hanting Chen, Xutao Wang, Zheyuan Bai, Qinghua Zhang, Ruifeng Li, Chao Xu, and Yunhe Wang. Multiscale positive-unlabeled detection of ai-generated texts. arXiv preprint arXiv:2305.18149, 2023

  19. [20]

    A watermark for large language models

    John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. In International Conference on Machine Learning, pages 17061–17084. PMLR, 2023

  20. [21]

    On Evaluating The Performance of Watermarked Machine-Generated Texts Under Adversarial Attacks

    Zesen Liu, Tianshuo Cong, Xinlei He, and Qi Li. On evaluating the performance of watermarked machine- generated texts under adversarial attacks. arXiv preprint arXiv:2407.04794, 2024

  21. [22]

    Wa- termarks in the sand: Impossibility of strong watermarking for generative models

    Hanlin Zhang, Benjamin L Edelman, Danilo Francati, Daniele Venturi, Giuseppe Ateniese, and Boaz Barak. Wa- termarks in the sand: Impossibility of strong watermarking for generative models. arXiv preprint arXiv:2311.04378, 2023

  22. [23]

    A Novel Multi-Task Teacher-Student Architecture with Self-Supervised Pretraining for 48-Hour Vasoactive-Inotropic Trend Analysis in Sepsis Mortality Prediction

    Houji Jin, Negin Ashrafi, Kamiar Alaei, Elham Pishgar, Greg Placencia, and Maryam Pishgar. A novel multi-task teacher-student architecture with self-supervised pretrain- ing for 48-hour vasoactive-inotropic trend analysis in sep- sis mortality prediction. arXiv preprint arXiv:2502.16834, 2025

  23. [24]

    Revisiting pre-trained models for chinese natural language processing

    Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. Revisiting pre-trained models for chinese natural language processing. arXiv preprint arXiv:2004.13922, 2020

  24. [25]

    Glu variants improve transformer

    Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020

  25. [26]

    Advanced predictive modeling for en- hanced mortality prediction in icu stroke patients using clinical data

    Armin Abdollahi, Negin Ashrafi, Xinghong Ma, Jia- hao Zhang, Daijia Wu, Tongshou Wu, Zizheng Ye, and Maryam Pishgar. Advanced predictive modeling for en- hanced mortality prediction in icu stroke patients using clinical data. PLoS One, 20(5):e0323441, 2025

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.