Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Tracing Thought: Using Chain-of-Thought Reasoning to Identify the LLM Behind AI-Generated Text

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Appending chain-of-thought explanations to BERT improves detection of AI-generated text and identifies the generating LLM.

desk verdict The reported CoT gain is an artifact of label leakage; the underlying idea is plausible but the evaluation as presented is broken. read the letter →

arxiv 2504.16913 v1 pith:HIM3VY6N submitted 2025-04-23 cs.CL cs.AI

Shifali Agrahari , Sanasam Ranbir Singh This is my paper · ORCID
classification cs.CLcs.AI
keywords chain-of-thoughtreasoningAI-generatedtextdetectionLLMattributionBERTfine-tuningclassificationinterpretabilityLLaMA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes COT_Finetuned, a dual-task framework that appends chain-of-thought reasoning to a fine-tuned BERT to classify text as AI-generated or human-written (Task A) and to identify which LLM produced an AI-generated document (Task B). The central claim is that the CoT reasoning, generated by LLaMA from a prompt that asks why the document was produced by its true label, substantially improves performance: BERT + CoT reaches 0.898 F1 on Task A and 0.307 on Task B, versus 0.742 and 0.249 for plain BERT. The authors argue the reasoning process adds interpretability as well as accuracy, and that CoT helps the model pick up stylistic patterns distinctive of different LLMs.

What carries the argument

The core mechanism is the chain-of-thought (CoT) explanation: for each document $d_i$, the prompt $p(d_i, y_i) = \text{'Why is this particular } d_i \text{ generated by } y_i\text{?'}$ is passed to LLaMA to produce reasoning $r_i$, and the document, its label, and $r_i$ are passed to a fine-tuned BERT whose output layer applies sigmoid for Task A and softmax for Task B. This CoT-augmented input is what carries the claimed improvement, together with a total loss that sums binary cross-entropy for Task A and cross-entropy for Task B.

What would settle it

Run the same COT_Finetuned pipeline on unlabeled test documents where the LLaMA reasoning is produced by a prompt that omits the true label, then compare the F1 scores; if the score falls back to the plain-BERT level, the reported CoT gain is an artefact of label leakage.

Watch

Extended reading notes

Core claim

The discovery is that a classifier can be made more accurate and more transparent by conditioning it on generated reasoning: for each training document, the method prompts LLaMA with 'Why is this particular document generated by this label?' and uses the resulting explanation as an additional input to a fine-tuned BERT. Trained with a joint loss over the binary human/AI decision and the multi-class LLM attribution, COT_Finetuned outperforms both RoBERTa and BERT alone. On the shared task's test set, BERT + CoT achieves an F1 of 0.898 for Task A and 0.307 for Task B, and the paper concludes that chain-of-thought reasoning improves both tasks.

Load-bearing premise

The reasoning fed into the classifier is generated from a prompt that already contains the true answer (the label), and the paper does not say how test documents get reasoning without leaking that label.

Editorial extensions

If this is right

  • If CoT-augmented classification works as claimed, adding explainable reasoning inputs is a cheap way to boost detector accuracy without changing the classifier architecture.
  • The dual-task framing means a single fine-tuned model can serve both human/AI filtering and forensic attribution of the generating model, which would matter for academic-integrity tools.
  • Since Task B scores remain low (0.307), the paper implies that identifying the exact LLM is inherently harder and needs more than current CoT inputs to reach usable accuracy.
  • The method's reliance on LLaMA-generated explanations implies detector quality depends on the explanatory capabilities of the reasoning generator, not just on the text classifier.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The prompt in Eq. (4) includes the true label $y_i$, so at test time the method would need an unstated label-free way to generate reasoning; if instead the label leaks through the reasoning, the reported improvement may be inflated.
  • A testable extension: generate reasoning from a label-free prompt (e.g., 'Why might this text be AI-generated?') and compare F1; a large drop would confirm label leakage.
  • CoT reasoning may act as a stylometric feature extractor: the explanations could encode the very stylistic cues that distinguish GPT-4.0 from DeBERTa or FalconMamba, so the method could be combined with contrastive learning to push Task B beyond 0.307.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes COT_Finetuned, a two-task framework for the DEFACTIFY 4.0 CT2 shared task: Task A classifies text as human-written or AI-generated, and Task B identifies which LLM generated an AI text. The method first uses LLaMA to generate chain-of-thought reasoning for each document, then fine-tunes BERT or RoBERTa on the document, its true label, and the reasoning. The authors report that adding CoT improves Task A F1 from 0.742 to 0.898 for BERT and Task B F1 from 0.249 to 0.307, and they conclude that CoT reasoning improves performance and interpretability.

Significance. The paper addresses a timely problem—detecting AI-generated text and attributing it to a specific model—and the idea of using free-form reasoning as an auxiliary feature is interesting. If the claims were valid, the approach would provide a transparent detection method. However, the central experimental result is undermined by a probable label leak in the reasoning-generation step, and the paper does not specify a reproducible test-time procedure. The reported F1 gains therefore cannot be attributed to CoT reasoning, which is the paper's core contribution. On the positive side, the authors participate in a shared task and report official leaderboard scores, but that does not compensate for the methodological flaws.

major comments (3)
  1. [3.2.4, Eqs. (4)-(5)] The prompt p(d_i, y_i) used to generate reasoning r_i contains the true Task A label y_i ("Why is this particular d_i generated by y_i?"). For Task B, the same construction is implied using y^B_i. The paper never specifies how reasoning is produced for unlabeled test documents; the only test-time instruction is to "pass the test dataset to the model." If the true label is used at test time, the classifier is effectively given the answer it is supposed to predict, so the reported F1 of 0.898 for Task A is not an honest detection score. If some unstated label-free reasoning generator is used instead, then the evaluated system is not the one described, and the comparison between BERT and BERT+COT is not reproducible. Either way, the claim that CoT improves performance is unsupported. Table 7 further confirms the leak: the reasoning outputs in the examples explicitly contain the classification labels ('Human' and 'LLM').
  2. [3.2.3, Eq. (3)] The text states that the combined loss includes a "reasoning loss component," but Eq. (3) sums only the two classification losses for Task A and Task B. No reasoning loss is defined or optimized anywhere else. Consequently, the claim that the model "learns to generate high-quality reasoning" is not backed by any training signal, and the reasoning strings function only as input features rather than as an output the model is trained to produce. This is a load-bearing discrepancy between the method description and the actual loss function.
  3. [4.2 and A.2] The experimental setup is internally inconsistent: the main text reports a batch size of 32 and epochs ranging from 50 to 250, while Appendix A.2 reports a batch size of 5 and epochs from 10 to 250. All results in Table 3 are single-run F1 scores with no variance, confidence intervals, or significance tests. Given the modest Task B improvement (0.249 to 0.307), the possibility that the observed gains are due to run-to-run variability cannot be ruled out. These issues make the central result difficult to evaluate or reproduce.
minor comments (6)
  1. [3.2.4] There is a typo in the final sentence: "Labe_A" should be "Label_A".
  2. [3.2.2, Eq. (1)] Equation (1) defines Label_B via a likelihood maximization over models, but this likelihood is never computed or used anywhere else in the paper; the actual Task B labels are simply taken from the dataset.
  3. [3.2.4, step 5] The phrase "the future text d_i" appears to be a typo; it should read "the text d_i".
  4. [Appendix A.2, Table 6] The hyperparameter row "k 6 layer" is unclear and should be removed or explained, and the learning rate format is inconsistent with the rest of the table.
  5. [Table 2] The column header says "Score" but the values are F1 scores; this should be stated explicitly in the caption.
  6. [References] Reference [13] has "????" in place of a year, and the email address in the footnote is corrupted; these should be corrected.

Circularity Check

3 steps flagged · score 7.0 of 10

The reported CoT gain is not evaluable because the reasoning input is generated from the true label and the test-time procedure is unspecified, so the central claim reduces to answer leakage by construction.

  1. self definitional [Section 3.2.4, Eqs. (4)-(5)]
    "For each document d_i, a prompt p(d_i, y_i) is created: p(d_i, y_i) = "Why is this particular d_i generated by y_i?". The prompt is passed to the LLaMA model, generating a reasoning r_i: o_i = LLaMA(p(d_i, y_i)) → r_i, where r_i provides insights into why d_i is classified as either AI-generated or human-written."

    The reasoning r_i is generated by conditioning on the true Task A label y_i, so the explanation text is constructed from the answer the classifier is supposed to predict. Feeding r_i to BERT as an input feature lets the classifier recover y_i from the reasoning itself. The reported BERT+COT F1 of 0.898 is therefore not an independent effect of chain-of-thought reasoning but a consequence of supplying the label inside the input. If the same prompt is used at test time, it requires the withheld labels, so the method cannot run as a blind detector; if a different label-free reasoning generator is used, it is never described.

  2. self definitional [Section 3.2.4, step 4]
    "For binary classification (Task A), the text d_i, label y_i, and reasoning r_i are passed to a pre-trained language model (PLM) such as BERT. The output layer applies a sigmoid activation for binary classification."

    The true Task A label y_i is explicitly listed as an input feature to the classifier whose output is the Task A prediction. If this protocol is applied at test time, the model is given the very answer it must output, making high accuracy forced by construction rather than by CoT reasoning. If y_i is intended only as a training-time target, the paper does not say so, and no test-time procedure is defined. Either interpretation invalidates the causal statement that CoT improves performance.

1 more flagged steps
  1. other [Section 3.2.4, final sentence; Section 4.1]
    "After completing the training, pass the test dataset to the model, which will return Labe_A and Label_B. ... testing data without labels for both tasks."

    This is the only description of test-time operation, but it never specifies how the reasoning r_i is produced for test documents when the true labels are withheld. If r_i is generated using the true label, the test answer leaks through the CoT prompt; if r_i is generated by some unstated label-free mechanism, that mechanism is absent from the paper and the ablation is unreproducible. The conclusion that CoT reasoning improves performance therefore rests on either leakage or an unverified, unspecified pipeline.

full rationale

The central claim of the paper is that adding LLaMA-generated chain-of-thought reasoning to BERT improves Task A F1 from 0.742 to 0.898 and Task B F1 from 0.249 to 0.307. The only described mechanism for generating that reasoning is Eq. (4)-(5), where the prompt contains the true label y_i: "Why is this particular d_i generated by y_i?". The resulting reasoning r_i is then passed to BERT along with d_i and, per step 4, the label y_i itself. Thus the input feature set is defined in terms of the target variable, so the classifier's output is not an independent prediction. At test time the paper simply says to pass the test dataset and receive labels, without explaining how reasoning is obtained for unlabeled test documents. Two possibilities remain: either the true label is used at test, leaking the answer, or a different reasoning generator is used, which is not described and would make the comparison non-reproducible. Either way, the statement "COT reasoning improves performance" is unsupported by the presented derivation. This is not a case of harmless self-citation or mere unorthodoxy; it is a structural circularity in the central evaluation. There is some independent content in the baseline BERT and RoBERTa results, and the method could in principle be repaired with a label-free reasoning generator, so the paper is not wholly vacuous. However, as written, the main claimed improvement reduces by construction to label leakage, warranting a score of 7.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The central claim rests on the unstated availability of label-free test-time reasoning and on the validity of the shared-task evaluation. The method introduces no new entities or fitted constants beyond standard hyperparameters, which are inconsistently reported.

assumptions (2)
  • domain assumption Chain-of-thought reasoning for test documents can be produced without the true labels, in a way consistent with training.
    Section 3.2.4 uses prompts with true labels to train; no label-free inference procedure is given.
  • domain assumption The DeFactify 4.0 F1 scores reflect a fair and stable evaluation.
    Single-run leaderboard scores without variance or significance testing.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tracing Thought: Using Chain-of-Thought Reasoning to Identify the LLM Behind AI-Generated Text." pith.science (2026). https://pith.science/paper/HIM3VY6N

@misc{pith2026250416913,
  author       = {Pith},
  title        = {Pith review of: Tracing Thought: Using Chain-of-Thought Reasoning to Identify the LLM Behind AI-Generated Text},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HIM3VY6N}},
  note         = {Machine review of arXiv:2504.16913}
}
read the original abstract

In recent years, the detection of AI-generated text has become a critical area of research due to concerns about academic integrity, misinformation, and ethical AI deployment. This paper presents COT Fine-tuned, a novel framework for detecting AI-generated text and identifying the specific language model. responsible for generating the text. We propose a dual-task approach, where Task A involves classifying text as AI-generated or human-written, and Task B identifies the specific LLM behind the text. The key innovation of our method lies in the use of Chain-of-Thought reasoning, which enables the model to generate explanations for its predictions, enhancing transparency and interpretability. Our experiments demonstrate that COT Fine-tuned achieves high accuracy in both tasks, with strong performance in LLM identification and human-AI classification. We also show that the CoT reasoning process contributes significantly to the models effectiveness and interpretability.

Figures

Figures reproduced from arXiv: 2504.16913 by the authors.

Figure 1
Figure 1. Proposed detector model for binary classification task A & multi classification task B. The key innovation in our approach is the use of Chain-of-Thought (CoT) reasoning, which allows the model to generate explanations for its decisions, enhancing interpretability and transparency. These explanations not only provide insights into the classification process but also assist in understanding the stylistic choices and … view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Agentic and Generative AI for Open-Source Intelligence and Cyber Investigations: Taxonomy, Evaluation, Challenges, and Future Directions

    cs.CR 2026-07 conditional novelty 6.0 of 10

    Across 74 OSINT/CTI AI studies, hallucination is widely named but end-to-end measured in only one non-reproducible system, so a human–AI co-pilot is the most defensible near-term architecture.

Reference graph

Works this paper leans on

21 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    R. Roy, G. Singh, A. Aziz, S. Bajpai, N. Imanpour, S. Biswas, K. Wanaskar, P. Patwa, S. Ghosh, S. Dixit, N. R. Pal, V. Rawte, R. Garimella, A. Das, A. Sheth, V. Sharma, A. N. Reganti, V. Jain, A. Chadha, Overview of text counter turing test: Ai generated text detection, in: proceedings of DeFactify 4: Fourth workshop on Multimodal Fact-Checking and Hate S...

    2025

  2. [2]

    Zellers, A

    R. Zellers, A. Holtzman, H. Rashkin, Y. Bisk, A. Farhadi, F. Roesner, Y. Choi, Defending against neural fake news, Advances in neural information processing systems 32 (2019)

    work page 2019

  3. [3]

    Gehrmann, H

    S. Gehrmann, H. Strobelt, A. M. Rush, Gltr: Statistical detection and visualization of generated text, arXiv preprint arXiv:1906.04043 (2019)

    arXiv 2019

  4. [4]

    Ippolito, D

    D. Ippolito, D. Duckworth, C. Callison-Burch, D. Eck, Automatic detection of generated text is easiest when humans are fooled, arXiv preprint arXiv:1911.00650 (2019)

    arXiv 2019

  5. [5]

    Z. Wang, J. Cheng, C. Cui, C. Yu, Implementing bert and fine-tuned roberta to detect ai generated news by chatgpt, ArXiv abs/2306.07401 (2023). URL: https://api.semanticscholar.org/CorpusID: 259145150

    arXiv 2023

  6. [6]

    Uchendu, Z

    A. Uchendu, Z. Ma, T. Le, R. Zhang, D. Lee, Turingbench: A benchmark environment for turing test in the age of neural text generation, arXiv preprint arXiv:2109.13296 (2021)

    arXiv 2021

  7. [7]

    Neural Deepfake Detection with Factual Structure of Text

    W. Zhong, D. Tang, Z. Xu, R. Wang, N. Duan, M. Zhou, J. Wang, J. Yin, Neural deepfake detection with factual structure of text, arXiv preprint arXiv:2010.07475 (2020)

    work page Pith review arXiv 2020

  8. [8]

    Y. Liu, Z. Zhang, W. Zhang, S. Yue, X. Zhao, X. Cheng, Y. Zhang, H. Hu, Argugpt: evaluating, understanding and identifying argumentative essays generated by gpt models, arXiv preprint arXiv:2304.07666 (2023)

    arXiv 2023

Show all 21 references
  1. [9]

    X. Liu, Z. Zhang, Y. Wang, H. Pu, Y. Lan, C. Shen, Coco: Coherence-enhanced machine-generated text detection under data limitation with contrastive learning, arXiv preprint arXiv:2212.10341 (2022)

    2022 arXiv

  2. [10]

    Mitchell, Y

    E. Mitchell, Y. Lee, A. Khazatsky, C. D. Manning, C. Finn, Detectgpt: Zero-shot machine-generated text detection using probability curvature, in: International Conference on Machine Learning, PMLR, 2023, pp. 24950–24962

    2023

  3. [11]

    J. Su, T. Y. Zhuo, D. Wang, P. Nakov, Detectllm: Leveraging log rank information for zero-shot detection of machine-generated text, arXiv preprint arXiv:2306.05540 (2023)

    2023 arXiv

  4. [12]

    Solaiman, M

    I. Solaiman, M. Brundage, J. Clark, A. Askell, A. Herbert-Voss, J. Wu, A. Radford, G. Krueger, J. W. Kim, S. Kreps, et al., Release strategies and the social impacts of language models, arXiv preprint arXiv:1908.09203 (2019)

    2019 arXiv

  5. [13]

    A. Hans, A. Schwarzschild, V. Cherepanova, H. Kazemi, A. Saha, M. Goldblum, J. Geiping, T. Gold- stein, Spotting llms with binoculars: Zero-shot detection of machine-generated text, 2024, URL: https://arxiv. org/abs/2401.12070 (????)

    2024 arXiv

  6. [14]

    Shijaku, E

    R. Shijaku, E. Canhasi, Chatgpt generated text detection, Publisher: Unpublished (2023)

    2023

  7. [15]

    Antoun, V

    W. Antoun, V. Mouilleron, B. Sagot, D. Seddah, Towards a robust detection of language model- generated text: Is chatgpt that easy to detect?, in: Actes de CORIA-TALN 2023. Actes de la 30e Conférence sur le Traitement Automatique des Langues Naturelles (TALN), volume 1: travaux...

    2023

  8. [16]

    Krishna, Y

    K. Krishna, Y. Song, M. Karpinska, J. Wieting, M. Iyyer, Paraphrasing evades detectors of ai- generated text, but retrieval is an effective defense, Advances in Neural Information Processing Systems 36 (2024)

    2024

  9. [17]

    A. Shah, P. Ranka, U. Dedhia, S. Prasad, S. Muni, K. Bhowmick, Detecting and unmasking ai- generated texts through explainable artificial intelligence using stylistic features, International Journal of Advanced Computer Science and Applications 14 (2023)

    2023

  10. [18]

    Nguyen-Son, N.-D

    H.-Q. Nguyen-Son, N.-D. T. Tieu, H. H. Nguyen, J. Yamagishi, I. E. Zen, Identifying computer- generated text using statistical analysis, in: 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), IEEE, 2017, pp. 1504–1511

    2017

  11. [19]

    Mindner, T

    L. Mindner, T. Schlippe, K. Schaaff, Classification of human-and ai-generated texts: Investigating features for chatgpt, in: International Conference on Artificial Intelligence in Education Technology, Springer, 2023, pp. 152–170

    2023

  12. [20]

    Kumarage, H

    T. Kumarage, H. Liu, Neural authorship attribution: Stylometric analysis on large language models, in: 2023 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC), IEEE Computer Society, 2023, pp. 51–54

    2023

  13. [21]

    R. Roy, G. Singh, A. Aziz, S. Bajpai, N. Imanpour, S. Biswas, K. Wanaskar, P. Patwa, S. Ghosh, S. Dixit, N. R. Pal, V. Rawte, R. Garimella, A. Das, A. Sheth, V. Sharma, A. N. Reganti, V. Jain, A. Chadha, Defactify-text: A comprehensive dataset for human vs. ai generated text d...

    2025

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.