REVIEW 4 major objections 6 minor 28 references
DART: An AIGT Detector using AMR of Rephrased Text
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read DART claims that a text's author can be identified by measuring how its meaning drifts when it is rephrased, reaching 96.5% F1 without using any model probabilities.
desk verdict DART’s AMR-based gap features and multi-candidate evaluation are a real step forward, but the GPT-4o rephraser-and-source confound and a mismatch in the reported leave-one-out numbers keep the generalization claim from holding yet. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the semantic-gap feature vector produced by a four-step pipeline. Rephrasing generates two semantic variants of the input text; semantic parsing converts each text into an AMR graph, a structured representation of who did what to whom; SEMA then scores precision and recall between the original graph and each rephrased graph. Those four scores are the only features given to an SVM or a decision tree. The mechanism turns an inaccessible property, which model or person generated the text, into an observable numerical signature that does not require logits or a proxy model.
What would settle it
Run the identical DART pipeline with a non-GPT rephraser, such as Llama 3 or Gemini, over the same four-source dataset; if the single-candidate F1 of about 96.5% or the leave-one-out F1 of about 85.6% collapses, the claimed origin signal is an artifact of GPT-4o rewriting itself rather than a general semantic difference between human and AI text.
Extended reading notes
Core claim
DART's central discovery is that the semantic distance between a text and its rephrased versions carries enough information to classify the text's origin. The input T0 is rephrased twice to T1 and T2, all three are parsed into AMR graphs, and SEMA precision and recall compare the original graph with each rephrased graph, producing the vector [p1, p2, r1, r2]. A decision tree or support vector machine trained on these four numbers assigns the text to Human, GPT-3.5-turbo, GPT-4o, Llama 3-70B, or Gemini-1.5-Flash. The paper reports that these vectors form loose clusters by origin and that the clusters are stable enough for a model trained without one source to still detect that source, with the largest drop when Gemini-1.5 is held out. The claim is therefore that origin information lives in meaning structure, not in token probabilities.
Load-bearing premise
The load-bearing premise is that the meaning gap between a text and its GPT-4o rephrasings is a stable, origin-specific fingerprint that transfers to unseen text domains and unseen writing models, even though GPT-4o serves both as the rephraser and as one of the candidate sources it is meant to detect.
Editorial extensions
If this is right
- Detectors for black-box models can be built without exposing probabilities: the input text plus a rephraser is enough.
- Origin attribution becomes a multiclass question, so the same framework can say which of several LLMs wrote a text, not just whether it is AI-generated.
- The leave-one-out result implies that a detector trained on known models can flag texts from a new model it has never seen.
- The decision-tree version keeps a similar F1 score down to roughly 500 training examples, suggesting the semantic signal is cheap to learn.
- The decision rule is readable: a threshold on rephrasing-induced semantic precision or recall separates human from machine writing, giving an interpretable audit trail.
Reading between the lines
- A testable extension the paper does not run is swapping the rephraser from GPT-4o to a cheaper open-weight model; a collapse in performance would mean the signal is a property of GPT-4o's rewriting style, not of human versus AI semantics.
- The same AMR-gap features could plausibly detect machine-translated text or data contamination, since both introduce systematic meaning drift.
- If the reported clusters are as separable as the principal-component plots suggest, a nearest-centroid classifier might reach near-decision-tree accuracy at lower cost and with a more auditable decision rule.
- Because the rephraser and one candidate generator are the same model, an explicit ablation removing GPT-4o from the training candidates would show whether the remaining models are still separated by the same gap.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DART, a four-step AIGT detector that rephrases an input text with GPT-4o, parses the original and paraphrases into AMR graphs, computes SEMA precision/recall features between them, and feeds these features to an SVM or decision-tree classifier. The authors evaluate DART on human texts from four domains and AIGTs from four LLMs under single-candidate, multi-candidate, and leave-one-out settings, reporting 96.5% average F1 for the decision-tree variant in the single-candidate setting, 81.2% macro F1 in the multi-candidate setting, and either 85.6% (prose) or 84.2% (table) in leave-one-out, with margins over the included baselines that the paper summarizes as more than 19 percentage points on average.
Significance. If the central assumption survives controls, DART is a useful contribution: it detects AIGT without requiring target-model logits, it is inexpensive at inference time compared with paraphrase-heavy baselines, and it tackles the relatively understudied multi-origin detection problem. The experimental protocol is a strength: 10 repetitions with reported standard deviations, per-domain checks, and training-efficiency curves make the results more interpretable than typical single-run AIGT evaluations. However, the paper's central claim that semantic gaps between an input and its GPT-4o rephrasing are an origin-specific signal is currently entangled with GPT-4o's dual role as both rephraser and one of the tested generators. The leave-one-out result and the numerical inconsistency in Table 3 add further uncertainty. The core idea remains promising and the required experiments are within the manuscript's scope, so revision, not rejection, is appropriate.
major comments (4)
- [Section 3, Steps 1-3; Section 4.1; Table 6; Limitations] The core semantic-gap signal is computed by rephrasing every input with GPT-4o and scoring AMR precision/recall with SEMA. Because GPT-4o is also one of the four AIGT generators evaluated in Section 4.1, a detector that uses GPT-4o rephrasing may succeed by detecting whether the input was produced by the same model that produced the paraphrase, rather than by detecting a general semantic property of AIGT. The per-subset checks in Table 5 and the feature averages in Table 6 do not remove this confound, since the rephrasing model is constant throughout all conditions. The Limitations section concedes that only a single rephraser was tested. This is load-bearing for the paper's generalization claim, so I ask for a control in which the rephraser is changed to a non-GPT model (e.g., Llama-3-70B or Gemini-1.5-Flash) and the single-candidate and leave-one-out experiments are rerun, or at minimum a direct comparison of feature distributions for GPT-4o versus a non-GPT rephraser is provided.
- [Section 5, Table 3] The prose reports that DARTDT achieved 85.6% average F1 in the leave-one-out experiment, but the values in Table 3 (99.3, 75.8, 99.1, 62.5) average to 84.2%, which is the number printed in the table. This internal inconsistency must be resolved and the prose corrected. In addition, the held-out Gemini-1.5 result of 62.5% shows that generalization is uneven; the claim that DARTDT 'can generalize trained knowledge to detect unseen source models' should be qualified to state which held-out models support the claim.
- [Footnote 1, Section 4.2, Table 1] The baseline comparison is asymmetric in a way that directly affects the reported margins. DetectGPT, Fast-DetectGPT, and SeqXGPT are run with GPT-2 as a proxy model (footnote 1), while DART uses GPT-4o as its rephraser. The average margin of more than 19 percentage points may therefore reflect the relative capability of the auxiliary models (GPT-4o versus GPT-2) rather than the AMR semantic-gap representation. To isolate DART's contribution, at least one comparison should pair DART with a weaker rephraser (e.g., a GPT-2-sized model) or pair the probabilistic baselines with a comparably capable auxiliary model; without such a control, the superiority claim in the contributions list is not yet established.
- [Section 3, Step 3; References Ki et al. (2024); Appendix C.1] The choice of SEMA as the AMR scorer rests on a validation paper (Ki et al., 2024) with an overlapping author, and no alternative AMR similarity metric is compared. Because the entire feature vector v=[p1,p2,r1,r2] is produced by SEMA, a systematic bias in SEMA's precision/recall would propagate directly into DART's classification. I recommend an ablation using Smatch or another standard AMR scorer to confirm that the reported results are not an artifact of the particular scorer.
minor comments (6)
- [Section 4, Table 2] The multi-candidate setting is described as 'multi-label classification,' but each text has exactly one origin, so the task is multiclass classification; please correct the terminology.
- [Appendix A.2] The rephrasing prompt says 'rewrite the following paragraph in {n} words,' but the value of n is never specified in the paper; state how n was chosen for each input length.
- [References] The Anchiêta et al. (2019) reference is incomplete, listing '(To appear)' and misspelling 'Publishg'; please update it to the published proceedings information.
- [Appendix B.1] The decision-tree depth of 5 is a heuristic free parameter; the paper would be stronger with a brief sensitivity analysis or a justification based on validation performance.
- [Appendix B.1] The paper says experiments were run 10 times but provides no code or data release and no seed information; adding a repository or at least fixed random seeds would strengthen the reproducibility claim.
- [Table 5] The header 'Xsum' should be 'XSum' and 'SQuad' should be 'SQuAD' for consistency with the dataset names used elsewhere.
Circularity Check
No significant circularity: DART's pipeline is empirically self-contained; the only self-citation (Ki et al., 2024) is not load-bearing, and the GPT-4o rephraser/source overlap is a confound, not a constructional reduction.
full rationale
DART is an empirical four-step pipeline (GPT-4o rephrasing, AMR parsing, SEMA scoring, and a shallow classifier) that is trained and evaluated on held-out splits; no parameter is fitted to the test labels, and the leave-one-out experiment trains on three sources and tests on a fourth unseen source, which provides genuine external validation. The only self-citation is the SEMA reference "(Anchiêta et al., 2019; Ki et al., 2024)" in Section 3, Step 3, where Ki et al. has an overlapping author, Bugeun Kim; however, SEMA itself is independently introduced by Anchiêta et al. (2019), and DART's F1 results do not rest on any claim unique to the self-citation, so the self-citation is not load-bearing. The overlap between the rephraser (GPT-4o) and one evaluated source (GPT-4o) is a real external-validity confound, explicitly acknowledged in the Limitations section: "we tested DART only with a single rephraser LLM, GPT-4o." This does not make the derivation circular because the feature vector v=[p1,p2,r1,r2] is computed from SEMA similarities and is not constructed from the source labels; moreover, the reported results go against a self-matching artifact explanation, since GPT-4o is the hardest class in Table 2 (76.6 F1) and its SEMA precision in Table 6 (0.636) is not the highest among the four sources. The text/table discrepancy in the leave-one-out average (85.6% claimed vs 84.2% in Table 3) is a reporting or arithmetic issue, not evidence of circularity. No equation in the paper reduces to its own input, no fitted parameter is renamed as a prediction, and no imported uniqueness theorem is invoked; the central claim therefore has independent empirical content, and the minor self-citation warrants at most a score of 2.
Assumptions & free parameters
free parameters (1)
- Decision tree max depth =
5
assumptions (4)
- domain assumption GPT-4o rephrasing preserves the core semantics of the input while exposing origin-specific style
- domain assumption DocAMR parser produces accurate semantic graphs for all text domains
- domain assumption SEMA precision and recall scores on AMR graphs measure meaningful semantic gaps
- domain assumption The four tested LLMs represent the space of black-box AIGT origins
Cite this review
Pith. "Pith review of DART: An AIGT Detector using AMR of Rephrased Text." pith.science (2026). https://pith.science/paper/JQWBR5NK
@misc{pith2026241211517,
author = {Pith},
title = {Pith review of: DART: An AIGT Detector using AMR of Rephrased Text},
year = {2026},
howpublished = {\url{https://pith.science/paper/JQWBR5NK}},
note = {Machine review of arXiv:2412.11517}
}
read the original abstract
As large language models (LLMs) generate more human-like texts, concerns about the side effects of AI-generated texts (AIGT) have grown. So, researchers have developed methods for detecting AIGT. However, two challenges remain. First, the performance of detecting black-box LLMs is low because existing models focus on probabilistic features. Second, most AIGT detectors have been tested on a single-candidate setting, which assumes that we know the origin of an AIGT and which may deviate from the real-world scenario. To resolve these challenges, we propose DART, which consists of four steps: rephrasing, semantic parsing, scoring, and multiclass classification. We conducted three experiments to test the performance of DART. The experimental result shows that DART can discriminate multiple black-box LLMs without probabilistic features and the origin of AIGT.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Harika Abburi, Michael Suesserman, Nirmala Pudota, Balaji Veeramani, Edward Bowen, and Sanmitra Bhattacharya. 2023. http://dblp.uni-trier.de/db/conf/sepln/iberlef2023.html#AbburiSPVBB23 Generative ai text classification using ensemble llm approaches. In IberLEF@SEPLN, volume 3496 of CEUR Workshop Proceedings. CEUR-WS.org
work page 2023
-
[2]
Rafael Torres Anchi\^ e ta, Marco Antonio Sobrevilla Cabezudo, and Thiago Alexandre Salgueiro Pardo. 2019. Sema: an extended semantic evaluation for amr. In (To appear) Proceedings of the 20th Computational Linguistics and Intelligent Text Processing. Springer International Publishg
work page 2019
-
[3]
Wissam Antoun, Beno \^ t Sagot, and Djam \'e Seddah. 2024. https://aclanthology.org/2024.lrec-main.665 From text to source: Results in detecting large language model-generated content . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 7531--7543, Torino, Ita...
work page 2024
-
[4]
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. https://aclanthology.org/W13-2322 A bstract M eaning R epresentation for sembanking . In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178--186...
2013
-
[5]
Guangsheng Bao, Yanbin Zhao, Zhiyang Teng, Linyi Yang, and Yue Zhang. 2024. https://openreview.net/forum?id=Bpcgcr8E8Z Fast-detectgpt: Efficient zero-shot detection of machine-generated text via conditional probability curvature . In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net
work page 2024
-
[6]
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783
arXiv 2024
-
[7]
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. https://doi.org/10.18653/v1/P18-1082 Hierarchical neural story generation . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889--898, Melbourne, Australia. Association for Computational Linguistics
-
[8]
Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. 2019. https://doi.org/10.18653/v1/P19-3019 GLTR : Statistical detection and visualization of generated text . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 111--116, Florence, Italy. Association for Computational Linguistics
Show all 28 references
-
[9]
Xiaomeng Hu, Pin - Yu Chen, and Tsung - Yi Ho. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/30e15e5941ae0cdab7ef58cc8d59a4ca-Abstract-Conference.html RADAR: robust ai-text detection via adversarial learning . In Advances in Neural Information Processing Systems 36:...
2023
-
[10]
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. https://doi.org/10.18653/v1/D19-1259 P ub M ed QA : A dataset for biomedical research question answering . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing an...
2019 doi
-
[11]
Kyung Seo Ki, Bugeun Kim, and Gahgene Gweon. 2024. https://doi.org/10.18653/v1/2024.starsem-1.32 Inspecting soundness of AMR similarity metrics in terms of equivalence and inequivalence . In Proceedings of the 13th Joint Conference on Lexical and Computational Semantics (*SEM ...
2024 doi
-
[12]
Zae Myung Kim, Kwang Lee, Preston Zhu, Vipul Raheja, and Dongyeop Kang. 2024. https://doi.org/10.18653/v1/2024.acl-long.298 Threads of subtlety: Detecting machine-generated texts through discourse motifs . In Proceedings of the 62nd Annual Meeting of the Association for Comput...
2024 doi
-
[13]
Linyang Li, Pengyu Wang, Ke Ren, Tianxiang Sun, and Xipeng Qiu. 2023. https://arxiv.org/abs/2304.14072 Origin tracing and detecting of llms . Preprint, arXiv:2304.14072
2023 arXiv
-
[14]
Chengzhi Mao, Carl Vondrick, Hao Wang, and Junfeng Yang. 2024. https://openreview.net/forum?id=bQWE2UqXmf Raidar: generative AI detection via rewriting
2024
-
[15]
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. 2023. https://proceedings.mlr.press/v202/mitchell23a.html D etect GPT : Zero-shot machine-generated text detection using probability curvature . In Proceedings of the 40th International Co...
2023
-
[16]
Cohen, and Mirella Lapata
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. https://doi.org/10.18653/v1/D18-1206 Don ' t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization . In Proceedings of the 2018 Conference on Empirical Methods in Natu...
2018 doi
-
[17]
Tahira Naseem, Austin Blodgett, Sadhana Kumaravel, Tim O ' Gorman, Young-Suk Lee, Jeffrey Flanigan, Ram \'o n Astudillo, Radu Florian, Salim Roukos, and Nathan Schneider. 2022. https://doi.org/10.18653/v1/2022.naacl-main.256 D oc AMR : Multi-sentence AMR representation and eva...
2022 doi
-
[18]
OpenAI. 2024 a . Hello gpt-4o. https://openai.com/index/hello-gpt-4o/. Accessed: 2024-10-13
2024
-
[19]
OpenAI. 2024 b . Introducing chatgpt. https://openai.com/index/chatgpt/. Accessed: 2024-10-13
2024
-
[20]
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. https://doi.org/10.18653/v1/P18-2124 Know what you don ' t know: Unanswerable questions for SQ u AD . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages ...
2018 doi
-
[21]
Yuhui Shi, Qiang Sheng, Juan Cao, Hao Mi, Beizhe Hu, and Danding Wang. 2024. https://doi.org/10.24963/ijcai.2024/55 Ten words only still help: Improving black-box ai-generated text detection via proxy-guided efficient re-sampling . In Proceedings of the Thirty-Third Internatio...
2024 doi
-
[22]
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, Miles McCain, Alex Newhouse, Jason Blazakis, Kris McGuffie, and Jasmine Wang. 2019. https://arxiv.org/abs/1908.09203 Release stra...
2019 arXiv
-
[23]
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024. https://arxiv.org/abs/2403.05530 Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context . Prepri...
2024 arXiv
-
[24]
Pengyu Wang, Linyang Li, Ke Ren, Botian Jiang, Dong Zhang, and Xipeng Qiu. 2023. https://aclanthology.org/2023.emnlp-main.73 S eq XGPT : Sentence-level AI -generated text detection . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pag...
2023
-
[25]
Xianjun Yang, Wei Cheng, Yue Wu, Linda Petzold, William Yang Wang, and Haifeng Chen. 2023. https://arxiv.org/abs/2305.17359 Dna-gpt: Divergent n-gram analysis for training-free detection of gpt-generated text
2023 arXiv
-
[26]
Xiao Yu, Yuang Qi, Kejiang Chen, Guoqiang Chen, Xi Yang, Pengyuan Zhu, Xiuwei Shang, Weiming Zhang, and Nenghai Yu. 2024. https://arxiv.org/abs/2305.12519 Dpic: Decoupling prompt and intrinsic characteristics for llm generated text detection . Preprint, arXiv:2305.12519
2024 arXiv
-
[27]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[28]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.