{"id":"0c997ee4-edc4-49a6-a3fc-71faf5c179ba","arxiv_id":"2412.11517","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A detector that scores semantic gaps between an original text and its GPT-4o rephrasings (via AMR) and classifies the source, hitting 96.5% F1 on seen models and 85.6% in leave-one-out.","lead":"DART is a detector for AI-generated text that compares a text with its rephrased versions using a semantic representation called AMR. It reports high accuracy in telling apart texts from four different language models without using their internal probabilities.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The semantic-gap signal is never tested against a non-GPT rephraser, so the reported margins may reflect GPT-4o's identity as rephraser-and-source rather than a general human-vs-AI semantic gap.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing concern: the semantic-gap signal is measured only through GPT-4o rephrasings, and GPT-4o is also one of the source models. This is a genuine confound for the central claim that the method detects origins via a general human-vs-AI semantic difference. The paper's own Limitations section concedes the single-rephraser issue, but does not run the control that would resolve it. My stress-test agrees with the reader's conditional verdict because the concern is concrete and testable, not because the paper is internally inconsistent. I also note two supporting issues: the baseline comparison is asymmetric in auxiliary model strength (GPT-2 proxy vs. GPT-4o rephraser), and the leave-one-out average is reported as 85.6% in the text while Table 3 shows 84.2%, which should be corrected. None of these issues overturn the qualitative result that DART_DT performs well in the tested setting, but they do prevent the stronger claim of a rephraser-independent semantic-gap mechanism from being accepted without further evidence. A single rephraser-replacement experiment would settle whether the concern lands.","tokens_in":11278,"tokens_out":9800,"duration_ms":88183,"concrete_test":"Run the multi-candidate and leave-one-out experiments again with a non-GPT rephraser—e.g., Llama 3-70B—replacing GPT-4o in Step 1, holding the AMR parser (Naseem et al., 2022), SEMA scoring, and the depth-5 decision tree fixed. If DART_DT's macro F1 remains within ~2 points of the GPT-4o-rephraser results (Tables 2 and 3), the semantic-gap signal is rephraser-independent. If macro F1 drops by more than ~10 points, or the held-out Gemini score collapses further, the reported performance is contingent on GPT-4o's identity as both source and rephraser, and the central claim must be qualified accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"DART's central claim is that the SEMA precision/recall gap between an input and its GPT-4o rephrasing is an origin-specific signal (Section 3, Steps 1-3). The only evidence for this is the F1 tables. But GPT-4o is simultaneously the rephraser and one of the four AIGT sources (Section 4.1), so the detector can succeed by exploiting self-matching (GPT-4o rephrasing GPT-4o text) instead of a general semantic property of AI text. The Limitations section explicitly concedes 'we tested DART only with a single rephraser LLM, GPT-4o,' and no control with any other rephraser is run. The baseline comparison is also asymmetric: probabilistic baselines use GPT-2 as a proxy (footnote 1), while DART uses GPT-4o, so the >19-point margin may reflect auxiliary-model strength rather than AMR semantics. Table 6 weakens but does not eliminate the concern: GPT-4o's mean SEMA precision (0.636) is not the highest among sources, and GPT-4o is the hardest class in Table 2 (76.6 F1), yet the overall human-vs-AI separation could still be driven by rephraser-specific behavior. The leave-one-out result is the best evidence for transfer, but the unseen Gemini column drops to 62.5 F1 (Table 3), and the text reports 85.6% average while the table shows 84.2%, an unresolved numeric mismatch. Without a rephraser-identity control, the central claim that semantic gaps reveal origin is not established beyond GPT-4o.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DART, a four-step AIGT detector that rephrases an input text with GPT-4o, parses the original and paraphrases into AMR graphs, computes SEMA precision/recall features between them, and feeds these features to an SVM or decision-tree classifier. The authors evaluate DART on human texts from four domains and AIGTs from four LLMs under single-candidate, multi-candidate, and leave-one-out settings, reporting 96.5% average F1 for the decision-tree variant in the single-candidate setting, 81.2% macro F1 in the multi-candidate setting, and either 85.6% (prose) or 84.2% (table) in leave-one-out, with margins over the included baselines that the paper summarizes as more than 19 percentage points on average.","tokens_in":11568,"tokens_out":8157,"duration_ms":66668,"significance":"If the central assumption survives controls, DART is a useful contribution: it detects AIGT without requiring target-model logits, it is inexpensive at inference time compared with paraphrase-heavy baselines, and it tackles the relatively understudied multi-origin detection problem. The experimental protocol is a strength: 10 repetitions with reported standard deviations, per-domain checks, and training-efficiency curves make the results more interpretable than typical single-run AIGT evaluations. However, the paper's central claim that semantic gaps between an input and its GPT-4o rephrasing are an origin-specific signal is currently entangled with GPT-4o's dual role as both rephraser and one of the tested generators. The leave-one-out result and the numerical inconsistency in Table 3 add further uncertainty. The core idea remains promising and the required experiments are within the manuscript's scope, so revision, not rejection, is appropriate.","major_comments":[{"comment":"The core semantic-gap signal is computed by rephrasing every input with GPT-4o and scoring AMR precision/recall with SEMA. Because GPT-4o is also one of the four AIGT generators evaluated in Section 4.1, a detector that uses GPT-4o rephrasing may succeed by detecting whether the input was produced by the same model that produced the paraphrase, rather than by detecting a general semantic property of AIGT. The per-subset checks in Table 5 and the feature averages in Table 6 do not remove this confound, since the rephrasing model is constant throughout all conditions. The Limitations section concedes that only a single rephraser was tested. This is load-bearing for the paper's generalization claim, so I ask for a control in which the rephraser is changed to a non-GPT model (e.g., Llama-3-70B or Gemini-1.5-Flash) and the single-candidate and leave-one-out experiments are rerun, or at minimum a direct comparison of feature distributions for GPT-4o versus a non-GPT rephraser is provided.","section":"Section 3, Steps 1-3; Section 4.1; Table 6; Limitations"},{"comment":"The prose reports that DARTDT achieved 85.6% average F1 in the leave-one-out experiment, but the values in Table 3 (99.3, 75.8, 99.1, 62.5) average to 84.2%, which is the number printed in the table. This internal inconsistency must be resolved and the prose corrected. In addition, the held-out Gemini-1.5 result of 62.5% shows that generalization is uneven; the claim that DARTDT 'can generalize trained knowledge to detect unseen source models' should be qualified to state which held-out models support the claim.","section":"Section 5, Table 3"},{"comment":"The baseline comparison is asymmetric in a way that directly affects the reported margins. DetectGPT, Fast-DetectGPT, and SeqXGPT are run with GPT-2 as a proxy model (footnote 1), while DART uses GPT-4o as its rephraser. The average margin of more than 19 percentage points may therefore reflect the relative capability of the auxiliary models (GPT-4o versus GPT-2) rather than the AMR semantic-gap representation. To isolate DART's contribution, at least one comparison should pair DART with a weaker rephraser (e.g., a GPT-2-sized model) or pair the probabilistic baselines with a comparably capable auxiliary model; without such a control, the superiority claim in the contributions list is not yet established.","section":"Footnote 1, Section 4.2, Table 1"},{"comment":"The choice of SEMA as the AMR scorer rests on a validation paper (Ki et al., 2024) with an overlapping author, and no alternative AMR similarity metric is compared. Because the entire feature vector v=[p1,p2,r1,r2] is produced by SEMA, a systematic bias in SEMA's precision/recall would propagate directly into DART's classification. I recommend an ablation using Smatch or another standard AMR scorer to confirm that the reported results are not an artifact of the particular scorer.","section":"Section 3, Step 3; References Ki et al. (2024); Appendix C.1"}],"minor_comments":[{"comment":"The multi-candidate setting is described as 'multi-label classification,' but each text has exactly one origin, so the task is multiclass classification; please correct the terminology.","section":"Section 4, Table 2"},{"comment":"The rephrasing prompt says 'rewrite the following paragraph in {n} words,' but the value of n is never specified in the paper; state how n was chosen for each input length.","section":"Appendix A.2"},{"comment":"The Anchiêta et al. (2019) reference is incomplete, listing '(To appear)' and misspelling 'Publishg'; please update it to the published proceedings information.","section":"References"},{"comment":"The decision-tree depth of 5 is a heuristic free parameter; the paper would be stronger with a brief sensitivity analysis or a justification based on validation performance.","section":"Appendix B.1"},{"comment":"The paper says experiments were run 10 times but provides no code or data release and no seed information; adding a repository or at least fixed random seeds would strengthen the reproducibility claim.","section":"Appendix B.1"},{"comment":"The header 'Xsum' should be 'XSum' and 'SQuad' should be 'SQuAD' for consistency with the dataset names used elsewhere.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the journal's scope and the four-step framework is presented clearly. The main risk is not internal inconsistency but the rephraser-identity confound: because GPT-4o is both rephraser and tested source, the headline results may overstate the generality of the semantic-gap signal. The absence of any non-GPT rephraser control, the SEMA validation being self-cited by an overlapping author, and the 85.6% versus 84.2% discrepancy in Table 3 all need to be addressed before publication. I do not see grounds for rejection if the authors can run the rephraser control; the method is simple enough that such a control is feasible within a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: DART is a serious, well-written attempt at black-box AIGT detection. The idea—rephrase the input twice, parse to AMR, score the semantic gaps with SEMA, and feed the four gap features to an SVM or decision tree—is simple and easy to reproduce, and the multi-candidate and leave-one-out evaluation is the right direction. The authors run each experiment 10 times, report standard deviations, and include per-subset checks. That is more care than most detector papers take. The Limitations section is candid about the single-rephraser issue and parser sensitivity.\n\nThe new contributions are real: the AMR-based gap features and the explicit multi-candidate / leave-one-out framing go beyond the RAIDAR rephrasing idea, which they cite. The per-subset robustness is genuinely useful.\n\nThe soft spots are concentrated around one confound and one baseline asymmetry. The rephraser is GPT-4o, which is also one of the four source models. The paper acknowledges this in Limitations but never runs the obvious control: rephrase with a non-GPT model (Llama or Gemini) and see whether the margins hold. Without that, the detector may be exploiting self-matching artifacts—GPT-4o rephrasing GPT-4o text—rather than a general human-vs-AI semantic gap. The Table 6 data provide some mitigation: GPT-4o’s SEMA precision is not the highest, and GPT-4o is the hardest class in the multi-candidate setting. But those observations do not rule out a self-matching contribution to the human-vs-AI boundary, which is where the main gain sits.\n\nSecond, the baselines are asymmetric: probabilistic detectors use GPT-2 as a proxy, while DART uses GPT-4o. The reported >19-point margin therefore mixes model strength with method. The multi-candidate and leave-one-out experiments compare only against SeqXGPT, and RAIDAR—the direct predecessor whose rephrasing idea the paper adopts—is not evaluated at all.\n\nThird, a concrete editorial problem: the text reports 85.6% average F1 for leave-one-out DART_DT, but Table 3 shows 84.2%. The SeqXGPT number in the text (77.9%) also differs from the table (78.5%). The abstract repeats 85.6%. Either the table or the text is wrong, and it needs fixing.\n\nThe paper works as an interesting method with a plausible but unproven core claim. It deserves serious peer review, but the referee should require the rephraser-identity control, a parity-fair baseline comparison (including RAIDAR), and corrected numbers. I would not cite it yet, but I would bring it to the reading group as a good example of a detector that gets the evaluation framing mostly right.","headline":"DART’s AMR-based gap features and multi-candidate evaluation are a real step forward, but the GPT-4o rephraser-and-source confound and a mismatch in the reported leave-one-out numbers keep the generalization claim from holding yet.","tokens_in":12174,"tokens_out":2949,"would_cite":false,"duration_ms":26033,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DART claims that a text's author can be identified by measuring how its meaning drifts when it is rephrased, reaching 96.5% F1 without using any model probabilities.","keywords":["AI-generated text detection","black-box LLM detection","Abstract Meaning Representation","rephrasing","semantic gap","source attribution","decision tree","SEMA"],"falsifier":"Run the identical DART pipeline with a non-GPT rephraser, such as Llama 3 or Gemini, over the same four-source dataset; if the single-candidate F1 of about 96.5% or the leave-one-out F1 of about 85.6% collapses, the claimed origin signal is an artifact of GPT-4o rewriting itself rather than a general semantic difference between human and AI text.","tokens_in":11024,"feed_emoji":"🤖","tokens_out":9941,"duration_ms":75575,"temperature":0.7,"pith_summary":"This paper proposes DART, a detector for AI-generated text that needs no access to token probabilities, which black-box commercial models do not expose. Instead of scoring word likelihoods, DART rewrites the input twice, parses the original and both rewrites into Abstract Meaning Representation graphs, and measures the semantic gap between the original and each rewrite. The claim is that this gap is a fingerprint of origin: it separates human writing from text produced by four different LLMs and can even say which LLM wrote a given passage. The decision-tree variant reaches 96.5% average F1 in the single-candidate setting, 81.2% macro F1 in the multi-candidate setting, and 85.6% macro F1 in the leave-one-out setting, beating the included baselines by more than 19 percentage points on average. The practical stake is that detecting AI text and attributing its source can be reframed as measuring meaning drift rather than estimating generation statistics.","feed_headline":"Meaning drift under rephrasing reveals the author at 96.5% F1","feed_subtitle":"DART scores meaning drift between text and rewrites, separating humans from four LLMs with no token probabilities.","key_machinery":"The load-bearing mechanism is the semantic-gap feature vector produced by a four-step pipeline. Rephrasing generates two semantic variants of the input text; semantic parsing converts each text into an AMR graph, a structured representation of who did what to whom; SEMA then scores precision and recall between the original graph and each rephrased graph. Those four scores are the only features given to an SVM or a decision tree. The mechanism turns an inaccessible property, which model or person generated the text, into an observable numerical signature that does not require logits or a proxy model.","core_discovery":"DART's central discovery is that the semantic distance between a text and its rephrased versions carries enough information to classify the text's origin. The input T0 is rephrased twice to T1 and T2, all three are parsed into AMR graphs, and SEMA precision and recall compare the original graph with each rephrased graph, producing the vector [p1, p2, r1, r2]. A decision tree or support vector machine trained on these four numbers assigns the text to Human, GPT-3.5-turbo, GPT-4o, Llama 3-70B, or Gemini-1.5-Flash. The paper reports that these vectors form loose clusters by origin and that the clusters are stable enough for a model trained without one source to still detect that source, with the largest drop when Gemini-1.5 is held out. The claim is therefore that origin information lives in meaning structure, not in token probabilities.","pith_inferences":["A testable extension the paper does not run is swapping the rephraser from GPT-4o to a cheaper open-weight model; a collapse in performance would mean the signal is a property of GPT-4o's rewriting style, not of human versus AI semantics.","The same AMR-gap features could plausibly detect machine-translated text or data contamination, since both introduce systematic meaning drift.","If the reported clusters are as separable as the principal-component plots suggest, a nearest-centroid classifier might reach near-decision-tree accuracy at lower cost and with a more auditable decision rule.","Because the rephraser and one candidate generator are the same model, an explicit ablation removing GPT-4o from the training candidates would show whether the remaining models are still separated by the same gap."],"forward_implications":["Detectors for black-box models can be built without exposing probabilities: the input text plus a rephraser is enough.","Origin attribution becomes a multiclass question, so the same framework can say which of several LLMs wrote a text, not just whether it is AI-generated.","The leave-one-out result implies that a detector trained on known models can flag texts from a new model it has never seen.","The decision-tree version keeps a similar F1 score down to roughly 500 training examples, suggesting the semantic signal is cheap to learn.","The decision rule is readable: a threshold on rephrasing-induced semantic precision or recall separates human from machine writing, giving an interpretable audit trail."],"supporting_citations":[{"why":"Supplies the rewriting idea that DART builds on: rephrasing exposes cues invisible in surface syntax.","marker":"(Mao et al., 2024)"},{"why":"Defines Abstract Meaning Representation, the graph formalism DART uses to abstract meaning.","marker":"(Banarescu et al., 2013)"},{"why":"Provides the AMR parser that turns original and rephrased texts into graphs.","marker":"(Naseem et al., 2022)"},{"why":"Defines SEMA, the precision/recall AMR similarity metric used to quantify semantic gaps.","marker":"(Anchiêta et al., 2019)"},{"why":"Validates the behavior of AMR similarity metrics, supporting the choice of SEMA for gap scoring.","marker":"(Ki et al., 2024)"},{"why":"Provides the DetectGPT baseline and the first-30-token continuation practice used to build AIGT data.","marker":"(Mitchell et al., 2023)"},{"why":"Provides the SeqXGPT multiclass baseline and the prompt style used for AIGT generation.","marker":"(Wang et al., 2023)"},{"why":"Supplies GPT-4o, the model used both as the rephraser and as one of the AIGT sources evaluated.","marker":"(OpenAI, 2024a)"}],"fun_headline_variants":["Rephrase and catch: DART detects AI by meaning drift, not token odds","Meaning drift after rephrasing IDs authors: DART beats token stats","No probabilities: DART rephrases text and reads semantic drift to find AI","Rephrase, parse, compare: DART exposes AI authors via AMR drift","DART catches AI by meaning structure, not token probabilities"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the meaning gap between a text and its GPT-4o rephrasings is a stable, origin-specific fingerprint that transfers to unseen text domains and unseen writing models, even though GPT-4o serves both as the rephraser and as one of the candidate sources it is meant to detect.","fun_headline_variants_meta":{"raw":{"variants":["Rephrase and catch: DART detects AI by meaning drift, not token odds","Meaning drift after rephrasing IDs authors: DART beats token stats","No probabilities: DART rephrases text and reads semantic drift to find AI","Rephrase, parse, compare: DART exposes AI authors via AMR drift","DART catches AI by meaning structure, not token probabilities"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000748,"raw_usage":{"total_tokens":3307,"prompt_tokens":898,"completion_tokens":2409,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":2307}},"tokens_in":514,"tokens_out":2409,"duration_ms":14481,"temperature":1.0,"reasoning_tokens":2307,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:50:41.615111+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical DART pipeline with a non-GPT rephraser, such as Llama 3 or Gemini, over the same four-source dataset; if the single-candidate F1 of about 96.5% or the leave-one-out F1 of about 85.6% collapses, the claimed origin signal is an artifact of GPT-4o rewriting itself rather than a general semantic difference between human and AI text.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the rewriting idea that DART builds on: rephrasing exposes cues invisible in surface syntax."}],"review_version":1}