{"id":"724565ba-a42b-4e6e-82d1-568be45236f6","arxiv_id":"2505.12405","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Repeated word-pattern similarity between a ChatGPT-generated reference and a suspicious article detects ChatGPT paraphrases of BBC news with roughly 96% accuracy, outperforming the RADAR detector on this benchmark.","lead":"This paper tests whether matching repeated word patterns can tell a ChatGPT paraphrase of a BBC news article apart from a Gemini paraphrase or the original. It reports about 96% accuracy on a new 4,448-article benchmark, without using deep learning or a trained classifier.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Phase II decision rule is underspecified: tie handling, pattern-length range, and normalization base are not stated; the worst-case example in §4.1 only yields +2 if lengths 13–15 are excluded, so the 96.23% result is not reproducible.","rationale":"The central claim is that the pattern-similarity method achieves 96.23% accuracy in detecting ChatGPT paraphrases and attributing the source. The evidence for this claim consists entirely of the confusion matrices in Fig. 20, which are produced by the Phase II scoring rule. If that rule is not uniquely specified, then the experimental outcome is not a mathematically definite result; it depends on implementation choices the paper does not disclose. The paper's own worst-case example is a concrete demonstration: under a literal reading of the stated rule, lengths 13-15 with zero patterns would contribute -3 and change the verdict. This is not a matter of generalizability or external validity; it is an internal ambiguity in the method as described. The reader's conditional verdict already asks for implementation details; my concern is more specific: even the algorithm as described is incomplete at the tie-handling step. Fixing this ambiguity is a precondition for accepting the numerical results. I do not see evidence of intentional misrepresentation, but the reported metrics cannot be independently verified without a precise rule. Therefore the reader's CONDITIONAL verdict is appropriate; no change is needed.","tokens_in":17064,"tokens_out":12344,"duration_ms":121429,"concrete_test":"Request the exact Phase II scoring rule from the authors: weight per pattern length, maximum pattern length considered, tie-handling procedure, and the denominator used for similarity ratios. Recompute the classifications on the 2,165-article benchmark using that rule and compare with Table 3. As a robustness check, re-run with an alternative tie-handling (e.g., ties count as -1) and with the maximum length extended to 15; if the overall accuracy changes by more than a few points or the worst-case Business article flips, the reported metrics depend on an unstated choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The Phase II decision rule (Fig. 8) is the core of the claimed 96.23% accuracy, but it is not completely specified. Section 3.2.4 says: for each pattern length, if REFERENCE-SUSPICIOUS similarity is greater than REFERENCE-ORIGINAL, record +1 times a weight; otherwise record -1 times the weight; sum; positive means ChatGPT. The paper never states (i) the weight values used in the experiments, (ii) the maximum pattern length included in the sum, (iii) how ties (equal similarities, e.g., both zero when no common patterns exist) are scored, or (iv) which text length is used as the denominator for the similarity ratio (Section 3.2.3 notes the ratio 'can have two values'). The worst-case Business article in Section 4.1 exposes the ambiguity: the authors report scores of -4 for lengths 3-6 and +6 for lengths 7-12, yielding +2. But for lengths 13-15, both REFERENCE-ORIGINAL and REFERENCE-SUSPICIOUS contain no common patterns, so the similarities are equal. Under the stated rule ('otherwise is recorded as minus one times the weight'), these lengths would contribute -3, flipping the classification to 'other source'. The example only works if such lengths are excluded or ties are scored as 0, which is not specified. Without a precise rule, the confusion matrices in Fig. 20 cannot be reproduced, and the reported 96.23% accuracy is not a well-defined result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a non-deep-learning method for detecting news articles paraphrased by ChatGPT and attributing the paraphrase to ChatGPT. The method creates a ChatGPT paraphrase (Reference) of a known original article at temperature 0, then compares n-gram pattern similarity (3-15 word patterns) between Reference and the suspicious text versus Reference and the Original; a weighted vote over pattern lengths classifies the suspicious text as 'ChatGPT' if the Reference-Suspicious similarity exceeds the Reference-Original similarity. The method is evaluated on 2,224 BBC articles paraphrased by ChatGPT (temperature 1) and 2,165 Gemini paraphrases as a negative class, reporting about 96% accuracy, precision, sensitivity, specificity, and F1. The paper also compares against RADAR, which fails to detect ChatGPT paraphrases as AI.","tokens_in":17350,"tokens_out":8671,"duration_ms":78094,"significance":"If the 96% figures survive a fully specified decision rule and a more representative negative class, the paper would offer a lightweight, training-free baseline for LLM paraphrase attribution, with clear practical value for plagiarism and copyright enforcement. The strengths are the algorithmic approach (ARPaD/LERP-RSA) with explicit complexity and no GPU requirements, and the large real-news benchmark. However, the reported result is currently not reproducible as stated, and the negative class (Gemini paraphrases) is too narrow to support the claim that the method 'identifies the source of infringement as ChatGPT' against the full space of human and non-ChatGPT alternatives.","major_comments":[{"comment":"The Phase II decision rule is not fully specified. The paper does not state the pattern-length range included in the sum (Section 3.2.2 says patterns are detected from 3 to 15 words, but the Business worst-case example in Section 4.1 uses only lengths 3-12), the tie-handling rule when REFERENCE-SUSPICIOUS equals REFERENCE-ORIGINAL (e.g., both zero), the weight values assigned to each pattern length in the reported experiments, or the text-length base used for the similarity ratio, which Section 3.2.3 notes 'can have two values.' The Business worst-case example reports scores of -4 for lengths 3-6 and +6 for lengths 7-12 and concludes +2; if lengths 13-15 are included under the stated rule, each contributes -1 (because both similarities are zero), which would flip the decision to 'other source.' Since no precise rule is given, the confusion matrices in Fig. 20 and the reported 96.23% accuracy are not reproducible. Please provide complete pseudocode of the decision rule, the exact parameter values used in the experiments, and clarify the example.","section":"§3.2.4 and Fig. 8, with §4.1"},{"comment":"The negative class 'Other' is materialized exclusively with Gemini paraphrases, not with human-written same-topic articles or human paraphrases of the original. The paper explicitly notes that human paraphrases were not used because of cost. This limits the external validity of Table 3: the claimed 96.23% accuracy is a ChatGPT-versus-Gemini discrimination score, not a ChatGPT-versus-any-other-source score. Since the abstract and Fig. 8 claim classification as 'ChatGPT' versus 'other source' (which Section 4.1 glosses as 'human or any other source'), the evaluation does not support the headline claim. Please either add a human-paraphrase (or human same-topic) condition, or explicitly scope the conclusion to 'ChatGPT versus Gemini paraphrases' and adjust the abstract accordingly.","section":"§4.2 and Definition 4"},{"comment":"The attribution logic relies on the premise that a ChatGPT (temperature-1) suspicious text shares more repeated n-gram patterns with a ChatGPT (temperature-0) reference of the same original than the original shares with that reference. Because the reference is generated by the same model with the same prompt, high REFERENCE-SUSPICIOUS similarity may reflect ChatGPT-specific lexical habits, but it may also simply reflect the fact that both are paraphrases of the same source and therefore share content words, named entities, and numbers. The Gemini control (Fig. 18) shows that REFERENCE-OTHER is lower than REFERENCE-ORIGINAL, but this does not establish that the rule separates ChatGPT from human paraphrases, which can be much more lexically similar to the original. A concrete test would be to run the same Phase II rule on (a) a human paraphrase of the same original and (b) a paraphrase from another LLM (e.g., Llama) and report the resulting confusion matrices. Until such a control is provided, the claim that the method 'identifies the source of infringement' should be treated as a correctness-risk concern rather than an established result.","section":"§3.2 Definitions 2-3 and Fig. 4"}],"minor_comments":[{"comment":"Please specify the exact model version (e.g., gpt-4, gpt-4-turbo, and date) and the OpenAI API call parameters so that the dataset generation is reproducible.","section":"§3.1"},{"comment":"The 'Longest Expected Repeated Pattern Theorem' is invoked without a precise statement or a reference to a proof; please add a statement or a citation to where the theorem is proven.","section":"§3.2.2"},{"comment":"The first sentence says 'dataset described in Section 4.1'; this should be Section 3.1.","section":"§4.1"},{"comment":"The caption uses 'worse' where 'worst' is intended.","section":"Fig. 12 caption"},{"comment":"The t-tests are reported only for pattern lengths 3-7, although the method detects lengths up to 15; please explain why longer lengths are omitted from the significance analysis or include them.","section":"Table 2"},{"comment":"The two possible similarity-ratio values are acknowledged but never disambiguated; the experimental section should state which text length is used as the denominator (e.g., the first text of each pair, the shorter text, or the longer text).","section":"§3.2.3"},{"comment":"No data or code availability statement is provided; releasing the benchmark and the implementation would materially help reproduction and comparison.","section":"Data/code availability"}],"recommendation":"major_revision","confidential_remarks":"The paper is an incremental extension of the authors' prior TripAdvisor work, and the dataset is not released. The main methodological idea, n-gram pattern similarity for LLM attribution, is plausible, but the Phase II rule must be fully specified before the numbers can be trusted. The editor may wish to request the dataset and code as part of the revision, and to ask for a human-paraphrase control condition to support the attribution claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a direct extension of the same group's earlier ARPaD-based similarity work, now applied to BBC news with a Gemini negative class. The headline 96.23% is internally consistent with the confusion matrices, and the t-tests show a substantial, statistically significant gap between Reference-Suspicious and Reference-Original similarity. That part is solid. The method is also genuinely lightweight: no deep learning, just repeated-pattern detection, and it comes with a clear one-to-one comparison rationale.\n\nThe problem is the decision rule. The stress-test note is right: Phase II as specified is not reproducible. The paper says patterns are detected from 3 to 15 words, but the worked example in Section 4.1 only uses lengths 3-6 and 7-12. If lengths 13-15 are included and both similarities are zero (a tie), the rule 'otherwise is recorded as minus one times the weight' would add -3, flipping the +2 decision to -1. The paper never states that ties are scored as 0 or that lengths beyond 12 are excluded. It also never gives the weight values used in the experiments, even though it says 'pattern length specific weight.' Without that, the confusion matrices are not derivable from the text. That's a real flaw, not a nitpick.\n\nThe other soft spots are proportionate. The negative class is a Gemini paraphrase, not human text or unrelated articles; the paper is honest about this, but it means 'other source' really means 'not ChatGPT in this benchmark.' The attribution claim is also limited: since the reference is itself a ChatGPT paraphrase, the method is measuring similarity to a same-generator sample, not an independent fingerprint. That's a defensible design choice, but the abstract overstates it as 'identify that the source of infringement is the ChatGPT.' No code or data are released, which makes the underspecification worse.\n\nOverall, the core pattern is real and the method is interesting, but the paper as submitted does not define its own decision rule precisely enough for the 96.23% claim to be verified. That is fixable with a revised section and a release of the rule and parameters. I'd send it to peer review rather than desk-reject, but I'd expect major revision before acceptance.","headline":"Solid lightweight detector with a real statistical gap, but the Phase II rule is underspecified and the headline accuracy is not reproducible as written.","tokens_in":17928,"tokens_out":3024,"would_cite":false,"duration_ms":27838,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Repeated word patterns shared by two ChatGPT paraphrases can identify a suspicious news article as ChatGPT-generated, with about 96% accuracy on a BBC benchmark.","keywords":["AI paraphrase detection","text similarity","algorithmic pattern detection","ChatGPT attribution","LLM-generated text","news plagiarism","ARPaD","LERP-RSA"],"falsifier":"Take a set of non-ChatGPT paraphrases of the same BBC articles (another LLM, a human rewriter, or an aggressive paraphrase prompt) and run Phase II unchanged. If a substantial fraction of these texts yield REFERENCE-SUSPICIOUS similarity greater than REFERENCE-ORIGINAL and are therefore labeled ChatGPT, the attribution claim collapses. Conversely, if altering the paraphrase prompt for ChatGPT to force synonym substitution and clause restructuring drops REFERENCE-SUSPICIOUS similarity below REFERENCE-ORIGINAL on the benchmark, the method fails on that input distribution.","tokens_in":16804,"feed_emoji":"📰","tokens_out":6540,"duration_ms":60256,"temperature":0.7,"pith_summary":"AI-paraphrased news is a copyright problem that current detectors handle poorly. This paper proposes a non-learning, pattern-based method that claims to do two things at once: decide whether a suspicious article is a paraphrase of a known original and attribute it specifically to ChatGPT. The core idea is a triangle comparison: generate a fresh ChatGPT paraphrase (the reference) of the original, then measure repeated word-pattern similarities between reference and original and between reference and suspicious text; if the reference-suspicious similarity wins, the suspicious text is classified as a ChatGPT paraphrase. On a benchmark of 2,224 BBC articles paired with ChatGPT paraphrases plus Gemini-generated other texts, the method reports 96.23% accuracy, 96.25% precision, 96.21% sensitivity, 96.25% specificity, and 96.23% F1. If the result holds, newsrooms and platforms gain a cheap, explainable, one-to-one screening tool for this specific infringement pattern.","feed_headline":"Word-pattern similarity catches ChatGPT paraphrases at 96 percent","feed_subtitle":"A no-deep-learning method compares a suspicious article with a fresh ChatGPT paraphrase and names the source.","key_machinery":"The load-bearing machinery is the combination of the LERP-RSA (Longest Expected Repeated Pattern Reduced Suffix Array) data structure and the ARPaD (All Repeated Patterns Detection) algorithm, which find every word n-gram of length 3 to 15 shared between any pair of texts. ARPaD runs on a suffix array built from the cleaned texts and returns the repeated patterns; the paper then converts pattern counts into per-length similarity ratios for each text pair. Phase I uses an absolute threshold on ORIGINAL-SUSPICIOUS similarity to catch near-copy cases, and Phase II uses the ratio-symmetric comparison between REFERENCE-SUSPICIOUS and REFERENCE-ORIGINAL, weighted by pattern length, as the decision rule. The whole scheme is deliberately one-to-one: the suspicious text is assumed to have been mapped to a single original article beforehand.","core_discovery":"The paper claims that two ChatGPT paraphrases of the same original article exhibit substantially higher repeated-word-pattern similarity with each other than either exhibits with the original, and this asymmetry is consistent enough to drive classification of a single suspicious text. The method's second phase computes, for each pattern length from 3 to 15 words, whether the REFERENCE-SUSPICIOUS similarity exceeds the REFERENCE-ORIGINAL similarity and sums weighted votes; a positive total classifies the suspicious text as a ChatGPT paraphrase. On the full benchmark the weighted vote reaches about 96% on all standard metrics across all five news categories, and the comparison with Gemini-generated texts shows the effect is specific enough to name ChatGPT as the source rather than merely flagging AI text.","pith_inferences":["Going beyond the paper: the same triangle comparison could be re-run with references from other LLMs, turning the method into a general attribution test; the paper only validates ChatGPT against Gemini as the negative class.","Going beyond the paper: because detection rests on lexical n-gram overlap, a paraphrase prompt that forces heavy synonym substitution or clause reordering would likely shrink the REFERENCE-SUSPICIOUS gap; this is a testable stress experiment.","Going beyond the paper: the one-to-one mapping assumption means a real deployment needs a retrieval step to find the original; the paper's benchmark supplies the original, so end-to-end performance including retrieval remains open."],"forward_implications":["A news agency can screen any suspect article by generating one ChatGPT paraphrase of the alleged original and comparing pattern similarities, with no training data or GPU required.","The method attributes the paraphrase to ChatGPT rather than merely labeling it AI: Gemini-generated texts show lower similarity to the reference than ChatGPT-generated ones do.","The weighted pattern-length voting makes long shared word sequences decisive, so even when short patterns favor the wrong pair and article lengths differ greatly, the overall score can still be correct.","Per-case comparisons keep computational cost low; the paper reports worst-case time and space complexity O(m n log n), with m effectively fixed at two or three texts.","Existing AI-text detectors are not a substitute: the paper finds its open-source comparison tool classifies almost all ChatGPT paraphrases as human."],"supporting_citations":[{"why":"Provides the BBC news article benchmark dataset used for all experiments.","marker":"(Greene & Cunningham, 2006)"},{"why":"Establishes the consistent similarity pattern between original text and ChatGPT paraphrases that motivates the detection rule.","marker":"(K. F. Xylogiannopoulos et al., 2024a)"},{"why":"Introduces the paraphrase-detection scheme on TripAdvisor reviews that this paper extends to news articles.","marker":"(K. F. Xylogiannopoulos et al., 2024b)"},{"why":"Supplies the ARPaD algorithm and the theorem bounding longest expected repeated patterns, which sets the 3-15 word pattern range.","marker":"(K. Xylogiannopoulos, 2017)"},{"why":"Provides the open-source AI-text detector used as comparison baseline, which fails to flag the ChatGPT paraphrases.","marker":"(Hu et al., 2023)"},{"why":"Demonstrates that paraphrasing evades AI detectors while retrieval helps, framing the problem this method targets.","marker":"(Krishna et al., 2023)"}],"fun_headline_variants":["Word-pattern similarity outs ChatGPT paraphrases at 96%","Pattern similarity fingerprints ChatGPT rewrites","No deep learning needed: pattern similarity catches ChatGPT","ChatGPT paraphrase? Word patterns give it away, 96% of the time","96% accurate: similarity test exposes ChatGPT paraphrases"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method stands or falls on the premise that a ChatGPT paraphrase of an article shares more repeated word patterns with a second ChatGPT paraphrase of the same article than it does with the original article, and that this gap is large and stable enough to classify each individual suspicious text.","fun_headline_variants_meta":{"raw":{"variants":["Word-pattern similarity outs ChatGPT paraphrases at 96%","Pattern similarity fingerprints ChatGPT rewrites","No deep learning needed: pattern similarity catches ChatGPT","ChatGPT paraphrase? Word patterns give it away, 96% of the time","96% accurate: similarity test exposes ChatGPT paraphrases"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000453,"raw_usage":{"total_tokens":2251,"prompt_tokens":892,"completion_tokens":1359,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":1280}},"tokens_in":508,"tokens_out":1359,"duration_ms":10037,"temperature":1.0,"reasoning_tokens":1280,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:34:59.337038+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of non-ChatGPT paraphrases of the same BBC articles (another LLM, a human rewriter, or an aggressive paraphrase prompt) and run Phase II unchanged. If a substantial fraction of these texts yield REFERENCE-SUSPICIOUS similarity greater than REFERENCE-ORIGINAL and are therefore labeled ChatGPT, the attribution claim collapses. Conversely, if altering the paraphrase prompt for ChatGPT to force synonym substitution and clause restructuring drops REFERENCE-SUSPICIOUS similarity below REFERENCE-ORIGINAL on the benchmark, the method fails on that input distribution.","supporting_citations":[],"review_version":1}