Pith. sign in

REVIEW 3 major objections 7 minor 8 references

The power of text similarity in identifying AI-LLM paraphrased documents: The case of BBC news articles and ChatGPT

T0 review · 3 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Repeated word patterns shared by two ChatGPT paraphrases can identify a suspicious news article as ChatGPT-generated, with about 96% accuracy on a BBC benchmark.

desk verdict Solid lightweight detector with a real statistical gap, but the Phase II rule is underspecified and the headline accuracy is not reproducible as written. read the letter →

arxiv 2505.12405 v1 pith:R2OMICJN submitted 2025-05-18 cs.CL cs.AI

Konstantinos Xylogiannopoulos , Petros Xanthopoulos , Panagiotis Karampelas , Georgios Bakamitsos This is my paper · ORCID
classification cs.CLcs.AI
keywords AIparaphrasedetectiontextsimilarityalgorithmicpatternChatGPTattributionLLM-generatednewsplagiarismARPaDLERP-RSA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AI-paraphrased news is a copyright problem that current detectors handle poorly. This paper proposes a non-learning, pattern-based method that claims to do two things at once: decide whether a suspicious article is a paraphrase of a known original and attribute it specifically to ChatGPT. The core idea is a triangle comparison: generate a fresh ChatGPT paraphrase (the reference) of the original, then measure repeated word-pattern similarities between reference and original and between reference and suspicious text; if the reference-suspicious similarity wins, the suspicious text is classified as a ChatGPT paraphrase. On a benchmark of 2,224 BBC articles paired with ChatGPT paraphrases plus Gemini-generated other texts, the method reports 96.23% accuracy, 96.25% precision, 96.21% sensitivity, 96.25% specificity, and 96.23% F1. If the result holds, newsrooms and platforms gain a cheap, explainable, one-to-one screening tool for this specific infringement pattern.

What carries the argument

The load-bearing machinery is the combination of the LERP-RSA (Longest Expected Repeated Pattern Reduced Suffix Array) data structure and the ARPaD (All Repeated Patterns Detection) algorithm, which find every word n-gram of length 3 to 15 shared between any pair of texts. ARPaD runs on a suffix array built from the cleaned texts and returns the repeated patterns; the paper then converts pattern counts into per-length similarity ratios for each text pair. Phase I uses an absolute threshold on ORIGINAL-SUSPICIOUS similarity to catch near-copy cases, and Phase II uses the ratio-symmetric comparison between REFERENCE-SUSPICIOUS and REFERENCE-ORIGINAL, weighted by pattern length, as the decision rule. The whole scheme is deliberately one-to-one: the suspicious text is assumed to have been mapped to a single original article beforehand.

What would settle it

Take a set of non-ChatGPT paraphrases of the same BBC articles (another LLM, a human rewriter, or an aggressive paraphrase prompt) and run Phase II unchanged. If a substantial fraction of these texts yield REFERENCE-SUSPICIOUS similarity greater than REFERENCE-ORIGINAL and are therefore labeled ChatGPT, the attribution claim collapses. Conversely, if altering the paraphrase prompt for ChatGPT to force synonym substitution and clause restructuring drops REFERENCE-SUSPICIOUS similarity below REFERENCE-ORIGINAL on the benchmark, the method fails on that input distribution.

Watch

Extended reading notes

Core claim

The paper claims that two ChatGPT paraphrases of the same original article exhibit substantially higher repeated-word-pattern similarity with each other than either exhibits with the original, and this asymmetry is consistent enough to drive classification of a single suspicious text. The method's second phase computes, for each pattern length from 3 to 15 words, whether the REFERENCE-SUSPICIOUS similarity exceeds the REFERENCE-ORIGINAL similarity and sums weighted votes; a positive total classifies the suspicious text as a ChatGPT paraphrase. On the full benchmark the weighted vote reaches about 96% on all standard metrics across all five news categories, and the comparison with Gemini-generated texts shows the effect is specific enough to name ChatGPT as the source rather than merely flagging AI text.

Load-bearing premise

The method stands or falls on the premise that a ChatGPT paraphrase of an article shares more repeated word patterns with a second ChatGPT paraphrase of the same article than it does with the original article, and that this gap is large and stable enough to classify each individual suspicious text.

Editorial extensions

If this is right

  • A news agency can screen any suspect article by generating one ChatGPT paraphrase of the alleged original and comparing pattern similarities, with no training data or GPU required.
  • The method attributes the paraphrase to ChatGPT rather than merely labeling it AI: Gemini-generated texts show lower similarity to the reference than ChatGPT-generated ones do.
  • The weighted pattern-length voting makes long shared word sequences decisive, so even when short patterns favor the wrong pair and article lengths differ greatly, the overall score can still be correct.
  • Per-case comparisons keep computational cost low; the paper reports worst-case time and space complexity O(m n log n), with m effectively fixed at two or three texts.
  • Existing AI-text detectors are not a substitute: the paper finds its open-source comparison tool classifies almost all ChatGPT paraphrases as human.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Going beyond the paper: the same triangle comparison could be re-run with references from other LLMs, turning the method into a general attribution test; the paper only validates ChatGPT against Gemini as the negative class.
  • Going beyond the paper: because detection rests on lexical n-gram overlap, a paraphrase prompt that forces heavy synonym substitution or clause reordering would likely shrink the REFERENCE-SUSPICIOUS gap; this is a testable stress experiment.
  • Going beyond the paper: the one-to-one mapping assumption means a real deployment needs a retrieval step to find the original; the paper's benchmark supplies the original, so end-to-end performance including retrieval remains open.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The manuscript proposes a non-deep-learning method for detecting news articles paraphrased by ChatGPT and attributing the paraphrase to ChatGPT. The method creates a ChatGPT paraphrase (Reference) of a known original article at temperature 0, then compares n-gram pattern similarity (3-15 word patterns) between Reference and the suspicious text versus Reference and the Original; a weighted vote over pattern lengths classifies the suspicious text as 'ChatGPT' if the Reference-Suspicious similarity exceeds the Reference-Original similarity. The method is evaluated on 2,224 BBC articles paraphrased by ChatGPT (temperature 1) and 2,165 Gemini paraphrases as a negative class, reporting about 96% accuracy, precision, sensitivity, specificity, and F1. The paper also compares against RADAR, which fails to detect ChatGPT paraphrases as AI.

Significance. If the 96% figures survive a fully specified decision rule and a more representative negative class, the paper would offer a lightweight, training-free baseline for LLM paraphrase attribution, with clear practical value for plagiarism and copyright enforcement. The strengths are the algorithmic approach (ARPaD/LERP-RSA) with explicit complexity and no GPU requirements, and the large real-news benchmark. However, the reported result is currently not reproducible as stated, and the negative class (Gemini paraphrases) is too narrow to support the claim that the method 'identifies the source of infringement as ChatGPT' against the full space of human and non-ChatGPT alternatives.

major comments (3)
  1. [§3.2.4 and Fig. 8, with §4.1] The Phase II decision rule is not fully specified. The paper does not state the pattern-length range included in the sum (Section 3.2.2 says patterns are detected from 3 to 15 words, but the Business worst-case example in Section 4.1 uses only lengths 3-12), the tie-handling rule when REFERENCE-SUSPICIOUS equals REFERENCE-ORIGINAL (e.g., both zero), the weight values assigned to each pattern length in the reported experiments, or the text-length base used for the similarity ratio, which Section 3.2.3 notes 'can have two values.' The Business worst-case example reports scores of -4 for lengths 3-6 and +6 for lengths 7-12 and concludes +2; if lengths 13-15 are included under the stated rule, each contributes -1 (because both similarities are zero), which would flip the decision to 'other source.' Since no precise rule is given, the confusion matrices in Fig. 20 and the reported 96.23% accuracy are not reproducible. Please provide complete pseudocode of the decision rule, the exact parameter values used in the experiments, and clarify the example.
  2. [§4.2 and Definition 4] The negative class 'Other' is materialized exclusively with Gemini paraphrases, not with human-written same-topic articles or human paraphrases of the original. The paper explicitly notes that human paraphrases were not used because of cost. This limits the external validity of Table 3: the claimed 96.23% accuracy is a ChatGPT-versus-Gemini discrimination score, not a ChatGPT-versus-any-other-source score. Since the abstract and Fig. 8 claim classification as 'ChatGPT' versus 'other source' (which Section 4.1 glosses as 'human or any other source'), the evaluation does not support the headline claim. Please either add a human-paraphrase (or human same-topic) condition, or explicitly scope the conclusion to 'ChatGPT versus Gemini paraphrases' and adjust the abstract accordingly.
  3. [§3.2 Definitions 2-3 and Fig. 4] The attribution logic relies on the premise that a ChatGPT (temperature-1) suspicious text shares more repeated n-gram patterns with a ChatGPT (temperature-0) reference of the same original than the original shares with that reference. Because the reference is generated by the same model with the same prompt, high REFERENCE-SUSPICIOUS similarity may reflect ChatGPT-specific lexical habits, but it may also simply reflect the fact that both are paraphrases of the same source and therefore share content words, named entities, and numbers. The Gemini control (Fig. 18) shows that REFERENCE-OTHER is lower than REFERENCE-ORIGINAL, but this does not establish that the rule separates ChatGPT from human paraphrases, which can be much more lexically similar to the original. A concrete test would be to run the same Phase II rule on (a) a human paraphrase of the same original and (b) a paraphrase from another LLM (e.g., Llama) and report the resulting confusion matrices. Until such a control is provided, the claim that the method 'identifies the source of infringement' should be treated as a correctness-risk concern rather than an established result.
minor comments (7)
  1. [§3.1] Please specify the exact model version (e.g., gpt-4, gpt-4-turbo, and date) and the OpenAI API call parameters so that the dataset generation is reproducible.
  2. [§3.2.2] The 'Longest Expected Repeated Pattern Theorem' is invoked without a precise statement or a reference to a proof; please add a statement or a citation to where the theorem is proven.
  3. [§4.1] The first sentence says 'dataset described in Section 4.1'; this should be Section 3.1.
  4. [Fig. 12 caption] The caption uses 'worse' where 'worst' is intended.
  5. [Table 2] The t-tests are reported only for pattern lengths 3-7, although the method detects lengths up to 15; please explain why longer lengths are omitted from the significance analysis or include them.
  6. [§3.2.3] The two possible similarity-ratio values are acknowledged but never disambiguated; the experimental section should state which text length is used as the denominator (e.g., the first text of each pair, the shorter text, or the longer text).
  7. [Data/code availability] No data or code availability statement is provided; releasing the benchmark and the implementation would materially help reproduction and comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the detection rule is the method under test and is validated against externally labeled ChatGPT vs Gemini texts.

full rationale

No circular step reduces the claimed result to its own inputs. The methodology is an empirical detection rule: generate a ChatGPT Reference for an Original, compute repeated-pattern similarities REFERENCE-SUSPICIOUS and REFERENCE-ORIGINAL, and classify the Suspicious text as ChatGPT when the former exceeds the latter (Section 3.2.4, Fig. 8). This rule is the method being tested, not a restatement of the benchmark labels; the positives and negatives in the evaluation are labeled by the actual generator (ChatGPT for Suspicious, Gemini for Other), independently of the similarity computation, so the reported 96.23% accuracy could have failed and is not forced by construction. The cornerstone observation that ChatGPT paraphrases are more similar to each other than to the original is attributed to the authors' previous work, but it is independently re-validated in this paper with bootstrap t-tests and descriptive statistics in Section 4.1 and Table 2, so the self-citations are not load-bearing. Using a same-generator ChatGPT reference as the fingerprint is an operational detector design, not a self-definitional equivalence: the classification is an empirical hypothesis about the similarity gap, not a definition of 'ChatGPT' in terms of the decision rule. The underspecified details of weights, tie handling, and normalization are reproducibility and correctness risks, but they do not constitute a circular reduction of the reported result to its inputs.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

The method has no fitted scalar weights only if uniform weights are assumed, which the paper never states. The central signal is an assumption about n-gram overlap between two ChatGPT outputs, and the strongest external grounding is the t-test separation on this corpus. All other burdens sit on the authors' prior ARPaD/LERP-RSA machinery and on the Gemini stand-in for 'not ChatGPT'.

free parameters (4)
  • minimum pattern length = 3 words
    Chosen to avoid trivial one- and two-word patterns; affects all similarity ratios and classification scores (Section 3.2.2).
  • maximum pattern length = 15 words
    Set from the authors' Longest Expected Repeated Pattern theorem rather than from data; if longer meaningful patterns exist, the similarity sums change (Section 3.2.2).
  • generation temperatures = Suspicious=1, Reference=0
    These values define the test conditions; the signal may depend on the random versus deterministic setting of the paraphrasing model (Section 3.1, Definitions 2 and 3).
  • pattern-length weights = not disclosed (examples imply 1)
    Phase II sums signed comparisons multiplied by 'pattern length specific weights' but no weight values are given; the overall score and final label depend on these (Sections 3.2.4 and 4.1).
assumptions (5)
  • domain assumption ARPaD and LERP-RSA correctly enumerate all repeated word patterns of length 3 to 15 in the paired texts.
    Invoked in Section 3.2.2; the correctness and efficiency of the detector are inherited from prior self-cited work rather than demonstrated here.
  • ad hoc to paper The Longest Expected Repeated Pattern theorem bounds meaningful repeated patterns at 15 words in these articles.
    Section 3.2.2 uses this theorem to stop pattern search at length 15; the theorem is from the first author's thesis and is not restated or proved.
  • domain assumption A temperature-0 ChatGPT reference and a temperature-1 ChatGPT suspicious text of the same original share more n-gram overlap than the reference shares with the original.
    Core premise of Phase II (Definitions 2 and 3, Fig. 8); without it the classifier has no signal.
  • domain assumption A suspicious article can be identified with a single known original article, enabling one-to-one comparison.
    Section 3.2 states the suspicious text must be tested against a single already identified article; real-world copycat text may be composite or unindexed.
  • ad hoc to paper Gemini paraphrases are a valid stand-in for all non-ChatGPT paraphrases in the negative class.
    Section 4.2 uses Gemini because human paraphrases and same-topic articles from another agency were unavailable; this narrows the meaning of 'Other' in the evaluation.
invented entities (1)
  • Reference text (temperature-0 ChatGPT paraphrase)
    purpose: Serves as the comparison yardstick; classification is positive when the suspicious text is more similar to the reference than the original is.
    The discriminative value of the reference is the empirical premise of the method; no external benchmark validates that the reference is a stable fingerprint of ChatGPT across prompts, versions, or domains.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The power of text similarity in identifying AI-LLM paraphrased documents: The case of BBC news articles and ChatGPT." pith.science (2026). https://pith.science/paper/R2OMICJN

@misc{pith2026250512405,
  author       = {Pith},
  title        = {Pith review of: The power of text similarity in identifying AI-LLM paraphrased documents: The case of BBC news articles and ChatGPT},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R2OMICJN}},
  note         = {Machine review of arXiv:2505.12405}
}
read the original abstract

Generative AI paraphrased text can be used for copyright infringement and the AI paraphrased content can deprive substantial revenue from original content creators. Despite this recent surge of malicious use of generative AI, there are few academic publications that research this threat. In this article, we demonstrate the ability of pattern-based similarity detection for AI paraphrased news recognition. We propose an algorithmic scheme, which is not limited to detect whether an article is an AI paraphrase, but, more importantly, to identify that the source of infringement is the ChatGPT. The proposed method is tested with a benchmark dataset specifically created for this task that incorporates real articles from BBC, incorporating a total of 2,224 articles across five different news categories, as well as 2,224 paraphrased articles created with ChatGPT. Results show that our pattern similarity-based method, that makes no use of deep learning, can detect ChatGPT assisted paraphrased articles at percentages 96.23% for accuracy, 96.25% for precision, 96.21% for sensitivity, 96.25% for specificity and 96.23% for F1 score.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

8 extracted references · 2 canonical work pages

  1. [1]

    Agarwal, B., Ramampiaro, H., Langseth, H., & Ruocco, M. (2018). A deep network model for paraphrase detection in short text messages. Information Processing & Management, 54(6), 922–937. https://doi.org/10.1016/j.ipm.2018.06.005 Alian, M., Awajan, A., Al-Hasan, A., & Akuzhia, R. (2021). Building Arabic Paraphrasing Benchmark based on Transformation Rules....

    work page Pith review arXiv doi:10.48550/arxiv.2303.13989 2018

  2. [5]

    https://doi.org/10.3390/info11050241 Vrbanec, T., & Meštrović, A. (2023). Comparison study of unsupervised paraphrase detection: Deep learning—The key for semantic similarity detection. Expert Systems, 40(9), e13386. https://doi.org/10.1111/exsy.13386 Wahle, J. P., Gipp, B., & Ruas, T. (2023). Paraphrase Types for Generation and Detection. Proceedings of ...

    arXiv 2023

  3. [6]

    https://proceedings.neurips.cc/paper_files/paper/2023/hash/575c450013d0e99e4b0ecf82b d1afaa4-Abstract-Conference.html Le, L., & Tran, D. (2025). A Metric-Based Detection System for Large Language Model Texts. ACM Trans. Manage. Inf. Syst., 16(1), 8:1-8:19. https://doi.org/10.1145/3704739 LegalZoom. (2024). Document Summarizer | LegalZoom. https://www.lega...

    arXiv doi:10.48550/arxiv.2405.12689 2025

  4. [32]

    https://proceedings.neurips.cc/paper_files/paper/2019/hash/5e2b66750529d8ae895ad259 1118466f-Abstract.html Ganitkevitch, J., & Callison-Burch, C. (2014). The Multilingual Paraphrase Database. In N. Calzolari, K. Choukri, T. Declerck, H. Loftsson, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, & S. Piperidis (Eds.), Proceedings of the Ninth International Co...

    arXiv 2014

  5. [36]

    https://proceedings.neurips.cc/paper_files/paper/2023/hash/30e15e5941ae0cdab7ef58cc8 d59a4ca-Abstract-Conference.html Huang, F., Kwak, H., & An, J. (2024). Token-Ensemble Text Generation: On Attacking the Automatic AI-Generated Text Detection (arXiv:2402.11167). arXiv. https://doi.org/10.48550/arXiv.2402.11167 Knibbs, K. (2024). Scammy AI-Generated Book R...

    arXiv doi:10.48550/arxiv.2402.11167 2024

  6. [472]

    https://doi.org/10.1162/COLI_a_00166 Brewster, J., Wang, M., & Palmer, C. (2023). How Copycat Sites Use AI to Plagiarize News Articles. https://www.newsweek.com/how-copycat-sites-use-ai-plagiarize-news-articles- 1835212 Chakraborty, M., Tonmoy, S. M. T. I., Zaman, S. M. M., Sharma, K., Barman, N. R., Gupta, C., Gautam, S., Kumar, T., Jain, V ., Chadha, A....

    arXiv doi:10.48550/arxiv.2310.05030 2023

  7. [2005]

    https://aclanthology.org/I05-5002 Dong, Q., Wan, X., & Cao, Y . (2021). ParaSCI: A Large Scientific Paraphrase Dataset for Longer Paraphrase Generation. In P. Merlo, J. Tiedemann, & R. Tsarfaty (Eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (pp. 424–434). Association for Com...

    doi:10.18653/v1/2021.eacl-main.33 2021

  8. [2015]

    (pp. 1–11). Association for Computational Linguistics. https://doi.org/10.18653/v1/S15- 2001 Xylogiannopoulos, K. (2017). Data Structures, Algorithms and Applications for Big Data Analytics: Single, Multiple and All Repeated Patterns Detection in Discrete Sequences. [University of Calgary]. Library and Archives Canada. 10.11575/prism/25522 Xylogiannopoulo...

    work page arXiv 2017

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.