REVIEW 3 major objections 3 minor 15 references
Your Language Model Can Secretly Write Like Humans: Contrastive Paraphrase Attacks on LLM-Generated Text Detectors
T0 review · 3 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read CoPA, a training-free contrastive paraphrase attack, reduces the detection rate of eight AI-text detectors to near-baseline levels while preserving semantics.
desk verdict CoPA is a genuinely effective training-free paraphrase attack with a serious theory-algorithm gap: the theorem analyzes an arithmetic mixture while the code does logit-space contrast, but the empirical results stand on their own. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the contrastive decoding distribution in Eq. (3), formed by subtracting the logits of a machine-like distribution from an amplified human-like distribution: $p_c \propto \exp((1+\lambda) f_\theta(\cdot|x_h,y_m,y_{<t}) - \lambda f_\theta(\cdot|x_m,y_m,y_{<t}))$. This is a training-free, self-corrective decoding mechanism: the machine-like prompt (in practice, asking the LLM to repeat the original text) provides a concentrated 'machine-style' token distribution that is subtracted to remove machine-inherent word preferences. An adaptive truncation constrains sampling to tokens with high probability under the human-like distribution, preserving plausibility. The theoretical support is a convexity/KL-divergence argument: $g(\lambda) = \mathrm{KL}(p_h \| (1+\lambda) p'_h - \lambda p_m)$ is convex, and if $g'(0)<0$ then a positive $\lambda$ strictly reduces the KL divergence to the human distribution.
What would settle it
Compute, for a fixed sample of human texts and a fixed paraphraser, the empirical KL divergence $\mathrm{KL}(p_h \| (1+\lambda) p'_h - \lambda p_m)$ for a grid of $\lambda \in [0,1]$ by estimating the distributions from token logits. If for the paper's default $\lambda=0.5$ the contrastive distribution is not closer to the human distribution than $p'_h$ on a majority of samples, or if $g'(0) \ge 0$ for many samples, the theorem's premise fails and the claimed bound does not hold for those samples. A second test is to train a detector on CoPA-paraphrased texts (beyond the paper's RoBERTa-large adaption) and see whether the attack's TPR remains near baseline.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that the machine-like statistical bias of an LLM can be turned against its own detectability. CoPA elicits two token distributions from the same off-the-shelf model, $p'_h$ from a human-style prompt and $p_m$ from a machine-style prompt, then samples from the contrastive distribution $p_c \propto \exp((1+\lambda) f_h - \lambda f_m)$. This penalizes tokens the LLM systematically prefers in its default mode while keeping tokens plausible under the human-style prompt, via an adaptive truncation. In experiments on XSum, SQuAD, and LongQA against eight detectors, CoPA drives the detection true positive rate at 5% FPR down to an average of about 12.63%, 11.06%, and 7.44% for GPT-3.5-turbo texts, versus 66.05%, 56.27%, and 57.09% without attack, with semantic similarity above 90%.
Load-bearing premise
The whole argument rests on the assumption that subtracting the machine-like token distribution in log space actually moves the rewritten text closer to how humans write, an assumption the paper justifies with a heuristic diagram rather than a direct measurement.
Editorial extensions
If this is right
- CoPA works across eight detector types, including zero-shot likelihood-based, curvature-based, and trained classifiers, with TPR near baseline at 5% FPR.
- It transfers across source LLMs (GPT-3.5-turbo, GPT-4, Claude-3.5, GPT-4o, Gemini-1.5 Pro) and across paraphrasers (Qwen2.5-72B, R1-Distill-32B, QwQ-32B, GLM-4-9B-hf), with QwQ-32B performing best.
- A single paraphrase suffices; multiple paraphrases give limited gains, and semantic similarity stays above 90%.
- An adaptive defense that adversarially trains RoBERTa-large with CoPA-paraphrased texts raises detection from 22.67% to 78.00% TPR at 5% FPR, indicating the attack can be mitigated by training on such texts.
- CoPA outperforms a surrogate-based RedTeaming attack even without using a surrogate detector, with 17.00% versus 30.00% TPR on Fast-DetectGPT on XSum.
Reading between the lines
- A broader principle suggested by this work is that detectors relying on LLM-inherent statistical biases can be fooled by making the generator sample away from its own mode, a strategy that may generalize to other modalities if a contrastive 'machine-style' distribution can be elicited.
- Because CoPA is training-free and uses off-the-shelf models, it lowers the bar for adversarial users: anyone with API access to a capable LLM can launch the attack, so detector robustness should be studied against adaptive attackers rather than static baselines.
- The theory's premise $g'(0)<0$ deserves direct measurement; the paper offers a heuristic diagram but no empirical check. A simple test is to compute KL divergences on real data for increasing $\lambda$ and verify that Eq. (6) holds as predicted.
- The attack's success depends on the choice of machine prompt; the paper uses 'repeat the paragraph' because it best concentrates machine-like features, but whether this holds for all domains and LLMs is an open question.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CoPA, a training-free contrastive paraphrase attack against LLM-generated text detectors. At each decoding step, CoPA combines a human-style prompt distribution with a machine-style prompt distribution via the logit-space contrast p_c ∝ exp((1+λ)f_h − λ f_m), after adaptive truncation. The authors report that CoPA reduces TPR at 5% FPR to near-baseline levels across eight detectors, three datasets, several source LLMs, and several paraphraser LLMs, while preserving high semantic similarity. A theoretical section (Sec. 3.3) presents a theorem intended to show that the contrastive distribution is closer to human text than the human-like prompt distribution alone. The empirical results are extensive, and the code is released. The main weakness is that the theorem analyzes an arithmetic mixture of distributions, whereas the implemented algorithm uses a logit-space contrast, so the stated theoretical guarantee does not cover the method as implemented.
Significance. If the empirical findings hold, CoPA is a practical and inexpensive attack that substantially weakens current LLM-text detectors, with implications for detector robustness and for the deployment of detection systems. The paper's strengths are its broad evaluation (three datasets, eight detectors, multiple source and paraphraser models), its release of code, and its inclusion of human and automated text-quality evaluations. The adaptive-defense experiment and the ethical discussion are also valuable. The theoretical claim, however, is currently not established for the actual algorithm, and the key premise of the theorem is not directly validated. These issues affect the paper's central advertised contribution and require substantial revision before publication.
major comments (3)
- [Sec. 3.2 Eq. (3) and Sec. 3.3 Eq. (5)] Theorem 1 analyzes the arithmetic mixture q_λ = (1+λ)p'_h − λ p_m, but the implemented CoPA samples from p_c ∝ exp((1+λ)f_h − λ f_m), which after normalization equals p_c ∝ (p'_h)^(1+λ) / p_m^λ. These are different distributions, so the theorem does not cover the algorithm evaluated in Section 4. I recommend deriving an analogous theorem for the logit-space geometric contrast, or explicitly repositioning Theorem 1 as a heuristic motivation and removing claims that the theory 'guarantees' or 'validates' the effectiveness of the implemented method.
- [Sec. 3.3, Fig. 4] The premise g′(0)<0 is never measured or verified. The only support is the |V|=3 contour diagram in Figure 4 and the assertion that p'_h is 'usually' between p_m and p_h. Since this condition is the basis of Theorem 1, the theoretical result is conditional on an unvalidated assumption. Please report direct estimates of g′(0) for the actual token distributions used in the experiments, or state clearly that this is an assumption rather than an established fact whose verification is left to the reader.
- [Sec. 4.2, Appendix B, Fig. 3] The machine prompt x_m was selected using detector likelihood as a proxy, and the contrast intensity λ was tuned on the first 50 XSum samples (Figure 6). This is detector-informed tuning, and it should be disclosed in the main text because the paper positions CoPA as training-free and surrogate-free. To support that positioning, the main results should either be confirmed with a fixed machine prompt and hyperparameters chosen without detector feedback, or the paper should acknowledge that the reported performance depends on detector-informed prompt selection.
minor comments (3)
- [Tables 1 and 2] Several cells in the Dipper rows are merged or missing delimiters (for example, Table 1 XSum shows '15.672.6727.33' and Table 2 for Claude-3.5 shows '18.170.3320.67'). Please fix the table formatting so that each metric is a separate entry.
- [Appendix A, Proposition 3] The proof of uniqueness of the minimum uses analyticity of g on the closed interval I, but g need not be analytic at points where the mixture assigns zero probability to a token with positive human probability, because the KL divergence becomes infinite there. Please clarify the domain of g and handle infinite KL values explicitly.
- [Sec. 3.3 and Sec. 4.4] The text claims that the trend of TPR over λ in Figure 6 'roughly aligns with our theoretical analysis,' but the theorem applies to the arithmetic mixture, not to the implemented logit-space contrast. The ablation result would be more appropriately presented as a heuristic observation unless the theory is re-derived for the actual sampling distribution.
Circularity Check
No significant circularity: the empirical attack is measured against detectors; the theory/algorithm gap is a rigor issue, not a circular reduction.
full rationale
The central claim that CoPA lowers detector TPR is supported by direct measurements against eight detectors (Tables 1, 2, 4, 5), with hyperparameters lambda=0.5 and alpha=1e-5 set by ablations rather than fitted to the reported final TPRs. The machine prompt is selected by maximizing the detector's LLM likelihood (Appendix B, Fig. 3), but this is a design choice about the contrastive reference, not a fitted parameter that is then renamed a prediction; the reported fooling rates remain independent measurements. The theoretical argument (Theorem 1, Eq. 5-6) analyzes KL(p_h || (1+lambda)p'_h - lambda p_m), whereas the implemented CoPA (Eq. 3) performs logit-space contrast, which after normalization is proportional to (p'_h)^(1+lambda)/(p_m)^lambda, a product-of-experts contrast not covered by the arithmetic-mixture theorem. The premise g'(0)<0 is also asserted to 'generally hold' without direct estimation. These are non-circular rigor gaps: the theorem does not reduce to the algorithm's inputs, and the empirical results stand independently of the theorem. The paper contains minor self-citations, e.g., Fang et al. (2025) in Sec. 3.2 for the token-penalty concern, but these are not load-bearing for the attack's success. Overall, no significant circularity; score 2 reflects only minor non-load-bearing self-citation and a mild in-sample lambda-selection concern.
Assumptions & free parameters
free parameters (2)
- λ (contrast intensity) =
0.5
- α (clipping factor) =
1e-5
assumptions (4)
- domain assumption The premise g'(0)<0 in Theorem 1 holds for the distributions in practice.
- domain assumption The prompts xh and xm faithfully elicit human-like and machine-like token distributions p'h and pm.
- domain assumption Detector likelihood (e.g., Fast-DetectGPT) is a valid proxy for machine-like style.
- standard math KL divergence is convex and the distance function g(λ) is analytic on its domain.
Cite this review
Pith. "Pith review of Your Language Model Can Secretly Write Like Humans: Contrastive Paraphrase Attacks on LLM-Generated Text Detectors." pith.science (2026). https://pith.science/paper/YYOKDKD3
@misc{pith2026250515337,
author = {Pith},
title = {Pith review of: Your Language Model Can Secretly Write Like Humans: Contrastive Paraphrase Attacks on LLM-Generated Text Detectors},
year = {2026},
howpublished = {\url{https://pith.science/paper/YYOKDKD3}},
note = {Machine review of arXiv:2505.15337}
}
read the original abstract
The misuse of large language models (LLMs), such as academic plagiarism, has driven the development of detectors to identify LLM-generated texts. To bypass these detectors, paraphrase attacks have emerged to purposely rewrite these texts to evade detection. Despite the success, existing methods require substantial data and computational budgets to train a specialized paraphraser, and their attack efficacy greatly reduces when faced with advanced detection algorithms. To address this, we propose \textbf{Co}ntrastive \textbf{P}araphrase \textbf{A}ttack (CoPA), a training-free method that effectively deceives text detectors using off-the-shelf LLMs. The first step is to carefully craft instructions that encourage LLMs to produce more human-like texts. Nonetheless, we observe that the inherent statistical biases of LLMs can still result in some generated texts carrying certain machine-like attributes that can be captured by detectors. To overcome this, CoPA constructs an auxiliary machine-like word distribution as a contrast to the human-like distribution generated by the LLM. By subtracting the machine-like patterns from the human-like distribution during the decoding process, CoPA is able to produce sentences that are less discernible by text detectors. Our theoretical analysis suggests the superiority of the proposed attack. Extensive experiments validate the effectiveness of CoPA in fooling text detectors across various scenarios.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Often difficult or impossible to understand
Very Poor: Severely broken or unnatural English with major grammar issues. Often difficult or impossible to understand. Prompt for GPT-4 evaluation: (1) Task: Evaluate the natural fluency of a given sentence. Use a 5-point scale (5 = highest). Natural Fluency (1-5):
-
[2]
Requires effort to understand in places
Poor: Frequent unnatural phrasing and grammatical errors make reading diffi- cult. Requires effort to understand in places
-
[3]
Some grammatical errors or awkward constructions occasionally disrupt flow
Fair: Generally understandable but contains noticeable unnatural phrasing. Some grammatical errors or awkward constructions occasionally disrupt flow
-
[4]
May contain 1-2 subtle non-native phrasings, but remains highly readable
Good: Text reads smoothly with only minor and infrequent awkwardness. May contain 1-2 subtle non-native phrasings, but remains highly readable
-
[5]
Grammar, word choice, and sentence structure appear completely native with zero awkwardness
Excellent: Text flows perfectly naturally with varied, idiomatic phrasing. Grammar, word choice, and sentence structure appear completely native with zero awkwardness
-
[10]
Does the rewritten sentence flow naturally, avoiding awkward phrasing or redundancy?
-
[11]
Assess grammar, word choice, and readability (e.g., smooth transitions between clauses)
-
[12]
The deadline got pushed back because of unexpected tech issues
Penalize unnatural idioms or register mismatches (e.g., mixing formal and colloquial terms) Output Format Please provide the score for the metric. Include a concise rationale (1-2 sentences per metric) highlighting specific strengths/weaknesses. Example: INPUT: "The deadline got pushed back because of unexpected tech issues." OUTPUT: 4/5 (Colloquial tone ...
Show all 15 references
-
[13]
Does the rewritten sentence preserve the original meaning?
-
[14]
Check for critical information retention, logical coherence, and absence of distortion
-
[15]
The project deadline was extended due to unforeseen technical challenges
Deduct points for omissions, additions, or ambiguous interpretations Output Format Provide the score for the metric. Include a concise rationale (1–2 sentences per metric) highlighting specific strengths/weaknesses. Example: INPUT: Original: "The project deadline was extended ...
-
[2002]
in the tone of an AI assis- tant
applies to open intervals, and g(λ) is con- tinuous at the endpoints, so it also applies to our closed intervals,[λ 1, λ2]andI. Definition 2.The minimum point ofg(λ)isλ ∗. Theorem 1.If g′(0)<0 , then λ∗ >0 and for any λ∈(0, λ ∗], we have KL(ph||(1 +λ)p′ h −λp m)<KL(p h||p′ h)....
2024
-
[2019]
Xiaomeng Hu, Pin-Yu Chen, and Tsung-Yi Ho
Unifying human and statistical evaluation for natural language generation.arXiv preprint arXiv:1904.02792. Xiaomeng Hu, Pin-Yu Chen, and Tsung-Yi Ho. 2023. Radar: Robust ai-text detection via adversarial learn- ing.Advances in Neural Information Processing Systems, 36:15077–15...
1904 arXiv
-
[2024]
InProceedings of the 2024 Conference on Empiri- cal Methods in Natural Language Processing, pages 16923–16936
Raft: Realistic attacks to fool text detectors. InProceedings of the 2024 Conference on Empiri- cal Methods in Natural Language Processing, pages 16923–16936. John Wieting and Kevin Gimpel. 2018. Paranmt-50m: Pushing the limits of paraphrastic sentence embed- dings with millio...
2024 arXiv
-
[2025]
Sebastian Gehrmann, Hendrik Strobelt, and Alexan- der M Rush
Grounding language with vision: A condi- tional mutual information calibrated decoding strat- egy for reducing hallucinations in lvlms.arXiv preprint arXiv:2505.19678. Sebastian Gehrmann, Hendrik Strobelt, and Alexan- der M Rush. 2019. Gltr: Statistical detection and visualiza...
2019
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.