Pith. sign in

REVIEW 3 major objections 3 minor 15 references

Your Language Model Can Secretly Write Like Humans: Contrastive Paraphrase Attacks on LLM-Generated Text Detectors

T0 review · 3 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read CoPA, a training-free contrastive paraphrase attack, reduces the detection rate of eight AI-text detectors to near-baseline levels while preserving semantics.

desk verdict CoPA is a genuinely effective training-free paraphrase attack with a serious theory-algorithm gap: the theorem analyzes an arithmetic mixture while the code does logit-space contrast, but the empirical results stand on their own. read the letter →

arxiv 2505.15337 v3 pith:YYOKDKD3 submitted 2025-05-21 cs.CL cs.AI

classification cs.CLcs.AI
keywords LLM-generatedtextdetectionparaphraseattackcontrastivedecodingadversarialAItraining-freeFast-DetectGPTauthenticity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a training-free 'contrastive paraphrase attack' (CoPA) can rewrite AI-generated text so that eight current detectors, including the strong Fast-DetectGPT, flag it as human at rates close to the random baseline. The trick is to have an off-the-shelf LLM generate two versions of a sentence: one prompted to sound human and one prompted to sound machine-like, then subtract the machine-like token distribution from the human-like one during decoding. The authors prove a convexity theorem showing that, under a premise they argue generally holds, this subtraction moves the paraphrase closer to true human word distributions than a human-style prompt alone. If correct, the method undercuts the reliability of current detectors while requiring no training data or specialized paraphraser, using only the attacker's own LLM.

What carries the argument

The central object is the contrastive decoding distribution in Eq. (3), formed by subtracting the logits of a machine-like distribution from an amplified human-like distribution: $p_c \propto \exp((1+\lambda) f_\theta(\cdot|x_h,y_m,y_{<t}) - \lambda f_\theta(\cdot|x_m,y_m,y_{<t}))$. This is a training-free, self-corrective decoding mechanism: the machine-like prompt (in practice, asking the LLM to repeat the original text) provides a concentrated 'machine-style' token distribution that is subtracted to remove machine-inherent word preferences. An adaptive truncation constrains sampling to tokens with high probability under the human-like distribution, preserving plausibility. The theoretical support is a convexity/KL-divergence argument: $g(\lambda) = \mathrm{KL}(p_h \| (1+\lambda) p'_h - \lambda p_m)$ is convex, and if $g'(0)<0$ then a positive $\lambda$ strictly reduces the KL divergence to the human distribution.

What would settle it

Compute, for a fixed sample of human texts and a fixed paraphraser, the empirical KL divergence $\mathrm{KL}(p_h \| (1+\lambda) p'_h - \lambda p_m)$ for a grid of $\lambda \in [0,1]$ by estimating the distributions from token logits. If for the paper's default $\lambda=0.5$ the contrastive distribution is not closer to the human distribution than $p'_h$ on a majority of samples, or if $g'(0) \ge 0$ for many samples, the theorem's premise fails and the claimed bound does not hold for those samples. A second test is to train a detector on CoPA-paraphrased texts (beyond the paper's RoBERTa-large adaption) and see whether the attack's TPR remains near baseline.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the machine-like statistical bias of an LLM can be turned against its own detectability. CoPA elicits two token distributions from the same off-the-shelf model, $p'_h$ from a human-style prompt and $p_m$ from a machine-style prompt, then samples from the contrastive distribution $p_c \propto \exp((1+\lambda) f_h - \lambda f_m)$. This penalizes tokens the LLM systematically prefers in its default mode while keeping tokens plausible under the human-style prompt, via an adaptive truncation. In experiments on XSum, SQuAD, and LongQA against eight detectors, CoPA drives the detection true positive rate at 5% FPR down to an average of about 12.63%, 11.06%, and 7.44% for GPT-3.5-turbo texts, versus 66.05%, 56.27%, and 57.09% without attack, with semantic similarity above 90%.

Load-bearing premise

The whole argument rests on the assumption that subtracting the machine-like token distribution in log space actually moves the rewritten text closer to how humans write, an assumption the paper justifies with a heuristic diagram rather than a direct measurement.

Editorial extensions

If this is right

  • CoPA works across eight detector types, including zero-shot likelihood-based, curvature-based, and trained classifiers, with TPR near baseline at 5% FPR.
  • It transfers across source LLMs (GPT-3.5-turbo, GPT-4, Claude-3.5, GPT-4o, Gemini-1.5 Pro) and across paraphrasers (Qwen2.5-72B, R1-Distill-32B, QwQ-32B, GLM-4-9B-hf), with QwQ-32B performing best.
  • A single paraphrase suffices; multiple paraphrases give limited gains, and semantic similarity stays above 90%.
  • An adaptive defense that adversarially trains RoBERTa-large with CoPA-paraphrased texts raises detection from 22.67% to 78.00% TPR at 5% FPR, indicating the attack can be mitigated by training on such texts.
  • CoPA outperforms a surrogate-based RedTeaming attack even without using a surrogate detector, with 17.00% versus 30.00% TPR on Fast-DetectGPT on XSum.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A broader principle suggested by this work is that detectors relying on LLM-inherent statistical biases can be fooled by making the generator sample away from its own mode, a strategy that may generalize to other modalities if a contrastive 'machine-style' distribution can be elicited.
  • Because CoPA is training-free and uses off-the-shelf models, it lowers the bar for adversarial users: anyone with API access to a capable LLM can launch the attack, so detector robustness should be studied against adaptive attackers rather than static baselines.
  • The theory's premise $g'(0)<0$ deserves direct measurement; the paper offers a heuristic diagram but no empirical check. A simple test is to compute KL divergences on real data for increasing $\lambda$ and verify that Eq. (6) holds as predicted.
  • The attack's success depends on the choice of machine prompt; the paper uses 'repeat the paragraph' because it best concentrates machine-like features, but whether this holds for all domains and LLMs is an open question.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes CoPA, a training-free contrastive paraphrase attack against LLM-generated text detectors. At each decoding step, CoPA combines a human-style prompt distribution with a machine-style prompt distribution via the logit-space contrast p_c ∝ exp((1+λ)f_h − λ f_m), after adaptive truncation. The authors report that CoPA reduces TPR at 5% FPR to near-baseline levels across eight detectors, three datasets, several source LLMs, and several paraphraser LLMs, while preserving high semantic similarity. A theoretical section (Sec. 3.3) presents a theorem intended to show that the contrastive distribution is closer to human text than the human-like prompt distribution alone. The empirical results are extensive, and the code is released. The main weakness is that the theorem analyzes an arithmetic mixture of distributions, whereas the implemented algorithm uses a logit-space contrast, so the stated theoretical guarantee does not cover the method as implemented.

Significance. If the empirical findings hold, CoPA is a practical and inexpensive attack that substantially weakens current LLM-text detectors, with implications for detector robustness and for the deployment of detection systems. The paper's strengths are its broad evaluation (three datasets, eight detectors, multiple source and paraphraser models), its release of code, and its inclusion of human and automated text-quality evaluations. The adaptive-defense experiment and the ethical discussion are also valuable. The theoretical claim, however, is currently not established for the actual algorithm, and the key premise of the theorem is not directly validated. These issues affect the paper's central advertised contribution and require substantial revision before publication.

major comments (3)
  1. [Sec. 3.2 Eq. (3) and Sec. 3.3 Eq. (5)] Theorem 1 analyzes the arithmetic mixture q_λ = (1+λ)p'_h − λ p_m, but the implemented CoPA samples from p_c ∝ exp((1+λ)f_h − λ f_m), which after normalization equals p_c ∝ (p'_h)^(1+λ) / p_m^λ. These are different distributions, so the theorem does not cover the algorithm evaluated in Section 4. I recommend deriving an analogous theorem for the logit-space geometric contrast, or explicitly repositioning Theorem 1 as a heuristic motivation and removing claims that the theory 'guarantees' or 'validates' the effectiveness of the implemented method.
  2. [Sec. 3.3, Fig. 4] The premise g′(0)<0 is never measured or verified. The only support is the |V|=3 contour diagram in Figure 4 and the assertion that p'_h is 'usually' between p_m and p_h. Since this condition is the basis of Theorem 1, the theoretical result is conditional on an unvalidated assumption. Please report direct estimates of g′(0) for the actual token distributions used in the experiments, or state clearly that this is an assumption rather than an established fact whose verification is left to the reader.
  3. [Sec. 4.2, Appendix B, Fig. 3] The machine prompt x_m was selected using detector likelihood as a proxy, and the contrast intensity λ was tuned on the first 50 XSum samples (Figure 6). This is detector-informed tuning, and it should be disclosed in the main text because the paper positions CoPA as training-free and surrogate-free. To support that positioning, the main results should either be confirmed with a fixed machine prompt and hyperparameters chosen without detector feedback, or the paper should acknowledge that the reported performance depends on detector-informed prompt selection.
minor comments (3)
  1. [Tables 1 and 2] Several cells in the Dipper rows are merged or missing delimiters (for example, Table 1 XSum shows '15.672.6727.33' and Table 2 for Claude-3.5 shows '18.170.3320.67'). Please fix the table formatting so that each metric is a separate entry.
  2. [Appendix A, Proposition 3] The proof of uniqueness of the minimum uses analyticity of g on the closed interval I, but g need not be analytic at points where the mixture assigns zero probability to a token with positive human probability, because the KL divergence becomes infinite there. Please clarify the domain of g and handle infinite KL values explicitly.
  3. [Sec. 3.3 and Sec. 4.4] The text claims that the trend of TPR over λ in Figure 6 'roughly aligns with our theoretical analysis,' but the theorem applies to the arithmetic mixture, not to the implemented logit-space contrast. The ablation result would be more appropriately presented as a heuristic observation unless the theory is re-derived for the actual sampling distribution.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the empirical attack is measured against detectors; the theory/algorithm gap is a rigor issue, not a circular reduction.

full rationale

The central claim that CoPA lowers detector TPR is supported by direct measurements against eight detectors (Tables 1, 2, 4, 5), with hyperparameters lambda=0.5 and alpha=1e-5 set by ablations rather than fitted to the reported final TPRs. The machine prompt is selected by maximizing the detector's LLM likelihood (Appendix B, Fig. 3), but this is a design choice about the contrastive reference, not a fitted parameter that is then renamed a prediction; the reported fooling rates remain independent measurements. The theoretical argument (Theorem 1, Eq. 5-6) analyzes KL(p_h || (1+lambda)p'_h - lambda p_m), whereas the implemented CoPA (Eq. 3) performs logit-space contrast, which after normalization is proportional to (p'_h)^(1+lambda)/(p_m)^lambda, a product-of-experts contrast not covered by the arithmetic-mixture theorem. The premise g'(0)<0 is also asserted to 'generally hold' without direct estimation. These are non-circular rigor gaps: the theorem does not reduce to the algorithm's inputs, and the empirical results stand independently of the theorem. The paper contains minor self-citations, e.g., Fang et al. (2025) in Sec. 3.2 for the token-penalty concern, but these are not load-bearing for the attack's success. Overall, no significant circularity; score 2 reflects only minor non-load-bearing self-citation and a mild in-sample lambda-selection concern.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or conceptual entities. It constructs distributions and prompts, but these are not invented postulates. The main free parameters are λ and α. The load-bearing axioms are about prompt fidelity and the unverified premise g'(0)<0, plus the standard math used in the proof.

free parameters (2)
  • λ (contrast intensity) = 0.5
    Chosen via ablation on the first 50 XSum samples (Figure 6). Controls the strength of the machine-distribution subtraction. Default set to 0.5.
  • α (clipping factor) = 1e-5
    Used in the adaptive truncation mechanism (Eq. 4). No sensitivity analysis is provided for this value.
assumptions (4)
  • domain assumption The premise g'(0)<0 in Theorem 1 holds for the distributions in practice.
    The paper asserts this 'generally holds' based on a heuristic |V|=3 diagram (Figure 4) and does not empirically measure g'(0). The theorem's conclusion depends on this premise.
  • domain assumption The prompts xh and xm faithfully elicit human-like and machine-like token distributions p'h and pm.
    The entire contrastive mechanism rests on these prompts producing distributions that reflect the intended styles. The prompts are hand-crafted and selected empirically.
  • domain assumption Detector likelihood (e.g., Fast-DetectGPT) is a valid proxy for machine-like style.
    Used both to select the machine prompt (Appendix B) and to measure attack success. If detectors respond to different features, the contrast may not transfer to other detectors.
  • standard math KL divergence is convex and the distance function g(λ) is analytic on its domain.
    Used in Proposition 2/3 to establish convexity and uniqueness of the minimum. Referenced to Krantz and Parks (2002).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Your Language Model Can Secretly Write Like Humans: Contrastive Paraphrase Attacks on LLM-Generated Text Detectors." pith.science (2026). https://pith.science/paper/YYOKDKD3

@misc{pith2026250515337,
  author       = {Pith},
  title        = {Pith review of: Your Language Model Can Secretly Write Like Humans: Contrastive Paraphrase Attacks on LLM-Generated Text Detectors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YYOKDKD3}},
  note         = {Machine review of arXiv:2505.15337}
}
read the original abstract

The misuse of large language models (LLMs), such as academic plagiarism, has driven the development of detectors to identify LLM-generated texts. To bypass these detectors, paraphrase attacks have emerged to purposely rewrite these texts to evade detection. Despite the success, existing methods require substantial data and computational budgets to train a specialized paraphraser, and their attack efficacy greatly reduces when faced with advanced detection algorithms. To address this, we propose \textbf{Co}ntrastive \textbf{P}araphrase \textbf{A}ttack (CoPA), a training-free method that effectively deceives text detectors using off-the-shelf LLMs. The first step is to carefully craft instructions that encourage LLMs to produce more human-like texts. Nonetheless, we observe that the inherent statistical biases of LLMs can still result in some generated texts carrying certain machine-like attributes that can be captured by detectors. To overcome this, CoPA constructs an auxiliary machine-like word distribution as a contrast to the human-like distribution generated by the LLM. By subtracting the machine-like patterns from the human-like distribution during the decoding process, CoPA is able to produce sentences that are less discernible by text detectors. Our theoretical analysis suggests the superiority of the proposed attack. Extensive experiments validate the effectiveness of CoPA in fooling text detectors across various scenarios.

Figures

Figures reproduced from arXiv: 2505.15337 by the authors.

Figure 1
Figure 1. Comparison of different paraphrasing strate [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed CoPA. The contrastive paraphrasing successfully penalizes the LLM-preferred [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Fast-DetectGPT detected LLM likelihood of [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Illustration of the premise of Theorem 1. Let |V| = 3 and thus P V is a triangle. We draw the contours of f(p) := KL(ph||p). The closer to ph, the lower the KL divergence with ph. If p ′ h−pm points to the inside of the contour at p ′ h , then f(p) decreases at p ′ h a…
Figure 5
Figure 5. Figure 5: Paraphrased sentences by Dipper and our CoPA. We use Fast-DetectGPT to provide the LLM likelihood. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Comparison of detection accuracy on the first 50 samples from XSum under different values of λ against Fast-DetectGPT (Bao et al., 2024). 4.3 Attacks on More Source LLMs. In addition to GPT-3.5-turbo, we also consider the machine texts generated by recently prevalent L…
Figure 8
Figure 8. Figure 8: ROC (0-5% FPR) for GPT-3.5-turbo on Fast [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Detection accuracy (at FPR=5%) of CoPA under different sampling parameters against Fast-DetectGPT [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Visualization of paraphrased sentences from human-like distribution [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 13 canonical work pages

  1. [1]

    Often difficult or impossible to understand

    Very Poor: Severely broken or unnatural English with major grammar issues. Often difficult or impossible to understand. Prompt for GPT-4 evaluation: (1) Task: Evaluate the natural fluency of a given sentence. Use a 5-point scale (5 = highest). Natural Fluency (1-5):

  2. [2]

    Requires effort to understand in places

    Poor: Frequent unnatural phrasing and grammatical errors make reading diffi- cult. Requires effort to understand in places

  3. [3]

    Some grammatical errors or awkward constructions occasionally disrupt flow

    Fair: Generally understandable but contains noticeable unnatural phrasing. Some grammatical errors or awkward constructions occasionally disrupt flow

  4. [4]

    May contain 1-2 subtle non-native phrasings, but remains highly readable

    Good: Text reads smoothly with only minor and infrequent awkwardness. May contain 1-2 subtle non-native phrasings, but remains highly readable

  5. [5]

    Grammar, word choice, and sentence structure appear completely native with zero awkwardness

    Excellent: Text flows perfectly naturally with varied, idiomatic phrasing. Grammar, word choice, and sentence structure appear completely native with zero awkwardness

  6. [10]

    Does the rewritten sentence flow naturally, avoiding awkward phrasing or redundancy?

  7. [11]

    Assess grammar, word choice, and readability (e.g., smooth transitions between clauses)

  8. [12]

    The deadline got pushed back because of unexpected tech issues

    Penalize unnatural idioms or register mismatches (e.g., mixing formal and colloquial terms) Output Format Please provide the score for the metric. Include a concise rationale (1-2 sentences per metric) highlighting specific strengths/weaknesses. Example: INPUT: "The deadline got pushed back because of unexpected tech issues." OUTPUT: 4/5 (Colloquial tone ...

Show all 15 references
  1. [13]

    Does the rewritten sentence preserve the original meaning?

  2. [14]

    Check for critical information retention, logical coherence, and absence of distortion

  3. [15]

    The project deadline was extended due to unforeseen technical challenges

    Deduct points for omissions, additions, or ambiguous interpretations Output Format Provide the score for the metric. Include a concise rationale (1–2 sentences per metric) highlighting specific strengths/weaknesses. Example: INPUT: Original: "The project deadline was extended ...

  4. [2002]

    in the tone of an AI assis- tant

    applies to open intervals, and g(λ) is con- tinuous at the endpoints, so it also applies to our closed intervals,[λ 1, λ2]andI. Definition 2.The minimum point ofg(λ)isλ ∗. Theorem 1.If g′(0)<0 , then λ∗ >0 and for any λ∈(0, λ ∗], we have KL(ph||(1 +λ)p′ h −λp m)<KL(p h||p′ h)....

  5. [2019]

    Xiaomeng Hu, Pin-Yu Chen, and Tsung-Yi Ho

    Unifying human and statistical evaluation for natural language generation.arXiv preprint arXiv:1904.02792. Xiaomeng Hu, Pin-Yu Chen, and Tsung-Yi Ho. 2023. Radar: Robust ai-text detection via adversarial learn- ing.Advances in Neural Information Processing Systems, 36:15077–15...

  6. [2024]

    InProceedings of the 2024 Conference on Empiri- cal Methods in Natural Language Processing, pages 16923–16936

    Raft: Realistic attacks to fool text detectors. InProceedings of the 2024 Conference on Empiri- cal Methods in Natural Language Processing, pages 16923–16936. John Wieting and Kevin Gimpel. 2018. Paranmt-50m: Pushing the limits of paraphrastic sentence embed- dings with millio...

  7. [2025]

    Sebastian Gehrmann, Hendrik Strobelt, and Alexan- der M Rush

    Grounding language with vision: A condi- tional mutual information calibrated decoding strat- egy for reducing hallucinations in lvlms.arXiv preprint arXiv:2505.19678. Sebastian Gehrmann, Hendrik Strobelt, and Alexan- der M Rush. 2019. Gltr: Statistical detection and visualiza...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.