REVIEW 10 cited by
Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense
read the original abstract
The rise in malicious usage of large language models, such as fake content creation and academic plagiarism, has motivated the development of approaches that identify AI-generated text, including those based on watermarking or outlier detection. However, the robustness of these detection algorithms to paraphrases of AI-generated text remains unclear. To stress test these detectors, we build a 11B parameter paraphrase generation model (DIPPER) that can paraphrase paragraphs, condition on surrounding context, and control lexical diversity and content reordering. Using DIPPER to paraphrase text generated by three large language models (including GPT3.5-davinci-003) successfully evades several detectors, including watermarking, GPTZero, DetectGPT, and OpenAI's text classifier. For example, DIPPER drops detection accuracy of DetectGPT from 70.3% to 4.6% (at a constant false positive rate of 1%), without appreciably modifying the input semantics. To increase the robustness of AI-generated text detection to paraphrase attacks, we introduce a simple defense that relies on retrieving semantically-similar generations and must be maintained by a language model API provider. Given a candidate text, our algorithm searches a database of sequences previously generated by the API, looking for sequences that match the candidate text within a certain threshold. We empirically verify our defense using a database of 15M generations from a fine-tuned T5-XXL model and find that it can detect 80% to 97% of paraphrased generations across different settings while only classifying 1% of human-written sequences as AI-generated. We open-source our models, code and data.
Forward citations
Cited by 10 Pith papers
-
ExaGPT: Example-Based Machine-Generated Text Detection for Human Interpretability
ExaGPT uses span-level similarity retrieval from human and LLM datastores to detect machine-generated text while supplying the matching spans as human-interpretable evidence, achieving up to 37-point accuracy gains ov...
-
Is This AI? Longitudinal Analysis of Strategies Used for AI Detection on Two Subreddits
Longitudinal mixed-method analysis of 13,098 posts and 222,060 comments from r/isthisAI and r/RealOrAI catalogs 12 AI detection strategies and documents their evolution with changing AI capabilities.
-
LLM Self-Recognition: Steering and Retrieving Activation Signatures
Steering LLM residual streams with random sparse vectors creates detectable self-recognition fingerprints that enable over 98% accurate attribution of generated text to specific models without degrading output quality.
-
Asking Back: Interaction-Layer Antidistillation Watermarks
Interaction-layer antidistillation watermarks use system-prompt-induced behavioral markers like explicit follow-up questions that transfer to distilled student models at 45-89% relative fidelity and can be audited via...
-
Small Is Enough: Per-User Style Rewriting of AI-Edited Text via LoRA Adapters
LoRA-adapted 0.5B-7B language models all reach the same automatic rewriting score (0.69), indicating model size does not change measured quality for this single-user style-rewriting task.
-
Is This AI? Longitudinal Analysis of Strategies Used for AI Detection on Two Subreddits
Users in r/isthisAI and r/RealOrAI employ 12 evolving strategies for AI detection that shift with model capabilities and online trends.
-
Paraphrasing Attack Resilience of Various AI-Generated Text Detection Methods
Binoculars-inclusive ensembles detect AI text best overall but suffer the largest performance drops under paraphrasing attacks.
-
READER: Reasoning-Enhanced AI-Generated Text Detection
A 1.5B LLM fine-tuned on a curated rationale dataset (READ) detects AI text with explanations and reportedly outperforms much larger prompted LLMs.
-
Findings of the Counter Turing Test: AI-Generated Text Detection
Shared task findings show near-perfect binary detection of AI-generated text but greater difficulty in attributing outputs to particular language models.
-
Findings of the Counter Turing Test: AI-Generated Text Detection
Shared task findings show F1=1.0000 for binary AI text detection and 0.9531 for model attribution using fine-tuned DeBERTa and BART transformers with ensembles.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.