{"id":"3e1bbc86-f662-46ed-824c-f2c587dcfda9","arxiv_id":"2411.12764","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SEFD combines retrieval-based semantic similarity with existing detectors and an adaptive pool to improve detection of paraphrased LLM-generated text in sequential streams.","lead":"This paper introduces SEFD, a tool that adds a semantic similarity lookup to existing AI-text detectors, so that rewritten AI text can still be caught. It matters because paraphrase attacks currently hide AI-generated content in forums, Q&A sites, and other real-world text streams.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline paraphrase gains are demonstrated only under an input ordering that guarantees every paraphrase arrives after its source can be added to the retrieval pool; the claimed real-world robustness is therefore conditional on that ordering, which is not tested.","rationale":"The reader's weakest assumption is exactly the ordering assumption I identified: SEFD's paraphrase gain depends on the original LLM text being in or entering the retrieval pool before the paraphrase arrives. My concern is not that the fusion function is mathematically wrong but that the empirical demonstration is built around an input sequence that guarantees this precondition, while the published claims are stated for general sequential real-world scenarios without that caveat. The paper's own Assumption A.2 (Section III-D) is plausible but narrower than the setting implied by the abstract; a paraphrase could originate from an LLM answer that was never processed by SEFD in the same stream. The authors also impose a stronger blocked order in the experiments, which ensures the pool is populated with all LLM originals before any paraphrase appears. This is the single most load-bearing threat to the central claim because if the order precondition fails, the framework reduces to the initial detector, and the paper's own Table III shows the initial detectors perform poorly on paraphrased text. A concrete test with a withheld-source or paraphrase-first condition would settle whether the improvement is an artifact of this favorable ordering or a genuine property of semantic retrieval across paraphrases. I also note a secondary correctness issue: Algorithm 1 lines 13-16 test s_det against epsilon_sim instead of s_sim, so the published pseudocode does not match Table I; this should be corrected for reproducibility, but it does not change my main concern. The paper's limitation section does not list the ordering dependency as a limitation, which supports treating this as a missing boundary condition rather than a merely cosmetic issue. Since the concern is addressable by an additional experiment and does not invalidate the framework under its stated assumptions, the existing CONDITIONAL verdict remains appropriate.","tokens_in":15538,"tokens_out":6322,"duration_ms":64777,"concrete_test":"Re-run the four dataset experiments with a modified input order that is realistic and still satisfies Assumption A.2: interleave each triplet as [LLM original, human answer, paraphrase] per question instead of blocking all originals, then all humans, then all paraphrases. Separately, run the same pipeline with the paraphrase placed before its source and with the source withheld from the stream. Using DetectGPT with an empty initial pool, report TPR at 1% FPR for paraphrased text. If the improvement over DetectGPT largely disappears in the withheld-source condition, the central claim should be explicitly restricted to streams where each paraphrase's source has already been processed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SEFD 'substantially enhances detection accuracy in paraphrasing scenarios' as a general wrapper. The mechanism that produces the gain is not the fusion function alone but the pool-update rule's ability to store an LLM original before its paraphrase arrives (Section III-D, Assumption A.2; Algorithm 1, Situation 2). The experiments enforce this maximally favorable ordering: every LLM original is processed before any paraphrase (Section IV-B, 'Input data summary'), so the pool can contain the exact source of each paraphrase at decision time. When that ordering is violated—for example, a paraphrase generated from an LLM answer that was never processed by SEFD in the same stream, or a paraphrase arriving before its source—the similarity score has no matching anchor and SEFD degenerates to the initial detector, for which the paper itself reports low paraphrased-text accuracy. The paper does not quantify performance in this case, and the limitation section (V-A) does not state that SEFD only helps paraphrases whose source text appeared earlier in the same processed stream. Because the abstract and Section V-B make an unqualified claim about real-world sequential scenarios, this missing boundary condition is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SEFD, a wrapper framework that augments any existing LLM-generated-text detector with a retrieval component. For each input text, SEFD computes an initial detection score s_det, a maximum-cosine semantic-similarity score s_sim against a retrieval pool of embeddings of previously seen LLM-generated texts, and fuses them via Eq. (4) to produce a final score. A four-case updating rule (Table I) adds or replaces entries in the retrieval pool. Experiments on long-form QA data from Reddit, using GPT-2 XL, OPT-13B, GPT-3.5, and GPT-4o-mini as source models and Log-Likelihood, DetectGPT, ID-MLE, and Soft Watermarking as initial detectors, report improved AUROC and detection accuracy for paraphrased text and improved resistance to recursive paraphrasing. The central claim is that SEFD substantially enhances detection accuracy in paraphrasing scenarios while maintaining robustness for standard LLM-generated content.","tokens_in":15755,"tokens_out":7345,"duration_ms":68315,"significance":"If the reported gains hold under realistic streaming conditions, SEFD is a practically useful, detector-agnostic wrapper whose mechanism is easy to understand: a paraphrase of a previously seen LLM output retains high semantic similarity to the stored embedding, so retrieval can compensate for the initial detector's degraded paraphrase score. The paper's strengths are its breadth (four initial detectors, four source LLMs, and a recursive-paraphrase attack scenario), its clear separation of the fusion and updating components, and the public code link. The main caveat is that the headline numbers are produced under deliberately favorable conditions: the input stream is ordered so that every source text precedes its paraphrase, and one of the two main configurations seeds the retrieval pool with one fifth of the test LLM texts. As a result, the experiments demonstrate feasibility under favorable conditions rather than a validated general solution.","major_comments":[{"comment":"The 'Input data summary' in Section IV-B structures each input sequence as: all LLM-generated texts, then all human-written answers, then all paraphrased versions. This guarantees Assumption A.2 in its strongest form: by the time any paraphrase is scored, the retrieval pool has already had the opportunity to store the exact source text (via Situation 2 of the updating rule when the pool starts empty, or via the initial pool when it is non-empty). The reported paraphrase gains therefore measure SEFD under the maximally favorable ordering. The paper does not test, report, or even state the boundary condition that SEFD's paraphrase benefit requires the source of the paraphrase to have appeared earlier in the same processed stream; when that condition fails, s_sim has no matching anchor and SEFD degenerates to the initial detector, whose paraphrase accuracy is low (e.g., DetectGPT at 8.72 DA for GPT-2 XL in Table III). Please add experiments with shuffled or realistic orderings and state this boundary condition in the limitation section (V-A).","section":"Section IV-B / III-D"},{"comment":"In the 'pool size 1/5' condition, the initial retrieval pool contains 1/5 of the LLM-generated test set (600 of 3000 responses per dataset). For those 600 original texts and for paraphrases derived from them, a high s_sim is obtained simply because the exact or near-identical source embedding is in the pool at decision time. The table does not separate the accuracy contribution from these 'seen' sources from the contribution for unseen sources, so the 'pool size 1/5' rows conflate retrieval from the test set with generalization. The same leakage affects the recursive-paraphrase experiment in Section IV-C3, which also uses a 1/5 initial pool. Please evaluate with a disjoint retrieval corpus or, at minimum, report accuracy separately for texts whose source is and is not contained in the initial pool.","section":"Section IV-C / Table III"},{"comment":"The four free parameters (epsilon_det, epsilon_sim, lambda1, lambda2) are fixed to single values per detector (Table II), selected on the same data used for evaluation, and Section IV-B states that they 'may not yield the optimal results.' No validation procedure, sensitivity analysis, or run-to-run variance is reported, so it is unclear whether the improvements in Table III are robust to reasonable parameter perturbations. Since the fusion function in Eq. (4) amplifies s_det by approximately 10^{lambda1/lambda2} when s_sim is near 1, the parameter sensitivity is potentially large. Please add a validation protocol, error bars or confidence intervals, and a sensitivity analysis over the thresholds.","section":"Section III-C / Table II"},{"comment":"The paper never reports the performance of the retrieval-only similarity score (thresholding s_sim alone) as a baseline. Because the entire claimed gain over the initial detector comes from the retrieval component, this baseline is necessary to determine whether the fusion function provides any benefit beyond simply using s_sim, and to contextualize the 'systematic integration' claim. Please add a retrieval-only row to Table III and, if applicable, to the recursive-paraphrase experiment.","section":"Section III-C / IV-C"},{"comment":"The abstract's statement that SEFD maintains robustness for standard LLM-generated content is not supported by the empty-pool configuration: in Table III, original-text detection accuracy decreases relative to the initial detector for Log-Likelihood (GPT-2 XL: 49.4 to 38.0), DetectGPT (GPT-2 XL: 58.54 to 54.56), and ID-MLE (GPT-3.5: 4.1 to 3.0). Section IV-C1 acknowledges this in passing, but the unqualified abstract claim should be revised to distinguish the empty-pool and non-empty-pool regimes.","section":"Abstract / IV-C1 / Table III"}],"minor_comments":[{"comment":"Algorithm 1, lines 13-16, has an inconsistency: the second and fourth branches of the updating rule compare si_det against both thresholds (e.g., 'if si_det >= epsilon_det and si_det < epsilon_sim'), but Table I and Section III-D clearly intend si_sim in the threshold comparisons. As printed, the pseudocode does not implement the stated updating rule.","section":"Algorithm 1"},{"comment":"The recursive-paraphrase subsection says 'The result is presented in Fig. ??'; the figure reference is unresolved and should be fixed to the actual figure number.","section":"Section IV-C3"},{"comment":"The metric called 'detection accuracy' is actually TPR at 1% FPR; please define this explicitly and consider renaming it to avoid confusion with standard classification accuracy, since the latter would be a different quantity.","section":"Section IV-A"},{"comment":"The use of maximum cosine similarity over a large pool can inflate s_sim for human text when the pool contains many answers to the same question; the paper should justify this max-pooling choice or report additional statistics (e.g., mean or top-k similarity) to show that the separation in Fig. 4 is not an artifact of the max operator.","section":"Section III-B"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable engineering contribution, but the evaluation protocol is too favorable to support the unqualified claims in the abstract. The main issues—source-before-paraphrase ordering, test-set seeding of the initial pool, missing retrieval-only baseline, and lack of parameter sensitivity analysis—are all addressable in a revision. The test-set-in-pool point should be handled head-on, since it bears on the validity of the headline numbers. I do not see evidence of misconduct, but the authors should be pushed to separate retrieval-memorization from generalization in their reported accuracy."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SEFD is a conditional result, not the robust wrapper the abstract implies. The framework fuses a retrieval similarity score with an existing detector, then updates a pool of LLM-text embeddings. The paraphrase gains in Table III are consistent across detectors and datasets, but they rest on Assumption A.2: the paraphrase always arrives after its source in the stream. The experiments enforce the strongest version of that ordering—all LLM originals, then all human answers, then all paraphrases—so the pool almost always contains the exact source when a paraphrase arrives. If a paraphrase arrives before its source, or the source was never in the stream, the retrieval term has no anchor and SEFD reduces to the base detector, which the paper itself shows fails on paraphrased text. The limitation section never states this boundary. That is the main thing to know.\n\nCredit where it's due: the adaptive update rule (Situation 2 adds new LLM originals, Situation 3 replaces a pool entry with the current paraphrase to handle recursive attacks), the fusion function, and the explicit sequential-stream framing are genuinely new relative to the retrieval defense in Krishna et al. [24], which the paper properly cites. The method is clearly described, and the experiments cover four LLMs and four detectors—more than most papers in this space. Code is promised on GitHub.\n\nSoft spots, in proportion: (1) the load-bearing ordering assumption above; (2) experimental hygiene—single runs with no error bars, thresholds set to 'reasonable values' without a validation split, and the initial pool holding 1/5 of the test LLM texts, so part of the effect is near-duplicate retrieval; (3) no retrieval-only baseline, so you can't isolate what the fusion adds; (4) a minor pseudocode typo in Algorithm 1 lines 13–14, where sdet is compared to εsim instead of ssim. None are deal-breakers on their own, but together they mean the abstract's 'substantially enhances' overstates the evidence.\n\nThis paper is for people working on AI-text detection in moderation pipelines and on evaluation design for retrieval-augmented detectors. I'd send it to peer review, with a request for reversed/interleaved ordering experiments, a retrieval-only ablation, error bars, and a validation protocol. The core idea is worth engaging with, but it needs to be scoped honestly.","headline":"SEFD's paraphrase gains are real but conditional on a sequential-ordering assumption; the experiments stack the deck and the paper never tests the failure case.","tokens_in":16304,"tokens_out":5965,"would_cite":true,"duration_ms":52515,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SEFD fuses retrieval similarity with existing detectors to catch paraphrased LLM text.","keywords":["LLM-generated text detection","paraphrasing attacks","semantic similarity retrieval","retrieval pool updating","sequential text","fusion function","watermarking","zero-shot detectors"],"falsifier":"Shuffle a test sequence so that paraphrased answers appear before their originals and keep every other setting identical; if paraphrase detection accuracy does not fall back to the level of the initial detector alone, then the retrieval-pool mechanism described is not the cause of the gain. A second check would be to insert a human-written answer that is semantically close to an LLM output already in the pool and see whether the fusion function misclassifies it.","tokens_in":15322,"feed_emoji":"🔍","tokens_out":10846,"duration_ms":147411,"temperature":0.7,"pith_summary":"The paper tries to establish that an existing LLM-generated-text detector can be made substantially more resistant to paraphrase attacks by adding a retrieval step that compares each incoming text, by semantic similarity, against a running pool of previously seen LLM outputs. The resulting wrapper, SEFD (Semantic-Enhanced Framework for Detecting LLM-Generated Text), fuses the detector's score with the best cosine similarity to anything in the pool and updates the pool as texts arrive, so a paraphrase can be matched to its original before the paraphrase is classified. Experiments across four source LLMs and four detection methods report gains in paraphrase detection accuracy while preserving performance on ordinary LLM-generated text. The intended setting is sequential text, such as chronologically ordered forum comments or Q&A answers.","feed_headline":"Retrieval pool rescues AI-text detectors from paraphrase attacks","feed_subtitle":"Fusing a similarity score with any detector lifts paraphrase detection accuracy across four LLMs.","key_machinery":"The load-bearing object is the retrieval pool together with the fusion rule. Each input text $x_i$ is embedded, and its similarity score $s_{\\mathrm{sim}}$ is the maximum cosine similarity against embeddings already in the pool. The final score is $s = s_{\\mathrm{det}}/(1+10^{-\\lambda_1}-s_{\\mathrm{sim}})^{1/\\lambda_2}$, which leaves the detector score nearly unchanged when similarity is near $0$ and amplifies it by about $10^{\\lambda_1/\\lambda_2}$ when similarity is near $1$. The updating rule compares detection and similarity scores against two thresholds and decides whether to ignore the text, add it to the pool, or replace the closest pool entry; the replacement case is what lets the pool track recursive paraphrases.","core_discovery":"SEFD's central claim is that paraphrasing weakens standard detection signals but leaves semantic content largely intact, and that a retrieval pool of previously seen LLM outputs supplies exactly the signal the detector loses. By combining a detector score with a maximum cosine-similarity score through the fusion function $s_{\\mathrm{det}}/(1+10^{-\\lambda_1}-s_{\\mathrm{sim}})^{1/\\lambda_2}$, the framework amplifies weak detector scores precisely when the input is semantically close to a known LLM output. The updating rule keeps the pool current, replacing old entries with their paraphrased versions so recursive paraphrases remain detectable. On the paper's datasets this recovers much of the paraphrase-induced accuracy drop across all four tested detectors and all four source models.","pith_inferences":["If the ordering assumption fails and a paraphrase arrives before its original is seen, or the original never appears in the stream, SEFD should fall back to the initial detector; this is testable by shuffling the input order.","The same wrapper logic could be transferred to other meaning-preserving transformations, such as machine translation or summarization, whenever a detector's score degrades but semantic similarity survives.","Contamination risk is inherent: a false positive in the initial detector adds a human text to the pool, which could raise later similarity scores for other human texts; low false-positive-rate operating points mitigate but do not remove this risk.","The four tuning parameters are detector-specific and the paper does not claim global optimality, so deployment would require a calibration step for each new detector and data distribution."],"forward_implications":["Paraphrase detection accuracy improves relative to the initial detector alone across all datasets tested, with the largest relative gains for detectors whose scores drop most under paraphrase.","An empty initial pool is sufficient for substantial gains, because the updating rule adds newly seen LLM-generated texts and later matches their paraphrases against them.","Larger initial pools generally improve detection accuracy, though a non-robust encoder can occasionally make a non-empty pool hurt human-text classification.","Recursive paraphrasing, which pushes baseline detection accuracy close to zero, causes only a moderate decline under SEFD.","SEFD is detector-agnostic: any detector with workable thresholds can be wrapped, so the framework does not depend on the four specific detectors tested."],"supporting_citations":[{"why":"Soft watermarking method used both as an initial detector and as evidence that watermark detection scores drop after paraphrase.","marker":"[7]"},{"why":"Curvature-based zero-shot detector used as the primary initial detector for pool-size and recursive-paraphrase experiments.","marker":"[16]"},{"why":"Log-likelihood detector used as an initial detector and as one of the four score-distribution baselines.","marker":"[18]"},{"why":"Intrinsic dimension detector used as an initial detector, with its MLE variant replacing the original PHD estimator.","marker":"[19]"},{"why":"Establishes that paraphrasing and recursive paraphrasing evade many detectors, motivating SEFD's robustness target and the recursive-paraphrase experiment.","marker":"[22]"},{"why":"Supplies the paraphrasing model for the data and the prior observation that retrieval is an effective defense against paraphrase attacks.","marker":"[24]"},{"why":"GPT-2 XL source model, one of the four LLMs whose outputs populate the datasets and retrieval pools.","marker":"[36]"},{"why":"Sentence embedding encoder used to compute the semantic similarity scores between input text and retrieval pool entries.","marker":"[37]"},{"why":"OPT-13B source model, another of the four LLMs used to generate the test datasets.","marker":"[42]"},{"why":"GPT-3.5 source model, a third LLM used to generate the test datasets.","marker":"[43]"}],"fun_headline_variants":["SEFD fuses retrieval to catch paraphrased AI text","Retrieval pool beats paraphrase attacks on AI-text detectors","Semantic similarity rescues AI-text detection from paraphrasing","Paraphrase-proof AI-text detection via semantic retrieval","Retrieval-based framework lifts AI-text detection under paraphrase"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gain depends on every paraphrased text arriving after its original LLM-generated text, so the retrieval pool has already seen (or can immediately store) a semantically matching anchor before the paraphrase is classified.","fun_headline_variants_meta":{"raw":{"variants":["SEFD fuses retrieval to catch paraphrased AI text","Retrieval pool beats paraphrase attacks on AI-text detectors","Semantic similarity rescues AI-text detection from paraphrasing","Paraphrase-proof AI-text detection via semantic retrieval","Retrieval-based framework lifts AI-text detection under paraphrase"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00025,"raw_usage":{"total_tokens":1501,"prompt_tokens":842,"completion_tokens":659,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":578}},"tokens_in":458,"tokens_out":659,"duration_ms":6595,"temperature":1.0,"reasoning_tokens":578,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:51:06.687052+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Shuffle a test sequence so that paraphrased answers appear before their originals and keep every other setting identical; if paraphrase detection accuracy does not fall back to the level of the initial detector alone, then the retrieval-pool mechanism described is not the cause of the gain. A second check would be to insert a human-written answer that is semantically close to an LLM output already in the pool and see whether the fusion function misclassifies it.","supporting_citations":[{"cited_title":"In- trinsic dimension estimation for robust detection of ai-generated texts,","cited_arxiv_id":null,"evidence_quote":"Intrinsic dimension detector used as an initial detector, with its MLE variant replacing the original PHD estimator."}],"review_version":1}