{"id":"082d9f20-e349-458c-b605-0a8748a0df4d","arxiv_id":"2608.05741","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"EchoPrompt detects LLM-generated text by measuring the likelihood gain from restoring a generic assistant-style prompt, calibrated against a base model.","lead":"This paper proposes EchoPrompt, a zero-shot detector that tells machine-written from human-written text by adding a generic assistant-style prefix and measuring whether an instruction-tuned model finds the text more likely than a base model does. It reports state-of-the-art accuracy across three benchmarks, but the prefix was selected on a subset of one benchmark, so the headline results need independent validation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on a prefix selected from the evaluation benchmark itself; if that selection is removed or transferred, the reported edge over IRM may shrink.","rationale":"The reader's weakest assumption identifies the same load-bearing point: the generic prefix is selected on a subset of the evaluation benchmark, so the central comparison is not a clean held-out test of a zero-shot detector. My read agrees with that concern and adds two supporting observations. First, the method's score in Eq. (7) is exactly IRM plus a within-instruction-model context gain, so the entire claimed improvement is carried by the choice of c_g; if that choice is dataset-specific, the method is a tuned variant of IRM rather than a new zero-shot signal. Second, the paper's own Appendix A limits the claim: the prompt is not shown to be globally optimal. The RealDet and RAID results do give partial out-of-distribution evidence that the method generalizes, which is why I would not move the verdict to REJECT; but the absence of error bars, code, and a held-out threshold procedure means the conditional verdict is appropriate. The proposed test would settle whether the reported gains survive an honest, pre-specified prefix choice.","tokens_in":21753,"tokens_out":8224,"duration_ms":74621,"concrete_test":"Hold out all three benchmarks during prefix selection: fix c_g to a pre-registered generic assistant prefix (e.g., 'You are a helpful assistant.') or select c_g on a disjoint corpus such as HC3, then recompute Table 1 for EchoPrompt and IRM with bootstrap 95% confidence intervals on the AUROC difference for each split. If the average gain over IRM falls below the reported 0.69 AUROC / 2.64 F1, or if the advantage on RealDet and RAID is not significantly positive, the headline state-of-the-art claim should be revised to be conditional on benchmark-specific prompt tuning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that the single generic prefix c_g, chosen in Appendix C by maximizing AUROC on a 500+500 labeled subset of DetectRL Multi-Domain, is a faithful restorer of latent prompt context rather than a benchmark-tuned hyperparameter. Because the headline results in Table 1 are reported on the same DetectRL benchmark (Multi-Domain, Multi-LLM, Multi-Attack), the claimed 0.69 AUROC / 2.64 F1 average gain over IRM partly reflects selection on the evaluation distribution. This is especially consequential because the entire novelty of EchoPrompt is the prompt-restoration term: Eq. (7) equals the IRM score plus log P_inst(x_t | c_g, x_<t) − log P_inst(x_t | x_<t). If c_g is simply the best of the 17+ candidate prefixes on one sample from DetectRL, and if the advantage does not replicate under a prefix chosen without seeing DetectRL labels, the method reduces to IRM plus a dataset-specific offset. Appendix A concedes the prompt is not shown to be globally optimal, and Section 4.5 reports F1-optimal thresholds without describing any held-out threshold selection, so the magnitude of the claimed advantage is not yet established. The RealDet and RAID rows provide some out-of-distribution support, but they use the same benchmark-selected prefix and are part of the same Table 1 average, so they do not by themselves validate the zero-shot claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EchoPrompt, a training-free detector for LLM-generated text. The method prepends a fixed assistant-style prefix c_g to an input passage, computes the average token-level log-likelihood gap between an instruction-tuned model on the restored sequence and its base model on the original text (Eq. 7), and thresholds this score. The authors motivate the score by the claim that machine-generated text retains a latent dependency on an assistant-response context, which can be reactivated by a generic prefix. Experiments on DetectRL, RealDet, and RAID report state-of-the-art results, including an average gain over IRM of 0.69 AUROC and 2.64 F1 with the Llama-3-8B proxy family, along with robustness analyses across attacks, proxy models, text lengths, and thresholds.","tokens_in":22034,"tokens_out":6140,"duration_ms":50337,"significance":"The proposed signal—latent prompt dependency measured by a calibrated likelihood gap—is interpretable and grounded in the post-training objective (Eq. 4). The evaluation is extensive: three public benchmarks, seven proxy families, five attack groups, length-binned analysis, and component ablations. If the reported gains are robust to properly held-out prefix selection, EchoPrompt would be a cheap, training-free, and broadly applicable detector with a clear mechanistic story. However, the headline improvement over IRM is not yet established because the root cause of the gain—the choice of c_g—is selected on a labeled subset of the main evaluation benchmark, which makes the zero-shot claim and the reported performance gains optimistically biased.","major_comments":[{"comment":"The global prefix c_g that defines the EchoPrompt score is selected by maximizing AUROC on a 500+500 labeled subset of DetectRL Multi-Domain, and this same prefix is then used to produce all reported results on DetectRL (Tables 1, 2, and 9). The statement in Appendix C that the subset is used only for prompt selection 'rather than for reporting main detection performance' is not accurate, because the selected prefix is a component of the score reported in those tables. The claimed average gains over IRM (0.69 AUROC, 2.64 F1) are therefore optimistically biased by selection on the evaluation distribution. To substantiate the zero-shot claim, the authors must re-select c_g without access to DetectRL labels (e.g., on a disjoint development set or on a different benchmark) and report both the tuned and untuned results, or demonstrate that the relative ordering of candidate prefixes is stable across benchmarks.","section":"Appendix C; Eq. (7)"},{"comment":"F1 scores throughout the paper are computed with the F1-optimal threshold τ* chosen on the test set itself, as described in the normalized-threshold analysis; no held-out threshold selection, cross-validation, or confidence intervals are reported. Because F1 is one of the two headline metrics, the reported 2.64-point average F1 improvement over IRM may partly reflect test-set threshold fitting. The authors should report F1 at a label-free threshold (e.g., a threshold chosen on a separate development set or the median of a calibration set) or provide bootstrapped confidence intervals and significance tests for the AUROC/F1 differences.","section":"Section 4.5; Figure 6"},{"comment":"The empirical evidence supporting the latent-prompt-dependency hypothesis—the claim that machine text receives a larger likelihood gain g(X; c_g) than human text under a restored prefix—is computed with the same c_g selected on the 500+500 DetectRL subset. This analysis therefore does not provide independent confirmation of the hypothesis; it shows only that the selected prefix discriminates on the benchmark from which it was chosen. Repeating the analysis with a prefix selected without reference to DetectRL, or with several randomly chosen prefixes, would strengthen the claim that the effect is a property of machine text rather than an artifact of prefix selection.","section":"Section 3.3; Figure 2"}],"minor_comments":[{"comment":"The y-axis label 'Base Prompt Gain g(X; c_g)' refers to Eq. (9), which is introduced later in Section 3.3; please define the quantity in the caption or move the definition earlier.","section":"Figure 2"},{"comment":"Describing c_g as 'task-agnostic' is potentially misleading given that it was chosen by an empirical search over candidate prefixes; consider describing it as an empirically chosen default prefix.","section":"Section 3.2.1"},{"comment":"The abstract's unqualified claim of 'state-of-the-art performance' is stronger than what the evidence supports; the main result is established only for the Llama-3-8B proxy family, and Table 9 shows that for some proxy families (e.g., Llama-3.2-1B) EchoPrompt does not beat IRM on average AUROC. Please qualify the claim accordingly.","section":"Abstract and Section 4.2"},{"comment":"The normalized threshold τ̂ = τ*/σ_pool is not fully defined; please specify what σ_pool denotes and explain why the sign of τ̂ is method-dependent.","section":"Section 4.5"},{"comment":"No code or reproducibility link is provided; adding a code repository would improve the paper's reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The central issue is that c_g, the key component of the score, is selected on a labeled subset of the main evaluation benchmark, which compromises both the zero-shot framing and the reported performance gains. If the authors cannot demonstrate with a decontaminated evaluation that the advantage over IRM persists, the paper's main claim should be substantially weakened. I would advise the editor to require a re-run with prefix selection performed without access to DetectRL labels before considering acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"EchoPrompt is a simple extension of IRM: add a fixed assistant-style prefix to the text and average the log-likelihood difference between an instruct model and its base model. The new part is not the score—it is essentially IRM plus a constant offset term—but the empirical claim behind it: machine text gets a larger likelihood gain than human text from a generic 'you are a helpful assistant' prefix. That observation is supported in Figure 2 and Table 8 across several proxy families, and it is a genuinely useful finding. The paper also does a thorough evaluation: multiple proxy families, attack types, length bins, and it reports efficiency. The method is training-free, cheap, and easy to implement.\n\nThe soft spots are real but not fatal. The global prefix c_g was selected by maximizing AUROC on a 500+500 sample from DetectRL Multi-Domain (Appendix C), and the headline results are on DetectRL. That means the DetectRL numbers partly reflect selection on the evaluation distribution. The authors acknowledge the prefix is not shown to be globally optimal, but they do not address the benchmark-selection issue. The RealDet and RAID results use the same prefix, so they are out-of-distribution and provide some independent support, but they are part of the same average and do not by themselves validate the zero-shot claim. F1 thresholds appear to be chosen on the test set (Section 4.5), and no error bars or significance tests are given. On several proxy families the gain over IRM is large; on Llama-3.2-1B EchoPrompt actually loses to IRM, so 'consistent gains' is an overstatement, though on the main Llama-3-8B result the gain is clear.\n\nWho is this for? Anyone working on zero-shot machine-text detection will want to know the prefix-gain observation. The method itself is a modest increment, but the empirical finding about latent prompt compatibility could inform future detectors. It deserves a serious referee, but the referee should ask for code release and for a re-running of the prefix selection on a fully disjoint validation set, or at least a presentation of DetectRL as a prompt-tuned result rather than a held-out comparison.","headline":"EchoPrompt is IRM plus a hand-picked assistant prefix; the prefix-gain observation is real and useful, but the headline DetectRL numbers are inflated by benchmark-tuned prompt selection.","tokens_in":22582,"tokens_out":3699,"would_cite":true,"duration_ms":27691,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A generic assistant-style prefix makes machine-written text give itself away, and the effect can be scored to detect AI text.","keywords":["AI-generated text detection","zero-shot detection","latent prompt restoration","assistant-style prefix","instruction-tuned models","EchoPrompt","likelihood scoring","machine text attribution"],"falsifier":"Take a held-out set of machine texts generated from user prompts that explicitly request non-assistant styles (for example, \"write as a diary entry\" or \"write without an assistant voice\"), prepend the chosen $c_g$, and measure the EchoPrompt score; if the machine-side gain does not exceed the human-side gain on such prompts, the latent prompt-restoration claim is falsified. More directly, replacing $c_g$ with any of the alternative prefixes from Appendix C and re-running the three benchmarks should make the reported AUROC edge shrink or vanish if the effect is an artifact of that specific wording.","tokens_in":21560,"feed_emoji":"🤖","tokens_out":5294,"duration_ms":43534,"temperature":0.7,"pith_summary":"EchoPrompt claims that machine-written text carries a hidden trace of the prompt it was generated under: it was written as an assistant-style response. Because the original prompt is gone, earlier detectors miss this trace. EchoPrompt restores a generic assistant-response prefix and measures how much an instruction-tuned model's token probabilities rise relative to the untuned base model, averaging the gap into a single score. The paper reports that this score separates machine from human text better than existing zero-shot detectors, beating the strongest prior method by 0.69 AUROC and 2.64 F1 on average across DetectRL, RealDet, and RAID. A sympathetic reader would care because detection works without training data and survives attacks, length changes, and proxy-model swaps.","feed_headline":"A generic prefix makes AI text give itself away","feed_subtitle":"A no-training detector catches AI text by measuring how much a restored assistant context boosts its likelihood.","key_machinery":"The load-bearing object is the restored assistant-style context, a fixed task-agnostic prefix $c_g$ (\"You are a helpful, versatile, and intelligent AI assistant...\") chosen by preliminary experiments, together with the EchoPrompt score of Eq. (7): an average over tokens of the instruction-tuned model's log-likelihood of the text given the restored prefix minus the base model's log-likelihood of the raw text. The prefix does the work of reactivating a latent \"this is a response\" dependency; the base-model subtraction removes ordinary fluency; the average makes the signal stable at sequence level. A threshold on this score produces the final machine/human decision.","core_discovery":"The central discovery is that the dependency of generated text on its upstream prompt is not erased when the prompt is removed: it can be partially reactivated by prepending a single fixed assistant-style opening. Formally, for tokens $x_1,\\dots,x_n$, the EchoPrompt score is $\\frac{1}{n-1}\\sum_{t=2}^{n}(\\log P_{\\text{inst}}(x_t\\mid c_g,x_{<t})-\\log P_{\\text{base}}(x_t\\mid x_{<t}))$, where $c_g$ is the generic assistant prefix, $P_{\\text{inst}}$ is an instruction-tuned proxy model, and $P_{\\text{base}}$ is the same family's base model. The instruction-tuned term asks how naturally the text continues under a restored assistant-response context; the base-model term calibrates out ordinary fluency. The paper shows empirically that the gain from injecting $c_g$ is consistently larger for machine text than for human text across proxy families, and reports that the resulting score reaches the best zero-shot detection accuracy among the compared methods.","pith_inferences":["If the mechanism is real, a natural attack is to frame machine text with a human-style preamble before generation, which might reduce the machine-side gain; detectors should therefore be tested against adversarially chosen prefixes.","The dependence on a single hand-picked prefix suggests a testable extension: searching over prefixes per domain or per generator family could improve accuracy further, but it would need a selection protocol that avoids overfitting.","Because the score relies on instruction-tuning strength, it should degrade gracefully as base models become more assistant-like; comparing models with different post-training intensities could quantify that relationship.","The same latent-dependency signal may transfer to other conditional-generation artifacts, such as machine-translated or summarized text, where a \"response to a source\" dependency is also hidden."],"forward_implications":["Zero-shot AI-text detection can be built from prompt restoration alone, with no labeled data and no detector training.","The detector remains usable when the original prompt is unknown, which is the usual setting in practice.","Performance holds across attack types, text lengths, and proxy model families, with the largest gains on direct prompting, perturbation, and data mixing.","The method is fast enough for practical screening, with reported inference under 0.26 seconds per sample on the tested hardware.","The prompt-component ablation indicates the core signal comes from restoring the prompt–response relation, not merely from assigning the model an assistant role."],"supporting_citations":[{"why":"Supplies the implicit-reward comparison between base and instruction-tuned models that EchoPrompt extends, and is the strongest zero-shot baseline it must beat.","marker":"[21]"},{"why":"Provides the conditional-curvature baseline Fast-DetectGPT, one of the main efficiency-oriented competitors.","marker":"[4]"},{"why":"Provides the cross-model perplexity-ratio baseline Binoculars that EchoPrompt compares against.","marker":"[15]"},{"why":"Supplies the DetectRL benchmark, including the 500+500 subset used to select the generic prefix and the main evaluation sets.","marker":"[33]"},{"why":"Supplies the RAID benchmark used for robustness evaluation across attacks and generators.","marker":"[9]"},{"why":"Supplies the RealDet benchmark used to test out-of-distribution false-positive reliability.","marker":"[36]"}],"fun_headline_variants":["EchoPrompt: one prefix betrays AI text","AI text can't hide from a restored prompt","Detect AI text by asking it to continue","A fixed prefix flushes out machine text","Zero-shot AI text detection via prompt echo"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The detector's edge rests on a single generic prefix whose wording was chosen on a 500-human/500-machine sample of one benchmark; if that prefix does not reactivate the hidden conditioning for other prompts, domains, or generators, EchoPrompt's advantage over IRM shrinks or disappears.","fun_headline_variants_meta":{"raw":{"variants":["EchoPrompt: one prefix betrays AI text","AI text can't hide from a restored prompt","Detect AI text by asking it to continue","A fixed prefix flushes out machine text","Zero-shot AI text detection via prompt echo"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000156,"raw_usage":{"total_tokens":1216,"prompt_tokens":943,"completion_tokens":273,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":203}},"tokens_in":559,"tokens_out":273,"duration_ms":2962,"temperature":1.0,"reasoning_tokens":203,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:34:43.485100+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of machine texts generated from user prompts that explicitly request non-assistant styles (for example, \"write as a diary entry\" or \"write without an assistant voice\"), prepend the chosen $c_g$, and measure the EchoPrompt score; if the machine-side gain does not exceed the human-side gain on such prompts, the latent prompt-restoration claim is falsified. More directly, replacing $c_g$ with any of the alternative prefixes from Appendix C and re-running the three benchmarks should make the reported AUROC edge shrink or vanish if the effect is an artifact of that specific wording.","supporting_citations":[{"cited_title":"Spotting LLMs with binoculars: Zero-shot detection of machine-generated text","cited_arxiv_id":null,"evidence_quote":"Provides the cross-model perplexity-ratio baseline Binoculars that EchoPrompt compares against."},{"cited_title":"Detectrl: Benchmarking llm-generated text detection in real-world scenarios.Advances in Neural Information Processing Systems, 37:100369–100401, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the DetectRL benchmark, including the 500+500 subset used to select the generic prefix and the main evaluation sets."},{"cited_title":"Raid: A shared benchmark for robust evaluation of machine-generated text detectors","cited_arxiv_id":null,"evidence_quote":"Supplies the RAID benchmark used for robustness evaluation across attacks and generators."},{"cited_title":"Reliably bounding false positives: A zero-shot machine-generated text detection framework via multiscaled conformal prediction","cited_arxiv_id":null,"evidence_quote":"Supplies the RealDet benchmark used to test out-of-distribution false-positive reliability."}],"review_version":2}