{"id":"c4fc8509-835d-4bb9-b065-86177f2885c3","arxiv_id":"2602.14161","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Standard train/test evaluation overstates prompt-attack classifier generalization by 8+ AUC points; leave-one-dataset-out reveals dataset shortcuts that CV hides.","lead":"This paper measures how much standard evaluation overstates how well prompt-attack detectors generalize, by training on all but one attack dataset at a time. It reports that leave-one-dataset-out scores are 8-16 points lower than cross-validation and that many learned features are dataset shortcuts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 8.4-point CV-vs-LODO gap measures benchmark-to-benchmark transfer, not deployment shift; the paper's own limitation ('LODO assumes datasets represent meaningfully different distributions') is untested, so the headline overgeneralizes.","rationale":"The reader's weakest assumption—that LODO's 18 datasets stand in for deployment distributions—is the same concern I find most load-bearing. The 8.4-point number is an internal benchmark property; calling it 'true distribution shift' requires an external validity argument the paper does not make. The limitation is acknowledged but not tested. I considered the abstract's four-LLM claim, which is unsupported by the full text, but that is a scope/reproducibility problem that can be fixed by re-scoping; it does not change what the Llama-8B experiment shows. The representativeness concern, by contrast, determines whether the headline quantity means what the title says. I give credit where due: LODO is a well-established domain-generalization protocol, the DeLong CIs are reported, the ablation and sensitivity analyses are honest, and the paper explicitly states that shortcut ablation does not explain the gap. Those strengthen the narrow methodological finding. The proposed test would either validate the deployment relevance of the 8.4-point gap or force a more modest claim.","tokens_in":19262,"tokens_out":10665,"duration_ms":104186,"concrete_test":"Use the released framework to recompute the headline gap on the six naturally mixed-class datasets only (BIPIA, deepset, jayavibhav, qualifire, safeguard, wildjailbreak), excluding the 11 single-class synthetic sets. If the pooled CV-minus-LODO AUC drops below, say, 3 points, then the 8.4-point headline is largely a benchmark-composition artifact. To test deployment relevance directly, additionally evaluate the same logistic-regression probe on a separately collected, annotated set of real agent prompts; if the CV-to-deployment AUC gap is not within a few points of the CV-to-LODO gap, the paper should re-scope its claim to 'leave-one-dataset-out on this benchmark' rather than 'true distribution shift.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claim is that standard CV overestimates 'true' out-of-distribution generalization by 8.4 points (0.996 vs 0.912 AUC). What LODO actually measures is transfer from 17 benchmark datasets to the 18th benchmark dataset. The leap from that to 'true distribution shift' depends on the untested assumption that these 18 curated datasets (11 of which are 100% single-class) are representative of the distribution shift a deployed detector will face. The Limitations section explicitly concedes: 'LODO assumes datasets represent meaningfully different distributions.' No evidence is offered that this holds; the benchmark contains no production agent traffic, and several datasets are synthetic red-team artifacts. Supporting this concern, Table 13 shows ablating all 14 identified shortcuts changes pooled LODO AUC by only 0.1pp, so the gap is not explained by the specific shortcut mechanism the paper emphasizes—it reflects broad distributional differences among artificial benchmarks. Heterogeneous per-dataset gaps (1.2–25.4%) reinforce that the aggregate number is a composition-weighted property of this particular benchmark. The recommendation to adopt LODO as 'the appropriate protocol' may still be sound, but the magnitude 8.4 and the word 'true' are not calibrated to deployment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Leave-One-Dataset-Out (LODO) evaluation for prompt-injection and jailbreak classifiers, arguing that standard cross-validation and same-dataset held-out splits overestimate out-of-distribution performance. Using linear probes on Llama-3.1-8B activations (and an SAE decomposition) over a benchmark of 18 datasets, it reports a pooled AUC of 0.996 under 5-fold CV versus 0.912 under LODO, an 8.4 percentage point gap, with per-dataset accuracy gaps between 1.2% and 25.4%. The paper then analyzes SAE feature coefficients to identify dataset shortcuts, reports that 28% of top features are dataset-dependent, and proposes LODO-weighted explanations to downweight such features. It also compares against PromptGuard 2, LlamaGuard, and Llama-as-judge, finding these baselines fail on indirect and agentic attacks. The central claim is that LODO is the appropriate protocol for measuring true out-of-distribution generalization in this domain.","tokens_in":19631,"tokens_out":9090,"duration_ms":75732,"significance":"If the central claim is accepted, the paper makes a useful methodological contribution: demonstrating that standard evaluation protocols can inflate prompt-security classifier performance by a substantial margin, and providing a concrete alternative protocol plus diagnostic tools based on LODO retention. The formalization of LODO in Eq. (4), the large multi-source benchmark (105K samples), the explicit per-dataset breakdown, the sensitivity analysis in Appendix D, and the public code release are all strengths. The shortcut-retention metric and the LODO-weighted explanation procedure are interesting and potentially reusable beyond this specific task. However, the paper's headline conclusions—'true distribution shift' and a multi-LLM cross-family generalization—are not fully supported by the evidence actually reported, and the gap magnitude is tied to specific configuration choices. The core idea is defensible, but the presentation and scope need substantial revision.","major_comments":[{"comment":"The abstract states that standard CV reports pooled AUC 8.0–16.5 points higher than LODO 'across four LLMs from three families (Llama-3.1-8B, Gemma-3-27B, Qwen-3.5-2B/4B)'. The experiments in the manuscript use only Llama-3.1-8B-Instruct (Section 4.1) and Llama-3.1-70B (Appendix E). No Gemma or Qwen results appear anywhere in the full text. This is a load-bearing overgeneralization: it converts a single-family result into a cross-family claim. Either add the claimed experiments or revise the abstract and all downstream claims to the supported Llama-only scope.","section":"Abstract; Section 4.1; Appendix E"},{"comment":"The paper repeatedly calls LODO a measure of 'true out-of-distribution generalization' and the abstract/conclusion frame the 8.4-point gap as evidence of benchmark deception about deployment behavior. The paper's own Limitations section concedes that 'LODO assumes datasets represent meaningfully different distributions', and no evidence is offered that the 18 curated benchmark datasets resemble production agent traffic. Moreover, Table 13 shows that ablating all identified shortcuts changes pooled LODO AUC by only 0.1pp, and Section 5.3 explicitly says the shortcut analysis is 'not explanatory' of the gap. Thus the aggregate gap is a composition-weighted property of this benchmark, not a demonstrated 'true' deployment shift. Please reframe the central claim as benchmark-to-benchmark transfer, or provide an external validation corpus to calibrate the gap.","section":"Section 3.3; Section 6 Limitations; Table 13"},{"comment":"The headline 8.4-point gap is computed for one configuration: raw activations at layer 31 and token position −5, selected 'for simplicity'. Appendix D shows that per-dataset LODO accuracy varies dramatically with layer and position (e.g., BIPIA 7% at L19 vs 63% at L31; llmail 29% vs 71% across positions at L31). The text asserts that the core finding persists across configurations, but no CV-vs-LODO gap is reported for the other configurations. The specific magnitude 8.4 is therefore configuration-dependent. Please report the CV-LODO gap across the layer/position configurations in Appendix D, or clearly state that only the qualitative gap is robust.","section":"Section 3.2; Appendix D"}],"minor_comments":[{"comment":"The text says 'Ablating all 14 shortcuts' but Table 13's last row is 'All (26)'. Clarify whether 14 or 26 features are ablated and reconcile the definition used in the main text with the appendix sensitivity analysis.","section":"Section 5.3; Table 13"},{"comment":"The BIPIA row is unreadable: '15000 9563.126.1 60.1' likely means N=15000, %Mal=95, Raw=63.1, SAE=26.1, but the lack of spacing makes it appear as a single number. Fix the table formatting.","section":"Table 3"},{"comment":"The manuscript mentions 18 datasets in the abstract and Table 6, but LODO evaluation merges Gandalf into Mosscap to yield 17 datasets. State explicitly that all LODO results use 17 datasets, to avoid apparent inconsistency.","section":"Abstract; Section 4.1"},{"comment":"The row 'CV-LODO Gap 8.4%' is labeled as a percentage, but the text correctly calls it a percentage-point inflation. Use 'pp' throughout to avoid unit confusion.","section":"Table 1"},{"comment":"The 70B comparison claims that 'the CV-LODO gap persists', but no CV or held-out AUC numbers are provided for 70B. Add the comparison values or soften the claim to 'per-dataset variation persists'.","section":"Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the discrepancy between the abstract's cross-LLM claim and the Llama-only experiments; this must be fixed before the paper can be considered. The 'true distribution shift' framing is also stronger than the evidence supports, though the underlying LODO protocol is a reasonable methodological proposal. If the authors add the missing multi-LLM results or remove the overclaim, and explicitly bound the headline gap to the reported configuration, the paper could be a useful contribution to the benchmark-evaluation literature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the methodological point is sound — cross-validation on same-source dataset splits overestimates how well prompt attack probes transfer to a held-out dataset. But the 8.4-point number is a property of this 18-dataset benchmark, not a measured deployment gap, and the abstract claims more than the full text currently delivers. Still worth a serious referee.\n\nWhat's actually new: the first systematic application of leave-one-dataset-out evaluation to activation-based prompt attack classifiers, plus a useful taxonomy of SAE features that are stable versus dataset-specific. The comparison with production guardrails and LLM-as-judge is also useful. The paper correctly credits the domain-generalization and shortcut-learning literature; it doesn't pretend LODO is new.\n\nWhere it's solid: the protocol is clearly defined, the main Llama-3.1-8B experiments support the direction of the claim, the per-dataset table makes the heterogeneity concrete, and the limitations section is unusually honest. Table 13 shows that ablating the identified shortcuts barely moves pooled LODO AUC, and the paper openly says the shortcut analysis is diagnostic, not explanatory. That is the right kind of self-assessment.\n\nSoft spots, in order of importance:\n\n1. The abstract claims results across four LLMs from three families and across several domain-generalization fixes, but the full text reports Llama-3.1-8B in the main body, an appendix run on 70B, and no Gemma/Qwen results. The adversarial training, reweighting, and class balancing experiments are also only described as preliminary. Claims need to match the evidence.\n\n2. \"True distribution shift\" is too strong. LODO measures transfer from 17 benchmark datasets to the 18th. Whether that approximates deployed traffic is an assumption, and the paper itself lists it in Limitations. It's a reasonable assumption for a methodological caution, but not for the title. The 8.4-point gap should be read as benchmark-to-benchmark transfer, not calibrated production risk.\n\n3. The shortcut analysis is real but not load-bearing for the main gap. Ablating all shortcuts changes pooled LODO AUC by only 0.1pp, so the gap remains mostly broad distributional difference rather than the specific features named. The paper says this, but the framing in the abstract and conclusion overstates the explanatory role.\n\n4. The explanation validation is partly self-referential: LODO-weighted attributions \"surface relevant features\" because relevance is defined by LODO retention. Fine as a diagnostic, but not independent evidence of interpretability quality.\n\n5. Minor: layer and token position choices matter; the paper reports sensitivity and doesn't hide it. Reproducibility artifacts lack a commit hash and complete model coverage, but that's fixable.\n\nWho it's for: anyone evaluating or building prompt injection and agentic security detectors. It deserves serious peer review, with a request to align claims with evidence and soften the deployment language.","headline":"The LODO evaluation idea is worth taking seriously for prompt attack classifiers, but the abstract claims more than the full text supports, and the headline 8.4-point gap is a benchmark-composition result, not a measured deployment gap.","tokens_in":20064,"tokens_out":2744,"would_cite":true,"duration_ms":26930,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Standard cross-validation overestimates prompt-attack classifier generalization: leaving an entire dataset out of training drops pooled AUC from 0.996 to 0.912, an 8.4-point gap.","keywords":["prompt injection detection","jailbreak detection","out-of-distribution generalization","leave-one-dataset-out evaluation","dataset shortcuts","sparse autoencoders","activation probing","LLM safety"],"falsifier":"Collect real deployment traffic (or a faithful proxy) for an agent that processes emails, documents, and tool outputs, run the same probe, and compare its actual AUC to both the CV estimate and the LODO estimate; if deployed AUC tracks CV rather than LODO, the shortcut explanation is wrong. A second check: remove all single-class datasets from the benchmark and retrain—if the CV-LODO gap collapses, the gap is an artifact of single-class benchmark design rather than a general out-of-distribution phenomenon.","tokens_in":19176,"feed_emoji":"🛡️","tokens_out":8163,"duration_ms":64132,"temperature":0.7,"pith_summary":"The paper is trying to establish that reported near-perfect results for prompt-injection and jailbreak detectors are an artifact of how benchmarks are split. When training and test samples come from the same datasets, classifiers can memorize dataset-specific formats instead of learning attack semantics. Replacing the standard split with Leave-One-Dataset-Out evaluation—training on all but one dataset and testing on the unseen one—drops pooled AUC from 0.996 to 0.912 for a logistic-regression probe on raw activations, with per-dataset accuracy losses from 1 to 25 points across four LLMs from three families. The authors further show that 28% of top sparse-autoencoder features are dataset shortcuts and that a dataset-identity classifier reaches 96.6% accuracy, confirming why the benchmark is so easy to game. The authors argue LODO should be the standard evaluation protocol for prompt-attack detection.","feed_headline":"Leave-one-dataset-out cuts prompt-attack detector AUC by 8 points","feed_subtitle":"Same-source test sets hide dataset shortcuts; whole-dataset holdouts reveal the true, lower performance.","key_machinery":"The load-bearing mechanism is Leave-One-Dataset-Out (LODO) evaluation: for each dataset D_i in a benchmark of K datasets, train a classifier on the other K−1 and test on D_i; pooling all held-out predictions gives the out-of-distribution metric. The companion diagnostic is the LODO coefficient retention r_j = min_i (w_j^{(-i)} / w_j), the minimum ratio between a feature's classifier coefficient when dataset i is held out and its full-data coefficient. Features with r_j ≈ 1 are stable across holdouts; features with r_j ≪ 1 or negative (sign flips) are dataset shortcuts. A dataset-identity classifier trained on SAE features (96.6% accuracy) shows why the shortcuts exist: datasets form separabl","core_discovery":"The central discovery is that standard evaluation protocols—both 5-fold cross-validation and same-source held-out test sets—systematically overestimate out-of-distribution performance of activation-based prompt-attack classifiers. On 18 benchmark datasets (105K samples, 47% malicious), logistic regression on layer-31 raw activations reports 0.996 AUC under cross-validation and 0.997 on a held-out test set, but only 0.912 pooled AUC under LODO. Per-dataset gaps between same-source held-out tests and LODO range from 1.2% (safeguard) to 25.4% (jayavibhav), showing heterogeneous failure modes. The paper attributes the gap to dataset shortcuts: features identifying dataset provenance rather than","pith_inferences":["If LODO becomes the reporting standard, many published detector results will need downward revision; a detector's practical value may hinge on its worst held-out dataset rather than its pooled AUC.","The 96.6% dataset-identity accuracy implies the gap is largely a property of single-class benchmark sources; a testable fix is building datasets with overlapping formats and multiple sources per attack type so provenance is not a class signal.","The retention metric could be applied to other aggregated benchmarks (toxicity, spam, medical QA) where single-class datasets make shortcut learning trivial.","The paper's own layer/position sensitivity results (per-dataset accuracy swings up to 42 points, e.g., BIPIA 7% at layer 19 vs 63% at layer 31) suggest LODO can double as a hyperparameter-selection tool, picking layers and token positions by their held-out stability rather than in-distribution accuracy."],"forward_implications":["Near-perfect AUCs (>0.99) reported for prompt-attack detectors should be re-read as upper bounds; LODO should be reported alongside cross-validation before deployment.","The 8.4-point pooled gap and 1–25-point per-dataset gaps persist across four LLMs from three families, indicating the problem is benchmark composition, not a single model's quirk.","Production guardrails detect only 7–37% of indirect and agentic injections, while the proposed activation probes trained under LODO reach 68–99% on those categories.","LODO retention scores filter SAE-feature attributions: 98.1% of per-prompt explanations change rank, with demoted features having mean retention 0.265 versus 0.990 for promoted ones.","Common domain-generalization fixes—adversarial training, subspace projection, sample reweighting, and class balancing—do not close the gap, so closing it needs new methods."],"fun_headline_variants":["When benchmarks lie: prompt-attack AUC inflated by 8-16.5 pts","LODO evaluation reveals prompt-attack detectors overrated by 8+ AUC","Cross-validation masks prompt-attack OOD failure by 8-16.5 AUC"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The 8.4-point gap estimate assumes that holding out an entire benchmark dataset reproduces the distribution shift a deployed detector actually faces; the paper itself notes that LODO assumes the 18 datasets represent meaningfully different distributions, so if they do not resemble production traffic, the gap measures benchmark composition rather than deployment generalization.","fun_headline_variants_meta":{"raw":{"variants":["When benchmarks lie: prompt-attack AUC inflated by 8-16.5 pts","LODO evaluation reveals prompt-attack detectors overrated by 8+ AUC","Cross-validation masks prompt-attack OOD failure by 8-16.5 AUC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000738,"raw_usage":{"total_tokens":3186,"prompt_tokens":848,"completion_tokens":2338,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":2269}},"tokens_in":592,"tokens_out":2338,"duration_ms":16945,"temperature":1.0,"reasoning_tokens":2269,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T23:16:40.894478+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect real deployment traffic (or a faithful proxy) for an agent that processes emails, documents, and tool outputs, run the same probe, and compare its actual AUC to both the CV estimate and the LODO estimate; if deployed AUC tracks CV rather than LODO, the shortcut explanation is wrong. A second check: remove all single-class datasets from the benchmark and retrain—if the CV-LODO gap collapses, the gap is an artifact of single-class benchmark design rather than a general out-of-distribution phenomenon.","supporting_citations":[],"review_version":1}