{"id":"2d126388-d7ed-4f8f-8073-a8faf6ac74b4","arxiv_id":"2608.02665","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Evaluating LLM safety with one canonical prompt understates unsafe behavior; across five meaning-preserving reformulations, 5-13% of safe-on-canonical seeds become unsafe, and the union exceeds the worst single form for all five models.","lead":"This paper tests whether safety benchmarks that read each prompt once in a single canonical wording give a fair picture of model safety. It finds that meaning-preserving reformulations such as translation and code-switching expose additional unsafe behavior in 5-13% of prompts, so single-form scores are optimistically biased.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero stochasticity floor is measured on only two of five models and only for the canonical prompt; if label flips occur for the other three models or for reformulated forms, part of the 5–13% new exposure is decoding/judge noise, not surface-form signal.","rationale":"The paper's central claim is a measurement-validity claim: a single canonical prompt is an optimistic estimator because unioning five meaning-preserving surface forms reveals 5.4–13.0% new unsafe exposure that cannot be explained by run-to-run stochasticity. The load-bearing control is the zero stochasticity floor in Section 4.2. The reader's weakest assumption correctly identifies that the floor is measured on only two of five models. I agree with that, and I add that the floor is also measured only for the canonical prompt, not for the reformulated forms that actually generate the new exposure. This matters because new exposure is defined on reformulations; if those forms have higher within-form label instability than canonical, some of the observed effect is noise. The paper acknowledges the model-scope limitation in its Limitations section, but the abstract and Table 3 caption overgeneralize it. This is not an internal inconsistency, and it is not grounds for rejection: the union-minus-canonical gaps are large enough that a small nonzero floor would likely not eliminate the effect, and the authors already disclose the limitation. The proposed test—resampling all forms on all models—would settle whether any correction is needed. Since the reader's CONDITIONAL verdict already reflects this uncertainty, my read does not change the verdict.","tokens_in":11532,"tokens_out":6924,"duration_ms":77851,"concrete_test":"Extend the stochasticity-floor measurement to all five models and all five surface forms. For each model, resample each of the five prompts (canonical, paraphrase, translation_zh, code_switch_zh_en, indirect_framing) five times at temperature 0, score with the same judge, and compute (i) the per-form label-flip rate and (ii) the floor for new exposure as the union across the repeated samples of each form. If the floor remains 0/370 for every model and form, the attribution stands. If not, recompute Table 3's new-exposure estimates after subtracting the per-model/per-form floor and report whether the Wilson CIs still exclude zero; also re-check Table 2's union-minus-worst gap with the noise-adjusted labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that canonical-prompt evaluation underestimates unsafe compliance by 5.4–13.0 pp—depends on attributing all 'new exposure' to surface form after subtracting a zero stochasticity floor (Section 4.2, Tables 2–3). But the floor is measured by resampling only the canonical prompt five times at temperature 0, and only on two of the five evaluated models (GPT-4o-mini, DeepSeek-V3; see Limitations). The abstract and Table 3 caption present this as 'the stochasticity floor' without that model restriction. If either (a) any of the three untested models (GPT-4o, Gemini 2.5 Flash, Gemini 2.5 Pro) has a nonzero label-flip rate when the same canonical prompt is resampled, or (b) any reformulated form (translation_zh, code_switch_zh_en, etc.) has a higher within-form label-flip rate than the canonical form does, then part of the reported new-exposure fraction is decoding/judge stochasticity rather than surface-form signal. The paper honestly discloses the floor's limited scope in Limitations, but the headline numbers and abstract do not carry that caveat. The largest reported new-exposure gap (Gemini 2.5 Pro, 13.0%) is on an untested model, so the magnitude of the central effect could shrink, even if it would likely not vanish completely.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates whether single-canonical-prompt safety benchmarks faithfully estimate LLM unsafe compliance. The authors construct 370 harmful seeds, generate four pre-authored reformulations (pivot back-translation paraphrase, Chinese translation, programmatic code-switching, and LLM indirect framing), run five frontier models at temperature 0, and score all 1,850 responses with a single vendor-neutral LLM judge (Claude) human-anchored on 185 items and cross-checked by GPT-4o on 250. They find no transformation is uniformly most dangerous, but the union of unsafe outcomes across the five forms exceeds the worst single form by 3.3–12.9 pp with seed-level bootstrap 95% CIs excluding zero for all five models, and 5.4–13.0% of seeds safe on canonical are unsafe under some reformulation. They attribute this new exposure to surface-form signal, above a 'zero stochasticity floor' measured by resampling the canonical prompt five times at temperature 0 on two of the five models.","tokens_in":11914,"tokens_out":8880,"duration_ms":95602,"significance":"If the claims hold, the paper advances evaluation methodology: safety scores should be treated as measurements with instrument bias, and robustness evaluation should report union/worst-case quantities over meaning-preserving surface forms rather than a single canonical point estimate. The strengths are real: pre-authored, mostly non-LLM perturbations sent identically to every model avoid meta-instruction confounds; the judge is vendor-neutral, human-anchored, and cross-checked by a second vendor; the paper verifies intent preservation; seed-level bootstrap CIs are used; and dataset, code, and per-response labels are released. The main caveat is that the noise-attribution claim — the 'above a zero floor' assertion — is directly supported on only two of five models and only for the canonical form, so the magnitude of the surface-form-specific signal for the other three models is not yet established. The union-minus-worst result, however, is independent of the floor and is the most robust contribution.","major_comments":[{"comment":"The stochasticity floor is measured only on GPT-4o-mini and DeepSeek-V3 (canonical prompt resampled five times at temperature 0), yet the abstract and Table 3 apply 'above a zero stochasticity floor' to all five models, and Section 4.2 concludes the cross-form new exposures are 'therefore surface-form-driven.' If any of the untested models (GPT-4o, Gemini 2.5 Flash, Gemini 2.5 Pro) has a nonzero label-flip rate under canonical resampling, or if any reformulated form has a higher within-form label-flip rate than canonical, part of the reported 5.4–13.0% new exposure is decoding/judge noise rather than surface-form signal. The largest new-exposure gap (Gemini 2.5 Pro, 13.0%) is on an untested model. The Limitations section honestly discloses the scope, but the headline claims do not carry the restriction. Please either extend the floor measurement to all five models and to the reformulated","section":"Abstract; Section 4.2; Table 3 caption; Limitations"}],"minor_comments":[{"comment":"The same Claude judge rates both intent preservation and unsafe compliance. The conclusion that indirect framing's lower severity is 'partly an artifact of intent neutralization' is therefore self-referential: the judge's own intent rating is used to explain the judge's own compliance rating. Independent human or second-vendor intent labels would strengthen this decomposition; the paper already flags this in Limitations, and the robustness check on intent-preserving forms mitigates the impact on the union result.","section":"Section 4.6; Limitations"},{"comment":"The benign control (XSTest) is not item-matched to the harmful pool, and the paper appropriately labels bidirectionality as secondary. Consider adding OR-Bench/PHTest or item-matched benign/harmful pairs if bidirectionality is to be promoted to a core claim.","section":"Section 4.3; Table 3"},{"comment":"Typesetting issue: 'zerostochasticityfloor' and similar concatenations lack word spacing in the abstract; also the abstract's stochasticity-floor parenthetical should name the two-model scope.","section":"Abstract"},{"comment":"Aggregate rates are reported without confidence intervals or per-cell counts; adding Wilson intervals would help readers assess the underpowered per-transformation comparisons.","section":"Table 1"},{"comment":"The McNemar power limitation is acknowledged; reporting exact discordant-pair counts and effect directions for all 20 tests in an appendix would make the 'no single transformation is uniformly most dangerous' claim easier to evaluate.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for cs.CR and makes a useful methodological contribution. The central union-vs-worst result is well supported; the main weakness is the overgeneralization of the stochasticity floor to models on which it was not measured. With additional floor measurements or careful restriction of claims, I would be happy to see this accepted. I would not reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short take: this paper earns its central claim. The union of five meaning-preserving reformulations exceeds the worst single form by 3.3–12.9 pp with bootstrap CIs excluding zero on all five models, and the result survives the paper's own noise floor and intent-preservation checks reasonably well. The protocol is the real contribution: pre-authored, mostly non-LLM reformulations sent identically to every model; a vendor-neutral judge anchored to a human subset and cross-checked by a second vendor; an explicit stochasticity floor; and a benign control. That is a genuinely careful measurement setup, and the paper ships the data, code, and per-response labels.\n\nWhat is new: prior work showed single transformations can jailbreak (translation, code-switching) and that multi-prompt evaluation matters for general NLP. This paper adds a confound-controlled protocol that separates surface-form signal from decoding/judge noise and shows the effect is distributional rather than transformation-specific: no single form is uniformly most dangerous, yet canonical-only evaluation is optimistic in both directions (missed unsafe compliance and missed over-refusals). The coverage curve ('one form ~53%, three forms ~85%') is descriptive and honestly framed as such.\n\nSoft spots, in proportion. The stochasticity floor is measured on two of five models, but the abstract and tables treat it as global. That is a real overreach, though a minor one: even if the other three models have a nonzero flip rate, it is unlikely to erase the union gap, but the 13% figure on Gemini 2.5 Pro could shrink. The intent-preservation check is done by the same judge that scores unsafe compliance, so the Section 4.6 claim that indirect framing's low severity is partly intent neutralization is self-referential; the paper admits this in Limitations. The benign/harmful pools are not item-matched, so bidirectionality is suggestive, not proven. All three are disclosed. None undermines the core result, but all should be tightened in revision.\n\nWho this is for: safety evaluation researchers and benchmark builders. It is a measurement-quality paper, not an attack paper. It deserves a serious referee: send it out, ask the authors to extend the floor to all five models, use an independent judge for intent preservation (or multi-annotator human ratings), and add item-matched benign controls. The central argument holds up and the artifacts are useful.","headline":"A careful measurement study that earns its central claim—canonical prompts are optimistic estimators of unsafe compliance—with one disclosed limitation (noise floor on two of five models) that should not block publication.","tokens_in":12360,"tokens_out":1847,"would_cite":true,"duration_ms":21574,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single canonical phrasing undercounts LLM unsafe compliance by up to 13 percentage points.","keywords":["LLM safety","benchmark validity","canonical prompt bias","surface-form sensitivity","meaning-preserving reformulation","unsafe compliance","over-refusal","stochasticity floor"],"falsifier":"Re-run the exact same canonical prompts five times at temperature 0 on the other three models (GPT-4o, Gemini 2.5 Flash, Gemini 2.5 Pro) and count how often a seed that was safe on the first run is unsafe on a repeat; a nonzero flip rate on any of those models, especially one large enough to cover the claimed new-exposure gap, would undercut the attribution to surface form.","tokens_in":11496,"feed_emoji":"🛡️","tokens_out":10013,"duration_ms":98889,"temperature":0.7,"pith_summary":"Safety benchmarks typically present each harmful request in one canonical English phrasing and report the refusal rate as if it were a property of the model. This paper argues that such a reading is a biased measurement instrument: when harmful intent is held fixed and only meaning-preserving surface form varies, different seeds fail under different phrasings, so the union of unsafe outcomes across five forms exceeds even the worst single form by 3.3–12.9 percentage points for all five models tested, and 5–13% of seeds that look safe on the canonical prompt turn out to be unsafe under some reformulation. The paper's protocol removes the usual confounds by pre-authoring all reformulations as fixed strings (mostly non-LLM: machine back-translation, Chinese translation, and a rule-based code-switch generator), scoring every response with one human-anchored, vendor-neutral judge, and checking intent preservation. A resampling control at temperature 0 produced zero label flips, so the effect is attributed to surface form rather than decoding/judge noise; a benign control suggests the instability is bidirectional, with comparable new over-refusals. If the claim holds, single-prompt safety scores are optimistic lower bounds on real-world unsafe behavior, and roughly three meaning-preserving forms would recover about 85% of the observed unsafe surface.","feed_headline":"Canonical prompts hide up to 13% of unsafe LLM answers","feed_subtitle":"A benchmark that reads one phrasing misses 5-13% of unsafe behavior and hides over-refusal too.","key_machinery":"The load-bearing mechanism is a pre-authored, refusal-free reformulation pipeline plus a stochasticity floor. Four meaning-preserving forms are generated as fixed strings—distant-pivot machine back-translation as paraphrase, machine translation to Chinese, a rule-based Matrix-Language-Frame code-switch generator (English matrix with Chinese noun phrases), and one LLM-produced indirect framing—so the identical surface form reaches every model and reformulation ability is not confounded with safety. A single human-anchored, vendor-neutral judge scores every response (kappa 0.86 vs. human on unsafe compliance), with an intent-preservation check. The stochasticity floor, measured by resampling t","core_discovery":"The central claim is that a single canonical surface form is an optimistically biased estimator of a model's unsafe-compliance surface. On 370 harmful seeds × 5 surface forms × 5 models, per-transformation effects are small and mostly non-significant or protective, but per-seed effects are idiosyncratic: seeds safe on canonical become unsafe under some reformulation in 5.4–13.0% of cases, and the union of unsafe outcomes across the five forms exceeds even the worst single form by 3.3–12.9 percentage points, with bootstrap 95% CIs excluding zero for every model. A stochasticity floor—re-running canonical prompts five times at temperature 0—produced 0/370 label flips on the two models tested,","pith_inferences":["If the zero stochasticity floor does not hold on the other three models, the magnitude of the surface-form effect shrinks; the paper's Limitations concede the floor was measured only on GPT-4o-mini and DeepSeek-V3, so re-measuring on all five is the most direct extension.","Because three of the four reformulations share the Chinese machine-translation pipeline, the coverage curve (one form ≈ 53%, three ≈ 85%) likely overstates the independent information added by each additional form; a mechanistically diverse form set—syntax-only restructuring, a different language family, a different paraphraser—would give a fairer redundancy estimate.","The union estimand is a lower-bound rather than a population estimate; a stronger claim would require sampling surface forms from a defined distribution, which the authors explicitly do not do.","The protocol could transfer to non-safety benchmarks with no gold label—capability, instruction-following, or reasoning—where single-prompt reading is also the norm and where a union estimand plus a noise floor would reveal how much of the score is instrument artifact."],"forward_implications":["Single-form safety scores are lower bounds: the five-form union is 1.3–2.2× the canonical rate, so a point estimate from one phrasing understates exposure.","A budget of about three meaning-preserving forms recovers roughly 85% of the observed five-form unsafe surface, making multi-form probing a practical check.","Reporting refusal and compliance rates alone hides per-seed instability; decomposed consistency (all-safe, all-unsafe, mixed) should accompany aggregate scores.","The canonical instrument is also optimistic about over-refusal: benign prompts flipped to refusal in 6–18% of cases, so single-form evaluation hides both safety failures and excess caution.","The underestimation is model-dependent and largest where baseline risk is already high, so single-form scores can misorder models relative to each other."],"supporting_citations":[{"why":"Supplies HarmBench, one of the two seed pools for the 370 harmful intents, and the validated-judge standard the protocol adopts.","marker":"[Mazeika et al., 2024]"},{"why":"Supplies AdvBench, the other seed pool for harmful intents.","marker":"[Zou et al., 2023]"},{"why":"Prior result that translation bypasses safety; the paper reconciles its mostly-null per-form finding with this attack-framed baseline.","marker":"[Yong et al., 2023]"},{"why":"Precedent for code-switched red-teaming, which the paper turns into a rule-based, refusal-free surface form.","marker":"[Yoo et al., 2025]"},{"why":"Basis for the stochasticity floor: LLMs are not text-deterministic even at temperature 0, so label-level stability must be measured.","marker":"[Zhou et al., 2026]"},{"why":"Best-of-N jailbreaking result used to argue the five-form union is a conservative lower bound on deployment-time exposure, not an upper bound.","marker":"[Hughes et al., 2024]"},{"why":"Supplies XSTest, the benign-prompt control for detecting new over-refusals.","marker":"[Röttger et al., 2024]"},{"why":"General evidence that semantically equivalent prompt formats change model performance, motivating the measurement stance imported into safety.","marker":"[Sclar et al., 2024]"},{"why":"Supplies the multi-prompt distributional evaluation framing that the paper adapts to safety with a union estimand.","marker":"[Mizrahi et al., 2024]"},{"why":"Provides the measurement-and-fairness framing that treats benchmark scores as instruments with bias and noise.","marker":"[Jacobs and Wallach, 2021]"}],"fun_headline_variants":["One phrasing hides up to 13% of unsafe LLM answers","Rephrasing finds 13% more unsafe behavior than single prompts","Safety scores are surface-sensitive: single prompts underreport risk","Up to 13% of unsafe answers missed by canonical prompt","Benchmark phrasings expose 5-13% more unsafe model behavior"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The claim rests on the assumption that re-running the exact same prompt five times never changes a model's safety label, measured on only two of the five models; if the other three models sometimes flip labels when nothing changes, part of the reported 5–13% 'new unsafe exposure' could be noise, not surface-form sensitivity.","fun_headline_variants_meta":{"raw":{"variants":["One phrasing hides up to 13% of unsafe LLM answers","Rephrasing finds 13% more unsafe behavior than single prompts","Safety scores are surface-sensitive: single prompts underreport risk","Up to 13% of unsafe answers missed by canonical prompt","Benchmark phrasings expose 5-13% more unsafe model behavior"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000369,"raw_usage":{"total_tokens":1911,"prompt_tokens":935,"completion_tokens":976,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":679,"completion_tokens_details":{"reasoning_tokens":886}},"tokens_in":679,"tokens_out":976,"duration_ms":12058,"temperature":1.0,"reasoning_tokens":886,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T00:42:51.298774+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the exact same canonical prompts five times at temperature 0 on the other three models (GPT-4o, Gemini 2.5 Flash, Gemini 2.5 Pro) and count how often a seed that was safe on the first run is unsafe on a repeat; a nonzero flip rate on any of those models, especially one large enough to cover the claimed new-exposure gap, would undercut the attribution to surface form.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies HarmBench, one of the two seed pools for the 370 harmful intents, and the validated-judge standard the protocol adopts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Precedent for code-switched red-teaming, which the paper turns into a rule-based, refusal-free surface form."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the multi-prompt distributional evaluation framing that the paper adapts to safety with a union estimand."}],"review_version":1}