REVIEW 1 major objections 5 minor 21 references
A single canonical phrasing undercounts LLM unsafe compliance by up to 13 percentage points.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-05 00:42 UTC pith:6HQECEZM
load-bearing objection A careful measurement study that earns its central claim—canonical prompts are optimistic estimators of unsafe compliance—with one disclosed limitation (noise floor on two of five models) that should not block publication. the 1 major comments →
Single Canonical Prompts Underestimate LLM Safety's Surface-Form Sensitivity
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that a single canonical surface form is an optimistically biased estimator of a model's unsafe-compliance surface. On 370 harmful seeds × 5 surface forms × 5 models, per-transformation effects are small and mostly non-significant or protective, but per-seed effects are idiosyncratic: seeds safe on canonical become unsafe under some reformulation in 5.4–13.0% of cases, and the union of unsafe outcomes across the five forms exceeds even the worst single form by 3.3–12.9 percentage points, with bootstrap 95% CIs excluding zero for every model. A stochasticity floor—re-running canonical prompts five times at temperature 0—produced 0/370 label flips on the two models tested,
What carries the argument
The load-bearing mechanism is a pre-authored, refusal-free reformulation pipeline plus a stochasticity floor. Four meaning-preserving forms are generated as fixed strings—distant-pivot machine back-translation as paraphrase, machine translation to Chinese, a rule-based Matrix-Language-Frame code-switch generator (English matrix with Chinese noun phrases), and one LLM-produced indirect framing—so the identical surface form reaches every model and reformulation ability is not confounded with safety. A single human-anchored, vendor-neutral judge scores every response (kappa 0.86 vs. human on unsafe compliance), with an intent-preservation check. The stochasticity floor, measured by resampling t
Load-bearing premise
The claim rests on the assumption that re-running the exact same prompt five times never changes a model's safety label, measured on only two of the five models; if the other three models sometimes flip labels when nothing changes, part of the reported 5–13% 'new unsafe exposure' could be noise, not surface-form sensitivity.
What would settle it
Re-run the exact same canonical prompts five times at temperature 0 on the other three models (GPT-4o, Gemini 2.5 Flash, Gemini 2.5 Pro) and count how often a seed that was safe on the first run is unsafe on a repeat; a nonzero flip rate on any of those models, especially one large enough to cover the claimed new-exposure gap, would undercut the attribution to surface form.
If this is right
- Single-form safety scores are lower bounds: the five-form union is 1.3–2.2× the canonical rate, so a point estimate from one phrasing understates exposure.
- A budget of about three meaning-preserving forms recovers roughly 85% of the observed five-form unsafe surface, making multi-form probing a practical check.
- Reporting refusal and compliance rates alone hides per-seed instability; decomposed consistency (all-safe, all-unsafe, mixed) should accompany aggregate scores.
- The canonical instrument is also optimistic about over-refusal: benign prompts flipped to refusal in 6–18% of cases, so single-form evaluation hides both safety failures and excess caution.
- The underestimation is model-dependent and largest where baseline risk is already high, so single-form scores can misorder models relative to each other.
Where Pith is reading between the lines
- If the zero stochasticity floor does not hold on the other three models, the magnitude of the surface-form effect shrinks; the paper's Limitations concede the floor was measured only on GPT-4o-mini and DeepSeek-V3, so re-measuring on all five is the most direct extension.
- Because three of the four reformulations share the Chinese machine-translation pipeline, the coverage curve (one form ≈ 53%, three ≈ 85%) likely overstates the independent information added by each additional form; a mechanistically diverse form set—syntax-only restructuring, a different language family, a different paraphraser—would give a fairer redundancy estimate.
- The union estimand is a lower-bound rather than a population estimate; a stronger claim would require sampling surface forms from a defined distribution, which the authors explicitly do not do.
- The protocol could transfer to non-safety benchmarks with no gold label—capability, instruction-following, or reasoning—where single-prompt reading is also the norm and where a union estimand plus a noise floor would reveal how much of the score is instrument artifact.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper evaluates whether single-canonical-prompt safety benchmarks faithfully estimate LLM unsafe compliance. The authors construct 370 harmful seeds, generate four pre-authored reformulations (pivot back-translation paraphrase, Chinese translation, programmatic code-switching, and LLM indirect framing), run five frontier models at temperature 0, and score all 1,850 responses with a single vendor-neutral LLM judge (Claude) human-anchored on 185 items and cross-checked by GPT-4o on 250. They find no transformation is uniformly most dangerous, but the union of unsafe outcomes across the five forms exceeds the worst single form by 3.3–12.9 pp with seed-level bootstrap 95% CIs excluding zero for all five models, and 5.4–13.0% of seeds safe on canonical are unsafe under some reformulation. They attribute this new exposure to surface-form signal, above a 'zero stochasticity floor' measured by resampling the canonical prompt five times at temperature 0 on two of the five models.
Significance. If the claims hold, the paper advances evaluation methodology: safety scores should be treated as measurements with instrument bias, and robustness evaluation should report union/worst-case quantities over meaning-preserving surface forms rather than a single canonical point estimate. The strengths are real: pre-authored, mostly non-LLM perturbations sent identically to every model avoid meta-instruction confounds; the judge is vendor-neutral, human-anchored, and cross-checked by a second vendor; the paper verifies intent preservation; seed-level bootstrap CIs are used; and dataset, code, and per-response labels are released. The main caveat is that the noise-attribution claim — the 'above a zero floor' assertion — is directly supported on only two of five models and only for the canonical form, so the magnitude of the surface-form-specific signal for the other three models is not yet established. The union-minus-worst result, however, is independent of the floor and is the most robust contribution.
major comments (1)
- [Abstract; Section 4.2; Table 3 caption; Limitations] The stochasticity floor is measured only on GPT-4o-mini and DeepSeek-V3 (canonical prompt resampled five times at temperature 0), yet the abstract and Table 3 apply 'above a zero stochasticity floor' to all five models, and Section 4.2 concludes the cross-form new exposures are 'therefore surface-form-driven.' If any of the untested models (GPT-4o, Gemini 2.5 Flash, Gemini 2.5 Pro) has a nonzero label-flip rate under canonical resampling, or if any reformulated form has a higher within-form label-flip rate than canonical, part of the reported 5.4–13.0% new exposure is decoding/judge noise rather than surface-form signal. The largest new-exposure gap (Gemini 2.5 Pro, 13.0%) is on an untested model. The Limitations section honestly discloses the scope, but the headline claims do not carry the restriction. Please either extend the floor measurement to all five models and to the reformulated
minor comments (5)
- [Section 4.6; Limitations] The same Claude judge rates both intent preservation and unsafe compliance. The conclusion that indirect framing's lower severity is 'partly an artifact of intent neutralization' is therefore self-referential: the judge's own intent rating is used to explain the judge's own compliance rating. Independent human or second-vendor intent labels would strengthen this decomposition; the paper already flags this in Limitations, and the robustness check on intent-preserving forms mitigates the impact on the union result.
- [Section 4.3; Table 3] The benign control (XSTest) is not item-matched to the harmful pool, and the paper appropriately labels bidirectionality as secondary. Consider adding OR-Bench/PHTest or item-matched benign/harmful pairs if bidirectionality is to be promoted to a core claim.
- [Abstract] Typesetting issue: 'zerostochasticityfloor' and similar concatenations lack word spacing in the abstract; also the abstract's stochasticity-floor parenthetical should name the two-model scope.
- [Table 1] Aggregate rates are reported without confidence intervals or per-cell counts; adding Wilson intervals would help readers assess the underpowered per-transformation comparisons.
- [Section 4.1] The McNemar power limitation is acknowledged; reporting exact discordant-pair counts and effect directions for all 20 tests in an appendix would make the 'no single transformation is uniformly most dangerous' claim easier to evaluate.
Circularity Check
One secondary self-referential judge-rating step; central union/new-exposure claim is empirical and not circular.
specific steps
-
other
[Section 4.6 and Limitations (Intent metric)]
"Intent preservation is itself judge-rated; no standard benchmark for intent preservation of harmful reformulations exists, and our 82.4% for indirect framing should be read as approximate. It is also rated by the same judge that scores unsafe compliance, so independent-judge re-scoring is needed to rule out circularity."
Section 4.6 concludes that indirect framing's lower apparent severity is 'partly an artifact of intent neutralization' by comparing the judge's unsafe-compliance rate for indirect framing with the same judge's intent-preservation rating. Since both labels come from the same judge under the same rubric, any systematic tendency of that judge to treat indirect or hypothetical wording as both 'intent not preserved' and 'less unsafe' would mechanically produce the reported association. The paper itself acknowledges this in the Limitations and asks for independent-judge re-scoring to rule out circularity. This is a secondary, explicitly caveated interpretation, not the paper's central claim.
full rationale
The central claim — that canonical single-form evaluation underreports unsafe compliance — is empirical and self-contained. The union, new-exposure, and stochasticity-floor statistics are computed directly from per-response judge labels with a human anchor (kappa 0.86) and an independent-vendor cross-check (GPT-4o). No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from prior work, and no ansatz is smuggled in via self-citation. The only self-citation (Zhou et al., 2026) motivates the stochasticity-floor idea, but the floor itself is measured in this paper, so it is not load-bearing. The zero-floor is measured on only two of five models and only for the canonical prompt; that is a generalization/validity limitation, not circularity. The one genuine self-referential step is the intent-preservation interpretation in Section 4.6, where the same judge supplies both the intent and unsafe-compliance labels; the paper explicitly asks for re-scoring to rule out circularity. Because this affects only a secondary, caveated interpretation and not the main union/new-exposure result, the overall circularity score is 2.
Axiom & Free-Parameter Ledger
free parameters (1)
- surface-similarity acceptance band =
(0.30, 0.88)
axioms (5)
- domain assumption The five surface forms preserve the harmful intent of each seed.
- domain assumption Claude's 'unsafe compliance' label is a valid operationalization of safety for all evaluated models.
- domain assumption The zero stochasticity floor measured on two models generalizes to all five.
- domain assumption HarmBench and AdvBench seeds are a meaningful sample of harmful intents.
- domain assumption XSTest benign prompts are comparable enough to the harmful pool to suggest bidirectionality.
Cite this review
Pith. "Pith review of Single Canonical Prompts Underestimate LLM Safety's Surface-Form Sensitivity." pith.science (2026). https://pith.science/paper/6HQECEZM
@misc{pith2026260802665,
author = {Pith},
title = {Pith review of: Single Canonical Prompts Underestimate LLM Safety's Surface-Form Sensitivity},
year = {2026},
howpublished = {\url{https://pith.science/paper/6HQECEZM}},
note = {Machine review of arXiv:2608.02665}
}
read the original abstract
A benchmark score is a measurement instrument, yet most benchmarks read each item at a single canonical surface form. We ask whether that reading is faithful: when an item's intent is held fixed and only its meaning-preserving surface form varies, does the canonical-form score estimate model behavior well, and how much of any variation is decoding/judge noise rather than signal? We instantiate this in safety, a high-stakes setting with no gold label to average toward. To avoid prior confounds, we pre-author the reformulations (refusal-free, mostly non-LLM: machine back-translation and a Matrix-Language-Frame code-switch generator) so an identical surface form reaches every model, score all responses with one human-anchored, vendor-neutral judge (Claude, kappa = 0.86 vs. human on unsafe compliance, stable across languages, cross-checked by GPT-4o), and verify intent preservation. On 370 seeds x 5 surface forms x 5 models, no single transformation is uniformly most dangerous (6 of 20 per-transformation McNemar tests survive correction, most protective). Yet evaluating only the canonical prompt underestimates unsafe compliance: the union of unsafe outcomes across forms exceeds even the worst single form by 3.3-12.9 pp, with bootstrap 95% CIs excluding zero for all five models, and 5-13% of seeds safe on canonical are unsafe under some reformulation -- above a zero stochasticity floor (canonical resampled five times at temperature 0 gives 0/370 new exposures). The size of this gap is model-dependent (largest on Gemini 2.5 Pro). One form recovers only ~53% of a model's observed unsafe surface and about three reach 85% -- a redundancy characterization of this form set, not of a defined population. A benign control (XSTest) suggests the instability is bidirectional, though the benign and harmful pools are not item-matched. We release the dataset, code, and per-response labels.
Reference graph
Works this paper leans on
-
[1]
Bang An, Sicheng Zhu, Ruiyi Zhang, Michael-Andrei Panaitescu-Liess, Yuancheng Xu, and Furong Huang. 2024. https://arxiv.org/abs/2409.00598 Automatically generating pseudo-harmful prompts for evaluating false refusals in large language models . arXiv preprint arXiv:2409.00598
Pith/arXiv arXiv 2024
-
[2]
Pappas, Florian Tramer, Hamed Hassani, and Eric Wong
Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramer, Hamed Hassani, and Eric Wong. 2024. https://papers.nips.cc/paper_files/paper/2024/hash/63092d79154adebd7305dfd498cbff70-Abstract-Datasets_and_Benchmarks_Track.html Jailbreakbench: ...
work page 2024
-
[3]
Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. 2024. https://arxiv.org/abs/2405.20947 OR-Bench : An over-refusal benchmark for large language models . arXiv preprint arXiv:2405.20947
Pith/arXiv arXiv 2024
-
[4]
Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2024. https://openreview.net/forum?id=vESNKdEMGp Multilingual jailbreak challenges in large language models . In International Conference on Learning Representations (ICLR)
work page 2024
-
[5]
John Hughes, Sara Price, Aengus Lynch, Rylan Schaeffer, Fazl Barez, Sanmi Koyejo, Henry Sleight, Erik Jones, Ethan Perez, and Mrinank Sharma. 2024. https://arxiv.org/abs/2412.03556 Best-of-n jailbreaking . arXiv preprint arXiv:2412.03556
Pith/arXiv arXiv 2024
-
[6]
Abigail Z. Jacobs and Hanna Wallach. 2021. https://dl.acm.org/doi/10.1145/3442188.3445901 Measurement and fairness . In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT), pages 375--385
-
[7]
Jie Li, Yi Liu, Chongyang Liu, Ling Shi, Xiaoning Ren, Yaowen Zheng, Yang Liu, and Yinxing Xue. 2024. https://arxiv.org/abs/2401.16765 A cross-language investigation into jailbreak attacks in large language models . arXiv preprint arXiv:2401.16765
Pith/arXiv arXiv 2024
-
[8]
Lifei Liu, Haoran Yu, Xiaochong Jiang, Su Wang, Pin Qian, and Yihang Chen. 2026. https://arxiv.org/abs/2607.07097 Operational reframing and approval-framed delegation in multi-agent LLM safety . Preprint, arXiv:2607.07097
work page internal anchor Pith review Pith/arXiv arXiv 2026
-
[9]
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. 2024. https://proceedings.mlr.press/v235/mazeika24a.html Harmbench: A standardized evaluation framework for automated red teaming and robust refusal . In Proceedings of the 41st International Conferen...
work page 2024
-
[10]
Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. https://aclanthology.org/2024.tacl-1.52/ State of what art? a call for multi-prompt LLM evaluation . Transactions of the Association for Computational Linguistics (TACL), 12
work page 2024
-
[11]
Yutao Mou, Shikun Zhang, and Wei Ye. 2024. https://papers.nips.cc/paper_files/paper/2024/hash/de7b99107c53e60257c727dc73daf1d1-Abstract-Datasets_and_Benchmarks_Track.html Sg-bench: Evaluating llm safety generalization across diverse tasks and prompt types . In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track
work page 2024
-
[12]
Carol Myers-Scotton. 1993. Duelling Languages: Grammatical Structure in Codeswitching. Oxford University Press
work page 1993
-
[13]
Paul R \"o ttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2024. XSTest : A test suite for identifying exaggerated safety behaviours in large language models. In Proceedings of NAACL
work page 2024
-
[14]
Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. https://arxiv.org/abs/2310.11324 Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting . In International Conference on Learning Representations (ICLR)
Pith/arXiv arXiv 2024
-
[15]
Su Wang, Pin Qian, Yihang Chen, Junxian You, Xiaoyuan Wang, Xiaochong Jiang, Lifei Liu, Haoran Yu, and Jingzhou Xu. 2026. https://arxiv.org/abs/2606.00448 When safe skills collide: Measuring compositional risk in agent skill ecosystems . Preprint, arXiv:2606.00448
Pith/arXiv arXiv 2026
-
[16]
Zheng-Xin Yong, Cristina Menghini, and Stephen H. Bach. 2023. https://arxiv.org/abs/2310.02446 Low-resource languages jailbreak gpt-4 . In NeurIPS Workshop on Socially Responsible Language Modelling Research (SoLaR). Best Paper Award
Pith/arXiv arXiv 2023
-
[17]
Haneul Yoo, Yongjin Yang, and Hwaran Lee. 2025. https://aclanthology.org/2025.acl-long.657/ Code-switching red-teaming: LLM evaluation for safety and multilingual understanding . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL)
work page 2025
-
[18]
Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. 2024. https://arxiv.org/abs/2401.06373 How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms . arXiv preprint arXiv:2401.06373
Pith/arXiv arXiv 2024
-
[19]
Yike Zhang, Zuodong Xiang, and Hailu Xu. 2026. Performance-efficiency trade-offs in human preference prediction: A comparative study of traditional machine learning and large language models. In Proceedings of the 31st IEEE Symposium on Computers and Communications (ISCC)
work page 2026
-
[20]
Yongxi Zhou, Lai Yun Choi, Jiaxi Wen, and Wenbo Ye. 2026. https://arxiv.org/abs/2606.00920 Accuracy, stability, and repeated-run reliability of large language models on deterministic programming tasks . Preprint, arXiv:2606.00920
work page internal anchor Pith review Pith/arXiv arXiv 2026
-
[21]
Zico Kolter, and Matt Fredrikson
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023. https://arxiv.org/abs/2307.15043 Universal and transferable adversarial attacks on aligned language models . arXiv preprint arXiv:2307.15043
Pith/arXiv arXiv 2023
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.