Pith. sign in

REVIEW 1 major objections 5 minor 21 references

A single canonical phrasing undercounts LLM unsafe compliance by up to 13 percentage points.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-05 00:42 UTC pith:6HQECEZM

load-bearing objection A careful measurement study that earns its central claim—canonical prompts are optimistic estimators of unsafe compliance—with one disclosed limitation (noise floor on two of five models) that should not block publication. the 1 major comments →

arxiv 2608.02665 v1 pith:6HQECEZM submitted 2026-08-01 cs.CR cs.AIcs.CL

Single Canonical Prompts Underestimate LLM Safety's Surface-Form Sensitivity

Yongxi Zhou , Junwei Yao , Yuanzhe Liu , Zihan Dong , Wenbo Ye , Jiaxi Wen , Lai Yun Choi This is my paper
classification cs.CR cs.AIcs.CL
keywords LLM safetybenchmark validitycanonical prompt biassurface-form sensitivitymeaning-preserving reformulationunsafe complianceover-refusalstochasticity floor
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Safety benchmarks typically present each harmful request in one canonical English phrasing and report the refusal rate as if it were a property of the model. This paper argues that such a reading is a biased measurement instrument: when harmful intent is held fixed and only meaning-preserving surface form varies, different seeds fail under different phrasings, so the union of unsafe outcomes across five forms exceeds even the worst single form by 3.3–12.9 percentage points for all five models tested, and 5–13% of seeds that look safe on the canonical prompt turn out to be unsafe under some reformulation. The paper's protocol removes the usual confounds by pre-authoring all reformulations as fixed strings (mostly non-LLM: machine back-translation, Chinese translation, and a rule-based code-switch generator), scoring every response with one human-anchored, vendor-neutral judge, and checking intent preservation. A resampling control at temperature 0 produced zero label flips, so the effect is attributed to surface form rather than decoding/judge noise; a benign control suggests the instability is bidirectional, with comparable new over-refusals. If the claim holds, single-prompt safety scores are optimistic lower bounds on real-world unsafe behavior, and roughly three meaning-preserving forms would recover about 85% of the observed unsafe surface.

Core claim

The central claim is that a single canonical surface form is an optimistically biased estimator of a model's unsafe-compliance surface. On 370 harmful seeds × 5 surface forms × 5 models, per-transformation effects are small and mostly non-significant or protective, but per-seed effects are idiosyncratic: seeds safe on canonical become unsafe under some reformulation in 5.4–13.0% of cases, and the union of unsafe outcomes across the five forms exceeds even the worst single form by 3.3–12.9 percentage points, with bootstrap 95% CIs excluding zero for every model. A stochasticity floor—re-running canonical prompts five times at temperature 0—produced 0/370 label flips on the two models tested,

What carries the argument

The load-bearing mechanism is a pre-authored, refusal-free reformulation pipeline plus a stochasticity floor. Four meaning-preserving forms are generated as fixed strings—distant-pivot machine back-translation as paraphrase, machine translation to Chinese, a rule-based Matrix-Language-Frame code-switch generator (English matrix with Chinese noun phrases), and one LLM-produced indirect framing—so the identical surface form reaches every model and reformulation ability is not confounded with safety. A single human-anchored, vendor-neutral judge scores every response (kappa 0.86 vs. human on unsafe compliance), with an intent-preservation check. The stochasticity floor, measured by resampling t

Load-bearing premise

The claim rests on the assumption that re-running the exact same prompt five times never changes a model's safety label, measured on only two of the five models; if the other three models sometimes flip labels when nothing changes, part of the reported 5–13% 'new unsafe exposure' could be noise, not surface-form sensitivity.

What would settle it

Re-run the exact same canonical prompts five times at temperature 0 on the other three models (GPT-4o, Gemini 2.5 Flash, Gemini 2.5 Pro) and count how often a seed that was safe on the first run is unsafe on a repeat; a nonzero flip rate on any of those models, especially one large enough to cover the claimed new-exposure gap, would undercut the attribution to surface form.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Single-form safety scores are lower bounds: the five-form union is 1.3–2.2× the canonical rate, so a point estimate from one phrasing understates exposure.
  • A budget of about three meaning-preserving forms recovers roughly 85% of the observed five-form unsafe surface, making multi-form probing a practical check.
  • Reporting refusal and compliance rates alone hides per-seed instability; decomposed consistency (all-safe, all-unsafe, mixed) should accompany aggregate scores.
  • The canonical instrument is also optimistic about over-refusal: benign prompts flipped to refusal in 6–18% of cases, so single-form evaluation hides both safety failures and excess caution.
  • The underestimation is model-dependent and largest where baseline risk is already high, so single-form scores can misorder models relative to each other.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the zero stochasticity floor does not hold on the other three models, the magnitude of the surface-form effect shrinks; the paper's Limitations concede the floor was measured only on GPT-4o-mini and DeepSeek-V3, so re-measuring on all five is the most direct extension.
  • Because three of the four reformulations share the Chinese machine-translation pipeline, the coverage curve (one form ≈ 53%, three ≈ 85%) likely overstates the independent information added by each additional form; a mechanistically diverse form set—syntax-only restructuring, a different language family, a different paraphraser—would give a fairer redundancy estimate.
  • The union estimand is a lower-bound rather than a population estimate; a stronger claim would require sampling surface forms from a defined distribution, which the authors explicitly do not do.
  • The protocol could transfer to non-safety benchmarks with no gold label—capability, instruction-following, or reasoning—where single-prompt reading is also the norm and where a union estimand plus a noise floor would reveal how much of the score is instrument artifact.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. The paper evaluates whether single-canonical-prompt safety benchmarks faithfully estimate LLM unsafe compliance. The authors construct 370 harmful seeds, generate four pre-authored reformulations (pivot back-translation paraphrase, Chinese translation, programmatic code-switching, and LLM indirect framing), run five frontier models at temperature 0, and score all 1,850 responses with a single vendor-neutral LLM judge (Claude) human-anchored on 185 items and cross-checked by GPT-4o on 250. They find no transformation is uniformly most dangerous, but the union of unsafe outcomes across the five forms exceeds the worst single form by 3.3–12.9 pp with seed-level bootstrap 95% CIs excluding zero for all five models, and 5.4–13.0% of seeds safe on canonical are unsafe under some reformulation. They attribute this new exposure to surface-form signal, above a 'zero stochasticity floor' measured by resampling the canonical prompt five times at temperature 0 on two of the five models.

Significance. If the claims hold, the paper advances evaluation methodology: safety scores should be treated as measurements with instrument bias, and robustness evaluation should report union/worst-case quantities over meaning-preserving surface forms rather than a single canonical point estimate. The strengths are real: pre-authored, mostly non-LLM perturbations sent identically to every model avoid meta-instruction confounds; the judge is vendor-neutral, human-anchored, and cross-checked by a second vendor; the paper verifies intent preservation; seed-level bootstrap CIs are used; and dataset, code, and per-response labels are released. The main caveat is that the noise-attribution claim — the 'above a zero floor' assertion — is directly supported on only two of five models and only for the canonical form, so the magnitude of the surface-form-specific signal for the other three models is not yet established. The union-minus-worst result, however, is independent of the floor and is the most robust contribution.

major comments (1)
  1. [Abstract; Section 4.2; Table 3 caption; Limitations] The stochasticity floor is measured only on GPT-4o-mini and DeepSeek-V3 (canonical prompt resampled five times at temperature 0), yet the abstract and Table 3 apply 'above a zero stochasticity floor' to all five models, and Section 4.2 concludes the cross-form new exposures are 'therefore surface-form-driven.' If any of the untested models (GPT-4o, Gemini 2.5 Flash, Gemini 2.5 Pro) has a nonzero label-flip rate under canonical resampling, or if any reformulated form has a higher within-form label-flip rate than canonical, part of the reported 5.4–13.0% new exposure is decoding/judge noise rather than surface-form signal. The largest new-exposure gap (Gemini 2.5 Pro, 13.0%) is on an untested model. The Limitations section honestly discloses the scope, but the headline claims do not carry the restriction. Please either extend the floor measurement to all five models and to the reformulated
minor comments (5)
  1. [Section 4.6; Limitations] The same Claude judge rates both intent preservation and unsafe compliance. The conclusion that indirect framing's lower severity is 'partly an artifact of intent neutralization' is therefore self-referential: the judge's own intent rating is used to explain the judge's own compliance rating. Independent human or second-vendor intent labels would strengthen this decomposition; the paper already flags this in Limitations, and the robustness check on intent-preserving forms mitigates the impact on the union result.
  2. [Section 4.3; Table 3] The benign control (XSTest) is not item-matched to the harmful pool, and the paper appropriately labels bidirectionality as secondary. Consider adding OR-Bench/PHTest or item-matched benign/harmful pairs if bidirectionality is to be promoted to a core claim.
  3. [Abstract] Typesetting issue: 'zerostochasticityfloor' and similar concatenations lack word spacing in the abstract; also the abstract's stochasticity-floor parenthetical should name the two-model scope.
  4. [Table 1] Aggregate rates are reported without confidence intervals or per-cell counts; adding Wilson intervals would help readers assess the underpowered per-transformation comparisons.
  5. [Section 4.1] The McNemar power limitation is acknowledged; reporting exact discordant-pair counts and effect directions for all 20 tests in an appendix would make the 'no single transformation is uniformly most dangerous' claim easier to evaluate.

Circularity Check

1 steps flagged

One secondary self-referential judge-rating step; central union/new-exposure claim is empirical and not circular.

specific steps
  1. other [Section 4.6 and Limitations (Intent metric)]
    "Intent preservation is itself judge-rated; no standard benchmark for intent preservation of harmful reformulations exists, and our 82.4% for indirect framing should be read as approximate. It is also rated by the same judge that scores unsafe compliance, so independent-judge re-scoring is needed to rule out circularity."

    Section 4.6 concludes that indirect framing's lower apparent severity is 'partly an artifact of intent neutralization' by comparing the judge's unsafe-compliance rate for indirect framing with the same judge's intent-preservation rating. Since both labels come from the same judge under the same rubric, any systematic tendency of that judge to treat indirect or hypothetical wording as both 'intent not preserved' and 'less unsafe' would mechanically produce the reported association. The paper itself acknowledges this in the Limitations and asks for independent-judge re-scoring to rule out circularity. This is a secondary, explicitly caveated interpretation, not the paper's central claim.

full rationale

The central claim — that canonical single-form evaluation underreports unsafe compliance — is empirical and self-contained. The union, new-exposure, and stochasticity-floor statistics are computed directly from per-response judge labels with a human anchor (kappa 0.86) and an independent-vendor cross-check (GPT-4o). No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from prior work, and no ansatz is smuggled in via self-citation. The only self-citation (Zhou et al., 2026) motivates the stochasticity-floor idea, but the floor itself is measured in this paper, so it is not load-bearing. The zero-floor is measured on only two of five models and only for the canonical prompt; that is a generalization/validity limitation, not circularity. The one genuine self-referential step is the intent-preservation interpretation in Section 4.6, where the same judge supplies both the intent and unsafe-compliance labels; the paper explicitly asks for re-scoring to rule out circularity. Because this affects only a secondary, caveated interpretation and not the main union/new-exposure result, the overall circularity score is 2.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 0 invented entities

No new physical or formal entities are introduced; the contributions are measurement controls. Free parameters are limited to hand-chosen generation thresholds. The main ledger entries are domain assumptions about judge validity, intent preservation, and the scope of the stochasticity floor.

free parameters (1)
  • surface-similarity acceptance band = (0.30, 0.88)
    Hand-chosen quality gate for back-translated paraphrases (Appendix B); not fitted to safety outcomes, but it determines which reformulations enter the dataset and could correlate with wording drift.
axioms (5)
  • domain assumption The five surface forms preserve the harmful intent of each seed.
    Section 3.4/4.6: intent preservation is rated by the same Claude judge that scores compliance; no independent benchmark for harmful rephrasings exists, so meaning preservation is an assumed construct.
  • domain assumption Claude's 'unsafe compliance' label is a valid operationalization of safety for all evaluated models.
    Section 3.4: human anchor on 185 responses gives kappa 0.86, but full-set labels come from one judge; cross-form reliability is stable per language, yet this is still a model-based instrument.
  • domain assumption The zero stochasticity floor measured on two models generalizes to all five.
    Section 4.2 and Limitations: floor is 0/370 for GPT-4o-mini and DeepSeek-V3; the 'above a zero floor' claim in the abstract extends this to unmeasured models.
  • domain assumption HarmBench and AdvBench seeds are a meaningful sample of harmful intents.
    Section 3.1: seeds are mapped to four harm categories; the paper does not claim population representativeness, but the union magnitudes are quoted as model properties.
  • domain assumption XSTest benign prompts are comparable enough to the harmful pool to suggest bidirectionality.
    Section 4.3 and Limitations: pools are not item-matched and differ in topic distribution; the paper downgrades this to a secondary observation.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Single Canonical Prompts Underestimate LLM Safety's Surface-Form Sensitivity." pith.science (2026). https://pith.science/paper/6HQECEZM

@misc{pith2026260802665,
  author       = {Pith},
  title        = {Pith review of: Single Canonical Prompts Underestimate LLM Safety's Surface-Form Sensitivity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6HQECEZM}},
  note         = {Machine review of arXiv:2608.02665}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

A benchmark score is a measurement instrument, yet most benchmarks read each item at a single canonical surface form. We ask whether that reading is faithful: when an item's intent is held fixed and only its meaning-preserving surface form varies, does the canonical-form score estimate model behavior well, and how much of any variation is decoding/judge noise rather than signal? We instantiate this in safety, a high-stakes setting with no gold label to average toward. To avoid prior confounds, we pre-author the reformulations (refusal-free, mostly non-LLM: machine back-translation and a Matrix-Language-Frame code-switch generator) so an identical surface form reaches every model, score all responses with one human-anchored, vendor-neutral judge (Claude, kappa = 0.86 vs. human on unsafe compliance, stable across languages, cross-checked by GPT-4o), and verify intent preservation. On 370 seeds x 5 surface forms x 5 models, no single transformation is uniformly most dangerous (6 of 20 per-transformation McNemar tests survive correction, most protective). Yet evaluating only the canonical prompt underestimates unsafe compliance: the union of unsafe outcomes across forms exceeds even the worst single form by 3.3-12.9 pp, with bootstrap 95% CIs excluding zero for all five models, and 5-13% of seeds safe on canonical are unsafe under some reformulation -- above a zero stochasticity floor (canonical resampled five times at temperature 0 gives 0/370 new exposures). The size of this gap is model-dependent (largest on Gemini 2.5 Pro). One form recovers only ~53% of a model's observed unsafe surface and about three reach 85% -- a redundancy characterization of this form set, not of a defined population. A benign control (XSTest) suggests the instability is bidirectional, though the benign and harmful pools are not item-matched. We release the dataset, code, and per-response labels.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 12 canonical work pages · 2 internal anchors

  1. [1]

    Bang An, Sicheng Zhu, Ruiyi Zhang, Michael-Andrei Panaitescu-Liess, Yuancheng Xu, and Furong Huang. 2024. https://arxiv.org/abs/2409.00598 Automatically generating pseudo-harmful prompts for evaluating false refusals in large language models . arXiv preprint arXiv:2409.00598

    Pith/arXiv arXiv 2024

  2. [2]

    Pappas, Florian Tramer, Hamed Hassani, and Eric Wong

    Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramer, Hamed Hassani, and Eric Wong. 2024. https://papers.nips.cc/paper_files/paper/2024/hash/63092d79154adebd7305dfd498cbff70-Abstract-Datasets_and_Benchmarks_Track.html Jailbreakbench: ...

    work page 2024

  3. [3]

    Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. 2024. https://arxiv.org/abs/2405.20947 OR-Bench : An over-refusal benchmark for large language models . arXiv preprint arXiv:2405.20947

    Pith/arXiv arXiv 2024

  4. [4]

    Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2024. https://openreview.net/forum?id=vESNKdEMGp Multilingual jailbreak challenges in large language models . In International Conference on Learning Representations (ICLR)

    work page 2024

  5. [5]

    John Hughes, Sara Price, Aengus Lynch, Rylan Schaeffer, Fazl Barez, Sanmi Koyejo, Henry Sleight, Erik Jones, Ethan Perez, and Mrinank Sharma. 2024. https://arxiv.org/abs/2412.03556 Best-of-n jailbreaking . arXiv preprint arXiv:2412.03556

    Pith/arXiv arXiv 2024

  6. [6]

    Jacobs and Hanna Wallach

    Abigail Z. Jacobs and Hanna Wallach. 2021. https://dl.acm.org/doi/10.1145/3442188.3445901 Measurement and fairness . In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT), pages 375--385

    work page arXiv 2021

  7. [7]

    Jie Li, Yi Liu, Chongyang Liu, Ling Shi, Xiaoning Ren, Yaowen Zheng, Yang Liu, and Yinxing Xue. 2024. https://arxiv.org/abs/2401.16765 A cross-language investigation into jailbreak attacks in large language models . arXiv preprint arXiv:2401.16765

    Pith/arXiv arXiv 2024

  8. [8]

    Lifei Liu, Haoran Yu, Xiaochong Jiang, Su Wang, Pin Qian, and Yihang Chen. 2026. https://arxiv.org/abs/2607.07097 Operational reframing and approval-framed delegation in multi-agent LLM safety . Preprint, arXiv:2607.07097

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  9. [9]

    Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. 2024. https://proceedings.mlr.press/v235/mazeika24a.html Harmbench: A standardized evaluation framework for automated red teaming and robust refusal . In Proceedings of the 41st International Conferen...

    work page 2024

  10. [10]

    Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. 2024. https://aclanthology.org/2024.tacl-1.52/ State of what art? a call for multi-prompt LLM evaluation . Transactions of the Association for Computational Linguistics (TACL), 12

    work page 2024

  11. [11]

    Yutao Mou, Shikun Zhang, and Wei Ye. 2024. https://papers.nips.cc/paper_files/paper/2024/hash/de7b99107c53e60257c727dc73daf1d1-Abstract-Datasets_and_Benchmarks_Track.html Sg-bench: Evaluating llm safety generalization across diverse tasks and prompt types . In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track

    work page 2024

  12. [12]

    Carol Myers-Scotton. 1993. Duelling Languages: Grammatical Structure in Codeswitching. Oxford University Press

    work page 1993

  13. [13]

    Paul R \"o ttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2024. XSTest : A test suite for identifying exaggerated safety behaviours in large language models. In Proceedings of NAACL

    work page 2024

  14. [14]

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. https://arxiv.org/abs/2310.11324 Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting . In International Conference on Learning Representations (ICLR)

    Pith/arXiv arXiv 2024

  15. [15]

    Su Wang, Pin Qian, Yihang Chen, Junxian You, Xiaoyuan Wang, Xiaochong Jiang, Lifei Liu, Haoran Yu, and Jingzhou Xu. 2026. https://arxiv.org/abs/2606.00448 When safe skills collide: Measuring compositional risk in agent skill ecosystems . Preprint, arXiv:2606.00448

    Pith/arXiv arXiv 2026

  16. [16]

    Zheng-Xin Yong, Cristina Menghini, and Stephen H. Bach. 2023. https://arxiv.org/abs/2310.02446 Low-resource languages jailbreak gpt-4 . In NeurIPS Workshop on Socially Responsible Language Modelling Research (SoLaR). Best Paper Award

    Pith/arXiv arXiv 2023

  17. [17]

    Haneul Yoo, Yongjin Yang, and Hwaran Lee. 2025. https://aclanthology.org/2025.acl-long.657/ Code-switching red-teaming: LLM evaluation for safety and multilingual understanding . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL)

    work page 2025

  18. [18]

    Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. 2024. https://arxiv.org/abs/2401.06373 How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms . arXiv preprint arXiv:2401.06373

    Pith/arXiv arXiv 2024

  19. [19]

    Yike Zhang, Zuodong Xiang, and Hailu Xu. 2026. Performance-efficiency trade-offs in human preference prediction: A comparative study of traditional machine learning and large language models. In Proceedings of the 31st IEEE Symposium on Computers and Communications (ISCC)

    work page 2026

  20. [20]

    Yongxi Zhou, Lai Yun Choi, Jiaxi Wen, and Wenbo Ye. 2026. https://arxiv.org/abs/2606.00920 Accuracy, stability, and repeated-run reliability of large language models on deterministic programming tasks . Preprint, arXiv:2606.00920

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  21. [21]

    Zico Kolter, and Matt Fredrikson

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023. https://arxiv.org/abs/2307.15043 Universal and transferable adversarial attacks on aligned language models . arXiv preprint arXiv:2307.15043

    Pith/arXiv arXiv 2023

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.