Pith. sign in

REVIEW 3 major objections 6 minor 17 references

Wrapper formatting alone can swing LLM benchmark accuracy by more than 30 times across models, mostly because outputs fail to parse.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 19:16 UTC pith:QSJIIV5B

load-bearing objection Useful multi-wrapper metrics and a large OpenRouter study; the 30× FSI claim is real under standard scoring, but the “compliance explains it” story is partly definitional. the 3 major comments →

arxiv 2607.09665 v1 pith:QSJIIV5B submitted 2026-05-02 cs.AI cs.CLcs.LG

Format Sensitivity Index: Token-Controlled Prompt Wrapper Robustness and Schema Compliance in LLM Benchmarking

Deep Pankajbhai Mehta This is my paper
classification cs.AI cs.CLcs.LG
keywords format sensitivityprompt wrappersschema complianceLLM benchmarkingparseabilitystructured outputstoken-controlled evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the formatting shell around a question—JSON instructions, key-value fields, delimiters, or a scratchpad—can change a model’s measured accuracy enough to reverse leaderboard rankings, even when the underlying task content is held fixed. The authors introduce two simple range metrics: Format Sensitivity Index (FSI), the gap between best and worst accuracy across wrappers, and Parseability Sensitivity Index (PSI), the same gap for whether the answer can be extracted at all. Under a protocol that pads prompts to similar length and logs token counts, they run 140,000 generations across seven common QA tasks, five wrapper families, and four instruction-tuned models from 7B to 72B. Mean FSI differs by more than thirtyfold between models; the largest swings track collapses in parseability, and a fixed-effects regression shows parseability still strongly predicts accuracy after controlling for task, model, and wrapper. The practical claim is that a single accuracy number without wrapper variance or compliance rates is statistically fragile, and both benchmark reports and production structured-output systems should measure and report these ranges.

Core claim

Across a token-controlled study of 140,000 generations, mean Format Sensitivity Index varies by more than 30 times across four instruct models (roughly 0.76 for the most sensitive versus 0.024 for the least). That range is largely mediated by schema-compliance failures: accuracy and parseability correlate at r = 0.825, and parseability remains a strong fixed-effects predictor of accuracy (β̂ ≈ 0.819) after controlling for task, model, and wrapper. Therefore, reporting point accuracy without wrapper variance and parseability rates is statistically fragile.

What carries the argument

Format Sensitivity Index (FSI) and Parseability Sensitivity Index (PSI): for each model–task pair, FSI is the accuracy range across five wrapper families and PSI is the corresponding parseability range; a normalized nFSI divides FSI by mean accuracy to highlight when wrappers dominate the score.

Load-bearing premise

Scoring every unparseable output as simply wrong under one fixed extractor is the right definition of accuracy for measuring wrapper effects; that choice mechanically ties accuracy to parseability.

What would settle it

Rerun the same models, tasks, and wrappers with a more lenient or partial-credit extractor, or with decoder-enforced JSON/schema modes, and check whether mean FSI collapses and the accuracy–parseability link disappears.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that prompt wrappers that differ mainly in formatting can change LLM benchmark scores enough to flip leaderboard conclusions, largely via schema-compliance failures. Under a token-controlled protocol (character padding, logged prompt tokens, matched subset with relative token spread ≤5%), it defines FSI as the accuracy range across wrappers and PSI as the corresponding parseability range, plus a normalized nFSI. Across 140,000 OpenRouter generations on 7 QA tasks, 5 wrapper families, and 4 instruct models (7B–72B), mean FSI varies by more than 30× (Phi-4 ≈0.763 vs Qwen-2.5-72B ≈0.024). Accuracy and parseability are strongly correlated (r=0.825); a cell-level fixed-effects regression yields β̂≈0.819 for parseability (partial R²≈0.82). The authors recommend reporting wrapper family, token counts, parseability, and sensitivity intervals alongside accuracy, and treating FSI/PSI as diagnostics for when constrained decoding is needed.

Significance. If the empirical pattern holds, the work is a useful methodological contribution: it quantifies an under-reported source of variance that already affects how leaderboards and structured-output systems are interpreted. Strengths include the scale of the study (140k generations), the token-control protocol with a matched-subset stability check (mean FSI 0.391 vs 0.394), bootstrap CIs over tasks, and concrete reporting recommendations. The metrics themselves are simple and reproducible. The result is incremental relative to prior format-sensitivity work (Sclar et al.; Chatterjee et al. POSIX) but is more tightly focused on structured-output compliance under a controlled evaluation protocol, which is practically relevant for both benchmarking and deployment.

major comments (3)
  1. [§3.5, §5.2] §3.5 and §5.2: Unparseable extractions are scored incorrect by construction, so Acc ≤ Parse for every cell. The reported Pearson r=0.825 and fixed-effects β̂=0.819 (partial R²=0.82) are therefore partly mechanical rather than pure behavioral mediation. The central claim that FSI is “largely explained by compliance failures” needs a decomposition that separates (i) parseability collapse from (ii) accuracy conditional on a successful parse. Please report Acc|parseable by model/wrapper/task, and/or an FSI decomposition into compliance-driven vs conditional-accuracy-driven components. Appendix B’s strict vs normalized extractors still zero unparseables and do not resolve this.
  2. [§3.1, §7, Table 2] §3.1 and §7: The headline “>30×” FSI contrast rests on four models that mix scale and family (7B–72B). Qwen-2.5-72B’s near-zero FSI may reflect size, post-training, or both; Phi-4’s extreme sensitivity may be idiosyncratic. With n=4 the cross-model claim is fragile. Either expand the model set (same scale, different families; or a scale ladder within family) or reframe the result as a demonstration that sensitivity can be extreme, not as a stable ranking of models by format robustness.
  3. [§5.2, Eq. (5)] Eq. (5) / §5.2: The regression is estimated at the (model, task, wrapper) cell level with fixed effects for all three factors. With one observation per cell, identification of β relies on additive residual structure; residual degrees of freedom and clustering (e.g., by example or seed) are not reported. Please state the effective sample, weighting scheme, and standard-error construction, and consider a hierarchical or example-level analysis so that the mediation claim is not driven by a few catastrophic cells (e.g., Phi-4 JSON: parseability 2.8%, accuracy 0.7%).
minor comments (6)
  1. [Table 1, Appendix A] Table 1 and Appendix A: Full wrapper templates are only summarized. For reproducibility, release the exact strings (including padding slots) used in the OpenRouter calls.
  2. [Eq. (4), Table 2] Eq. (4): nFSI divides by mean accuracy + 1e-6; when mean accuracy is near zero (as for Phi-4 under JSON), nFSI becomes very large and hard to interpret. Clarify intended use and consider a bounded alternative (e.g., range relative to chance-adjusted accuracy).
  3. [Figure 2] Figure 2: Overplotting of 140 cells makes density hard to read; a small-multiples or binned heatmap by model would help.
  4. [§2] Related work: Differentiate FSI more explicitly from Chatterjee et al. (POSIX) and Sclar et al., especially regarding structured-output compliance vs few-shot format sensitivity.
  5. [§3.4] §3.4: max_tokens=24 is aggressive for the deliberate (scratchpad) wrapper and may truncate reasoning; report completion-length distributions and sensitivity to this cap.
  6. [Abstract, header] Typo/style: “varies by over 30x” / “more than 30 times” is fine, but keep “×” vs “x” consistent; arXiv id line says 2 May 2026 (future date)—verify metadata.

Circularity Check

1 steps flagged

Mild definitional circularity in the mediation story: unparseable outputs are scored incorrect by construction, so Acc ≤ Parse and the strong accuracy–parseability link is partly forced by the scoring rule rather than pure behavioral mediation.

specific steps
  1. self definitional [§3.5 (Answer Extraction); §5.2 (Parseability mediates); Eq. (5)]
    "Outputs that cannot be mapped are marked unparseable and scored as incorrect. Accuracy is the mean of the 0/1 correctness indicator over all generations. We additionally compute parseability as the fraction of generations with a valid extracted answer. ... Across the 140 model, task, and wrapper cells, accuracy and parseability are strongly correlated with Pearson r = 0.825. ... Am,t,f = β Pm,t,f + αm + γt + δf + ε ... Parseability remains a strong predictor, with β̂ = 0.819 ± 0.034, explaining substantial residual variance with partial R² = 0.82."

    By the §3.5 scoring rule, every unparseable generation contributes 0 to accuracy, so Acc ≤ Parse holds identically for every cell. Accuracy is therefore Parse times conditional accuracy given parseability. When parseability collapses (e.g., Phi-4 JSON: parseability 2.8%, accuracy 0.7%), large FSI and the high r/β are largely the definitional cap rather than independent evidence that compliance mediates capability. The fixed-effects regression still conditions on model/task/wrapper, but cannot escape the bound Acc ≤ P that the paper itself imposed; the 'largely explained by compliance' claim is therefore partly true by construction of the outcome.

full rationale

This is an empirical methodology paper, not a first-principles derivation. FSI and PSI are descriptive range statistics over measured accuracies and parse rates; the >30× cross-model FSI contrast and the token-controlled protocol are independent empirical content, not fits recycled as predictions. There are no load-bearing self-citations, uniqueness theorems, or smuggled ansatzes. The only circularity is structural and partial: §3.5 defines unparseable extractions as incorrect, which forces Acc ≤ Parse for every cell, so the reported r = 0.825, fixed-effects β̂ ≈ 0.819, and the claim that compliance 'largely explains' FSI are mechanically inflated by the scoring bound (Acc = Parse × Acc|parseable). Appendix B’s strict/normalized variants still zero unparseables and do not break the tautology. That said, the paper’s practical warning about leaderboard fragility under standard extractors remains non-circular once the scoring convention is granted, and residual identification after model/task/wrapper fixed effects still has some independent content. Score 3 reflects one mild self-definitional step that does not collapse the central empirical findings.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

This is an empirical evaluation paper, not a first-principles derivation. The central claim rests on operational definitions (unparseable=incorrect; range over five hand-designed wrappers), a small model/task sample, and API-mediated generation rather than on free physical constants or new particles. Free parameters are protocol knobs; axioms are standard eval assumptions; invented entities are the two sensitivity indices as measurement constructs.

free parameters (5)
  • max_tokens = 24
    Fixed at 24 for all wrappers (§3.4). Can truncate deliberate/scratchpad outputs and interact with parseability; not swept.
  • character padding budget per task = task-specific (not numerically tabulated)
    Prompts padded to a fixed character budget with whitespace (§3.3). Exact budgets are protocol choices that affect residual token spread.
  • Δtok matched-subset threshold = 5%
    Secondary analysis filters groups with relative token spread ≤5% (§3.3, §5.4). Threshold is chosen, not derived.
  • nFSI epsilon = 1e-6
    Denominator uses mean accuracy + 10^{-6} (Eq. 4) to avoid division by zero; small ad hoc stabilizer.
  • examples per task and seeds = 200 examples, 5 seeds
    200 examples × 5 seeds define the sampling design (§3.1, §3.4); finite-sample noise affects range statistics.
axioms (5)
  • ad hoc to paper Unparseable extractions are scored as incorrect (accuracy is a 0/1 correctness indicator over generations).
    Stated in §3.5; load-bearing for the claim that compliance failures explain FSI. Standard in some harnesses but still a design axiom that forces Acc ≤ Parse.
  • domain assumption The five wrapper families (plain, json, structured, structured delim, deliberate) adequately represent format-induced variance of interest.
    §3.2 and Table 3; FSI/PSI are ranges over this finite set F, so sensitivity is relative to the chosen family, not all possible prompts.
  • domain assumption Task-aware answer extraction (first letter for MC, boolean map, last number for numeric) defines the ground-truth parse.
    §3.5 and Appendix B; strict vs normalized variants are similar, but results still depend on the extractor.
  • domain assumption OpenRouter-served instruct checkpoints behave as stable evaluation targets for the study window.
    §3.1; API routing, quantization, or backend changes could alter compliance rates.
  • standard math Fixed-effects linear model Am,t,f = β Pm,t,f + αm + γt + δf + ε is an appropriate cell-level summary of mediation.
    §5.2 weighted least-squares with model/task/wrapper fixed effects; standard but assumes additive structure on accuracy rates.
invented entities (3)
  • Format Sensitivity Index (FSI) independent evidence
    purpose: Quantify wrapper-induced accuracy range for a model-task pair (Eq. 2).
    New measurement construct; independent_evidence is operational (recomputable from any multi-wrapper eval) rather than physical.
  • Parseability Sensitivity Index (PSI) independent evidence
    purpose: Quantify wrapper-induced range in answer parseability (Eq. 3).
    Companion metric to separate compliance collapse from reasoning effects.
  • Normalized FSI (nFSI) independent evidence
    purpose: Scale FSI by mean accuracy to reduce dependence on absolute score level (Eq. 4).
    Derived normalization of FSI; still an author-defined sensitivity variant.

pith-pipeline@v1.1.0-grok45 · 11872 in / 3913 out tokens · 41632 ms · 2026-07-14T19:16:07.819962+00:00 · methodology

0 comments
read the original abstract

Prompt wrappers often differ only in formatting, yet they can change model scores enough to flip leaderboard conclusions. We study this variance under a token-controlled protocol and introduce two complementary metrics: the Format Sensitivity Index (FSI), the accuracy range induced by wrapper choice, and the Parseability Sensitivity Index (PSI), the corresponding range in answer parseability. Across 140,000 OpenRouter generations spanning 7 QA tasks, 5 wrapper families, and 4 instruct models from 7B to 72B parameters, we find that mean FSI varies by over 30x across models and is largely explained by compliance failures. A fixed-effects regression shows that parseability remains a strong predictor of accuracy even after controlling for task, model, and wrapper. We argue that reporting accuracy without wrapper variance and compliance is statistically fragile, and we give practical recommendations for both benchmarking and structured-output deployments.

Figures

Figures reproduced from arXiv: 2607.09665 by Deep Pankajbhai Mehta.

Figure 1
Figure 1. Figure 1: Mean FSI and PSI across tasks, with bootstrap confidence intervals over tasks from 10,000 [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Accuracy versus parseability across all model, task, and wrapper cells. Lower parseability [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Wrapper win counts over 28 model and task combinations. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of prompt-token spread ∆tok across wrappers, after padding to equal character budgets. compliance. In our study, sensitivity ranges from near zero to dominating the mean score. We argue that reliable benchmarking and reliable structured-output systems both require measuring, and often reducing, format sensitivity. Impact Statement This paper is about evaluation methodology. More transparent re… view at source ↗
Figure 5
Figure 5. Figure 5: Accuracy by model, wrapper, and task. 9 [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: FSI and PSI by model and task. Points are wrapper ranges. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

17 extracted references · 1 canonical work pages

  1. [1]

    Abdin, M. et al. Phi-4 technical report. Technical Report MSR-TR-2024-57, Microsoft,

    2024

  2. [2]

    Anthropic

    https://arxiv.org/abs/2412.08905. Anthropic. Tool use,

    Pith/arXiv arXiv

  3. [3]

    https://ojs.aaai.org/index.php/ AAAI/article/view/6239

    doi: 10.1609/aaai.v34i05.6239. https://ojs.aaai.org/index.php/ AAAI/article/view/6239. Chang, Y., Wang, X., Wang, J., Wu, Y., Zhu, K., Chen, H., Yang, L., Yi, X., Wang, C., Wang, Y., Ye, W., Zhang, Y., Chang, Y., Yu, P. S., Yang, Q., and Xie, X. A survey on evaluation of large language models.arXiv preprint arXiv:2307.03109,

    Pith/arXiv arXiv doi:10.1609/aaai.v34i05.6239

  4. [4]

    https://arxiv.org/abs/2307.03109

    doi: 10.48550/arXiv.2307.03109. https://arxiv.org/abs/2307.03109. Chatterjee, A., Renduchintala, H. S. V. N. S. K., Bhatia, S., and Chakraborty, T. Posix: A prompt sensitivity index for large language models. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 14550–14565. Association for Computational Linguistics,

    Pith/arXiv arXiv doi:10.48550/arxiv.2307.03109 2024

  5. [5]

    https://aclanthology.org/2024.findings-emnlp

    doi: 10.18653/v1/2024.findings-emnlp.852. https://aclanthology.org/2024.findings-emnlp. 852/. Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ: Exploring the surprising difficulty of natural yes/no questions. InProceedings of NAACL-HLT, pages 2924–2936. Association for Computational Linguistics,

    doi:10.18653/v1/2024.findings-emnlp.852 2024

  6. [6]

    https://aclanthology.org/N19-1300/

    doi: 10.18653/v1/N19-1300. https://aclanthology.org/N19-1300/. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? Try ARC, the AI2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018.https://arxiv.org/abs/1803.05457. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Ju...

    Pith/arXiv arXiv doi:10.18653/v1/n19-1300 2018

  7. [7]

    Geng, S., Josifoski, M., Peyrard, M., and West, R

    https: //arxiv.org/abs/2408.00118. Geng, S., Josifoski, M., Peyrard, M., and West, R. Grammar-constrained decoding for structured NLP. InProceedings of EMNLP, pages 12383–12414. Association for Computational Linguistics,

    Pith/arXiv arXiv

  8. [8]

    Jiang, A

    doi: 10.18653/v1/2023.emnlp-main.769.https://aclanthology.org/2023.emnlp-main.769/. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. M. Mistral 7B,

    work page doi:10.18653/v1/2023.emnlp-main.769.https://aclanthology.org/2023.emnlp-main.769/ 2023

  9. [9]

    11 Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al

    https: //arxiv.org/abs/2310.06825. 11 Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. Holistic evaluation of language models.arXiv preprint arXiv:2211.09110, 2022.https://arxiv.org/abs/2211.09110. OpenAI. Introducing structured outputs in the API,

    Pith/arXiv arXiv 2022

  10. [10]

    https://openai.com/index/ introducing-structured-outputs-in-the-api/. Park, J. S. et al. Grammar-aligned decoding.arXiv preprint arXiv:2405.21047,

    arXiv

  11. [11]

    Poesia, G

    doi: 10.48550/arXiv.2405.21047.https://arxiv.org/abs/2405.21047. Poesia, G. et al. Synchromesh: Constrained decoding for language models,

    doi:10.48550/arxiv.2405.21047.https://arxiv.org/abs/2405.21047

  12. [12]

    com/stanfordnlp/synchromesh

    https://github. com/stanfordnlp/synchromesh. Qwen Team. Qwen2.5: A party of foundation models, 2024.https://qwen2.org/qwen2.5/. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. WinoGrande: An adversarial Winograd schema challenge at scale. InProceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740,

    2024

  13. [13]

    https://ojs.aaai.org/ index.php/AAAI/article/view/6399

    doi: 10.1609/aaai.v34i05.6399. https://ojs.aaai.org/ index.php/AAAI/article/view/6399. Sclar, M. et al. Quantifying language models’ sensitivity to spurious features in prompt design or: How I learned to start worrying about prompt formatting. InInternational Conference on Learning Representations, 2024.https://arxiv.org/abs/2310.11324. Talmor, A., Herzig...

    Pith/arXiv arXiv doi:10.1609/aaai.v34i05.6399 2024

  14. [14]

    https://aclanthology.org/ N19-1421/

    doi: 10.18653/v1/N19-1421. https://aclanthology.org/ N19-1421/. Ugare, A. et al. Syncode: Efficient and general syntactical decoding for large language models. arXiv preprint arXiv:2403.01632,

    Pith/arXiv arXiv doi:10.18653/v1/n19-1421

  15. [15]

    https://arxiv.org/ abs/2403.01632

    doi: 10.48550/arXiv.2403.01632. https://arxiv.org/ abs/2403.01632. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800. Association for Computational Linguistics,

    Pith/arXiv arXiv doi:10.48550/arxiv.2403.01632

  16. [16]

    Zhu, K., Wang, J., Zhou, J., Wang, Z., Chen, H., Wang, Y., Yang, L., Ye, W., Gong, N

    doi: 10.18653/v1/P19-1472.https://aclanthology.org/P19-1472/. Zhu, K., Wang, J., Zhou, J., Wang, Z., Chen, H., Wang, Y., Yang, L., Ye, W., Gong, N. Z., Zhang, Y., and Xie, X. PromptBench: Towards evaluating the robustness of large language models on adversarial prompts.arXiv preprint arXiv:2306.04528,

    Pith/arXiv arXiv doi:10.18653/v1/p19-1472.https://aclanthology.org/p19-1472/

  17. [17]

    https://arxiv.org/abs/2306.04528

    doi: 10.48550/arXiv.2306.04528. https://arxiv.org/abs/2306.04528. 12

    Pith/arXiv arXiv doi:10.48550/arxiv.2306.04528