Pith. sign in

REVIEW 4 major objections 5 minor 47 references

LLM answers flip under meaning-preserving paraphrases even when average accuracy barely moves.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 13:40 UTC pith:ZSYNGQLO

load-bearing objection I buy the central finding—stable aggregate accuracy can mask large instance-level instability under paraphrase—though the judge-invariance gap needs a control before the headline numbers fully land. the 4 major comments →

arxiv 2607.22554 v1 pith:ZSYNGQLO submitted 2026-05-18 cs.AI cs.CLcs.LG

Same Question, Different Answers: Evaluating LLM Reliability Beyond Accuracy

Kazem Faghih , Yize Cheng , Shoumik Saha , Mobina Pournemat , Armin Gerami , Soheil Feizi This is my paper
classification cs.AI cs.CLcs.LG
keywords LLM reliabilityparaphrase consistencysemantic equivalenceaccuracy metricslatent capabilityreliability gapquestion answeringmathematical reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that standard accuracy numbers hide a large amount of wording-induced instability in large language models. Using rigorously filtered paraphrase sets across four benchmarks and thirteen models, the authors show that while average accuracy changes only slightly (usually 0–5%), individual questions frequently switch between correct and incorrect answers depending on phrasing. They propose that model knowledge is better described by a band: latent capability (correct on at least one paraphrase) is often far higher than reliable capability (correct on all paraphrases), with single-prompt accuracy sitting in between. If right, this reframes benchmark evaluation: a model's score on a fixed wording is a fragile snapshot, and consistency across equivalent inputs should be measured and reported. The paper also shows that a simple self-paraphrasing step can recover part of the latent knowledge at inference time.

Core claim

Across four benchmarks and 13 models, with deterministic decoding and paraphrase sets filtered for meaning preservation, average accuracy changes only about 0–5% (max 7.3%) when questions are reworded, yet up to 23% of questions receive different correctness labels and up to 47% of paraphrase sets contain conflicting predictions. Conditioning on questions answered correctly in the original wording, models often flip to incorrect on a paraphrase (flip rates frequently above 10%); conversely, paraphrases often elicit correct answers where the original failed. The authors formalize this as a reliability gap: Aany (correct on at least one paraphrase) sits above original accuracy, while Astrict (

What carries the argument

The framework's load-bearing object is the validated paraphrase set: each original question is rewritten several ways, then passed through lexical-overlap filtering (below 0.8), an LLM judge, and bidirectional NLI entailment (threshold 0.75) to guarantee semantic equivalence. On top of these sets the paper defines a label-distribution suite—mismatch rate, non-determinism, flip rates, entropy, and the capability extremes Aany and Astrict—so that stability is measured per instance rather than through aggregate accuracy. The crucial comparison is original-wording correctness versus the distribution of correctness across paraphrases, which exposes offsetting errors that average accuracy hides.

Load-bearing premise

The claim collapses if any retained paraphrase changes the question's constraints, assumptions, or answer, because then the measured flips would come from invalid rewording rather than model instability.

What would settle it

Take every question where a model is correct on the original wording but wrong on a paraphrase and have multiple human annotators re-check those paraphrase pairs for strict semantic equivalence; if a sizable fraction are judged to differ in constraints or answer, the mismatch and flip rates are inflated by invalid paraphrases.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Benchmark scores computed from one fixed wording are not a reliable measure of what a model can consistently do; reporting reliable capability (Astrict) alongside ordinary accuracy would reveal a reliability gap that accuracy alone hides.
  • Deployment evaluations should sample several meaning-preserving phrasings, because a model that answers the original wording correctly can fail on equivalent questions from users.
  • Apparent stability of aggregate accuracy can be a statistical artifact of offsetting correct-to-incorrect and incorrect-to-correct flips, so stable leaderboard ranks do not imply stable instance-level behavior.
  • A lightweight inference-time self-paraphrasing step—having the model generate paraphrases and then answer—improves correctness for most tested models, partially bridging the gap between latent and reliable capability.
  • Larger scale and chain-of-thought reasoning reduce but do not eliminate paraphrase sensitivity.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If answer flips under valid paraphrases are truly bidirectional, then contamination detectors that flag performance drops on reworded questions may confound memorization with paraphrase sensitivity; a model could appear contaminated simply because retrieval is wording-dependent, and consistency metrics might offer a contamination-robust alternative.
  • The self-paraphrasing result suggests a testable training-time corollary: objectives that reward paraphrase-consistent answers could compress the reliability gap more directly than optimizing single-prompt accuracy alone.
  • For math tasks, value-level non-determinism means correctness labels can agree while the model emits different numerical answers; this could be used as a probe for whether a model is genuinely computing a solution or pattern-matching a memorized result.
  • The reliability gap predicts that at inference time, sampling multiple paraphrases and taking a majority vote should already beat single-prompt accuracy on some tasks, making multi-paraphrase evaluation a cheap deployment pattern for high-stakes question answering.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a framework for evaluating LLM reliability under meaning-preserving paraphrase variation. For each of four benchmarks (SimpleQA, TruthfulQA, GSM8K, MATH-500), the authors generate multiple paraphrases per question with GPT-4.1-mini, filter them by lexical-overlap and NLI-entailment thresholds, and validate a subset through human annotation. They then evaluate 13 models under deterministic decoding, using metrics such as average paraphrase accuracy, majority-vote accuracy, mismatch rate, directional flip rates, non-determinism P(ND), and latent/reliable capability (Aany/Astrict). The central finding is that aggregate accuracy changes by only a few points under paraphrasing, while instance-level labels are far less stable: mismatch rates up to about 23% and P(ND) up to about 47%. The paper also reports a reliability gap between Aany and Astrict and proposes a self-paraphrasing strategy as a partial remedy.

Significance. If the findings hold, the paper makes a valuable contribution to LLM evaluation: it provides a concrete, human-validated protocol for constructing paraphrase sets and a metric suite that separates aggregate performance from instance-level consistency. The robustness checks with stricter multi-judge semantic filtering (Appendix C), the broad model coverage, and the absence of fitted parameters in the main metrics are strengths. The study also articulates a practical consequence for benchmark design: single-prompt accuracy can materially misrepresent reliable capability. However, the headline conclusions currently rest on label-based metrics whose own sensitivity to wording is not controlled, so the quantitative claims need additional validation before the paper's conclusions can be fully accepted.

major comments (4)
  1. [Appendix E.2; Tables 2 and 9-12] The headline mismatch, P(ND), and flip rates for SimpleQA and TruthfulQA are computed from labels assigned by a four-LLM judge protocol. The judge prompt fixes the evaluation template but not the question text, so a judge can be sensitive to the same paraphrase manipulation the paper attributes to the model. The agreement rule (≥3/4 judges, including GPT-4.1-mini and GPT-3.5-turbo) filters idiosyncratic disagreement but not a systematic wording bias shared by all judges. No control is reported in which the same model output is labeled under the original and paraphrased questions. Please add such a judge-invariance test and, if judge flips are non-negligible, report corrected bounds on the headline quantities.
  2. [Section 3.2; Appendix A] The paper attributes observed variation to paraphrasing because decoding is deterministic, but no same-prompt repeat baseline is reported. For locally hosted open-weight models determinism can be verified, but for proprietary API models temperature-0 sampling is not guaranteed to be deterministic. Without measuring same-prompt label variance, some of the reported P(ND) and flip rates could reflect sampling noise rather than paraphrase sensitivity. Please report a same-prompt repeat control for at least the API models, with the same evaluation pipeline, and state the resulting same-prompt P(ND).
  3. [Section 4.5; Table 3] The claim that self-paraphrasing improves performance for 6/7 models is based solely on the Correct rate. For GPT-3.5-turbo, Correct rises from 9.2% to 12.3%, but Incorrect rises from 39.5% to 68.2% while Not Attempted falls from 51.3% to 19.4%; the intervention converts many abstentions into wrong answers. If abstention is preferable to hallucination in deployment, this is not an improvement. Please report error rates and a severity-sensitive analysis, or qualify the claim as 'increases correct rate at the cost of more incorrect answers.'
  4. [Section 4.2; Tables 16-22] Many quantitative claims are reported without confidence intervals or significance tests. MATH-500 sample sizes are only n≈113-146, and several conditional statistics such as P(Aany|¬Aorig)=100% for Qwen-3 4B +Thinking and Qwen-3 14B +Thinking are based on very small denominators. Please add bootstrap confidence intervals for the main metrics (accuracy gaps, mismatch, P(ND), Aany, Astrict), especially for MATH-500 and conditional flip rates, and re-state cross-model comparisons that are not statistically separable.
minor comments (5)
  1. [Title page/Figure 1] The GitHub link appears as '/gtbProject:' and should be 'GitHub Project:'; numbered callouts in Figure 1 are referenced as ' in Figure.1' with missing labels. Typos and formatting errors of this kind should be cleaned up.
  2. [Appendix C; Tables 4 and Figures 4-5] Mistral 7B v0.2 and Phi-1.5 appear in the stricter-filtering robustness tables and figures but are not part of the 13 models in the main experiments. Please clarify whether these are auxiliary models or add a sentence explaining their role.
  3. [Section 4.3; Appendix F] The claim that Aany 'approximates the upper bound of the model's latent knowledge' is presented as a brief justification in Appendix F, but the argument is heuristic and depends on the number and diversity of paraphrases. Consider calling Aany a 'best-case observed accuracy' or 'latent-capability heuristic' and state explicitly that it is not a formal upper bound on knowledge.
  4. [Section 4.5] The two-stage and one-stage self-paraphrasing implementations are described, but the preliminary experiment showing 'no meaningful performance difference' is not reported. Please provide the comparison or remove the claim.
  5. [References] GPT-4.1 and GPT-4.1-mini are cited to the GPT-4 technical report [1]; a dedicated reference or model card citation for GPT-4.1 would be more appropriate. Also, 'GPT-5.4-mini' in Appendix C is an unusual name and should be clarified.

Circularity Check

1 steps flagged

Central instability finding is empirically self-contained; only minor definitional relabeling of Aany/Astrict.

specific steps
  1. self definitional [Section 3.2.3 (Latent Knowledge Boundaries) and Section 4.3 (The Reliability Gap)]
    "Maximum Capability (Aany): The proportion of instances where the model generates the correct answer for at least one variation in Qi. This quantity, which we also refer to as best-case accuracy, approximates the upper bound of the model’s latent knowledge. [...] Reliable Capability (Astrict): The proportion of instances where the model answers correctly for all variations in Qi, representing robust and reliable knowledge. [...] Taken together, these results reveal a consistent ordering across models, Astrict < Aorig ≈ Adist < Aany."

    Because Qi always contains the original Q0_i, Aany ≥ Aorig and Astrict ≤ Aorig follow immediately from the any-correct/all-correct definitions. Thus the existence of a nonnegative reliability gap and the bounding of Aorig by Astrict and Aany are true by construction, not discovered. The empirical content is only the magnitude of the gaps; presenting the ordering as a 'revealed' result is a relabeling of the metrics. Appendix F's chance-correctness argument partially justifies interpreting Aany as latent knowledge, which is why this is minor rather than central.

full rationale

The paper's headline quantities (mismatch rate, P(ND), flip rates) are computed directly from deterministic model outputs across paraphrase sets and do not depend on any fitted parameter or on a self-citation chain. The paraphrase sets are externally validated (human agreement 92–98%, stricter multi-judge filtering preserves conclusions), and the main instability finding would stand even if the Aany/Astrict terminology were removed. The only notable self-citation ([6] in Appendix H.1) is used to support a background premise about contamination and is not load-bearing for the central claim. The multi-judge evaluation includes GPT-3.5-turbo, GPT-4.1-mini, Qwen2.5-7B, and Qwen3-8B as judges for models that include those same systems; this is a real validity threat (judge wording sensitivity could contribute to label flips) but it is an empirical confound, not a by-construction circularity, and the paper's agreement rule and robustness checks partially mitigate it. Overall, the derivation is self-contained; the only circular element is the definitional interpretation of the capability bounds, so the score is low.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

No physical entities are postulated. The main load-bearing design choices are paraphrase retention thresholds, judge agreement rules, and paraphrase count; these are hand-set and affect the magnitude of the reported metrics.

free parameters (4)
  • lexical_overlap_threshold = 0.8
    Paraphrases with overlap ≥0.8 against the original are discarded; chosen by hand, affects which paraphrases are tested (§3.1).
  • nli_entailment_threshold = 0.75
    Bidirectional NLI entailment threshold for semantic retention; chosen by hand (§3.1).
  • judge_agreement_requirement = 3 of 4 (incl. GPT-4.1-mini and GPT-3.5-turbo)
    Answer labels are accepted only when three of four judges agree; this protocol choice determines the label distributions used in all metrics (§E.2).
  • paraphrase_count_per_question = not fixed; avg 8.44 in stricter setting
    The number of paraphrases per instance affects Aany and Astrict and is not held constant across validation settings (Table 4).
axioms (5)
  • domain assumption Retained paraphrases are semantically equivalent to the original question.
    Relies on GPT-4.1-mini judge, BART-MNLI, and ~1250 human checks; supports causal attribution of flips to wording (§3.1, §C).
  • domain assumption Deterministic decoding removes sampling noise.
    No same-prompt repeat baseline is reported; API models at temperature=0 are not guaranteed deterministic (§3.2).
  • domain assumption Multi-judge labels are unbiased ground truth for correctness.
    Labels are accepted when 3/4 judges agree, but judges include models under evaluation (GPT-3.5-turbo and GPT-4.1-mini) (§E.2).
  • ad hoc to paper Random correct guessing is negligible, so Aany reflects latent knowledge.
    Appendix F argues chance correct is small; for math answers with normalized numerical labels, the effective chance may be larger (§F).
  • standard math Standard probability calculus for label distributions.
    Definitions of mismatch, flip rates, entropy, and majority vote use standard probability (§3.2).

pith-pipeline@v1.3.0-alltime-deepseek · 34543 in / 12175 out tokens · 124544 ms · 2026-08-02T13:40:03.356305+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) often achieve strong accuracy on benchmarks, yet it remains unclear how reliably they apply this knowledge when the same question is phrased in different but equivalent ways. In this work, we study how model answers change under meaning-preserving paraphrases across factual question answering and mathematical reasoning tasks. Across four benchmarks and 13 models, we find that model outputs frequently depend on the exact wording of the prompt. While overall accuracy typically changes only modestly across paraphrases, instance-level behavior is far less stable: for many questions, models alternate between correct and incorrect answers depending on phrasing, with mismatch rates reaching more than 23%. Conditioning on questions that are answered correctly in their original form reveals even larger failures measured by answer flip rates, showing that single-prompt correctness is often a poor indicator of reliability. At the same time, we find that models often produce a correct answer for at least one paraphrase of a question, suggesting that the underlying knowledge is present but inconsistently retrieved. Building on this observation, we show that a simple self-paraphrasing strategy can partially recover this latent knowledge and improve performance at inference time. Together, these findings suggest that standard accuracy metrics can mask substantial instability, and that evaluating consistency across equivalent inputs provides a clearer picture of LLM reliability.

Figures

Figures reproduced from arXiv: 2607.22554 by Armin Gerami, Kazem Faghih, Mobina Pournemat, Shoumik Saha, Soheil Feizi, Yize Cheng.

Figure 1
Figure 1. Figure 1: Overview of the evaluation framework. For each input question, we generate multiple [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Left: Accuracy varies across paraphrases on TRUTHFULQA. Bars show range; orange = Adist, red = Aorig. Right: Relationship between Aorig, Adist, Aany, and Astrict, revealing the reliability gap between latent and reliable capability. Detailed results are provided in Tables 19–22 [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Effect of scale and CoT on accuracy (solid) and mismatch (dashed). To decouple the effects of parameter count and whether or not the model operates in reason￾ing mode, we conduct controlled experiments using six variants of the Qwen 3 model fam￾ily [42] (spanning 0.6B, 1.7B, 4B, 8B, 14B, and 32B). Each model is evaluated both with and without long Chain-of-Thought (CoT) rea￾soning [PITH_FULL_IMAGE:figures… view at source ↗
Figure 4
Figure 4. Figure 4: Performance ranges across paraphrases under progressively stricter semantic filtering. The [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Relationship between single-prompt accuracy ( [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Relationship between single-prompt accuracy ( [PITH_FULL_IMAGE:figures/full_fig_p020_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Accuracy ranges across meaning-preserving paraphrases for all remaining evaluated datasets. [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The effect of model scale and reasoning mode on performance. Accuracy (solid lines) [PITH_FULL_IMAGE:figures/full_fig_p021_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 1 canonical work pages

  1. [1]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Alzahrani, H

    N. Alzahrani, H. Alyahya, Y . Alnumay, S. Alrashed, S. Alsubaie, Y . Almushayqih, F. Mirza, N. Alotaibi, N. Al-Twairesh, A. Alowisheq, et al. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1...

    2024

  3. [3]

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amo...

    Pith/arXiv arXiv 2020

  4. [4]

    Burnell, W

    R. Burnell, W. Schellaert, J. Burden, T. D. Ullman, F. Martinez-Plumed, J. B. Tenenbaum, D. Rutar, L. G. Cheke, J. Sohl-Dickstein, M. Mitchell, et al. Rethink reporting of evaluation results in ai.Science, 380(6641):136–138, 2023

    2023

  5. [5]

    Carlini and D

    N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In2017 ieee symposium on security and privacy (sp), pages 39–57. Ieee, 2017

    2017

  6. [6]

    Cheng, W

    Y . Cheng, W. Wang, M. Moayeri, and S. Feizi. DyePack: Provably flagging test set con- tamination in LLMs using backdoors. In C. Christodoulopoulos, T. Chakraborty, C. Rose, and V . Peng, editors,Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 15356–15373, Suzhou, China, Nov. 2025. Association for Compu- tatio...

    work page doi:10.18653/v1/2025.emnlp-main.776 2025

  7. [7]

    M. Choi. Roparq: Paraphrase-aware alignment of large language models towards robustness to paraphrased questions.arXiv preprint arXiv:2511.21568, 2025

    arXiv 2025

  8. [8]

    Chowdhery, S

    A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y . Tay, N. Shazeer, V . Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghemawat, S. Dev,...

    Pith/arXiv arXiv 2022

  9. [9]

    Cobbe, V

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems, 2021. URLhttps://arxiv.org/abs/2110.14168

    Pith/arXiv arXiv 2021

  10. [10]

    Dekoninck, M

    J. Dekoninck, M. Müller, and M. Vechev. Constat: Performance-based contamination detection in large language models.Advances in Neural Information Processing Systems, 37:92420– 92464, 2024

    2024

  11. [11]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding, 2019. URLhttps://arxiv.org/abs/1810.04805

    Pith/arXiv arXiv 2019

  12. [12]

    Elazar, N

    Y . Elazar, N. Kassner, S. Ravfogel, A. Ravichander, E. Hovy, H. Schütze, and Y . Goldberg. Measuring and improving consistency in pretrained language models.Transactions of the Association for Computational Linguistics, 9:1012–1031, 2021

    2021

  13. [13]

    W. C. Gan and H. T. Ng. Improving the robustness of question answering systems to question paraphrasing. InProceedings of the 57th annual meeting of the association for computational linguistics, pages 6065–6075, 2019

    2019

  14. [14]

    Ghosh, S

    B. Ghosh, S. Hasan, N. A. Arafat, and A. Khan. Logical consistency of large language models in fact-checking.arXiv preprint arXiv:2412.16100, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Golchin and M

    S. Golchin and M. Surdeanu. Time travel in llms: Tracing data contamination in large language models.arXiv preprint arXiv:2308.08493, 2023

    Pith/arXiv arXiv 2023

  16. [16]

    Golchin and M

    S. Golchin and M. Surdeanu. Data contamination quiz: A tool to detect and estimate contami- nation in large language models, 2024. URLhttps://arxiv.org/abs/2311.06233

    Pith/arXiv arXiv 2024

  17. [17]

    I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014

    Pith/arXiv arXiv 2014

  18. [18]

    Grattafiori, A

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Y . Han, H. Seo, H. Namgoong, and S. Jung. An analysis of the impact of problem paraphrasing on llm-based mathematical problem solving. InProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pages 383–395, 2025

    2025

  21. [21]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxiv. org/abs/2103.03874

    Pith/arXiv arXiv 2021

  22. [22]

    Jiang, Y

    L. Jiang, Y . Chai, M. Li, M. Liu, R. Fok, N. Dziri, Y . Tsvetkov, M. Sap, A. Albalak, and Y . Choi. Artificial hivemind: The open-ended homogeneity of language models (and beyond), 2025. URLhttps://arxiv.org/abs/2510.22954

    arXiv 2025

  23. [23]

    Khashabi, S

    D. Khashabi, S. Min, T. Khot, A. Sabharwal, O. Tafjord, P. Clark, and H. Hajishirzi. Unifiedqa: Crossing format boundaries with a single qa system, 2020. URL https://arxiv.org/abs/ 2005.00700. 11

    Pith/arXiv arXiv 2020

  24. [24]

    W. Li, L. Li, T. Xiang, X. Liu, W. Deng, and N. Garcia. Can multiple-choice questions really be useful in detecting the abilities of LLMs? In N. Calzolari, M.-Y . Kan, V . Hoste, A. Lenci, S. Sakti, and N. Xue, editors,Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024),...

    2024

  25. [25]

    S. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods,

  26. [26]

    Lunardi, V

    R. Lunardi, V . Della Mea, S. Mizzaro, and K. Roitero. On robustness and reliability of benchmark-based evaluation of llms.arXiv preprint arXiv:2509.04013, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Meier, J

    D. Meier, J. P. Wahle, T. L. Ruas, and B. Gipp. Towards human understanding of paraphrase types in large language models. InProceedings of the 31st International Conference on Computational Linguistics, pages 6298–6316, 2025

    2025

  28. [28]

    Mirzadeh, K

    I. Mirzadeh, K. Alizadeh, H. Shahrokhi, O. Tuzel, S. Bengio, and M. Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models.arXiv preprint arXiv:2410.05229, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Mizrahi, G

    M. Mizrahi, G. Kaplan, D. Malkin, R. Dror, D. Shahaf, and G. Stanovsky. State of what art? a call for multi-prompt llm evaluation.Transactions of the Association for Computational Linguistics, 12:933–949, 2024

    2024

  30. [30]

    Nalbandyan, R

    G. Nalbandyan, R. Shahbazyan, and E. Bakhturina. Score: Systematic consistency and robust- ness evaluation for large language models.arXiv preprint arXiv:2503.00137, 2025

    Pith/arXiv arXiv 2025

  31. [31]

    Pezeshkpour and E

    P. Pezeshkpour and E. Hruschka. Large language models sensitivity to the order of options in multiple-choice questions. InFindings of the Association for Computational Linguistics: NAACL 2024, pages 2006–2017, 2024

    2024

  32. [32]

    F. M. Polo, R. Xu, L. Weber, M. Silva, O. Bhardwaj, L. Choshen, A. F. M. de Oliveira, Y . Sun, and M. Yurochkin. Efficient multi-prompt evaluation of llms.arXiv preprint arXiv:2405.17202, 2024

    Pith/arXiv arXiv 2024

  33. [33]

    Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y . Fan, Y . Su, Y . Zhang, Y . Wan, Y . Liu, Z. Cui, Z. Zhang, ...

    Pith/arXiv arXiv 2025

  34. [34]

    Raffel, N

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. URL https://arxiv.org/abs/1910.10683

    Pith/arXiv arXiv 2023

  35. [35]

    H. Raj, V . Gupta, D. Rosati, and S. Majumdar. Improving consistency in large language models through chain of guidance.arXiv preprint arXiv:2502.15924, 2025

    Pith/arXiv arXiv 2025

  36. [36]

    M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh. Beyond accuracy: Behavioral testing of nlp models with checklist.arXiv preprint arXiv:2005.04118, 2020

    Pith/arXiv arXiv 2005

  37. [37]

    Sclar, Y

    M. Sclar, Y . Choi, Y . Tsvetkov, and A. Suhr. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. arXiv preprint arXiv:2310.11324, 2023

    Pith/arXiv arXiv 2023

  38. [38]

    A. K. Singh, M. Y . Kocyigit, A. Poulton, D. Esiobu, M. Lomeli, G. Szilvasy, and D. Hupkes. Evaluation data contamination in llms: how do we measure it and (when) does it matter?, 2024. URLhttps://arxiv.org/abs/2411.03923

    Pith/arXiv arXiv 2024

  39. [39]

    Szegedy, W

    C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks.arXiv preprint arXiv:1312.6199, 2013. 12

    Pith/arXiv arXiv 2013

  40. [40]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023. URL https: //arxiv.org/abs/2201.11903

    Pith/arXiv arXiv 2023

  41. [41]

    J. Wei, N. Karina, H. W. Chung, Y . J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus. Measuring short-form factuality in large language models, 2024. URL https://arxiv.org/ abs/2411.04368

    Pith/arXiv arXiv 2024

  42. [42]

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. ...

    Pith/arXiv arXiv 2025

  43. [43]

    ask the exact same thing

    Y . Zhou, Y . Zhu, D. Antognini, Y . Kim, and Y . Zhang. Paraphrase and solve: Exploring and exploiting the impact of surface form on mathematical reasoning in large language models. arXiv preprint arXiv:2404.11500, 2024. 13 A Limitations Our analysis relies on a finite set of automatically generated and filtered paraphrases, and therefore does not fully ...

    Pith/arXiv arXiv 2024

  44. [45]

    Generate several paraphrases of the question

  45. [46]

    Use the original question and paraphrases as context to better understand the user’s intent

  46. [47]

    any-correct

    Provide a single final answer. Question: [original question] H Additional Discussion H.1 Paraphrase Sensitivity and Benchmark Contamination Our findings also affect how benchmark contamination is interpreted. It’s often argued that existing popular benchmarks suffer from test set contamination, and that current models have effectively “seen” almost all pu...

    1985

  47. [2022]

    URLhttps://arxiv.org/abs/2109.07958

    Pith/arXiv arXiv