Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

The paper's central claim: in models without native reasoning, prompt complexity does not help; only minimal role framing (expert/inductive) beats a one-line prompt, while elaborate scaffolds match or underperform it by up to 30.7 points.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 14:37 UTC pith:BTNOSP4Z

load-bearing objection Large matched study with a real finding, but the CoT implementation is a mislabeled zero-shot trigger and the stats ignore clustering — fixable, worth refereeing. the 3 major comments →

arxiv 2607.14109 v1 pith:BTNOSP4Z submitted 2026-05-07 cs.CL cs.AI

Simplicity Paradox: Debunking myths about prompting and datasets for LLM evaluation

Inder Preet , Shuxin Lin , Dhaval Patel This is my paper
classification cs.CL cs.AI
keywords LLM evaluationprompting techniquesmultiple-choice question answeringchain-of-thoughtreasoning budgetsElo rankingbenchmark saturationparameter efficiency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish a 'simplicity paradox' in multiple-choice LLM evaluation: sophisticated prompting techniques do not reliably beat a one-line direct prompt. On a fully balanced comparison of 8 techniques across the same 10 non-reasoning model configurations and 10 datasets, plain baseline scores 52.08% mean accuracy, behind only CoT-Expert (55.71%) and CoT-Inductive (55.40%), while Chain-of-Thought, Plan-and-Solve, Self-Generate, and Self-Analogical match or underperform it. The authors also find that enabling reasoning at all is a much larger lever than scaling the reasoning budget, that parameter count poorly predicts Elo rank, and that most benchmarks are far from saturated. If correct, the paper redirects evaluation effort from prompt engineering toward base-model capability and harder benchmarks.

Core claim

Across 430,738 model–question evaluations spanning 8 prompting techniques, 10 MCQA datasets, and 27 model configurations, the paper claims that within the regime of models without a vendor-exposed reasoning toggle, prompt complexity is not what helps. The headline evidence is a matched 100-cell comparison in which every technique is run on the same 10 datasets and the same 10 non-reasoning configurations: baseline prompting ranks third at 52.08%, behind only two single-line role framings, CoT-Expert (55.71%) and CoT-Inductive (55.40%), by a statistically significant ~3 pp; every elaborate scaffold matches or underperforms baseline, with Self-Analogical 30.70 pp worse. Two further claims: tur

What carries the argument

The central mechanism is the matched comparison: a fully balanced 10×10 grid of (dataset, model) cells in which all 8 prompting techniques are evaluated on exactly the same questions and the same 10 model configurations that lack a vendor-exposed reasoning toggle. This design licenses paired t-tests on 100 cell-level accuracies, isolating the effect of the prompt from the effect of model capability. The technique application policy — external prompting applied only to non-reasoning configurations — keeps the elicited prompt from being conflated with a model's internal reasoning trace. Reasoning-budget analysis maps vendor labels to explicit token caps (nothink, low = 1,024 reasoning tokens,

Load-bearing premise

The central claim rests on treating each technique's implementation as faithful — notably CoT-Standard as a single zero-shot trigger rather than few-shot demonstrations — and on generalizing from the 10 non-reasoning configurations in the matched comparison to models with native reasoning.

What would settle it

Re-run the matched 100-cell comparison with a faithful few-shot CoT (3–5 curated demonstrations per dataset, as in the original method) and a tuned Self-Analogical template; if few-shot CoT beats baseline by more than the role-framing gain, or if tuned Self-Analogical clears the random floor, the claim that elaborate prompting does not help would need to be restricted to these specific instantiations.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • No multi-step reasoning scaffold reliably beats a one-line direct prompt on non-reasoning models; the only reliable gains come from minimal role framing (expert or inductive), worth about 3 percentage points.
  • For models with exposed thinking budgets, the dominant lever is enabling reasoning at all (up to +22.5 pp for the GPT-5 family), not increasing the token budget within the on-state (only 1–4 pp for an 8× increase).
  • Parameter count is a poor predictor of MCQA performance near the frontier: a 30B-parameter model can sit at the top of an Elo leaderboard ahead of models 4–13× larger, with the top eight configurations within 14.3 Elo points.
  • MCQA saturation claims are premature: 6 of 10 datasets average below 70% accuracy, 4 of 10 stay below 70% even for their best model, and the easiest-to-hardest spread is 47.5 percentage points.
  • Evaluation effort should shift from prompt engineering to model capability, architecture, and harder benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The CoT-Standard implementation uses a single zero-shot trigger rather than the few-shot exemplars of the originally cited method; a faithful few-shot CoT might erase or reverse the 9.45 pp deficit, so the paper's headline may be about prompt instantiation rather than the CoT technique class.
  • The finding that a small reasoning budget can underperform no reasoning (Claude low vs nothink) suggests adaptive, question-difficulty-based budget allocation could beat fixed budgets; the paper does not test this.
  • Baseline's low variance ('everywhere good, nowhere best') and the high variance of complex prompts suggest that ensembling or reranking a few role framings could combine consistency with peak performance, a testable extension the paper leaves implicit.
  • An Elo gap of ~4 points between ranks 1 and 2 corresponds to a 50.6% expected win rate; rankings that ignore this uncertainty will overstate differences among top models.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper reports a large-scale empirical evaluation of eight prompting techniques on ten MCQA datasets across 27 model configurations, totaling roughly 430,000 model–question evaluations. The central result is a balanced 'matched' comparison on 100 (dataset, model) cells, restricted to the ten configurations without a vendor-exposed reasoning toggle. There, baseline direct prompting achieves 52.08% mean accuracy and ranks third, behind CoT-Expert (55.71%) and CoT-Inductive (55.40%), while Prompt-Repeat is statistically indistinguishable from baseline; the more elaborate scaffolds — Self-Generate, Plan-and-Solve, CoT, and Self-Analogical — match or underperform baseline, with deficits up to 30.7 pp. The paper also presents Elo rankings across models, a reasoning-budget analysis for thinking-enabled models, and a dataset-difficulty analysis. The authors conclude that, within the non-reasoning regime, prompt complexity is not what helps; that parameter count is a poor predictor of MCQA performance; that enabling reasoning matters more than scaling its budget; and that benchmarks retain substantial headroom.

Significance. If the matched comparison is accepted, this is a useful and unusually well-powered empirical contribution: it uses a unified protocol, evaluates every technique on the same cells, repeats the same ~4,300 questions across many configurations to enable variance estimation, logs token costs, and includes a failure-mode audit for Self-Analogical. The finding that two minimal role-framings give a small but consistent gain while several multi-step scaffolds do not beat baseline is a valuable corrective to the 'more prompting is better' assumption. The main caveats are implementation fidelity — especially the CoT-Standard condition — and the statistical treatment of clustered cells. The paper's stated scope is narrow ('models without native reasoning'), and the limitations section is honest about that scope; the contribution would be strengthened if the implementation and inference concerns below are addressed.

major comments (3)
  1. [§3.3, Table 2; §4.1] Table 2 implements 'CoT-Standard [34]' with the single trigger 'Let me think step by step.' This is the zero-shot CoT trigger from Kojima et al. [9], not the few-shot exemplar-based CoT protocol of Wei et al. [34], which is the cited source. Since CoT underperforms baseline by 9.45 pp on the matched set (Table 3) and is one of the four 'elaborate scaffolds' supporting Finding 1, the headline comparison may be between baseline and a one-line variant rather than chain-of-thought prompting as originally proposed. The Limitations section's statement that templates were not tuned does not resolve the mismatch. Please either include the few-shot exemplar condition from [34] with full templates in the appendix, or relabel the condition as 'zero-shot CoT' and adjust the claim that chain-of-thought prompting underperforms baseline.
  2. [§3.4, §4.1] The paired t-test treats the 100 matched cells as independent observations. But the cells share the same 10 datasets and the same 10 models, so the cell accuracies are clustered; the p-values in Table 3 are likely anti-conservative. The comparison also tests seven techniques without any multiple-comparison control. This matters for the claims that CoT-Expert and CoT-Inductive are 'statistically significant' and that Prompt-Repeat is 'statistically indistinguishable' from baseline. Please provide clustered inference (e.g., block bootstrap by dataset or by model, or a mixed model with random effects for dataset and model) and report multiplicity-adjusted p-values or confidence intervals. The heterogeneity in dataset sizes (e.g., AIME 2025 has 30 questions, PhysicsQA has 45) further argues against unweighted cell means as independent observations.
  3. [§3.2, Table 4, §5] The matched set is described as the ten configurations 'without a vendor-exposed reasoning toggle,' but Table 4 lists gpt-oss-20b-medium and gpt-oss-120b-medium with thinking budget 'medium' and marks both as 'in matched set.' If these are reasoning-enabled configurations, the matched set is not confined to the non-reasoning regime that scopes Finding 1. If 'medium' is a model variant label rather than a reasoning budget, the table should say so explicitly. Please clarify, and if necessary re-run the matched analysis excluding or reclassifying these two configurations.
minor comments (4)
  1. [Abstract/Appendix D] The abstract says 'roughly 4,300 unique questions,' but the record counts in Appendix Table 5 sum to 4,848. Please specify any subsampling or filtering and report the per-dataset evaluation counts actually used, since the Self-Analogical audit implies about 435 questions per dataset per model.
  2. [§3.3/Appendix A] No code, data, or full prompt templates are released; Table 2 gives only trigger summaries. Given that the central result depends on faithful implementations of named techniques, releasing the exact prompts and exemplars would substantially strengthen reproducibility.
  3. [§4.1] The claim that baseline has 'the lowest standard deviation (19.83)' among non-trivial techniques is given only in the text; reporting per-technique standard deviations in Table 3 or Figure 3 would make the consistency claim easier to verify.
  4. [Appendix A] The implementation note that Self-Analogical was not tuned and was originally validated on math reasoning is appropriately cautious, but the paper should consider whether the 21.38% matched mean should be presented as the headline deficit or as a lower bound with the failure-mode breakdown more prominently in the main text.

Circularity Check

0 steps flagged

No significant circularity: the headline result is measured accuracy over independent/external benchmarks; only a minor, non-load-bearing self-cited dataset is present.

full rationale

The central derivation chain is empirical rather than deductive. Finding 1 rests on measured mean accuracies across the matched 100 (dataset, model) cells in Table 3 and Figure 3; no predicted quantity is defined in terms of fitted constants. CoT-Expert/CoT-Inductive beat baseline by ~3 pp and CoT, Plan-and-Solve, Self-Generate, and Self-Analogical match or underperform baseline; these are observed outcomes, not consequences of the definitions of the techniques. The Elo analysis uses standard settings ("K= 32, initial rating 1500", §3.5) on pairwise per-question outcomes, not parameters fit to produce the Qwen surprise. The reasoning-budget analysis maps vendor labels to explicit token caps (nothink / low=1,024 / medium=8,192) and reports measured deltas such as +22.5 pp for GPT-5; the "enabling reasoning is the dominant lever" conclusion is an observation, not an artifact of construction. The only self-citation is FailureSensorIQ [3] in §3.1, co-authored by two of the present authors; it contributes one of ten datasets and is not used as a premise for the headline, so excluding it would not materially change the matched comparison. The paper's Limitations honestly concede implementation exposure ("we did not tune exemplar count, prompt wording, or output format per model", §5), including the CoT-Standard trigger "Let me think step by step" in Table 2/§3.3; this is a prompt-fidelity/validity risk rather than a circular reduction, since the result would be falsifiable under a different faithful CoT implementation. No equation in the paper reduces a prediction to its own input. Score 2 is assigned only because the rubric treats one minor, non-load-bearing self-citation as a low-level flag; no circular step was identified.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

No physical or conceptual entities are postulated. The central claims rest on several domain assumptions about parser fidelity, model representativeness, technique implementation fidelity, and statistical independence; these are listed above. The self-cited dataset FailureSensorIQ is one of ten and not load-bearing for the headline result.

free parameters (3)
  • Elo K = 32
    Standard Elo update constant chosen by the authors, not fitted to data. It affects the spread of ratings but not the qualitative claim that parameter count is a poor predictor.
  • Reasoning token caps (low=1,024, medium=8,192) = 1024 / 8192 tokens
    Hand-chosen caps mapped to vendor labels; the step-vs-plateau result depends on these specific values, but they are experimental design, not fitted to optimize accuracy.
  • Elo initial rating = 1500
    Conventional Elo initialization, not fitted; does not affect relative ordering materially.
axioms (5)
  • domain assumption Accuracy as measured by a single rule-based parser (first letter after fixed marker, regex fallback) reflects the model's true answer on MCQA items.
    The paper uses one parser across all techniques; if a technique produces valid answers in formats the parser cannot extract, its accuracy is understated. Self-Analogical format-collapse is explicitly acknowledged in Appendix A.
  • domain assumption The 10 model configurations without vendor-exposed reasoning toggles are representative enough to support conclusions about prompting effectiveness for non-reasoning models.
    Matched comparison is restricted to these 10 configs (§3.2, §3.4); results may not generalize to other non-reasoning models or to reasoning-capable models in nothink mode.
  • standard math Paired t-tests on 100 (dataset, model) cell means can treat cells as independent observations.
    Cells share model and dataset effects, so the effective sample size is smaller than 100; p-values may be overconfident. The paper does not use clustered or mixed-effect tests.
  • domain assumption Technique implementations faithfully represent the cited techniques.
    CoT-Standard is implemented with the single trigger 'Let me think step by step' (Table 2, §3.3), closer to zero-shot CoT [9] than to the few-shot exemplar-based CoT [34]; Self-Analogical is implemented without tuning exemplar count (Appendix A).
  • domain assumption Reasoning-token caps from vendor labels correspond to actual reasoning compute budgets.
    Maps low=1,024 and medium=8,192 reasoning tokens; vendor implementations may not honor exact caps, affecting the budget analysis in §4.3.

pith-pipeline@v1.3.0-alltime-deepseek · 15863 in / 12291 out tokens · 125654 ms · 2026-08-02T14:37:39.282773+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Simplicity Paradox: Debunking myths about prompting and datasets for LLM evaluation." pith.science (2026). https://pith.science/paper/BTNOSP4Z

@misc{pith2026260714109,
  author       = {Pith},
  title        = {Pith review of: Simplicity Paradox: Debunking myths about prompting and datasets for LLM evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BTNOSP4Z}},
  note         = {Machine review of arXiv:2607.14109}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Probing the capabilities of Large Language Models (LLMs) and building robust solutions for Multiple-Choice Question Answering (MCQA) remain central challenges in natural language understanding. Furthermore, the rapid proliferation of LLMs has created the implicit assumption that more sophisticated prompting techniques yield better performance. Several studies claim better performance with more sophisticated prompting techniques, but do not provide a comprehensive evaluation. We address this gap through a comprehensive empirical study of 8 prompting techniques across 10 multiple-choice question answering (MCQA) datasets, encompassing 27 model configurations and roughly 4,300 unique questions evaluated more than 430,000 times. Our findings reveal a striking paradox that baseline prompting consistently outperforms complex reasoning techniques on various benchmarks. Only minimal expert and inductive role framing (CoT-Expert and CoT-Inductive) yields a small but statistically significant $\sim$3 percentage-point (pp) gain over baseline whereas every other elaborate technique we tested matches or under-performs it, often by large margins (up to 31~pp for Self-Analogical). We further investigate three critical phenomena: (1) the unexpected victory of Qwen3-30B-A3B-Thinking-2507 in Elo ratings, (2) the performance-efficiency trade-offs across model variants with different thinking budgets, revealing model-dependent optimal configurations, and (3) the substantial variation in dataset difficulty, with 60% of benchmarks below 70% accuracy and a 47.5~pp spread from easiest to hardest, indicating considerable room for model improvement. These results suggest that the LLM evaluation community may be overcomplicating prompt engineering and that substantial performance gaps remain across diverse benchmarks, offering opportunities for genuine model improvements rather than prompt optimization.

Figures

Figures reproduced from arXiv: 2607.14109 by Dhaval Patel, Inder Preet, Shuxin Lin.

Figure 1
Figure 1. Figure 1: End-to-end ReasonLab evaluation pipeline. A (technique, model, dataset) cell is dispatched [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Dataset suite at a glance. (a) Design space: mean question length vs. mean number of answer options; bubble area is proportional to record count, colour separates single-domain from multi-domain benchmarks. (b) Answer-distribution skew: proportion of correct answers in the most￾frequent letter class; the black tick marks the uniform baseline (1/k for k options). Bars past the tick indicate answer-key bias.… view at source ↗
Figure 3
Figure 3. Figure 3: Matched comparison of 8 prompting techniques. Restricted to the 100 (dataset, model) cells in which every technique was evaluated against the same 10 model configurations. (a) Mean accuracy per technique with 95% confidence intervals. Baseline ranks third, behind CoT-Expert and CoT-Inductive. (b) Per-cell distribution of (baseline − technique) accuracy in pp; positive values indicate baseline wins. W/T/L g… view at source ↗
Figure 4
Figure 4. Figure 4: Mean accuracy by technique × dataset on the matched set. Per-dataset best technique is outlined in red. CoT-Expert wins 6 datasets, CoT-Inductive 3, and Self-Generate 1 (AIME 2025); baseline wins none outright despite ranking third overall. 1100 1200 1300 1400 1500 1600 1700 Elo rating qwen3-30b-medium claude-4-sonnet-low gpt-5-4-medium gpt-oss-120b-medium claude-4-sonnet-medium claude-4-sonnet-nothink gpt… view at source ↗
Figure 5
Figure 5. Figure 5: Elo leaderboard across all 27 model configurations. Pairwise comparisons across 10 datasets and 8 techniques (K = 32, initial rating 1500, 85K–101K matches per model). The top eight configurations cluster within 14.3 Elo points; the rank-1 lead corresponds to a 50.6% expected head-to-head win rate. Colour denotes model family. 4.2 The Qwen Surprise [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: reports the resulting accuracy–cost curves alongside paired variant deltas. Two regimes 0 250 500 750 1000 1250 1500 Mean reasoning tokens / question (matched datasets) 55 60 65 70 75 Mean accuracy (%) nothink low medium low medium nothink low medium nothink low medium (a) Accuracy vs reasoning budget claude-4-sonnet gemini-2.5-pro gpt-5-2025-08-07 gpt-5-4 0 10 20 Δ accuracy (pp, paired) claude-4-sonnet me… view at source ↗
Figure 7
Figure 7. Figure 7: Dataset difficulty under baseline prompting. Each row shows the range of accuracies across all evaluated model configurations (grey bar = min–max), the cross-model mean (blue circle), and the best-performing model (green triangle). 6/10 datasets have mean accuracy below 70%; 4/10 remain below 70% even for their best model. The easiest-to-hardest spread is 47.5 pp. than the discourse suggests, and it sits i… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B

    cs.CL 2026-07 accept novelty 5.5

    Self-inspection methods (Self-Refine, Reflexion, Best-of-N self-verify) lose to equal-token repeated sampling on math from 1.5B to 7B; no tested method reliably wins.

Reference graph

Works this paper leans on

41 extracted references · 18 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Matharena: Evaluating llms on uncontaminated math competitions, February 2025

    Mislav Balunovi ´c, Jasper Dekoninck, Ivo Petrov, Nikola Jovanovi ´c, and Martin Vechev. Matharena: Evaluating llms on uncontaminated math competitions, February 2025. URL https://matharena.ai/

    2025

  2. [2]

    On the measure of intelligence.arXiv preprint arXiv:1911.01547, 2019

    François Chollet. On the measure of intelligence.arXiv preprint arXiv:1911.01547, 2019

    Pith/arXiv arXiv 1911

  3. [3]

    Failuresensoriq: A multi-choice qa dataset for understanding sensor relationships and failure modes, 2025

    Christodoulos Constantinides, Dhaval Patel, Shuxin Lin, Claudio Guerrero, Sunil Dagajirao Patil, and Jayant Kalagnanam. Failuresensoriq: A multi-choice qa dataset for understanding sensor relationships and failure modes, 2025. URLhttps://arxiv.org/abs/2506.03278

    Pith/arXiv arXiv 2025

  4. [4]

    DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    DeepSeek-AI, Daya Guo, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    The language model evaluation harness, July 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The languag...

    arXiv 2024

  6. [6]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Pith/arXiv arXiv 2009

  7. [7]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences, 11(14):6421, 2021

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams.Applied Sciences, 11(14):6421, 2021

    2021

  8. [8]

    Dynabench: Rethinking benchmarking in NLP

    Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. Dynabench: Rethinking benchmarking in NLP. InProceedings of the 202...

    2021

  9. [9]

    Large language models are zero-shot reasoners.Advances in Neural Information Processing Systems, 35:22199–22213, 2022

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners.Advances in Neural Information Processing Systems, 35:22199–22213, 2022

    2022

  10. [10]

    Time-mqa: Time series multi-task question answering with context enhancement, 2025

    Yaxuan Kong, Yiyuan Yang, Yoontae Hwang, Wenjie Du, Stefan Zohren, Zhangyang Wang, Ming Jin, and Qingsong Wen. Time-mqa: Time series multi-task question answering with context enhancement, 2025. URLhttps://arxiv.org/abs/2503.01875

    Pith/arXiv arXiv 2025

  11. [11]

    Prompt repetition improves non-reasoning llms.arXiv preprint arXiv:2512.14982, 2025

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Prompt repetition improves non-reasoning llms.arXiv preprint arXiv:2512.14982, 2025. URLhttps://arxiv.org/abs/2512.14982

    arXiv 2025

  12. [12]

    Holistic evaluation of language models.arXiv preprint arXiv:2211.09110, 2022

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models.arXiv preprint arXiv:2211.09110, 2022

    Pith/arXiv arXiv 2022

  13. [13]

    Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 8086–8098, Dublin, Ireland, 2022. Association for Comput...

    doi:10.18653/v1/2022.acl-long.556 2022

  14. [14]

    SuperGPQA: Scaling llm evaluation across 285 graduate disciplines

    M-A-P Team, Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, Kang Zhu, et al. SuperGPQA: Scaling llm evaluation across 285 graduate disciplines. InAdvances in Neural Information Processing Systems Datasets and Benchmarks Track, 2025. arXiv:2502.14739. 10

    Pith/arXiv arXiv 2025

  15. [15]

    McIntosh, Teo Susnjak, Tong Liu, Paul Watters, and Malka N

    Timothy R. McIntosh, Teo Susnjak, Tong Liu, Paul Watters, and Malka N. Halgamuge. Inad- equacies of large language model benchmarks in the era of generative artificial intelligence. arXiv preprint arXiv:2402.09880, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. InEMNLP, 2018

    2018

  17. [17]

    State of what art? a call for multi-prompt LLM evaluation.Transactions of the Association for Computational Linguistics, 12:933–949, 2024

    Moran Mizrahi, Guy Kaplan, Dan Malkin, Rotem Dror, Dafna Shahaf, and Gabriel Stanovsky. State of what art? a call for multi-prompt LLM evaluation.Transactions of the Association for Computational Linguistics, 12:933–949, 2024. doi: 10.1162/tacl_a_00681

    doi:10.1162/tacl_a_00681 2024

  18. [18]

    s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    Large language models sensitivity to the order of options in multiple-choice questions

    Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of options in multiple-choice questions. InFindings of the Association for Computational Linguistics: NAACL 2024, pages 2006–2017, Mexico City, Mexico, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-naacl.130

    doi:10.18653/v1/2024.findings-naacl.130 2024

  20. [20]

    Smith, and Mike Lewis

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 5687–5711, Singapore, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.378

    doi:10.18653/v1/2023.findings-emnlp.378 2023

  21. [21]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Qwen Team. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling (COLM), 2024. arXiv:2311.12022

    Pith/arXiv arXiv 2024

  23. [23]

    Leveraging large language models for multiple choice question answering

    Joshua Robinson, Christopher Michael Rytting, and David Wingate. Leveraging large language models for multiple choice question answering. InThe Eleventh International Conference on Learning Representations (ICLR), 2023

    2023

  24. [24]

    Rogers, Inna Goncearenco, Giuseppe Sarli, Igor Galynker, Denis Peskoff, Marine Carpuat, Jules White, Shyamal Anadkat, Alexander Hoyle, and Philip Resnik

    Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, HyoJung Han, Sevien Schulhoff, Pranav Sandeep Dulepet, Saurav Vidyadhara, Dayeon Ki, Sweta Agrawal, Chau Pham, Gerson Kroiz, Feileen Li, Hudson Tao, Ashay Srivastava, Hevander Da Costa, Saloni Gupta, Megan L. Rogers, Inna Goncearenco, G...

    2024

  25. [25]

    Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. InThe Twelfth International Conference on Learning Representations (ICLR), 2024

    2024

  26. [26]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. InThe Thirteenth International Conference on Learning Representations (ICLR), 2025. arXiv:2408.03314

    Pith/arXiv arXiv 2025

  27. [27]

    To CoT or not to CoT? chain- of-thought helps mainly on math and symbolic reasoning.arXiv preprint arXiv:2409.12183, 2024

    Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To CoT or not to CoT? chain- of-thought helps mainly on math and symbolic reasoning.arXiv preprint arXiv:2409.12183, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rastogi, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022. 11

    Pith/arXiv arXiv 2022

  29. [29]

    On the self-verification limitations of large language models on reasoning and planning tasks

    Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. On the self-verification limitations of large language models on reasoning and planning tasks. InThe Thirteenth International Conference on Learning Representations (ICLR), 2025

    2025

  30. [30]

    Correctbench: A benchmark of self-correction in llms

    Guiyao Tie, Zenghui Yuan, Zeli Zhao, Chaoran Hu, Tianhe Gu, Ruihang Zhang, Sizhe Zhang, Junran Wu, Xiaoyue Tu, Ming Jin, Qingsong Wen, Lixing Chen, Pan Zhou, and Lichao Sun. Correctbench: A benchmark of self-correction in llms. InProceedings of the NeurIPS 2025 Datasets and Benchmarks Track, 2025

    2025

  31. [31]

    Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models

    Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee- Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2609–2634, Toronto, Canada, 2023. Association ...

    doi:10.18653/v1/2023.acl-long.147 2023

  32. [32]

    Self-consistency improves chain of thought reasoning in language models, 2023

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023. URLhttps://arxiv.org/abs/2203.11171

    Pith/arXiv arXiv 2023

  33. [33]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.arXiv preprint arXiv:2406.01574, 2024

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.arXiv preprint arXiv:2406.01574, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. InAdvances in Neural Information Processing Systems, volume 35, pages 24824–24837, 2022. URL https://proceedings.neurips.cc/paper/2022/hash/ 9d5609613524ecf4f15af0f7b31abca4-...

    2022

  35. [35]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems, volume 36, 2023

    2023

  36. [36]

    Chi, and Denny Zhou

    Michihiro Yasunaga, Xinyun Chen, Yujia Li, Panupong Pasupat, Jure Leskovec, Percy Liang, Ed H. Chi, and Denny Zhou. Large language models as analogical reasoners. InThe Twelfth International Conference on Learning Representations (ICLR), 2024

    2024

  37. [37]

    Generate rather than retrieve: Large language models are strong context generators.arXiv preprint arXiv:2209.10063, 2022

    Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. Generate rather than retrieve: Large language models are strong context generators.arXiv preprint arXiv:2209.10063, 2022. URL https://arxiv. org/abs/2209.10063

    Pith/arXiv arXiv 2022

  38. [38]

    Large language models are not robust multiple choice selectors

    Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. InThe Twelfth International Conference on Learning Representations (ICLR), 2024

    2024

  39. [39]

    Scaling physical reasoning with the physics dataset, 2025

    Shenghe Zheng, Qianjia Cheng, Junchi Yao, Mengsong Wu, Haonan He, Ning Ding, Yu Cheng, Shuyue Hu, Lei Bai, Dongzhan Zhou, Ganqu Cui, and Peng Ye. Scaling physical reasoning with the physics dataset, 2025. URLhttps://arxiv.org/abs/2506.00022

    arXiv 2025

  40. [40]

    PromptBench: A unified library for evaluation of large language models.Journal of Machine Learning Research, 2024

    Kaijie Zhu, Qinlin Zhao, Hao Chen, Jindong Wang, and Xing Xie. PromptBench: A unified library for evaluation of large language models.Journal of Machine Learning Research, 2024. arXiv:2312.07910

    Pith/arXiv arXiv 2024

  41. [41]

    Avg. opt

    Zitnik Lab. CURE-Bench: Competition on reasoning models for drug decision-making in precision therapeutics. NeurIPS 2025 Competition Track, 2025. URL https:// curebench.ai. Hugging Face dataset: https://huggingface.co/datasets/update0909/ cure-bench-reasoning-traces. 12 A Self-Analogical failure-mode analysis Self-Analogical’s 21.38% mean accuracy on the ...

    2025