Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

REST: Stress Testing Large Reasoning Models by Asking Multiple Problems at Once

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Asking large reasoning models to solve several problems in one prompt causes large accuracy drops, exposing a hidden weakness that single-question benchmarks miss.

desk verdict Central claim survives the extraction concern; a solid empirical contribution that deserves referee time, with extraction and order-robustness checks to add. read the letter →

arxiv 2507.10541 v2 pith:CMJIYLIE submitted 2025-07-14 cs.CL

Zhuoshi Pan , Qizhi Pei , Yu Li , Qiyao Sun , Zinan Tang , H. Vicky Zhao , Conghui He , Lijun Wu This is my paper · ORCID
classification cs.CL
keywords stresstestingmulti-questionpromptinglargereasoningmodelsbenchmarksaturationoverthinkinglong2shorttrainingdiscriminativepowerevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

REST (Reasoning Evaluation through Simultaneous Testing) claims that current large reasoning models are not reliable multi-problem solvers: when several questions are concatenated into one prompt, even state-of-the-art models lose substantial accuracy, with DeepSeek-R1 dropping 29.1% on AIME24. The paper argues this effect is not just output truncation but reflects deeper misbehaviors, including question omission, summary errors, and a tendency to overthink early questions at the expense of later ones. If true, REST gives a low-cost way to make saturated benchmarks discriminative again and a new axis for evaluating real-world multi-context reasoning. It also identifies a promising remedy: models trained with long2short (length-rewarded) techniques preserve more of their single-question accuracy under stress.

What carries the argument

The load-bearing object is the stress-level-$s$ prompt set $P^s$, built by composing $s$ consecutive questions into one instruction with cyclic indexing, so each benchmark question appears exactly once in every position. The protocol measures the model's accuracy on each $P^s$ by extracting every answer from the single generated response, then compares accuracy across stress levels. This construction is what converts a saturated benchmark into a graded stress test: it imposes controlled multi-problem cognitive load, makes positional and cross-question interference visible, and produces an error taxonomy (question omission, summary error, output truncation, endless repetition, format violation, reasoning error) that single-question testing cannot reveal.

What would settle it

Re-score all REST responses with an LLM-based extractor and compare the stress-level accuracy curves for verbose overthinking models against concise long2short models; if the extraction gap is systematically larger for verbose outputs and closes the performance gap, the overthinking-based explanation is a measurement artifact.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that REST, which concatenates $s$ consecutive benchmark questions into one prompt with cyclic indexing, reveals a reasoning-capability gap invisible to single-question evaluation. Across 34 models and 7 benchmarks, accuracy falls as $s$ grows, and models that look nearly identical on single questions separate sharply under stress: R1-7B and R1-32B score 93.0% and 94.6% on MATH500 singly but 66.75% and 88.97% under REST. Error analysis attributes the drop to a small set of failure modes, mainly question omission and reasoning errors in large models and endless repetition and truncation in small ones. The paper also finds a clear position effect, with earlier questions answered far more accurately, and an order effect, with easy-to-hard prompts beating hard-to-easy ones, and it identifies overthinking of early questions as a central mechanism: models that burn too many tokens on the first question leave too little for the rest.

Load-bearing premise

The reported REST accuracy gaps are real model behavior rather than artifacts of the rule-based answer extractor, which the paper's own appendix shows scores 3 to 5 percentage points below LLM-based extraction.

Editorial extensions

If this is right

  • Existing benchmarks like MATH500 and AMC23 can be made challenging again by concatenation, without requiring new human-written questions.
  • REST rank-orders models that appear tied at near-ceiling single-question accuracy, giving evaluation signal where standard benchmarks have saturated.
  • The overthinking trap is a concrete failure mechanism: models that allocate reasoning effort poorly across multiple questions suffer the largest drops.
  • Long2short or length-rewarded training appears to be a practical mitigation, preserving more single-question accuracy under stress.
  • Post-training gains measured on single questions do not automatically transfer to multi-question scenarios, so REST offers a complementary check on model improvement claims.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors do not claim this, but REST's format should make memorized single-question answers harder to apply directly, which could give existing benchmarks a partial, cheap defense against contamination.
  • Because REST accuracy depends strongly on question position and order, REST scores are prompt-sensitive; comparing models fairly will require position-balanced and order-controlled prompt construction.
  • A direct test of the measurement pipeline would be to re-score all responses with an LLM-based extractor; the paper's own appendix reports 3 to 5 percentage point gaps, so if verbose models are systematically under-extracted, part of the overthinking penalty is a measurement artifact.
  • REST could be extended to other reasoning domains and to more stress levels to map how degradation scales with model size and training objective.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. REST (Reasoning Evaluation through Simultaneous Testing) transforms existing reasoning benchmarks by concatenating s consecutive questions into a single prompt and measures accuracy at each stress level s, reporting the average over s > 1. The paper evaluates 34 large reasoning models on seven benchmarks and reports three main findings: (i) even DeepSeek-R1 drops substantially under REST (29.1 points on AIME24); (ii) REST distinguishes models that are near-tied in single-question accuracy; and (iii) verbose "overthinking" behavior and the absence of long2short compression training are associated with larger degradation. It also analyzes error types, output truncation, question position and order, and compares rule-based vs LLM-based answer extraction.

Significance. The main observation that multi-question prompting sharply lowers accuracy on challenging reasoning benchmarks is large and practically important, and the paper's breadth (34 models, 7 benchmarks), public code, 128K token-limit control in Appendix D, and the detailed error taxonomy in Appendix G are genuine strengths. The enhanced-discriminative-power and long2short conclusions are, however, only as credible as the answer-extraction pipeline, and the validation of that pipeline covers only two models. If the extraction gap is resolved and uncertainty is reported, REST would be a valuable evaluation contribution; as it stands, the artifact-free claim in Section 5.1 is too strong.

major comments (3)
  1. [§5.1; Appendix C, Table 4] The default rule-based extractor under-scores REST responses relative to the LLM-based extractor by 3-5 points for R1-7B on MATH500 and AMC23 (Table 4: 66.75 vs 70.59; 36.06 vs 41.05) and by smaller amounts for R1-32B. This is large enough to affect some rankings in Tables 1 and 2. Section 5.1's statement that the degradation is intrinsic because no Format Violation errors were found only rules out one artifact class; it does not address parsing or answer-to-question alignment. The long2short advantage (Section 4.2, Figures 3-4) could be partly an extraction artifact if concise outputs are easier to parse. The headline drops are much larger than the extraction gap, so the central observation is not overturned, but the paper should either adopt LLM-based extraction for the main tables or report parser-error bounds and re-check the key conclusions for all 34 models.
  2. [§4.1; Tables 1-2] Tables 1-2 report single point estimates for most stress accuracies; only AIME24, AIME25, and AMC23 use 8 sampling runs, and no confidence intervals or significance tests are reported anywhere. Some of the discriminative-power claims rest on margins of a few points (e.g., Table 1: Qwen-QwQ-32B vs R1-32B on MATH500 differ by 3.52 points; Table 2: O3-mini vs Gemini-2.5-Flash-Thinking on LiveCodeBench differ by 0.02 points). Without an estimate of run-to-run variance, "stronger discriminative power" is not established for these small margins. Please add bootstrap confidence intervals or per-seed results, at least for the models used in the headline comparisons.
  3. [§5.2, Figure 8] Section 5.2 uses R1-7B's fail rate as a difficulty proxy but does not specify whether the proxy is computed from single-question or REST responses or how the ordering is generated for each stress level. Because the hard-first condition always places the hard question at position 1, the observed effect is confounded with the positional bias documented in Figure 7; a design that counterbalances difficulty and position (or conditions on first-question difficulty) is needed to conclude that presenting questions from easy to hard consistently yields better overall accuracy.
minor comments (5)
  1. [Table 2] The header says "GQPA Diamond"; it should be "GPQA Diamond".
  2. [§4.2] "supervised fune-tuning" should be "supervised fine-tuning".
  3. [Appendix C] The statement that LLM-based extraction "may introduce additional instability" is not supported by any measurement; either report an agreement or stability statistic or omit the claim.
  4. [§5.1, Figure 5] The no-Format-Violation statement is made for the four models in Figure 5 only; the following sentence generalizes to all 34 models and should be qualified.
  5. [Figure 5] The text refers to "Nemotron-7B" while Table 1 uses "Nemotron-Nano-8B"; use one consistent model name throughout.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: REST's accuracy drops are direct empirical measurements, and self-citations are not load-bearing.

full rationale

REST is an empirical evaluation protocol. It concatenates externally defined benchmark questions and measures model accuracy with an explicit scoring rule, so the reported degradations (e.g., DeepSeek-R1's 29.17% AIME24 drop) are direct measurements rather than quantities derived from fitted parameters or self-referential definitions. The 'Stress' score is the explicit average over stress levels greater than 1 defined in Sec. 3.2 and Sec. 4.1, making Single-vs-Stress a protocol comparison rather than a definitional identity. Findings on overthinking, position bias, question order, and long2short training are empirical comparisons between conditions or model families; no equation in the paper reduces to its own inputs by construction. The paper's self-citations (MathFusion [36], Metal-adder [26], Lemma [35], CipherBank [25]) appear only in related-work and model-selection context and are not load-bearing for REST's central claims. Appendix C's rule-based vs LLM-based extraction differences (3-5 points on two models) are a measurement-validity concern about parsing long or verbose responses, not a circularity: they do not make any reported accuracy equal to an input by construction. I therefore find no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities, parameters, or mathematical constructs beyond experimental design choices. The stress levels, sampling runs, token limits, and averaging rule are hand-selected and affect the reported outcomes, while the listed assumptions are necessary for interpreting the measurements as valid evidence about model behavior.

free parameters (4)
  • Stress levels per benchmark = GSM8K: {1,3,6,9,12}; MATH500/AMC23: {1,3,5,7,9}; AIME/GPQA/LiveCodeBench: {1,2,3,4,5}
    Chosen by hand based on benchmark difficulty; the reported stress accuracy depends on this selection.
  • Number of sampling runs = 8 for AIME24/AIME25/AMC23; 1 otherwise
    Affects variance of reported accuracies; no error bars are provided for single-run benchmarks.
  • Max output token limit = 32K for reasoning models, 8K for non-reasoning; 128K in ablation
    Can truncate lengthy multi-question responses; the ablation suggests negligible impact, but the default could still affect results.
  • Reporting metric: average over stress levels >1 = Average of levels II, III, IV, V
    The choice of summary statistic changes the headline 'Stress' numbers; reporting each stress level separately would preserve more information.
assumptions (4)
  • domain assumption The prompt 'Answer the above questions one by one' elicits a representative sample of the model's multi-problem reasoning ability.
    The validity of REST as a stress test depends on this prompt being a fair elicitation; no comparison against other multi-question prompt formats is provided.
  • domain assumption Rule-based extraction is a sufficiently accurate measurement of correctness.
    Table 4 shows LLM-based extraction yields 3-5 point higher accuracy, indicating rule-based extraction misclassifies some correct answers; the paper nonetheless uses it as default.
  • domain assumption The order of questions within concatenated prompts does not systematically bias stress-level comparisons.
    Section 5.2 shows question order affects accuracy (easy-first vs hard-first), yet the main REST evaluation uses fixed consecutive ordering; cyclic shifts do not cover all order permutations.
  • domain assumption The evaluated benchmarks are not contaminated, or contamination does not interact with stress level.
    The paper motivates REST by data contamination but does not test whether memorized answers survive concatenation; if contamination is present, REST may not escape it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of REST: Stress Testing Large Reasoning Models by Asking Multiple Problems at Once." pith.science (2026). https://pith.science/paper/CMJIYLIE

@misc{pith2026250710541,
  author       = {Pith},
  title        = {Pith review of: REST: Stress Testing Large Reasoning Models by Asking Multiple Problems at Once},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CMJIYLIE}},
  note         = {Machine review of arXiv:2507.10541}
}
read the original abstract

Recent Large Reasoning Models (LRMs) have achieved remarkable progress on task-specific benchmarks, yet their evaluation methods remain constrained by isolated problem-solving paradigms. Existing benchmarks predominantly assess single-question reasoning through sequential testing, resulting critical limitations: (1) vulnerability to data contamination and less challenging (e.g., DeepSeek-R1 achieves 97.0% on MATH500), forcing costly creation of new questions with large human efforts, (2) failure to evaluate models under multi-context pressure, a key requirement for real-world deployment. To bridge this gap, we present REST (Reasoning Evaluation through Simultaneous Testing), a stress-testing framework that exposes LRMs to multiple problems simultaneously. Beyond basic reasoning, REST evaluates several under-tested capabilities: contextual priority allocation, cross-problem interference resistance, and dynamic cognitive load management. Our evaluation reveals several striking findings: Even state-of-the-art (SOTA) models like DeepSeek-R1 exhibit substantial performance degradation under stress testing. Crucially, REST demonstrates stronger discriminative power than existing benchmarks, revealing pronounced performance differences among models that exhibit similar, near-ceiling performance under single-question evaluations. Some key insights emerge from our analysis: (1) the "overthinking trap" is a critical factor contributing to the performance degradation; (2) the models trained with "long2short" technique preserve more accuracy of their single-problem performance under REST, outperforming standard-trained counterparts. These results establish REST as a cost-efficient, future-proof evaluation paradigm that better reflects real-world reasoning demands while reducing reliance on continuous human annotation. Code and results are available at https://opendatalab.github.io/REST.

Figures

Figures reproduced from arXiv: 2507.10541 by the authors.

Figure 1
Figure 1. Illustration of REST evaluation compared with single-question evaluation. REST concatenates [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Performance comparison of LRMs of different sizes under various stress levels. REST reveals [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Limitation of post training on REST. Post training exacerbates performance degradation [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: The effect of Long2Short training on REST. Long2Short training mitigates the performance [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: The error type distribution for various LRMs on AIME24 under REST. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: The reasoning token count for questions at different positions on AIME24. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: The effect of question position under stress tests. The performance at each position is [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: The effect of question order on overall performance under stress tests. LRMs consistently [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Evaluation prompt for math tasks. Prompt 2: Prompt for GQPA Diamond {Questions}. Answer the above multiple-choice question one by one. Remember to give each answer in the following format: ‘ANSWER: \boxed{LETTER}’ (without quotes) where LETTER is one of ABCD [PITH_FUL…
Figure 10
Figure 10. Figure 10: Evaluation prompt for GPQA Diamond. Prompt 3: Prompt for Code Generation {Questions}. Answer the above questions one by one. Enclose the code for each question within delimiters as follows. ```python #YOUR CODE HERE ```. ### Answer: (use the provided format with backt…
Figure 11
Figure 11. Figure 11: Evaluation prompt for code generation tasks. [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Prompt for extracting answers from the response. [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: Performance degradation across different question types on MATH500. [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]
Figure 14
Figure 14. Figure 14: Example of Output Truncation error on DS-R1-Distill-Qwen-7B. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_14.png]
Figure 15
Figure 15. Figure 15: Example of Endless Repetition error on DS-R1-Distill-Qwen-7B. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_15.png]
Figure 16
Figure 16. Figure 16: Example of Question Omission error on DS-R1-Distill-Qwen-7B. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_16.png]
Figure 17
Figure 17. Figure 17: Example of Summary Error on DS-R1-Distill-Qwen-7B. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_17.png]
Figure 18
Figure 18. Figure 18: Example of Reasoning Error on DS-R1-Distill-Qwen-7B. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_18.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Thinking Hard, Not Smart: Reasoning Models Fail to Ration Test-Time Compute Across Questions

    cs.CL 2026-08 conditional novelty 7.0 of 10

    Across seven reasoning models and two domains, shared test-time compute is allocated by presentation position rather than by question value or difficulty, and planning prompts widen coverage without improving prioritization.

  2. ConPress: Learning Efficient Reasoning from Multi-Question Contextual Pressure

    cs.CL 2026-02 conditional novelty 6.0 of 10

    Multi-question prompts elicit shorter chain-of-thought traces, and fine-tuning on those traces transfers the compression to single-question reasoning.

  3. Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning

    cs.AI 2025-07 conditional novelty 6.0 of 10

    Training a Qwen2.5-7B model with GRPO on math and puzzle data improves both domains, code transfer depends on the starting model, and template or reward mismatches sharply hurt performance.

  4. From Atomic Actions to Standard Operating Procedures: Iterative Tool Optimization for Self-Evolving LLM Agents

    cs.AI 2026-07 conditional novelty 5.0 of 10

    EvoSOP enables LLM agents to self-evolve by iteratively synthesizing atomic tool actions into reusable Standard Operating Procedures, improving success rates and reducing reasoning rounds on ACEBench and Tau2Bench.

Reference graph

Works this paper leans on

69 extracted references · 26 canonical work pages · cited by 4 Pith papers

  1. [1]

    L1: Controlling how long a reasoning model thinks with reinforcement learning.arXiv preprint arXiv:2503.04697, 2025

    Pranjal Aggarwal and Sean Welleck. L1: Controlling how long a reasoning model thinks with reinforcement learning.arXiv preprint arXiv:2503.04697, 2025

    arXiv 2025

  2. [2]

    AIMO Validation AIME Dataset

    AI-MO. AIMO Validation AIME Dataset

    work page

  3. [3]

    Training language models to reason efficiently.arXiv preprint arXiv:2502.04463, 2025

    Daman Arora and Andrea Zanette. Training language models to reason efficiently.arXiv preprint arXiv:2502.04463, 2025

    arXiv 2025

  4. [4]

    Llama-nemotron: Efficient reasoning models, 2025

    Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, Ido Shahaf, Oren Tropp, Ehud Karpas, Ran Zilberstein, Jiaqi Zeng, Soumye Singhal, Alexander Bukharin, Yian Zhang, Tugrul Konuk, Gerald Shen, Ameya Sunil Mahabalesh- warkar, Bilal Kartal, Yoshi Suhara, Olivier Delalleau, Zi...

    arXiv 2025

  5. [5]

    Do not think that much for 2+ 3=? on the overthinking of o1-like llms.arXiv preprint arXiv:2412.21187, 2024

    Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of o1-like llms.arXiv preprint arXiv:2412.21187, 2024

    arXiv 2024

  6. [6]

    Batch prompting: Efficient inference with large language model apis

    Zhoujun Cheng, Jungo Kasai, and Tao Yu. Batch prompting: Efficient inference with large language model apis. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry T rack, pages 792–810, 2023

    work page 2023

  7. [8]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  8. [9]

    Process reinforcement through implicit rewards.arXiv preprint arXiv:2502.01456, 2025

    Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et al. Process reinforcement through implicit rewards.arXiv preprint arXiv:2502.01456, 2025

    arXiv 2025

Show all 69 references
  1. [10]

    Open r1: A fully open reproduction of deepseek-r1, January 2025

    Hugging Face. Open r1: A fully open reproduction of deepseek-r1, January 2025. URL https://github.com/ huggingface/open-r1

    2025

  2. [11]

    Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.arXiv preprint arXiv:2503.01307, 2025

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.arXiv preprint arXiv:2503.01307, 2025

    2025 arXiv

  3. [12]

    Omni-math: A universal olympiad level mathematic benchmark for large language models

    Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic benchmark for large language models. arXiv preprint arXiv:2410.07985, 2024. 12 REST: Stress Testing Large R...

    arXiv

  4. [13]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  5. [14]

    Token-budget- aware llm reasoning.arXiv preprint arXiv:2412.18547, 2024

    Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token-budget- aware llm reasoning.arXiv preprint arXiv:2412.18547, 2024

    2024 arXiv

  6. [15]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad- level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad- level bilingual multimodal scientific problems. InProceedings of the 62nd Annu...

    2024

  7. [16]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In J. Vanschoren and S. Yeung, editors,Proceedings of the Neural Information Processing Systems T ...

    2021

  8. [17]

    Measuring mathematical problem solving with the math dataset, 2021

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https: //arxiv.org/abs/2103.03874

    2021 arXiv

  9. [18]

    A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility.arXiv preprint arXiv:2504.07086, 2025

    Andreas Hochlehnert, Hardik Bhatnagar, Vishaal Udandarao, Samuel Albanie, Ameya Prabhu, and Matthias Bethge. A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility.arXiv preprint arXiv:2504.07086, 2025

    2025

  10. [19]

    Compound-qa: A benchmark for evaluating llms on compound questions.arXiv preprint arXiv:2411.10163, 2024

    Yutao Hou, Yajing Luo, Zhiwen Ruan, Hongru Wang, Weifeng Ge, Yun Chen, and Guanhua Chen. Compound-qa: A benchmark for evaluating llms on compound questions.arXiv preprint arXiv:2411.10163, 2024

    2024

  11. [20]

    Open- reasoner-zero: An open source approach to scaling up reinforcement learning on the base model.arXiv preprint arXiv:2503.24290, 2025

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open- reasoner-zero: An open source approach to scaling up reinforcement learning on the base model.arXiv preprint arXiv:2503.24290, 2025

    2025 arXiv

  12. [21]

    Qwen2.5-coder technical report, 2024

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, An Yang, Rui Men, Fei Huang, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. Qwen2.5-coder technical report, 2024. URLhttps://arxiv.org/abs/2409.12186

    2024 arXiv

  13. [22]

    Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  14. [23]

    Swe-bench: Can language models resolve real-world github issues? InThe Twelfth International Conference on Learning Representations

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? InThe Twelfth International Conference on Learning Representations

  15. [24]

    Mosaic-it: Free compositional data augmentation improves instruction tuning.arXiv preprint arXiv:2405.13326, 2024

    Ming Li, Pei Chen, Chenguang Wang, Hongyu Zhao, Yijun Liang, Yupeng Hou, Fuxiao Liu, and Tianyi Zhou. Mosaic-it: Free compositional data augmentation improves instruction tuning.arXiv preprint arXiv:2405.13326, 2024

    2024 arXiv

  16. [25]

    Cipherbank: Exploring the boundary of llm reasoning capabilities through cryptography challenges.arXiv preprint arXiv:2504.19093, 2025

    Yu Li, Qizhi Pei, Mengyuan Sun, Honglin Lin, Chenlin Ming, Xin Gao, Jiang Wu, Conghui He, and Lijun Wu. Cipherbank: Exploring the boundary of llm reasoning capabilities through cryptography challenges.arXiv preprint arXiv:2504.19093, 2025

    2025 arXiv

  17. [26]

    Metal- adder: Ascending mathematical solution quality via analogical-problem reasoning transfer.arXiv preprint arXiv:2503.14891, 2025

    Honglin Lin, Zhuoshi Pan, Yu Li, Qizhi Pei, Xin Gao, Mengzhang Cai, Conghui He, and Lijun Wu. Metal- adder: Ascending mathematical solution quality via analogical-problem reasoning transfer.arXiv preprint arXiv:2503.14891, 2025

    2025 arXiv

  18. [27]

    Aime 2025 dataset, 2025

    Yen-Ting Lin. Aime 2025 dataset, 2025. URL https://huggingface.co/datasets/yentinglin/aime_2025. Accessed: 2025-03-29. 13 REST: Stress Testing Large Reasoning Models by Asking Multiple Problems at Once

    2025

  19. [28]

    Lost in the middle: How language models use long contexts.T ransactions of the Association for Computational Linguistics, 12, 2024

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.T ransactions of the Association for Computational Linguistics, 12, 2024

    2024

  20. [29]

    O1-pruner: Length-harmonizing fine-tuning for o1-like reasoning pruning.arXiv preprint arXiv:2501.12570, 2025

    Haotian Luo, Li Shen, Haiying He, Yibo Wang, Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. O1-pruner: Length-harmonizing fine-tuning for o1-like reasoning pruning.arXiv preprint arXiv:2501.12570, 2025

    2025 arXiv

  21. [30]

    Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica

    Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl, 2025. Notion Blog

    2025

  22. [31]

    Real: Efficient rlhf training of large language models with parameter reallocation

    Zhiyu Mei, Wei Fu, Kaiwei Li, Guangju Wang, Huanchen Zhang, and Yi Wu. Real: Efficient rlhf training of large language models with parameter reallocation. InProceedings of the Eighth Conference on Machine Learning and Systems, MLSys 2025, Santa Clara, CA, USA, May 12-15, 2025....

    2025

  23. [32]

    s1: Simple test-time scaling, 2025

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettle- moyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, 2025. URL https://arxiv.org/abs/2501.19393

    2025 arXiv

  24. [33]

    Concise thoughts: Impact of output length on llm reasoning and cost.arXiv preprint arXiv:2407.19825, 2024

    Sania Nayab, Giulio Rossolini, Marco Simoni, Andrea Saracino, Giorgio Buttazzo, Nicolamaria Manes, and Fabrizio Giacomelli. Concise thoughts: Impact of output length on llm reasoning and cost.arXiv preprint arXiv:2407.19825, 2024

    2024 arXiv

  25. [34]

    Openai o3 and o4-mini system card, Apr 2025

    OpenAI. Openai o3 and o4-mini system card, Apr 2025. URL https://openai.com/index/ o3-o4-mini-system-card/

    2025

  26. [35]

    Lemma: Learning from errors for mathematical advancement in llms.arXiv preprint arXiv:2503.17439, 2025

    Zhuoshi Pan, Yu Li, Honglin Lin, Qizhi Pei, Zinan Tang, Wei Wu, Chenlin Ming, H Vicky Zhao, Conghui He, and Lijun Wu. Lemma: Learning from errors for mathematical advancement in llms.arXiv preprint arXiv:2503.17439, 2025

    2025 arXiv

  27. [36]

    Mathfusion: Enhancing mathematic problem-solving of llm through instruction fusion.arXiv preprint arXiv:2503.16212, 2025

    Qizhi Pei, Lijun Wu, Zhuoshi Pan, Yu Li, Honglin Lin, Chenlin Ming, Xin Gao, Conghui He, and Rui Yan. Mathfusion: Enhancing mathematic problem-solving of llm through instruction fusion.arXiv preprint arXiv:2503.16212, 2025

    2025 arXiv

  28. [37]

    Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings.arXiv preprint arXiv:2501.01257, 2025

    Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, et al. Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings.arXiv preprint arXiv:2501.01257, 2025

    2025 arXiv

  29. [38]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling, 2024

    2024

  30. [39]

    Code llama: Open foundation models for code, 2024

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong...

    2024 arXiv

  31. [40]

    A practitioners’ guide to transfer learning for text classification using convolutional neural networks

    Tushar Semwal, Promod Yenigalla, Gaurav Mathur, and Shivashankar B Nair. A practitioners’ guide to transfer learning for text classification using convolutional neural networks. InProceedings of the 2018 SIAM international conference on data mining, pages 513–521. SIAM, 2018

    2018

  32. [41]

    Rethinking reflection in pre-training.arXiv preprint arXiv:2504.04022, 2025

    Darsh J Shah, Peter Rushton, Somanshu Singla, Mohit Parmar, Kurt Smith, Yash Vanjani, Ashish Vaswani, Adarsh Chaluvaraju, Andrew Hojel, Andrew Ma, et al. Rethinking reflection in pre-training.arXiv preprint arXiv:2504.04022, 2025

    2025 arXiv

  33. [42]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URLhttps://arxiv.org/abs/2402.03300. 14 REST: Stress T...

    2024 arXiv

  34. [43]

    Structuredrag: Json response formatting with large language models.arXiv preprint arXiv:2408.11061, 2024

    Connor Shorten, Charles Pierse, Thomas Benjamin Smith, Erika Cardenas, Akanksha Sharma, John Trengrove, and Bob van Luijt. Structuredrag: Json response formatting with large language models.arXiv preprint arXiv:2408.11061, 2024

    2024 arXiv

  35. [44]

    Guijin Son, SangWon Baek, Sangdae Nam, Ilgyun Jeong, and Seungone Kim. Multi-task inference: Can large language models follow multiple instructions at once? InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages...

    2024

  36. [45]

    Stop overthinking: A survey on efficient reasoning for large language models.arXiv preprint arXiv:2503.16419, 2025

    Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, et al. Stop overthinking: A survey on efficient reasoning for large language models.arXiv preprint arXiv:2503.16419, 2025

    2025 arXiv

  37. [46]

    Commonsenseqa: A question answer- ing challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answer- ing challenge targeting commonsense knowledge. InProceedings of the 2019 Conference of the North, page 4149. Association for Computational Linguistics, 2019

    2019

  38. [47]

    Let me speak freely? a study on the impact of format restrictions on performance of large language models.arXiv preprint arXiv:2408.02442, 2024

    Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, and Yun-Nung Chen. Let me speak freely? a study on the impact of format restrictions on performance of large language models.arXiv preprint arXiv:2408.02442, 2024

    2024 arXiv

  39. [48]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  40. [49]

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786, 2025

    2025 arXiv

  41. [50]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms.arXiv preprint arXiv:2501.12599, 2025

    2025 arXiv

  42. [51]

    Open Thoughts, January 2025

    Open Thoughts Team. Open Thoughts, January 2025. URLhttps://www.open-thoughts.ai/

    2025

  43. [52]

    Qwq-32b: Embracing the power of reinforcement learning, March 2025

    Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https://qwenlm. github.io/blog/qwq-32b/

    2025

  44. [53]

    Thoughts are all over the place: On the underthinking of o1-like llms.arXiv preprint arXiv:2501.18585, 2025

    Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, et al. Thoughts are all over the place: On the underthinking of o1-like llms.arXiv preprint arXiv:2501.18585, 2025

    2025 arXiv

  45. [54]

    Evaluating llms with multiple problems at once: A new paradigm for probing llm capabilities.arXiv e-prints, pages arXiv–2406, 2024

    Zhengxiang Wang, Jordan Kodner, and Owen Rambow. Evaluating llms with multiple problems at once: A new paradigm for probing llm capabilities.arXiv e-prints, pages arXiv–2406, 2024

    2024

  46. [55]

    Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond.arXiv preprint arXiv:2503.10460, 2025

    Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, and Xiangzheng Zhang. Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond.arXiv preprint arXiv:2503.10...

    2025 arXiv

  47. [56]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  48. [58]

    Qwen2.5-math technical report: Toward mathematical expert model via self-improvement.arXiv preprint arXiv:2409.12122, 2024

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement.arXiv preprint arXiv:2409.12122, 2024

    2024 arXiv

  49. [59]

    Aime-preview: A rigorous and immediate evalua- tion framework for advanced mathematical reasoning

    Yixin Ye, Yang Xiao, Tiantian Mi, and Pengfei Liu. Aime-preview: A rigorous and immediate evalua- tion framework for advanced mathematical reasoning. https://github.com/GAIR-NLP/AIME-Preview, 2025. GitHub repository. 15 REST: Stress Testing Large Reasoning Models by Asking Mul...

    2025

  50. [60]

    Mitigate position bias in large language models via scaling a single dimension

    Yijiong Yu, Huiqiang Jiang, Xufang Luo, Qianhui Wu, Chin-Yew Lin, Dongsheng Li, Yuqing Yang, Yongfeng Huang, and Lili Qiu. Mitigate position bias in large language models via scaling a single dimension. InFirst Workshop on Long-Context Foundation Models@ ICML 2024

    2024

  51. [61]

    Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837, 2025

    Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?arXiv preprint arXiv:2504.13837, 2025

    2025 arXiv

  52. [62]

    Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025

    Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild, 2025. URL https: //arxiv.org/abs/2503.18892

    2025 arXiv

  53. [63]

    Marco-o1: Towards open reasoning models for open-ended solutions, 2024

    Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tianqi Shi, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang. Marco-o1: Towards open reasoning models for open-ended solutions, 2024. URL https://arxiv.org/abs/2411.14405. 16 REST: Stress Testing Large Reasoning Models by Asking ...

    2024 arXiv

  54. [64]

    Your task is to extract the final answer from the prediction as it is, even if it is incorrect

    Don’t try to answer the original question. Your task is to extract the final answer from the prediction as it is, even if it is incorrect

  55. [65]

    Prediction sometimes involves lengthy thinking processes, you don’t need to consider these, just extract the final answer

  56. [66]

    You should set the final answer to None (e.g., \boxed{None})

    If there is question that has not been answered, don’t answer it yourself. You should set the final answer to None (e.g., \boxed{None})

  57. [67]

    Ensure the number of final answers you extract is exactly the same as the number of the given questions

  58. [68]

    For example, if there are three questions, the output should be Answer to Q1: \boxed{answer 1} Answer to Q2: \boxed{answer 2} Answer to Q3: \boxed{answer 3}

    Extract the final answer for each question one by one and enclose each final answer within an \boxed. For example, if there are three questions, the output should be Answer to Q1: \boxed{answer 1} Answer to Q2: \boxed{answer 2} Answer to Q3: \boxed{answer 3}. Here is your task...

  59. [70]

    We need to find this distance, express it in a specific form, and then compute m+n+p where the distance is m√n/p

    There’s a point I inside this tetrahedron such that the distances from I to each face are equal. We need to find this distance, express it in a specific form, and then compute m+n+p where the distance is m√n/p. Hmm, okay. So, first, I remember that in a tetrahedron, the point ...

  60. [89]

    This distance can be written in the form m√n p , where m, n, and p are positive integers, m and p are relatively prime, andnis not divisible by the square of any prime

    There exists a point I inside the tetrahedron such that the distances from I to each of the faces of the tetrahedron are all equal. This distance can be written in the form m√n p , where m, n, and p are positive integers, m and p are relatively prime, andnis not divisible by t...

  61. [307]

    tikz\"); label(\

    307 − 289 = 18. So, 554 ≡ 18 mod 289. 18 + 1 = 19 ̸= 0. n= 55 doesn’t work. Wait, n= 68. 682 = 4, 624. 4, 624 mod 289: 289 · 15 = 4, 335. 4, 624− 4, 335 = 289. 289 ≡ 0 mod 289. So, 682 ≡ 0 mod 289. So, 684 ≡ 0 mod 289. Doesn’t work. n= 69. 69 2 = 4, 761. 4, 761 mod 289: 289 · ...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.