Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

LongReasonArena: A Long Reasoning Benchmark for Large Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that long reasoning can be measured by algorithmic execution, and that current LLMs fail as the required reasoning steps grow logarithmically.

desk verdict A well-built scalable long-reasoning benchmark with a real log-linear finding; the steps-vs-input confound should be analyzed before the benchmark becomes standard. read the letter →

arxiv 2508.19363 v1 pith:XRNI2SIU submitted 2025-08-26 cs.CL cs.AI

Jiayu Ding , Shuming Ma , Lei Cui , Nanning Zheng , Furu Wei This is my paper
classification cs.CLcs.AI
keywords longreasoningLLMevaluationbenchmarkalgorithmicexecutionstepsscalingretrievalbacktrackingchain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LongReasonArena asks models to solve algorithmic problems by actually executing multi-step procedures, with input sizes chosen so that the expected reasoning length reaches roughly 1K, 100K, and 1M tokens. The paper's central claim is that this controlled scaling tests a distinct ability—long reasoning—that long-input comprehension benchmarks miss. Results across 12 models show accuracy falling linearly with the logarithm of expected reasoning steps; the strongest tested reasoning model scores only 7.5% at the million-token level. The paper also isolates two failure modes inside long chains: basic retrieval operations and backtracking, both of which current models handle poorly even though the inputs fit in ordinary context windows. A sympathetic reader would take the core proposal as: make reasoning length the independent variable, then measure how models degrade.

What carries the argument

The machinery is the execution-line proxy. For each problem, a reference solution is run on a generated input and the number of executed lines is counted; dividing samples into levels 1/2/3 at roughly 1K/100K/1M expected reasoning tokens creates an arbitrary scaling knob. Around this sit the supporting components: an input generator that must pass stability and code-coverage criteria (>90%), an output verifier that handles order-independent answers, a 'guessing program' filter that removes samples solvable by trivial heuristics, and a rule that a problem is dropped if a strong reasoning model solves five of five hardest samples. The proxy is what lets the authors claim that accuracy can be p

What would settle it

Measure a model's actual reasoning tokens on a fixed set of samples while varying only the algorithm's loop structure (e.g., vectorized vs. scalar) at the same input size. If accuracy tracks the execution-line proxy rather than the actual problem complexity, or if the log-linear decline vanishes when steps are counted from the model's own trace, the central scaling claim is refuted.

Watch

Extended reading notes

Core claim

At the center is a benchmark that turns 'long reasoning' into a controllable quantity. The authors collect algorithmic problems, generate unbounded valid inputs programmatically, and label sample difficulty by the number of execution lines the reference solution runs on that input—a proxy for the reasoning steps the model must perform. Because inputs are capped at 32K tokens while the expected reasoning length reaches a million tokens, the task isolates reasoning depth from context length. The reported discovery is a regular degradation pattern: for every reasoning model tested, accuracy falls linearly with the logarithm of the expected step count (R²>0.9), and no model comes close to solvin

Load-bearing premise

The load-bearing premise is that the number of executed lines in the reference solution equals the number of reasoning steps the model must perform, with each line costing at least ten tokens; if that mapping is inaccurate, the difficulty levels and the log-linear accuracy decline are artifacts of the proxy.

Editorial extensions

If this is right

  • Accuracy on samples with expected reasoning length L can be extrapolated from shorter runs, so benchmark builders can estimate million-token performance without paying for it.
  • Long-input benchmarks are not enough: a model can pass retrieval over a long prompt and still fail the same operation when it must be repeated inside a reasoning chain.
  • Any model that aspires to solve Level 3 must sustain state over roughly 10^5–10^6 steps; current reasoning models do not, so the bottleneck is step-count, not context window.
  • Since incorrect chains are longer than correct ones, simply scaling test-time computation does not produce correctness; some internal quality signal is missing.
  • The benchmark provides a standardized way to compare models on a property that can be scaled arbitrarily, up to a million reasoning tokens.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because each sample has a machine-checkable reference answer, the benchmark could be used directly as a reward source for reinforcement learning on long reasoning, not only as an evaluation set.
  • Editorial inference: the log-linear decline suggests a memory or state-tracking bottleneck; a testable extension would interleave explicit checkpoints or scratchpad refreshes in the prompt and see whether the slope flattens.
  • Editorial inference: the execution-line proxy ties difficulty to one particular algorithm; a harder stress test would vary algorithms with the same asymptotic complexity but different memory patterns and check whether model accuracy tracks lines or structure.
  • Editorial inference: because sources are public coding problems, contamination is a live risk; a temporal holdout using newly added problems would reveal whether scores reflect memorized solutions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces LongReasonArena, a benchmark for evaluating long reasoning by requiring LLMs to execute algorithmic problems drawn from LeetCode on generated inputs. Difficulty levels are defined by the number of execution lines of the reference solution, converted to token counts via a hand-set 10-token-per-line rule, yielding Level 1/2/3 at roughly 1K/100K/1M tokens. The paper evaluates twelve models, reports low accuracy at higher levels (e.g., DeepSeek-R1 7.5% at Level 3), and claims a log-linear decline of accuracy with expected reasoning steps. It also presents case studies on Two Sum and Word Search to attribute failures to retrieval and backtracking limitations.

Significance. If the claims hold, the benchmark is a valuable complement to long-context benchmarks: it offers controllable scaling of algorithmic complexity, generated inputs with validity/coverage checks, removal of guessable samples, order-independent output verification, and public code/data. The accuracy measurement is not circular, since it is judged against external reference solutions and verifiers. However, the headline scaling law rests on an unvalidated proxy—execution line count—which is confounded with input length and depends on the reference algorithm rather than on the model's actual reasoning process. The case studies are informative but do not by themselves validate the general scaling claim. With additional control analyses, this could be a useful resource for the long-reasoning evaluation community.

major comments (4)
  1. [§3.2, Figure 2, Table 1] The central claim that accuracy declines linearly with log(expected reasoning steps) is not separated from input-length difficulty. The execution-line proxy is generated by loops over input size, so larger arrays, strings, or boards naturally produce more executed lines; Table 1 shows median input length increasing across levels (64, 524, 1,983). Figure 2(b) already shows accuracy dropping sharply with input length. To support the claim that reasoning steps, not input length, drive the trend, the paper should report a partial correlation controlling for input length, matched-input analyses (e.g., samples with similar input length but differing line counts), or the analogous R² for accuracy vs. log(input length). The text also states that at least 50% of Level 3 samples have input length within 1K tokens, which contradicts the median of 1,983 in Table 1; this needs correction or explanati
  2. [§4.2, Appendix E] The regression analysis behind the log-linear law is underspecified. Figure 2(a) and Table 6 appear to summarize fits over the three difficulty-level aggregates; with only three levels, a high R² is weak evidence for a log-linear law, and the reported p-values <1e-4 are suspect if indeed only three points were used (they are not attainable under standard assumptions with three noisy means). The paper should describe the binning procedure, report the number of fitted points, give confidence intervals, and compare against plausible alternative functional forms (e.g., step, quadratic, or input-length-only fits). Without this, the scaling claim is not established.
  3. [§3.2 and §5] The 'expected number of reasoning steps' is a property of the reference implementation, not of the model. The paper acknowledges in Section 5 that models may use different strategies, but does not mitigate the consequences for the difficulty levels. A model could solve a problem with a more efficient algorithm (e.g., a hash map instead of nested loops) and therefore require far fewer reasoning steps than the execution-line count suggests; conversely, a less efficient model-level simulation could require more. Since the benchmark does not verify the model's process, the levels are a property of the reference solution, not a controlled measure of cognitive demand. The 10-token-per-line conversion is also hand-set with no empirical calibration; the '100K/1M token reasoning' description should be either justified with evidence or downgraded to an upper-bound estimate. At minimum, all scaling
  4. [§4.3, §4.4] The retrieval and backtracking claims are based on two hand-picked case studies with manually classified error types (index errors vs. full errors; distinct paths in Word Search). While the examples are suggestive, no inter-annotator agreement or quantitative protocol is provided for the error taxonomy, and the 'dominant source of failure' conclusion in Two Sum relies on equation correctness (98%) without showing that retrieval errors are not partly due to arithmetic slips or format issues. The authors should either provide a more rigorous coding protocol or temper the general claim that current models struggle with 'basic retrieval operations' based on a single task.
minor comments (4)
  1. [Appendix B vs. Table 2] The seed-sweep evaluation of QwQ on Level 1 reports a mean accuracy of 50.9 (Table 5), while Table 2 reports 49.4 for the same model and level. This discrepancy should be reconciled (e.g., different seed or subset).
  2. [Throughout] Spelling of 'DeepSeek-R1' is inconsistent ('Deepseek-R1' in the abstract and Section 1). There are also grammatical errors, e.g., 'the model need to attempt' and 'the model need to retain' in Section 1; a careful proofread is needed.
  3. [§3.2] The 'Input Length Constraint' says samples are retained only if input length is within 32K tokens, but the relation of this constraint to the token-count estimate of reasoning length is unclear. Clarify whether the 32K refers to tokens in the prompt serialization or a different unit.
  4. [References] Some references are missing venue or version information (e.g., [Tea25], [DAGY+25]). This is a presentation issue but should be cleaned up before publication.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the benchmark difficulty proxy is external to model outputs, and the scaling result is an empirical measurement.

full rationale

LongReasonArena's derivation chain is not circular. Difficulty is assigned by counting execution lines of the reference solution code on a given input (§3.2), which is an external oracle independent of any model's responses. The headline result—accuracy declining linearly with log expected reasoning steps—is measured over 12 models and is not forced by the benchmark construction; the construction would be equally compatible with flat accuracy. The 10-token-per-line conversion in §3.2 is a hand-set calibration, not a parameter fitted to model accuracy, and the levels are bins over an external oracle. Section 4.2's statement that the step-count proxy is 'effective' is a predictive-validity claim, not a tautology: nothing guarantees that model accuracy will track execution-line counts. The acknowledged limitation in Section 5—that execution lines may deviate from the model's actual reasoning steps—is an external-validity caveat, not a circular step. The skeptic's concern that input length may confound the scaling trend (Figures 2a vs 2b) is a missing control analysis, not a reduction of the claim to its own inputs. The benchmark also filters out guessable samples and problems where QwQ solves all Level 3 samples; this is deliberate difficulty curation, not circular derivation. There are no load-bearing self-citations or imported uniqueness theorems. The central claim retains independent empirical content.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The benchmark's central claims depend on a handful of hand-set thresholds and the execution-line proxy. None of these are hidden, and the limitations section acknowledges the main proxy issue. The most consequential assumption is the line-to-step mapping, because it underpins the difficulty scaling and the log-linear finding. The QwQ-based filtering also introduces a dependency on a particular model's abilities, which could bias the benchmark's difficulty upward.

free parameters (4)
  • Token-to-line ratio = 10 tokens per execution line (estimated)
    Used in Section 3.2 to convert execution line counts into token-based difficulty levels (1K, 100K, 1M). This hand-set multiplier is not empirically justified and directly determines the advertised reasoning length scaling.
  • Input generator coverage threshold = 90%
    Section 3.2 requires generators to achieve over 90% code coverage. This threshold filters which problems enter the benchmark and affects the diversity of generated inputs.
  • Consecutive valid input count = 10
    Section 3.2 requires a generator to produce ten valid inputs consecutively to qualify. This stability criterion influences which problems are retained.
  • QwQ exclusion sample count = 5
    Section 3.2 discards a problem if QwQ solves all five generated Level 3 samples. This hand-set number directly shapes the upper difficulty of the benchmark and depends on a specific model's capability.
assumptions (3)
  • domain assumption Execution line count is proportional to the number of reasoning steps required by a model.
    Section 3.2 defines difficulty levels by execution lines, and Section 5 admits this is an approximation. The log-linear accuracy trend is a consequence of this proxy.
  • domain assumption LeetCode algorithmic problems, after filtering, are representative of long reasoning tasks.
    The paper motivates algorithmic execution as capturing retrieval, backtracking, and memory operations, but this is a modeling choice about what constitutes 'long reasoning'.
  • ad hoc to paper A simple guessing program and QwQ's performance reliably identify samples that do not require long reasoning.
    Section 3.2 uses a generated guessing program and QwQ as filters. These criteria are specific to the paper's construction and are not independently validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LongReasonArena: A Long Reasoning Benchmark for Large Language Models." pith.science (2026). https://pith.science/paper/XRNI2SIU

@misc{pith2026250819363,
  author       = {Pith},
  title        = {Pith review of: LongReasonArena: A Long Reasoning Benchmark for Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XRNI2SIU}},
  note         = {Machine review of arXiv:2508.19363}
}
read the original abstract

Existing long-context benchmarks for Large Language Models (LLMs) focus on evaluating comprehension of long inputs, while overlooking the evaluation of long reasoning abilities. To address this gap, we introduce LongReasonArena, a benchmark specifically designed to assess the long reasoning capabilities of LLMs. Our tasks require models to solve problems by executing multi-step algorithms that reflect key aspects of long reasoning, such as retrieval and backtracking. By controlling the inputs, the required reasoning length can be arbitrarily scaled, reaching up to 1 million tokens of reasoning for the most challenging tasks. Extensive evaluation results demonstrate that LongReasonArena presents a significant challenge for both open-source and proprietary LLMs. For instance, Deepseek-R1 achieves only 7.5% accuracy on our task. Further analysis also reveals that the accuracy exhibits a linear decline with respect to the logarithm of the expected number of reasoning steps. Our code and data is available at https://github.com/LongReasonArena/LongReasonArena.

Figures

Figures reproduced from arXiv: 2508.19363 by the authors.

Figure 1
Figure 1. (a) The construction process of LONGREASONARENA (b) Algorithmic problems abstract key elements of long reasoning process, including retrieval, backtracking, and other essential cognitive operations. single response. Wu et al.’s LongGenBench [WHHL25] tasks the model with sequentially completing a series of subtasks, incorporating specific details at designated positions in the output text. Both benchmarks construct l… view at source ↗
Figure 2
Figure 2. Model accuracy with respect to (a) expected number of reasoning steps and (b) input [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. The performance trend and error type analysis for the Two Sum task. (a) The accuracy [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The performance trend for the Word Search task. The accuracy exhibits a linear decline [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Relationship between the expected number of reasoning steps and reasoning length [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction

    cs.CL 2026-06 conditional novelty 6.5 of 10

    Structured cognitive-episode features from LRM reasoning traces, combined with item semantics, improve human item-difficulty prediction and show harder items drive more implementation-centered, iterative solving.

  2. Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    Epi2Diff extracts cognitive episode sequences from LRM reasoning traces and combines them with semantic features to predict human item difficulty, outperforming baselines on four educational datasets.

  3. Efficient Evaluation of LLM Performance with Statistical Guarantees

    stat.ML 2026-01 unverdicted novelty 6.0 of 10

    Factorized Active Querying (FAQ) provides up to 5 times more effective samples for LLM accuracy estimation by using Bayesian factor models and adaptive querying under a fixed budget with guaranteed coverage.

Reference graph

Works this paper leans on

32 extracted references · 19 canonical work pages · cited by 2 Pith papers

  1. [1]

    L-eval: Instituting standardized evaluation for long context language models

    Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 14388--14411, 2024

    work page 2024

  2. [2]

    Claude 3.7 sonnet system card

    Anthropic. Claude 3.7 sonnet system card. 2025

    2025

  3. [3]

    Program synthesis with large language models, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021

    2021

  4. [4]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 3119--3137, 2024

    work page 2024

  5. [5]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    2021

  6. [6]

    LR B ench: Evaluating long-chain reflective reasoning capabilities of large language models via constraint satisfaction problems

    Jianghao Chen, Zhenlin Wei, Zhenjiang Ren, Ziyong Li, and Jiajun Zhang. LR B ench: Evaluating long-chain reflective reasoning capabilities of large language models via constraint satisfaction problems. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics: ACL 202...

    work page 2025

  7. [7]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

    2025

  8. [8]

    Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian L...

    work page 2024

Show all 32 references
  1. [9]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 , 2024

    2024 arXiv

  2. [10]

    Measuring coding challenge competence with apps

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Ben...

    2021

  3. [11]

    Ruler: What’s the real context size of your long-context language models? In First Conference on Language Modeling , 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models? In First Conference on Language Modeling , 2024

    2024

  4. [12]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186 , 2024

    2024 arXiv

  5. [13]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learn...

    2024

  6. [14]

    Needle In A Haystack - pressure testing LLM s

    Gregory Kamradt. Needle In A Haystack - pressure testing LLM s. Github , 2023

    2023

  7. [15]

    Babilong: Testing the limits of llms with long context reasoning-in-a-haystack

    Yury Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of llms with long context reasoning-in-a-haystack. Advances in Neural Information Processing Systems , 37:106519--106554, 2024

    2024

  8. [16]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles , ...

    2023

  9. [17]

    Longgenbench: Long-context generation benchmark

    Xiang Liu, Peijie Dong, Xuming Hu, and Xiaowen Chu. Longgenbench: Long-context generation benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2024 , pages 865--883, 2024

    2024

  10. [18]

    Codei/o: Condensing reasoning patterns via code input-output prediction, 2025

    Junlong Li, Daya Guo, Dejian Yang, Runxin Xu, Yu Wu, and Junxian He. Codei/o: Condensing reasoning patterns via code input-output prediction, 2025

    2025

  11. [19]

    Longreason: A synthetic long-context reasoning benchmark via context expansion

    Zhan Ling, Kang Liu, Kai Yan, Yifan Yang, Weijian Lin, Ting-Han Fan, Lingfeng Shen, Zhengyin Du, and Jiecao Chen. Longreason: A synthetic long-context reasoning benchmark via context expansion. arXiv preprint arXiv:2501.15089 , 2025

    2025

  12. [20]

    ALR^2 : A retrieve-then-reason framework for long-context question answering

    Huayang Li, Pat Verga, Priyanka Sen, Bowen Yang, Vijay Viswanathan, Patrick Lewis, Taro Watanabe, and Yixuan Su. ALR^2 : A retrieve-then-reason framework for long-context question answering. arXiv preprint arXiv:2410.03227 , 2024

    2024 arXiv

  13. [21]

    Large language models as code executors: An exploratory study

    Chenyang Lyu, Lecheng Yan, Rui Xing, Wenxi Li, Younes Samih, Tianbo Ji, and Longyue Wang. Large language models as code executors: An exploratory study. arXiv preprint arXiv:2410.06667 , 2024

    2024 arXiv

  14. [22]

    Gpt-4o system card

    OpenAI . Gpt-4o system card. arXiv preprint arXiv:2410.21276 , 2024

    2024 arXiv

  15. [23]

    Openai o1 system card

    OpenAI . Openai o1 system card. arXiv preprint arXiv:2412.16720 , 2024

    2024 arXiv

  16. [24]

    QwQ-32B : Embracing the power of reinforcement learning, March 2025

    Qwen Team. QwQ-32B : Embracing the power of reinforcement learning, March 2025

    2025

  17. [25]

    Longgenbench: Benchmarking long-form generation in long context llms, 2025

    Yuhao Wu, Ming Shan Hee, Zhiqing Hu, and Roy Ka-Wei Lee. Longgenbench: Benchmarking long-form generation in long context llms, 2025

    2025

  18. [26]

    Thoughts are all over the place: On the underthinking of o1-like llms, 2025

    Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Thoughts are all over the place: On the underthinking of o1-like llms, 2025

    2025

  19. [27]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824--24837, 2022

    2022

  20. [28]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems , 36:11809--11822, 2023

    2023

  21. [29]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 , 2024

    2024 arXiv

  22. [30]

    bench: Extending long context evaluation beyond 100k tokens

    Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. bench: Extending long context evaluation beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ...

    2024

  23. [31]

    Docpuzzle: A process-aware benchmark for evaluating realistic long-context reasoning capabilities

    Tianyi Zhuang, Chuqiao Kuang, Xiaoguang Li, Yihua Teng, Jihao Wu, Yasheng Wang, and Lifeng Shang. Docpuzzle: A process-aware benchmark for evaluating realistic long-context reasoning capabilities. arXiv preprint arXiv:2502.17807 , 2025

    2025 arXiv

  24. [32]

    Gsm-infinite: How do your llms behave over infinitely increasing context length and reasoning complexity? arXiv preprint arXiv:2502.05252 , 2025

    Yang Zhou, Hongyi Liu, Zhuoming Chen, Yuandong Tian, and Beidi Chen. Gsm-infinite: How do your llms behave over infinitely increasing context length and reasoning complexity? arXiv preprint arXiv:2502.05252 , 2025

    2025 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.