Pith. sign in

REVIEW 4 major objections 5 minor 42 references

FinanceReasoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challenging

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FinanceReasoning re-annotates 15.6% of public finance questions and, under a strict 0.2% error margin, finds the best reasoning model achieves 89.1% on hard problems.

desk verdict A genuinely useful benchmark resource whose knowledge-augmentation headline is undermined by a circular retrieval setup. read the letter →

arxiv 2506.05828 v2 pith:CNSIIPVU submitted 2025-06-06 cs.CL cs.CE

Zichen Tang , Haihong E , Ziyan Ma , Haoyang He , Jiacheng Liu , Zhongjun Yang , Zihua Rong , Rongjin Li
show 5 more authors
Kun Ji Qing Huang Xinyang Hu Yang Liu Qianhe Zheng
This is my paper
classification cs.CLcs.CE
keywords financialnumericalreasoningbenchmarklargemodelschain-of-thoughtprogram-of-thoughtdifficultyheuristicknowledgeretrievalerroranalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper builds FinanceReasoning, a financial numerical reasoning benchmark of 2,238 problems, to test large reasoning models (LRMs) more fairly and more strictly than earlier datasets. It re-annotates 15.6% of questions from four public finance datasets, adds 908 expert-reviewed problems generated with a new 3,133-function Python library, and evaluates answers under a stringent 0.2% error margin. The headline finding is that even the best model, OpenAI o1 with program-of-thought prompting, reaches only 89.1% accuracy on the 238 hardest problems, and error analysis shows LRMs still struggle with applying the right financial formula and with numerical precision. The paper argues that earlier datasets were saturated and too lenient, hiding these weaknesses.

What carries the argument

The benchmark's difficulty taxonomy is driven by a heuristic $r_c = \ln(\max(o,1)) + \ln(\max(l+p,1))$, where $o$ counts operators, $p$ pairs of parentheses, and $l$ lines of code in a problem's Python solution; this score sorts the 2,238 problems into Easy, Medium, and Hard tiers. The other central instrument is the 3,133-function library: each function is a Python implementation of one financial calculation, with docstring, parameters, and step-by-step code, which the paper uses both to generate the 908 new problems and as retrievable knowledge that improves model accuracy. Evaluation pairs chain-of-thought and program-of-thought prompting with a strict 0.2% error-margin comparison against re-verified ground truths.

What would settle it

Ask a panel of finance experts to independently rate the difficulty of a stratified sample of FinanceReasoning problems and compare their ratings to the $r_c$ tiers; alternatively, move the tier thresholds by one unit and re-run the model accuracy table. If the ranking of models, or the conclusion that LRMs dominate only on Hard, changes materially, the benchmark's central difficulty claim is an artifact of the heuristic.

Watch

Extended reading notes

Core claim

The central claim is that credible evaluation of financial numerical reasoning requires cleaned questions, a strict error margin, and genuinely hard multi-formula problems, and that with those conditions current LRMs show a clear but incomplete advantage. The authors report that after their re-annotation, DeepSeek-R1 improves sharply over DeepSeek-V3 on FinanceMath under gold criteria, and that on the Hard subset only OpenAI o1 with PoT breaks 89% accuracy. They further show that supplying the model with retrieved Python-formatted functions lifted GPT-4o to 91.6% on Hard, and that pairing DeepSeek-R1 as reasoner with Claude 3.5 Sonnet as programmer corrected 91.7% of numerical calculation errors.

Load-bearing premise

The Easy/Medium/Hard split, and therefore the claim that LRMs show their real advantage only on Hard problems, rests on the assumption that counting operators, parentheses, and lines of code in the Python solution faithfully measures how hard a financial problem is, an assumption the paper does not validate against human difficulty ratings.

Editorial extensions

If this is right

  • If the 0.2% error margin is adopted for other finance benchmarks, reported accuracies will drop relative to the 1%-margin and unit-flexible evaluations used previously.
  • Retrieving Python-formatted functions, rather than encyclopedia passages, converts a strong LLM (GPT-4o) into the best performer on the Hard subset, at 91.6% versus 89.1% for OpenAI o1 with PoT alone.
  • Pairing DeepSeek-R1 as reasoner with Claude 3.5 Sonnet as programmer reaches 87.82% on Hard and corrects 91.7% of the numerical calculation errors the reasoner alone makes.
  • PoT achieves accuracy comparable to CoT-driven reasoning models at a fraction of the token cost, since GPT-4o with PoT matches DeepSeek-R1 with CoT on Hard while using 54k tokens versus 742k.
  • Formula application errors grow as difficulty rises, while numerical extraction errors stay low, so improving formula selection and numerical precision should yield the largest gains for the hardest problems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The difficulty heuristic could be validated against human error rates: if a tier of problems with high $r_c$ does not show higher human error or longer solving time, the benchmark's claim that Hard problems are genuinely harder should be revised.
  • The reasoner/programmer split is directly transferable to other quantitative domains with heavy arithmetic, such as actuarial, engineering, or scientific computation, where separating decision-making from calculation may sidestep a single model's precision ceiling.
  • A natural stress test of the strict 0.2% margin is to attach economic cost to errors: weighting large-dollar errors more heavily than small ones would tell whether the remaining failures are material for real portfolio decisions rather than purely numerical.
  • Because FinanceReasoning evaluates only perfect-information scenarios, an obvious complementary benchmark would inject missing or contradictory data and measure whether models ask for clarification; the authors flag this as future work, and it would test a different failure mode from the one studied here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. FinanceReasoning is presented as a benchmark for financial numerical reasoning that combines 1,330 re-annotated questions from four public datasets (CodeFinQA, CodeTAT-QA, FinCode, FinanceMath) with 908 newly generated questions, yielding 2,238 total problems. The authors report updating 15.6% of the re-annotated questions (208 examples) through disambiguation, elaboration, and correction, and enforce a strict 0.2% error margin in evaluation. They also construct a library of 3,133 Python-formatted financial functions extracted from Investopedia articles, use it to generate the new questions, and claim 67.8% coverage of financial concepts and formulas. Problems are graded into Easy (1,000), Medium (1,000), and Hard (238) using a code-complexity heuristic. The evaluation covers six LRMs and seven LLMs under CoT and PoT, reporting OpenAI o1 with PoT as best on Hard (89.1%), a knowledge-augmentation gain for GPT-4o from 83.2% to 91.6%, and an 87.8% result for DeepSeek-R1 combined with Claude 3.5 Sonnet.

Significance. The resource is potentially valuable: the re-annotation effort, the strict evaluation protocol, the public release of the benchmark and function library, and the comparative evaluation of reasoning models address a real saturation problem in existing financial QA benchmarks. The error analysis (Table 4) and the Reasoner+Programmer combination are useful empirical contributions. If the internal inconsistencies and the circularity of the knowledge-augmentation experiment were resolved, the benchmark would be a credible addition to the domain. However, several headline claims are currently not supported by the evidence presented.

major comments (4)
  1. [§2.1 and Appendix A] The paper gives conflicting counts for the number of updated questions. The Introduction states '1,420 problems have been reviewed and revised' and Appendix A repeats '1,420 updated existing problems,' yet Table 1(a) sums to 208 updated questions across 1,330 test/validation examples, and 1,420 + 908 = 2,328, not the stated 2,238 total. The correct number appears to be 1,330. This discrepancy must be fixed because the credibility claim (15.6% updated) and the dataset composition are central to the paper.
  2. [§2.2, Figure 1, Abstract] The claim that FinanceReasoning 'covers 67.8% of financial concepts and formulas' is not backed by a precise definition. The Figure 1 caption describes coverage as the proportion of financial calculations involved in the questions relative to a financial encyclopedia, but the denominator, the matching method, and the comparison numbers for existing datasets are not given. As a headline comprehensiveness claim, this needs a reproducible definition.
  3. [§2.5, Eq. (1)] The difficulty heuristic rc = ln(max(o,1)) + ln(max(l+p,1)) is used to partition the benchmark into Easy/Medium/Hard, and the Hard subset of 238 problems is the cornerstone of the 'challenge' claim and model ranking. However, the thresholds for the three tiers are not disclosed, the proxy (code operators, parentheses, lines) is not validated against any external difficulty judgment, and no sensitivity analysis is provided. The authors should report the threshold values and at least a correlation with expert difficulty ratings.
  4. [§4.3, Table 5, with §2.3] The knowledge-augmentation experiment is circular. The 908 new questions were generated by prompting GPT-4o with seed functions selected from the same 3,133-function library that §4.3 later retrieves from. For a question built from a specific seed function, Top-3 retrieval by question-function similarity can return that exact function, whose docstring and code encode the solution recipe. The observed gain (83.2% to 91.6% for GPT-4o on Hard) may therefore result from answer leakage rather than from general financial knowledge. The passage-retrieval baseline is also not matched, since the questions were generated from functions, not from Investopedia article passages. Please rerun the augmentation with a held-out function set (e.g., functions not used in generation) and/or measure the fraction of retrieved sets that contain the seed function; also report augmentation results on the 1,330 re-annotated public questions.
minor comments (5)
  1. [Table 2 vs. Abstract/Appendix A] Table 2 reports '# Total Functions 3,313' while the abstract and Appendix A state 3,133; one of these is a typo and must be corrected.
  2. [Table 20 and Limitations] Table 20 lists token usage for OpenAI o1 PoT as '505 / - / -', yet Table 3 reports Hard accuracy for o1 PoT (89.1%); reconcile the token table with the statement in Limitations that o1 PoT was not evaluated on Easy/Medium.
  3. [Table 3] The 'Avg.' column in Table 3 is an unweighted macro-average across Easy/Medium/Hard; state this explicitly or use a weighted average, since subset sizes differ.
  4. [§3.2] The 0.2% error margin is not defined as absolute or relative; specify how it is applied to rounded or integer answers (e.g., relative to the ground truth).
  5. [§4.1] The claim that OpenAI o1 'significantly outperforms other LRMs and LLMs' is not supported by a significance test; with 238 Hard problems, the 3.8-point gap over DeepSeek-R1 PoT (89.1 vs. 85.3) is within the margin of error.

Circularity Check

1 steps flagged · score 6.0 of 10

Knowledge-augmentation gain is partly circular: retrieval returns the seed functions used to generate the test questions.

  1. fitted input called prediction [Section 2.3 'Expansion of Data Annotation'; Section 4.3 'Knowledge Augmentation', Table 5]
    "During the data expansion process, we leveraged the structured financial functions to guide GPT-4o in generating new financial numerical reasoning problems and Python solutions. ... Seed Function Selection We selected 1,250 financial functions from the library ... For each seed function, GPT-4o was prompted to generate the complex reasoning problem with the necessary financial tabular data, using the financial terms and the computational processes of the function."

    The 908 new questions were authored from 1,250 seed functions drawn from the same 3,133-function library that Section 4.3 indexes for retrieval. For a question generated from a seed function, the question text and context are constructed directly from that function's semantics and computational steps. Retrieving Top-3 functions by question-to-description similarity will therefore tend to return the exact seed function, effectively supplying the model with the solution recipe. The reported gain from 83.19% to 91.60% on the Hard subset (Table 5) thus conflates knowledge augmentation with answer-template leakage; it is not independent evidence that function-formatted knowledge per se helps.

full rationale

The static benchmark construction and the main accuracy ranking (Tables 1 and 3) are not circular: they re-annotate existing public datasets, add expert-verified problems, and evaluate model outputs against fixed ground truths with a strict 0.2% error margin. The difficulty heuristic in Eq. (1) is an unvalidated proxy, not a circular step. The one load-bearing circularity is the RQ3 knowledge-augmentation experiment: the 908 new problems were generated from 1,250 seed functions selected from the same 3,133-function library that Section 4.3 later retrieves from. Because each seed function's semantics and computational steps are the source of the corresponding question, retrieving Top-3 functions by semantic similarity can hand the model the exact function used to generate the question, so the 83.19 to 91.60 improvement is at least partly answer-template leakage rather than evidence for the general value of function-formatted knowledge. This does not invalidate the benchmark's static evaluation, but it undercuts the headline 'refined knowledge augmentation' claim. No load-bearing self-citation or imported uniqueness theorem was found.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The benchmark rests on the correctness of inherited and corrected ground truths, the authority of Investopedia as a knowledge source, the reliability of expert review, and the validity of a code-metric difficulty proxy. The most fragile elements are the unvalidated difficulty heuristic and the lack of inter-annotator agreement metrics.

free parameters (3)
  • Accuracy error margin = 0.2%
    Chosen threshold for judging a model answer correct; all reported accuracies depend on it. Not justified by sensitivity analysis.
  • Difficulty heuristic = rc = ln(max(o,1)) + ln(max(l+p,1))
    Ad hoc formula used to assign Easy/Medium/Hard; coefficients and log form are hand-chosen and thresholds for the split are not disclosed (Section 2.5).
  • Retrieval top-k values = Top-3 functions; Top-10 passages; Top-30 for LLM judge
    Hyperparameters in knowledge augmentation experiments (Section 4.3); no tuning or ablation on k.
assumptions (5)
  • domain assumption The four source datasets (CodeFinQA, CodeTAT-QA, FinCode, FinanceMath) are valid seeds and their non-updated questions remain correct.
    The benchmark inherits 1,330 questions from these datasets; if untouched examples contain errors, the benchmark credibility suffers. Section 2.1.
  • domain assumption Expert annotation by 8 graduate students and 2 CFA holders yields correct ground truth.
    No inter-annotator agreement statistics or independent verification are reported; correctness of revised answers is assumed (Section 2.4).
  • domain assumption Investopedia articles are an authoritative and comprehensive source of financial formulas.
    The function library and the 67.8% coverage claim are built entirely from Investopedia (Section 2.2).
  • domain assumption GPT-4o can generate valid financial problems and Python solutions when provided a seed function.
    908 new problems are generated by GPT-4o and only reviewed, not re-derived, by experts (Section 2.3).
  • ad hoc to paper Code metrics (operators, parentheses, lines) monotonically proxy reasoning difficulty.
    Eq. (1) defines difficulty; no validation against human ratings or independent difficulty measures (Section 2.5).

how reviews work

0 comments
Cite this review

Pith. "Pith review of FinanceReasoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challenging." pith.science (2026). https://pith.science/paper/CNSIIPVU

@misc{pith2026250605828,
  author       = {Pith},
  title        = {Pith review of: FinanceReasoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challenging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CNSIIPVU}},
  note         = {Machine review of arXiv:2506.05828}
}
abstract

We introduce FinanceReasoning, a novel benchmark designed to evaluate the reasoning capabilities of large reasoning models (LRMs) in financial numerical reasoning problems. Compared to existing benchmarks, our work provides three key advancements. (1) Credibility: We update 15.6% of the questions from four public datasets, annotating 908 new questions with detailed Python solutions and rigorously refining evaluation standards. This enables an accurate assessment of the reasoning improvements of LRMs. (2) Comprehensiveness: FinanceReasoning covers 67.8% of financial concepts and formulas, significantly surpassing existing datasets. Additionally, we construct 3,133 Python-formatted functions, which enhances LRMs' financial reasoning capabilities through refined knowledge (e.g., 83.2% $\rightarrow$ 91.6% for GPT-4o). (3) Challenge: Models are required to apply multiple financial formulas for precise numerical reasoning on 238 Hard problems. The best-performing model (i.e., OpenAI o1 with PoT) achieves 89.1% accuracy, yet LRMs still face challenges in numerical precision. We demonstrate that combining Reasoner and Programmer models can effectively enhance LRMs' performance (e.g., 83.2% $\rightarrow$ 87.8% for DeepSeek-R1). Our work paves the way for future research on evaluating and improving LRMs in domain-specific complex reasoning tasks.

Figures

Figures reproduced from arXiv: 2506.05828 by the authors.

Figure 1
Figure 1. Statistics and an example of FinanceRea [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of FinanceReasoning’s construction. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. An example of financial function from the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The difficulty distribution of FinanceReason [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Results of different model combinations and [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Relationship between token consumption and accuracy across different LLMs and LRMs using CoT and [PITH_FULL_IMAGE:figures/full_fig_p021_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 13 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    AI@Meta. 2024 a . https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md Llama 3.1 model card

    work page 2024

  4. [4]

    AI@Meta. 2024 b . https://github.com/meta-llama/llama-models/blob/main/models/llama3_3/MODEL_CARD.md Llama 3.3 model card

    work page 2024

  5. [5]

    Anthropic. 2024. https://www.anthropic.com/claude/sonnet Claude 3.5 sonnet

    2024

  6. [6]

    Liang Chen, Yang Deng, Yatao Bian, Zeyu Qin, Bingzhe Wu, Tat-Seng Chua, and Kam-Fai Wong. 2023 a . https://doi.org/10.18653/v1/2023.emnlp-main.390 Beyond factuality: A comprehensive evaluation of large language models as knowledge generators . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6325--6341, Sing...

    work page doi:10.18653/v1/2023.emnlp-main.390 2023

  7. [7]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    arXiv 2021

  8. [8]

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023 b . https://openreview.net/forum?id=YfZ4ZPt8zd Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks . Transactions on Machine Learning Research

    work page 2023

Show all 42 references
  1. [9]

    Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. 2023 c . https://doi.org/10.18653/v1/2023.emnlp-main.489 T heorem QA : A theorem-driven question answering dataset . In Proceedings of the 2023 Conference on Empirical Methods in...

    2023 doi

  2. [10]

    Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021 b . https://doi.org/10.18653/v1/2021.emnlp-main.300 F in QA : A dataset of numerical reasoning over financia...

    2021 doi

  3. [11]

    Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.421 C onv F in QA : Exploring the chain of numerical reasoning in conversational finance question answering . In Proceedings of the 2022 ...

    2022 doi

  4. [12]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei F...

    2025 arXiv

  5. [13]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei L...

    2024 arXiv

  6. [14]

    Gemini. 2025. https://developers.googleblog.com/en/gemini-2-family-expands/ Gemini 2.0: Flash, flash-lite and pro

    2025

  7. [15]

    Jian Guan, Wei Wu, zujie wen, Peng Xu, Hongning Wang, and Minlie Huang. 2024. https://openreview.net/forum?id=jImXgQEmX3 AMOR : A recipe for building adaptable modular knowledge agents through process feedback . In The Thirty-eighth Annual Conference on Neural Information Proc...

    2024

  8. [16]

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. https://openreview.net/forum?id=jKN1pXi7b0 Unsupervised dense information retrieval with contrastive learning . Transactions on Machine Learning Research

    2022

  9. [17]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. https://openreview.net/forum?id=chfJJYC3iL Livecodebench: Holistic and contamination free evaluation of large language models for code . I...

    2025

  10. [18]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. https://arxiv.org/abs/2001.08361 Scaling laws for neural language models . Preprint, arXiv:2001.08361

    2020 arXiv

  11. [19]

    Michael Krumdick, Rik Koncel-Kedziorski, Viet Dac Lai, Varshini Reddy, Charles Lovering, and Chris Tanner. 2024. https://doi.org/10.18653/v1/2024.acl-long.452 B iz B ench: A quantitative reasoning benchmark for business and finance . In Proceedings of the 62nd Annual Meeting o...

    2024 doi

  12. [20]

    Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. 2025. https://arxiv.org/abs/2501.05366 Search-o1: Agentic search-enhanced large reasoning models . Preprint, arXiv:2501.05366

    2025 arXiv

  13. [21]

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. https://openreview.net/forum?id=v8L0pN6EOi Let's verify step by step . In The Twelfth International Conference on Learning Re...

    2024

  14. [22]

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. https://openreview.net/forum?id=KUNzEQMWU7 Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts . I...

    2024

  15. [23]

    Yujun Mao, Yoon Kim, and Yilun Zhou. 2024. https://doi.org/10.18653/v1/2024.findings-acl.785 CHAMP : A competition-level dataset for fine-grained analyses of LLM s' mathematical reasoning capabilities . In Findings of the Association for Computational Linguistics: ACL 2024, pa...

    2024 doi

  16. [24]

    OpenAI. 2024 a . https://openai.com/index/hello-gpt-4o/ Hello gpt-4o

    2024

  17. [25]

    OpenAI. 2024 b . https://openai.com/index/learning-to-reason-with-llms/ Learning to reason with llms

    2024

  18. [26]

    OpenAI. 2024 c . https://openai.com/index/openai-o1-mini-advancing-cost-efficient-reasoning/ Openai o1-mini

    2024

  19. [27]

    OpenAI. 2024 d . https://openai.com/index/openai-o1-system-card/ Openai o1 system card

    2024

  20. [28]

    OpenAI. 2025. https://openai.com/index/o3-mini-system-card/ Openai o3-mini system card

    2025

  21. [29]

    Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023. https://arxiv.org/abs/2302.12813 Check your facts and try again: Improving large language models with external knowledge and automate...

    2023 arXiv

  22. [30]

    Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Back. 2024. https://arxiv.org/abs/2407.11511 Reasoning with large language models, a survey . Preprint, arXiv:2407.11511

    2024

  23. [31]

    Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le ...

    2025 arXiv

  24. [32]

    Pawan Kumar, Emilien Dupont, Francisco J

    Bernardino Romera - Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. 2024. https://doi.org/10.1038/S41586-023-06924-6 Mat...

    2024 doi

  25. [33]

    Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, ...

    2025 arXiv

  26. [34]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qwq-32b-preview/ Qwq: Reflect deeply on the boundaries of the unknown

    2024

  27. [35]

    Prakhar Verma, Sukruta Prakash Midigeshi, Gaurav Sinha, Arno Solin, Nagarajan Natarajan, and Amit Sharma. 2025. https://openreview.net/forum?id=gi9aqlYdBk Plan ^ rag: Efficient test-time planning for retrieval augmented generation . In Workshop on Reasoning and Planning for La...

    2025

  28. [36]

    Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. 2024. https://openreview.net/forum?id=bq1JEgioLr Scibench: Evaluating college-level scientific problem-solving abilities of large language m...

    2024

  29. [37]

    Yuqing Wang and Yun Zhao. 2024. https://doi.org/10.18653/v1/2024.naacl-long.106 Metacognitive prompting improves understanding in large language models . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human...

    2024 doi

  30. [38]

    Chi, Quoc V Le, and Denny Zhou

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. https://openreview.net/forum?id=_VjQlMeSB_J Chain of thought prompting elicits reasoning in large language models . In Advances in Neural Information Proc...

    2022

  31. [39]

    Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, Chenyang Shao, Yuwei Yan, Qinglong Yang, Yiwen Song, Sijian Ren, Xinyuan Hu, Yu Li, Jie Feng, Chen Gao, and Yong Li. 2025. https://arxiv.org/a...

    2025 arXiv

  32. [40]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024. https://openaccess.thecvf.com/content/CVPR2024/html/Yue_MMMU_A_Massive_Multi-discipline_Multimodal_Understanding_and_Reasoning_Benchmark_f...

    2024

  33. [41]

    Yilun Zhao, Hongjun Liu, Yitao Long, Rui Zhang, Chen Zhao, and Arman Cohan. 2024. https://doi.org/10.18653/v1/2024.acl-long.693 Financemath: Knowledge-intensive math reasoning in finance domains . In Proceedings of the 62nd Annual Meeting of the Association for Computational L...

    2024 doi

  34. [42]

    Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. https://doi.org/10.18653/v1/2021.acl-long.254 TAT - QA : A question answering benchmark on a hybrid of tabular and textual content in finance . In Proceedings of...

    2021 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.