Pith. sign in

REVIEW 4 major objections 5 minor 48 references

Understanding Financial Reasoning in AI: A Multimodal Benchmark and Error Learning Approach

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Error feedback from past mistakes lifts financial reasoning accuracy in multimodal AI models by up to 12.44 percentage points.

desk verdict FinMR is a plausible benchmark resource, but the EFL gains are confounded by answer-plus-explanation leakage in the retrieved prompt. read the letter →

arxiv 2506.06282 v1 pith:QQ5QDDV3 submitted 2025-04-22 cs.AI

Shuangyan Deng , Haizhou Peng , Jiachen Xu , Chunhou Liu , Ciprian Doru Giurcuaneanu , Jiamou Liu This is my paper · ORCID
classification cs.AI
keywords FinMRbenchmarkfinancialreasoningmultimodallargelanguagemodelserrorfeedbacklearningchain-of-thoughtpromptingretrieval-augmentedimagerecognitionerrorsmath
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces FinMR, a benchmark of 3,200 expert-level financial question-answer pairs that pair text with charts, tables, and other visuals across 15 finance topics. Its central claim is that multimodal models reason better on finance when they receive both the image and a retrieved example of a similar past mistake with corrective feedback, a training-free method the paper calls Error Feedback Learning (EFL). The paper reports that EFL improves accuracy over plain chain-of-thought prompting across every model tested, with the largest reported gain 12.44 percentage points for Qwen-VL and with Gemini-1.5-Pro reaching 82.06 percent. It also finds that direct image input beats image captions, that math-heavy financial questions are roughly 10 points harder than expertise questions, and that image recognition failure accounts for the majority of errors. If these results hold, retrieving past errors is a cheap way to push financial reasoning accuracy without fine-tuning.

What carries the argument

The load-bearing machinery is the error database plus top-1 retrieval. For each development-set question a model got wrong, the paper has the model produce step-by-step reasoning, then prompts a financial expert with the wrong reasoning, the correct answer, and the gold explanation to generate feedback: corrected steps, a comparison of right versus wrong, and a reusable hint. At test time, EFL encodes the current question, retrieves the most semantically similar negative example from that database, and inserts the example and its feedback into the prompt before asking the model to reason step by step. This is what carries the claimed improvement: no weights change, only the prompt content changes.

What would settle it

A control experiment that retrieves a similar solved example with the same correct answer and explanation but no error feedback, or that strips the correct answer and explanation out of the feedback, would settle it: if the control matches EFL, the claimed error-learning mechanism is not the cause.

Watch

Extended reading notes

Core claim

The paper's central claim is that financial reasoning in large models is best measured with a multimodal benchmark and best improved, without retraining, by showing the model a similar solved-with-mistakes example and its feedback. FinMR supplies that measurement: 3,200 questions, each with text, at least one image, a correct option, and a manually verified expert explanation, drawn from professional certification-style university exams across 15 topics, split 80/20 into development and test sets. The EFL method builds an error database from the development set by taking the models' wrong answers, attaching AI-generated feedback that uses the gold explanation, and then, at test time, retrieving the most semantically similar error case and feeding it into the prompt. On this benchmark, the paper reports that EFL beats chain-of-thought prompting for every model surveyed, that the best multimodal result is Gemini-1.5-Pro at 82.06% accuracy, and that most failures, 72.84%, are image recognition errors rather than financial knowledge errors.

Load-bearing premise

The load-bearing premise is that the accuracy gain attributed to EFL comes from learning from past errors, rather than from the retrieved prompt already containing the correct answer and expert explanation.

Editorial extensions

If this is right

  • Multimodal financial QA systems can be improved on a fixed model simply by maintaining a growing database of its own past failures and retrieving the nearest one at inference time.
  • Because EFL works across all tested models, prompt-level error feedback could be folded into production financial assistants without GPU retraining.
  • Direct image access matters: the same Gemini-1.5-Pro model scores 82.06% with images versus 61.37% with captions, so finance benchmarks that omit images underestimate multimodal models.
  • Financial math remains a separate bottleneck, roughly 10 points below expertise questions, so gains in visual input do not remove the need for stronger calculation and formula application.
  • Image recognition failure, not financial knowledge, dominates errors at 72.84%, directing future work toward chart and table reading.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct ablation of EFL, comparing feedback that includes the correct answer against feedback that only says where the reasoning went wrong, would test whether the mechanism is error-based learning or answer leakage; the paper does not report this comparison.
  • If the gain is mostly leakage, the practical takeaway changes from 'models learn from mistakes' to 'models benefit from retrieving solved examples,' which is a retrieval-augmented generation result rather than a self-reflection result.
  • The same error database could be used to build few-shot curricula, retrieving not one but a sequence of errors of increasing similarity, which the paper does not explore.
  • The benchmark's manually verified explanations could support fine-grained per-topic diagnosis, for example separating formula errors from visual extraction errors within quantitative topics.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces FinMR, a multimodal financial reasoning benchmark with 3,200 (also stated as 3,700 in Table 1) expert-level QA pairs across 15 topics, and proposes Error Feedback Learning (EFL), a retrieval-augmented prompting method that retrieves a similar negative example with model-generated feedback from a development-set error database. The authors evaluate several LLMs and MLLMs under CoT and EFL prompting, reporting that EFL consistently improves accuracy over CoT (up to 12.44% for Qwen-VL) and that Gemini-1.5-Pro with EFL reaches 82.06%. They also present an error-type analysis identifying image recognition as the dominant failure mode. The paper claims three contributions: the FinMR benchmark, the EFL method, and a systematic evaluation/error analysis.

Significance. If the benchmark is validated and the EFL gains are shown to arise from error feedback, the paper would be a useful contribution to multimodal financial reasoning: FinMR covers a broader topic set than FAMMA, includes manual explanations, and the EFL method is training-free and simple. Credit is due for the expert-verification pipeline, the manual annotation of explanations, and the breadth of models and topics evaluated. However, the central EFL claim is currently confounded: the retrieved example embeds the correct answer and gold explanation, so the comparison to zero-shot CoT cannot separate the effect of error feedback from the effect of receiving a solved in-context example. The benchmark also has an internal counting inconsistency and the data are not released, which weakens reproducibility. The findings are therefore promising but not yet established.

major comments (4)
  1. [Section 4.1–4.2, Figures 3–4] The EFL evaluation is structurally confounded. The feedback prompt in Figure 3 explicitly includes the Correct Answer and Explanation as inputs, and the EFL test prompt in Figure 4 supplies a Retrieved Example drawn from the resulting database. Thus the EFL condition gives the model a fully solved similar problem, while the zero-shot CoT baseline provides no such demonstration. The reported improvements (e.g., 12.44% for Qwen-VL) could therefore be entirely due to few-shot in-context learning of a solved example rather than to learning from errors. To support the paper's central claim, the authors must add control conditions, such as retrieving a positive example, retrieving an example without the answer/explanation, or ablating the feedback component while keeping the retrieved problem.
  2. [Table 1 vs. Abstract, Section 1, Table 3] The number of examples in FinMR is inconsistent: Table 1 reports 3,700, while the abstract, Section 1, and Table 3 all report 3,200. Since the benchmark size is a headline statistic and the paper relies on it in multiple places, this discrepancy must be resolved and the correct count stated consistently throughout.
  3. [Section 4.2, retrieval setup] The retrieval index is built from the development set, which is drawn from the same exam sources as the test set. If a test question is a near-duplicate of a development question (e.g., the same template with different numbers), the top-1 retrieved example could effectively leak the solution. The paper does not report retrieval similarity distributions, does not exclude near-duplicates, and does not analyze how often retrieved examples are near-duplicates. This leakage risk directly affects the validity of the EFL results and needs to be addressed.
  4. [Section 5.3, error type analysis] The claim that image recognition failures account for 72.84% of total errors is presented without describing the categorization protocol, the number of annotators, the coding instructions, or inter-annotator agreement. Given that the error analysis is one of the three stated contributions, the reliability of this statistic should be documented, and the paper should clarify whether the percentage is computed over all errors or a sampled subset.
minor comments (5)
  1. [Table 3] The image statistics are internally inconsistent: 2,643 + 1,118 = 3,761, not 3,764, and the '30%' label for multiple-image QAs does not match either the QA proportion (557/3,200 ≈ 17.4%) or the image proportion (1,118/3,764 ≈ 29.7%). Please correct the totals and percentages.
  2. [Table 4] Several entries in the LLaMa-3.2-Vision rows appear to be typos (e.g., '43.00' in the PM column and '77.00' in the FRA column), and they are formatted inconsistently with the other entries. Please verify all numbers in the table.
  3. [References] The FAMMA reference is listed as 'Famma' in the bibliography; please unify the capitalization and ensure all citations match the reference list.
  4. [Abstract and data availability] The anonymous link in the abstract is not a functional data release; for reproducibility, the paper should state where and when the data and code will be made available, or explicitly describe any restrictions.
  5. [Section 5.2] The claim that 'multimodal inputs significantly enhance performance' is only directly supported by the within-model comparison for Gemini-1.5-Pro (text+caption vs. text+image); the other comparisons are between different model families. Please qualify the claim accordingly.

Circularity Check

1 steps flagged · score 6.0 of 10

EFL's reported gains are confounded: the retrieved 'error feedback' is generated from, and includes, the correct answer and gold explanation, so the improvement over zero-shot CoT may be in-context answer leakage rather than error learning.

  1. fitted input called prediction [Section 4.1 Error Database Construction (Fig. 3); Section 4.2 Evaluation Process (Fig. 4 and Fig. 5 caption)]
    "For incorrect responses, we employ a feedback prompt (depicted in Figure 3) to guide the model in refining its reasoning steps. This process is further supported by manually annotated explanations. ... Correct Answer: {answer}; Explanation: {explanation}. ... The latter retrieves the most similar (i.e., top-1 semantic similarity) negative examples and error feedback for learning."

    The error database is constructed by feeding the model the ground-truth answer and gold explanation (Fig. 3 gives 'Correct Answer: {answer}; Explanation: {explanation}'). At test time, EFL inserts the retrieved stored example—including this answer-bearing feedback—into the prompt (Fig. 4 'Retrieved Example: {example}'), while the CoT baseline receives no such demonstration. The paper then attributes the consistent EFL gains to 'leveraging negative examples with feedback' (Section 5.2). Because the feedback already contains the solution and correct reasoning, the gain over CoT is indistinguishable from standard few-shot in-context learning of a solved example; the causal mechanism 'learning from past errors' is not identified.

full rationale

The FinMR benchmark itself is not circular: the QA pairs are sourced from external exam papers, expert-verified, and the benchmark comparisons do not presuppose the conclusions. The circularity is confined to the EFL method claim. The error database construction (Section 4.1, Fig. 3) explicitly supplies the model with 'Correct Answer: {answer}' and 'Explanation: {explanation}' when generating feedback; the EFL prompt (Section 4.2, Fig. 4) then inserts the retrieved example built from this database into the final prompt. Thus, compared with zero-shot CoT, EFL includes a solved similar problem with its correct option and reasoning steps. The consistent accuracy gains reported in Table 4 and Section 5.2 (up to 12.44% for Qwen VL; Gemini-1.5-Pro at 82.06%) cannot be attributed to 'error feedback' specifically: a control retrieving positive examples, or stripping the answer and explanation from the retrieved example, would be needed. The paper provides no such control and no retrieval-similarity statistics, so the central mechanism claim is unsupported by construction. This is a partial circularity: the benchmark evaluation and multimodal/textual comparisons are independent, but the headline EFL result reduces to a confounded few-shot demonstration. Score 6.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The paper's central claims rest on the benchmark's annotation quality and the validity of the EFL evaluation protocol, both listed as axioms above.

free parameters (1)
  • Retrieval count k = 1
    EFL always retrieves the top-1 similar negative example; no ablation on k is provided, so the reported gains may depend on this arbitrary choice.
assumptions (3)
  • domain assumption The benchmark annotations are correct and free of systematic bias from the manual filtering process.
    The paper relies on two PhD students and four annotators to verify correctness and filter 4,470 QA pairs down to 3,200, but no inter-annotator agreement or audit is reported.
  • domain assumption The 80/20 topic-wise split prevents leakage of question content between the development error database and the test set.
    Retrieval in EFL uses the development set; if any test questions share source exams or near-duplicate content with development questions, the retrieved feedback could reveal answers.
  • ad hoc to paper The feedback generated by the model from the gold explanation accurately represents the reasoning error and provides useful guidance, rather than just restating the answer.
    The generated feedback is a key component of EFL, yet its quality is not measured or compared to human-written feedback.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding Financial Reasoning in AI: A Multimodal Benchmark and Error Learning Approach." pith.science (2026). https://pith.science/paper/QQ5QDDV3

@misc{pith2026250606282,
  author       = {Pith},
  title        = {Pith review of: Understanding Financial Reasoning in AI: A Multimodal Benchmark and Error Learning Approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QQ5QDDV3}},
  note         = {Machine review of arXiv:2506.06282}
}
read the original abstract

Effective financial reasoning demands not only textual understanding but also the ability to interpret complex visual data such as charts, tables, and trend graphs. This paper introduces a new benchmark designed to evaluate how well AI models - especially large language and multimodal models - reason in finance-specific contexts. Covering 3,200 expert-level question-answer pairs across 15 core financial topics, the benchmark integrates both textual and visual modalities to reflect authentic analytical challenges in finance. To address limitations in current reasoning approaches, we propose an error-aware learning framework that leverages historical model mistakes and feedback to guide inference, without requiring fine-tuning. Our experiments across state-of-the-art models show that multimodal inputs significantly enhance performance and that incorporating error feedback leads to consistent and measurable improvements. The results highlight persistent challenges in visual understanding and mathematical logic, while also demonstrating the promise of self-reflective reasoning in financial AI systems. Our code and data can be found at https://anonymous/FinMR/CodeData.

Figures

Figures reproduced from arXiv: 2506.06282 by the authors.

Figure 1
Figure 1. FinMR provides diverse visual data, as shown in panel (a). Evaluation of financial rea￾soning abilities of LLMs and MLLMs covers mathematical and expertise-based tasks (see panel (b)), and performance varies across 15 financial domain topics (see panel (c), the abbreviation list of topics provided in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Panel (a) provides a typical example of FinMR. Panel (b) exhibits three stages of error [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. EFL Prompt Template 4.2 EVALUATION PROCESS The evaluation process consists of four stages: test data input, reasoning, output, and evaluation, as displayed in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Four stages of the evaluation process. For LLMs with no visual ability, we leverage GPT [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 9 canonical work pages

  1. [1]

    Llama 3.2: Revolutionizing edge AI and vision with open, customizable models, 2024

    AI@Meta. Llama 3.2: Revolutionizing edge AI and vision with open, customizable models, 2024. URL https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/

    work page 2024

  2. [2]

    Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku anthropic, 2024

    Anthropic. Introducing computer use, a new claude 3.5 sonnet, and claude 3.5 haiku anthropic, 2024. URL https://www.anthropic.com/news/3-5-models-and-computer-use

    work page 2024

  3. [3]

    Qwen- VL : A versatile vision-language model for understanding, localization, text reading, and beyond, 2023

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen- VL : A versatile vision-language model for understanding, localization, text reading, and beyond, 2023. URL http://arxiv.org/abs/2308.12966

    arXiv 2023

  4. [4]

    Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression, 2022 a

    Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression, 2022 a . URL https://arxiv.org/abs/2212.02746

    arXiv 2022

  5. [5]

    Xing, and Liang Lin

    Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P. Xing, and Liang Lin. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning, 2022 b . URL https://arxiv.org/abs/2105.14517

    arXiv 2022

  6. [6]

    M ^3 CoT : A novel benchmark for multi-domain multi-step multi-modal chain-of-thought

    Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M ^3 CoT : A novel benchmark for multi-domain multi-step multi-modal chain-of-thought. arXiv, 2024. URL http://arxiv.org/abs/2405.16473

    arXiv 2024

  7. [7]

    See, think, confirm: Interactive prompting between vision and language models for knowledge-based visual reasoning, 2023

    Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Hao Zhang, and Chuang Gan. See, think, confirm: Interactive prompting between vision and language models for knowledge-based visual reasoning, 2023. URL https://arxiv.org/abs/2301.05226

    arXiv 2023

  8. [8]

    FinQA : A dataset of numerical reasoning over financial data

    Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. FinQA : A dataset of numerical reasoning over financial data. pp.\ 3697--3711. Association for Computational Linguistics, 2021. doi:10.18653/v1/2021.emnlp-main.300. URL https://aclanthology.org...

    doi:10.18653/v1/2021.emnlp-main.300 2021

Show all 48 references
  1. [9]

    FinQA : A dataset of numerical reasoning over financial data

    Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. FinQA : A dataset of numerical reasoning over financial data. arXiv, 2022 c . URL http://arxiv.org/abs/2109.00122

    2022 arXiv

  2. [10]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025

    DeepSeek-AI, Daya Guo, and Dejian Yang. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948

    2025 arXiv

  3. [11]

    The llama 3 herd of models, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and Kadian. The llama 3 herd of models, 2024. URL http://arxiv.org/abs/2407.21783

    2024 arXiv

  4. [12]

    Gemini: A family of highly capable multimodal models, 2024

    GeminiTeam, Natalie Clay, Tomas Kocisky, Bartek Perz, Dian Yu, and Howard. Gemini: A family of highly capable multimodal models, 2024. URL http://arxiv.org/abs/2312.11805

    2024 arXiv

  5. [13]

    U Verfassung und Recht in \

    Morten Jerven. Poor numbers--how we are misled by african development statistics and what to do about it (uzuazo etemire). VR \"U Verfassung und Recht in \"U bersee , 46 0 (3): 0 336--340, 2013. URL http://www.jstor.org/stable/43239700

    2013

  6. [14]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp.\ 2...

    2022

  7. [15]

    LLaVA-NeXT : Stronger LLMs supercharge multimodal capabilities in the wild, 2024 a

    Kaichen Li, Hao Zhang, Renrui Zhang, Dong Guo, Feng Li, Yuanhan Zhang, Ziwei Liu, Chun Yuan, and Bo Li. LLaVA-NeXT : Stronger LLMs supercharge multimodal capabilities in the wild, 2024 a . URL https://llava-vl.github.io/blog/2024-05-10-llava-next-stronger-llms/

    2024

  8. [16]

    A survey of multimodal composite editing and retrieval, 2024 b

    Suyan Li, Fuxiang Huang, and Lei Zhang. A survey of multimodal composite editing and retrieval, 2024 b . URL http://arxiv.org/abs/2409.05405

    2024 arXiv

  9. [17]

    Zhenwen Liang, Tianyu Yang, Jipeng Zhang, and Xiangliang Zhang. Unimath. pp.\ 7126--7133. Association for Computational Linguistics, 2023. doi:10.18653/v1/2023.emnlp-main.440. URL https://aclanthology.org/2023.emnlp-main.440

    2023 doi

  10. [18]

    Retrieval-augmented multi-modal chain-of-thoughts reasoning for large language models, 2024

    Bingshuai Liu, Chenyang Lyu, Zijun Min, Zhanyu Wang, Jinsong Su, and Longyue Wang. Retrieval-augmented multi-modal chain-of-thoughts reasoning for large language models, 2024. URL http://arxiv.org/abs/2312.01714

    2024 arXiv

  11. [19]

    A survey of deep learning for mathematical reasoning

    Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Chang. A survey of deep learning for mathematical reasoning. arXiv, 2023. doi:10.48550/arXiv.2212.10535. URL http://arxiv.org/abs/2212.10535

    2023 doi arXiv

  12. [20]

    MathVista : Evaluating mathematical reasoning of foundation models in visual contexts, 2024

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista : Evaluating mathematical reasoning of foundation models in visual contexts, 2024. URL http://arxiv.org/abs/2310.02255

    2024 arXiv

  13. [21]

    An engine, not a camera: How financial models shape markets, volume 48

    Donald MacKenzie. An engine, not a camera: How financial models shape markets, volume 48. Mit Press, 2008. doi:https://doi.org/10.1353/tech.2007.0154

    2008

  14. [22]

    Introducing GPT-4o and more tools to ChatGPT free users OpenAI , 2024 a

    OpenAI. Introducing GPT-4o and more tools to ChatGPT free users OpenAI , 2024 a . URL https://openai.com/index/gpt-4o-and-more-tools-to-chatgpt-free/

    2024

  15. [23]

    Introduction OpenAI o1, 2024 b

    OpenAI. Introduction OpenAI o1, 2024 b . URL https://openai.com/o1/

    2024

  16. [24]

    Qwen2.5- LLM : Extending the boundary of LLMs , 2024

    Team Qwen. Qwen2.5- LLM : Extending the boundary of LLMs , 2024. URL http://qwenlm.github.io/blog/qwen2.5-llm/

    2024

  17. [25]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath : Pushing the limits of mathematical reasoning in open language models, 2024. URL http://arxiv.org/abs/2402.03300

    2024 arXiv

  18. [26]

    A review of multimodal explainable artificial intelligence: Past, present and future

    Shilin Sun, Wenbin An, Feng Tian, Fang Nan, Qidong Liu, Jun Liu, Nazaraf Shah, and Ping Chen. A review of multimodal explainable artificial intelligence: Past, present and future. arXiv, 2024. doi:10.48550/arXiv.2412.14056. URL http://arxiv.org/abs/2412.14056

    2024 doi arXiv

  19. [27]

    Cheng Tan, Jingxuan Wei, Linzhuang Sun, Zhangyang Gao, Siyuan Li, Bihui Yu, Ruifeng Guo, and Stan Z. Li. Retrieval meets reasoning: Even high-school textbook knowledge benefits multimodal reasoning, 2024. URL http://arxiv.org/abs/2405.20834

    2024 arXiv

  20. [28]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, and Gulati. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL http://arxiv.org/abs/2403.05530

    2024 arXiv

  21. [29]

    Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. arXiv, 2021. URL http://arxiv.org/abs/2106.13884

    2021 arXiv

  22. [30]

    Measuring multimodal mathematical reasoning with MATH-vision dataset, 2024 a

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with MATH-vision dataset, 2024 a . URL http://arxiv.org/abs/2402.14804

    2024 arXiv

  23. [31]

    Exploring the reasoning abilities of multimodal large language models ( MLLMs ): A comprehensive survey on emerging trends in multimodal reasoning, 2024 b

    Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu, Bohan Zhai, Jianbo Yuan, Quanzeng You, and Hongxia Yang. Exploring the reasoning abilities of multimodal large language models ( MLLMs ): A comprehensive survey on emerging trends in multimodal reason...

    2024 arXiv

  24. [32]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2...

    2022 arXiv

  25. [33]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. 2022 b . URL http://arxiv.org/abs/2201.11903

    2022 arXiv

  26. [34]

    Famma: A benchmark for financial domain multilingual multimodal question answering, 2024

    Siqiao Xue, Tingting Chen, Fan Zhou, Qingyang Dai, Zhixuan Chu, and Hongyuan Mei. Famma: A benchmark for financial domain multilingual multimodal question answering, 2024. URL https://arxiv.org/abs/2410.04526

    2024 arXiv

  27. [35]

    TabMedBERT : A tabular knowledge enhanced biomedical pretrained language model

    Xu Yan, Lei Geng, Ziqiang Cao, Juntao Li, Wenjie Li, Sujian Li, Xinjie Zhou, Yang Yang, and Jun Zhang. TabMedBERT : A tabular knowledge enhanced biomedical pretrained language model. IOS Press, 2024 a . URL https://ebooks.iospress.nl/doi/10.3233/FAIA240674

    2024 doi

  28. [36]

    A survey of mathematical reasoning in the era of multimodal large language model: Benchmark, method & challenges, December 2024 b

    Yibo Yan, Jiamin Su, Jianxiang He, Fangteng Fu, Xu Zheng, Yuanhuiyi Lyu, Kun Wang, Shen Wang, Qingsong Wen, and Xuming Hu. A survey of mathematical reasoning in the era of multimodal large language model: Benchmark, method & challenges, December 2024 b . URL http://arxiv.org/a...

    2024 arXiv

  29. [37]

    VisRAG : Vision-based retrieval-augmented generation on multi-modality documents, 2024

    Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, and Maosong Sun. VisRAG : Vision-based retrieval-augmented generation on multi-modality documents, 2024. URL http://arxiv.org/abs/2410.10594

    2024 arXiv

  30. [38]

    MMMU : A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI , 2024

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU : A ...

    2024 arXiv

  31. [39]

    Multimodal chain-of-thought reasoning in language models, 2024

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models, 2024. URL http://arxiv.org/abs/2302.00923

    2024 arXiv

  32. [40]

    Retrieving multimodal information for augmented generation: A survey, 2023 a

    Ruochen Zhao, Hailin Chen, Weishi Wang, Fangkai Jiao, Xuan Long Do, Chengwei Qin, Bosheng Ding, Xiaobao Guo, Minzhi Li, Xingxuan Li, and Shafiq Joty. Retrieving multimodal information for augmented generation: A survey, 2023 a . URL http://arxiv.org/abs/2303.10868

    2023 arXiv

  33. [41]

    MultiHiertt : Numerical reasoning over multi hierarchical tabular and textual data, 2022

    Yilun Zhao, Yunxiang Li, Chenying Li, and Rui Zhang. MultiHiertt : Numerical reasoning over multi hierarchical tabular and textual data, 2022. URL http://arxiv.org/abs/2206.01347

    2022 arXiv

  34. [43]

    FinanceMath : Knowledge-intensive math reasoning in finance domains, 2024 a

    Yilun Zhao, Hongjun Liu, Yitao Long, Rui Zhang, Chen Zhao, and Arman Cohan. FinanceMath : Knowledge-intensive math reasoning in finance domains, 2024 a . URL http://arxiv.org/abs/2311.09797

    2024 arXiv

  35. [44]

    DocMath-eval : Evaluating math reasoning capabilities of LLMs in understanding long and specialized documents, 2024 b

    Yilun Zhao, Yitao Long, Hongjun Liu, Ryo Kamoi, Linyong Nan, Lyuhao Chen, Yixin Liu, Xiangru Tang, Rui Zhang, and Arman Cohan. DocMath-eval : Evaluating math reasoning capabilities of LLMs in understanding long and specialized documents, 2024 b . URL http://arxiv.org/abs/2311.09805

    2024 arXiv

  36. [45]

    TAT-QA : A question answering benchmark on a hybrid of tabular and textual content in finance, 2021

    Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. TAT-QA : A question answering benchmark on a hybrid of tabular and textual content in finance, 2021. URL http://arxiv.org/abs/2105.07624

    2021 arXiv

  37. [46]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  38. [47]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  39. [48]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  40. [49]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.