Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces FinMME, an 11,099-question financial multimodal benchmark, and reports that GPT-4o scores only 46.56 average accuracy on it.

desk verdict A valuable financial MLLM benchmark that needs its scoring formula and error-rate claim cleaned up before the numbers can be trusted. read the letter →

arxiv 2505.24714 v1 pith:BKHNO3VP submitted 2025-05-30 cs.CL

Junyu Luo , Zhizhuo Kou , Liming Yang , Xiao Luo , Jinsheng Huang , Zhiping Xiao , Jingshu Peng , Chengzhong Liu
show 5 more authors
Jiaming Ji Xuanzhe Liu Sirui Han Ming Zhang Yike Guo
This is my paper
classification cs.CL
keywords financialmultimodalbenchmarkchartunderstandinglargelanguagemodelsreasoninghallucinationpenaltydomain-normalizedevaluationFinMMEScore
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FinMME is a new benchmark dataset for testing multimodal AI on professional financial documents, built from more than 11,000 questions anchored to real research-report charts. The authors claim it is the largest and most carefully curated financial multimodal evaluation set to date, with 18 knowledge domains, 6 asset classes, and a three-tier question design that spans basic perception, fine-grained reading, and analytical reasoning. They report that the best proprietary and open-source models, including GPT-4o, score only about 46.6 on average, and that a new FinScore metric with hallucination penalties drops even strong models below 20. The paper's point is that current multimodal models are far from reliable enough for realistic financial analysis, and that FinMME provides the measurement instrument to show it.

What carries the argument

The load-bearing machinery is the dataset and its scoring protocol. Each FinMME sample pairs a financial chart (organized into 10 main and 21 sub chart types) with captions, fine-grained labels, and a question at one of three cognitive levels: comprehensive perception, fine-grained perception, or analysis and reasoning. On top of this, the paper defines FinScore, a domain-normalized score $F = \frac{1}{K}\sum_{k=1}^K \frac{1}{N_k}\sum_{i=1}^{N_k} S_{k,i}$ multiplied by a hallucination penalty $(1 - P_H)$, where $P_H$ is the mean ratio of incorrect selections in multiple-answer questions. This makes every knowledge domain count equally and makes over-selection costly, which is what lets the benchmark separate raw accuracy from financially usable reliability.

What would settle it

Audit a random sample of, say, 200 FinMME questions: have two independent finance professionals answer them from chart plus caption with no prior exposure to model outputs, and measure agreement with the published ground truth. If agreement falls well short of the claimed below-1% annotation error rate, the benchmark's reliability and difficulty conclusions would need revision.

Watch

Extended reading notes

Core claim

The central claim is that FinMME constitutes a comprehensive, high-quality, and difficult benchmark for financial multimodal reasoning, and that current state-of-the-art multimodal large language models (MLLMs) fail it. On FinMME's 11,099 expert-annotated samples, GPT-4o reaches 46.56 average accuracy and a FinScore of 15.34, while the best tested system, Qwen2.5-VL 72B, reaches 52.54 average accuracy and FinScore 20.87. Calculation questions are the hardest block, with top models below 40%, and economics and fixed-income domains are consistently weak. The benchmark is presented as robust because repeated runs with different prompts stay within about 1% standard deviation.

Load-bearing premise

The load-bearing premise is that the expert-adjudicated answers in FinMME are correct and unambiguous; if even a few percent of the 11,099 labels are wrong or open to interpretation, the reported model scores and the claim that the benchmark is hard and stable would be distorted.

Editorial extensions

If this is right

  • Even the strongest tested models leave a large accuracy gap, so FinMME sets a clear target for financial multimodal research.
  • Fine-grained perception is the easiest of the three levels, while calculation and multi-answer questions are the bottlenecks; future models should focus there.
  • Domain-normalized FinScore shows that a model with decent average accuracy can still be unusable if it hallucinates, since FinScore collapses for models with high incorrect-selection rates.
  • The sub-1% prompt-variation standard deviation implies that FinMME can serve as a stable leaderboard without prompt-engineering noise.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If FinMME's labels are as clean as claimed, the benchmark could be extended to live data or perturbation tests, where the current stability result is explicitly limited.
  • The gap between average accuracy and FinScore suggests that a model's financial value depends more on suppressing wrong selections than on raw knowledge; future model development might target calibration rather than only accuracy.
  • A natural next test is whether performance on FinMME predicts success on real financial analyst tasks, since the paper itself notes MCQ format does not capture full workflow complexity.
  • The lack of reported inter-annotator agreement means the below-1% error claim should be verified by an independent audit before relying on exact model rankings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces FinMME, a financial multimodal question-answering benchmark with 11,099 samples drawn from professional research reports, covering 18 financial domains, 6 asset classes, and 10 chart types. The dataset is organized into three cognitive levels (comprehensive perception, fine-grained perception, analysis and reasoning) and uses single-answer, multiple-answer, and calculation questions. The authors describe a quality-control pipeline that combines human annotators, multiple LLM annotators, and expert adjudication, and they propose FinScore, a domain-normalized evaluation score with a hallucination penalty. They evaluate 17 proprietary and open-source MLLMs, report that the best models achieve only about 50% average performance, and claim high robustness with prediction variation below 1% across prompts. The dataset and evaluation code are publicly released.

Significance. If validated, FinMME would fill a clear gap: there are few specialized multimodal financial benchmarks, and existing ones are smaller or less finely labeled. The paper's strengths include the large released dataset, the effort to involve financial experts in annotation, the explicit hierarchical taxonomy, and a parameter-free scoring proposal. The FinScore formula and the robustness analysis are also directly checkable from the paper, which is a positive feature. However, the central reliability claims are currently not established. The scoring formula in Eq. (1) is inconsistent with the reported numerical results, and the asserted sub-1% annotation error rate is not supported by any measurement. These issues affect the validity of the model rankings and difficulty conclusions, so the benchmark's value as a reliable evaluation resource is not yet confirmed.

major comments (4)
  1. [Section 4.1, Eq. (1)] The scoring formula as printed cannot produce the scores reported in Table 3. If n is the total number of options, then a correct single-answer response to a 4-option question gives Sq = 1/4 = 0.25, and a perfect response to a 2-correct multiple-answer question with 4 options gives Sq = 2/4 = 0.5. This would cap the reported percentages far below 100%, yet Table 3 reports values such as 58.49 for Single and 54.26 for Multi. Either the formula is misprinted (for example, n should be the number of correct options rather than the total number of options), or Table 3 is computed with a different metric. Since FinScore and all domain averages are derived from Sq, the model rankings and the conclusion that state-of-the-art MLLMs score around 46.56 cannot be interpreted until this inconsistency is resolved. The authors should state the exact per-question computation used for Table 3 and correct Eq. (1) and its verbal definition accordingly.
  2. [Section 3.6 and Limitations] The claim that 'annotation error rates remain below 1%' is asserted in the abstract and Section 1, but Section 3.6 provides no error audit: there is no inter-annotator agreement statistic, no random sample re-labeled by independent experts, and no report of how many questions required expert adjudication or how often the expert review changed the LLM/human consensus. The Limitations section itself states that 'complex financial concepts posed interpretation difficulties even for knowledgeable annotators, potentially introducing subtle biases despite our quality control protocols.' The consistency-based pipeline can reduce obvious errors, but agreement among LLM annotators does not guarantee correctness, since shared biases can produce consistent wrong answers. Because label noise directly shifts every model score, domain comparison, and difficulty conclusion in Table 3 and Table 4, the sub-1% error claim is load-bearing and currently unverified. The authors should either provide a concrete audit (e.g., an independent expert relabeling of a random sample with reported agreement and error counts) or consistently describe the error rate as an estimate rather than an established fact.
  3. [Section 5.6 and abstract] The abstract and Section 1 claim that 'prediction variations under different prompts remain below 1%', but Section 5.6 reports standard deviations over five runs without describing any variation of the prompts. Repeating the same evaluation with the same prompt measures run-to-run sampling variance, not prompt sensitivity. The evidence in Table 5 supports only the weaker statement that scores are stable across repeated runs under a fixed protocol. To support the prompt-robustness claim, the authors should run an experiment with rephrased instructions, different option orders, or other prompt perturbations and report the resulting score distributions; otherwise the claim should be revised to 'run-to-run standard deviation below 1%'.
  4. [Section 3.2, Section 5.4, Eq. (2), Appendix E] The number of knowledge domains used for the domain-normalized FinScore is inconsistent across the paper. Section 3.2 lists 18 core financial domains, Section 5.4 and Table 4 use 16 columns including an 'Others' category, and Appendix E and Figure 7 state 15 knowledge domains. Equation (2) defines FinScore as the mean over K domain averages, so the exact value of K and the domain assignments matter for reproducibility. If 'Others' aggregates multiple listed domains, those domains do not contribute equally, which contradicts the stated goal that 'each knowledge domain contributes equally to the final score.' The authors should align the counts, list the exact domains used in Eq. (2), and report per-domain sample sizes.
minor comments (6)
  1. [Section 1] The model name 'Germini Flash' should be 'Gemini Flash'.
  2. [Table 5 caption] The caption reads 'with5runs'; it should be 'with 5 runs'. The authors should also state whether temperature and decoding settings were identical across the five runs.
  3. [Equations (2) and (4)] The symbol F is used both for the domain-normalized average score in Eq. (2) and for the final FinScore in Eq. (4). Using two different symbols would reduce confusion.
  4. [Section 3.2] The list of 18 domains includes 'Military & Defense', but Table 4 has no column for this domain; the paper should state where this category falls in the domain-level analysis or provide a complete mapping to the 16 columns.
  5. [Figure 4 and Section 5.2] The evaluation prompt is shown only as a figure; for reproducibility, the exact prompt text should be included in an appendix or the repository.
  6. [Appendix D] Model names such as 'Qwen25vl72b' should be written in the same consistent notation used elsewhere in the paper (e.g., 'Qwen2.5-VL 72B').

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FinScore is defined by explicit formulas, model scores are external measurements, and the dataset's quality claims do not reduce to the paper's own outputs.

full rationale

The paper's central derivation chain is FinScore from Eqs. (1)-(4), which are explicit arithmetic definitions with no fitted parameters and no dependence on the evaluated models' outputs beyond the raw scores being aggregated. The reported model performances in Tables 3-5 are external measurements on the released benchmark, not predictions derived from the benchmark's construction. The annotation pipeline in Section 3.6 uses human and LLM annotators with expert adjudication; while LLM agreement is used as a QC signal, the final ground truth is determined by expert review, so no label is defined in terms of a model being evaluated. The robustness claim (standard deviations below 1%) is an empirical observation from repeated runs, and the annotation-error claim is an asserted quality-control outcome, not a quantity that is definitionally equal to any input. The Limitations section even concedes potential annotator bias, which is a validity caveat rather than a circular step. Self-citations in the references (e.g., Luo et al., 2025) are background and not load-bearing for any derivation. Therefore no circular step can be exhibited from the paper's text, and the appropriate score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The benchmark's contributions rest on annotation correctness and dataset representativeness. No fitted parameters appear in FinScore, but tolerances and label correctness are unverified assumptions.

free parameters (1)
  • Numeric tolerance for calculation answers = Not reported
    Section 3.1 defines standard answers with unit and error tolerance ranges, which determine pass/fail for calculation questions; tolerances are chosen by annotators and not published.
assumptions (4)
  • domain assumption The expert-adjudicated ground-truth answers are correct and unambiguous.
    The quality-control protocol in Section 3.6 assumes that consistency among human and LLM annotations, plus expert review, yields true answers; no inter-annotator agreement or independent error audit is reported.
  • domain assumption All collected research reports are authorized for public dissemination.
    Section 3.4 states copyright compliance was prioritized, but no provenance, licenses, or permissions are provided.
  • domain assumption Multiple-choice and calculation questions can measure the financial reasoning levels defined in Section 3.3.
    The authors themselves note in the Limitations that MCQs may not capture real-world financial analysis complexity.
  • domain assumption The chart image plus supplied caption contains enough information to answer each question.
    QA generation used report text and captions; if external knowledge is required, the benchmark would test priors rather than chart reasoning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation." pith.science (2026). https://pith.science/paper/BKHNO3VP

@misc{pith2026250524714,
  author       = {Pith},
  title        = {Pith review of: FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BKHNO3VP}},
  note         = {Machine review of arXiv:2505.24714}
}
read the original abstract

Multimodal Large Language Models (MLLMs) have experienced rapid development in recent years. However, in the financial domain, there is a notable lack of effective and specialized multimodal evaluation datasets. To advance the development of MLLMs in the finance domain, we introduce FinMME, encompassing more than 11,000 high-quality financial research samples across 18 financial domains and 6 asset classes, featuring 10 major chart types and 21 subtypes. We ensure data quality through 20 annotators and carefully designed validation mechanisms. Additionally, we develop FinScore, an evaluation system incorporating hallucination penalties and multi-dimensional capability assessment to provide an unbiased evaluation. Extensive experimental results demonstrate that even state-of-the-art models like GPT-4o exhibit unsatisfactory performance on FinMME, highlighting its challenging nature. The benchmark exhibits high robustness with prediction variations under different prompts remaining below 1%, demonstrating superior reliability compared to existing datasets. Our dataset and evaluation protocol are available at https://huggingface.co/datasets/luojunyu/FinMME and https://github.com/luo-junyu/FinMME.

Figures

Figures reproduced from arXiv: 2505.24714 by the authors.

Figure 1
Figure 1. The Comprehensive Taxonomy, Data Examples and Statistical Characteristics of [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Representative examples of different question types in F [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The annotation pipeline of FINMME. The process consists of three main stages: (1) Raw Multimodal Data collection, (2) Annotation through parallel human and LLM annotators to ensure external and internal consistency, and (3) Quality Control checking where expert reviewers validate consistent annotations and resolve inconsistencies. of work from the 20-member team, with time esti￾mates aggregated from individual contr… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Illustration of the evaluation prompt template. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The radar chart of the asset class distribution [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The heatmap of the knowledge domain distribution of the dataset. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Data Comparison with related works [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A new benchmark shows LLMs' financial calculation accuracy collapses without explicit formulas and degrades further when they must generate multi-metric tables.

Reference graph

Works this paper leans on

45 extracted references · 9 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, et al. 2024. https://arxiv.org/abs/2404.14219 Phi-3 technical report: A highly capable language model locally on your phone . Preprint, arXiv:2404.14219

    arXiv 2024

  4. [4]

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966

    arXiv 2023

  5. [5]

    Gagan Bhatia, Hasan Cavusoglu, Muhammad Abdul-Mageed, et al. 2024. Fintral: A family of gpt-4 level multimodal financial large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 13064--13087

    work page 2024

  6. [6]

    Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. 2010. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pages 333--342

    work page 2010

  7. [7]

    Jian Chen, Peilin Zhou, Yining Hua, Yingxin Loh, Kehui Chen, Ziyuan Li, Bing Zhu, and Junwei Liang. 2024 a . https://arxiv.org/abs/2405.09980 Fintextqa: A dataset for long-form financial question answering . Preprint, arXiv:2405.09980

    arXiv 2024

  8. [8]

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. 2024 b . Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330

    arXiv 2024

Show all 45 references
  1. [9]

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Doll \'a r, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325

    2015 arXiv

  2. [10]

    Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. 2022. Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Pr...

    2022

  3. [11]

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. 2024 a . https://arxiv.org/abs/2306.13394 Mme: A comprehensive evaluation benchmark for multimodal large language models . Preprin...

    2024 arXiv

  4. [12]

    Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Shaoqi Dong, Xiong Wang, Di Yin, Long Ma, et al. 2024 b . Vita: Towards open-source interactive omni multimodal llm. arXiv preprint arXiv:2408.05211

    2024 arXiv

  5. [13]

    Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Yangze Li, Zuwei Long, Heting Gao, Ke Li, et al. 2025. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction. arXiv preprint arXiv:2501.01957

    2025 arXiv

  6. [14]

    Chaoyou Fu, Yi-Fan Zhang, Shukang Yin, Bo Li, Xinyu Fang, Sirui Zhao, Haodong Duan, Xing Sun, Ziwei Liu, Liang Wang, et al. 2024 c . Mme-survey: A comprehensive survey on evaluation of multimodal llms. arXiv preprint arXiv:2411.15296

    2024 arXiv

  7. [15]

    Ziliang Gan, Yu Lu, Dong Zhang, Haohan Li, Che Liu, Jian Liu, Ji Liu, Haipang Wu, Chaoyou Fu, Zenglin Xu, et al. 2024. Mme-finance: A multimodal finance benchmark for expert-level understanding and reasoning. arXiv preprint arXiv:2411.03314

    2024 arXiv

  8. [16]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904--6913

    2017

  9. [17]

    Shengding Hu, Yuge Tu, Xu Han, et al. 2024. https://arxiv.org/abs/2404.06395 Minicpm: Unveiling the potential of small language models with scalable training strategies . Preprint, arXiv:2404.06395

    2024 arXiv

  10. [18]

    Jinsheng Huang, Liang Chen, Taian Guo, Fu Zeng, Yusheng Zhao, Bohan Wu, Ye Yuan, Haozhe Zhao, Zhihui Guo, Yichi Zhang, et al. 2024. Mmevalpro: Calibrating multimodal benchmarks towards trustworthy and efficient evaluation. arXiv preprint arXiv:2407.00468

    2024 arXiv

  11. [19]

    Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700--6709

    2019

  12. [20]

    Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. 2023. Financebench: A new benchmark for financial question answering. arXiv preprint arXiv:2311.11944

    2023 arXiv

  13. [21]

    Yang Lei, Jiangtong Li, Ming Jiang, Junjie Hu, Dawei Cheng, Zhijun Ding, and Changjun Jiang. 2023. Cfbenchmark: Chinese financial assistant benchmark for large language model. arXiv preprint arXiv:2311.05812

    2023 arXiv

  14. [22]

    Bohao Li, Yuying Ge, Yi Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan. 2024 a . Seed-bench-2-plus: Benchmarking multimodal large language models with text-rich visual comprehension. arXiv preprint arXiv:2404.16790

    2024 arXiv

  15. [23]

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023 a . Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125

    2023 arXiv

  16. [24]

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2024 b . Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36

    2024

  17. [25]

    Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557

    2019 arXiv

  18. [26]

    Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. 2023 b . Large language models in finance: A survey. In Proceedings of the fourth ACM international conference on AI in finance, pages 374--382

    2023

  19. [27]

    Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. 2024. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26689--26699

    2024

  20. [28]

    Chenghao Liu, Arunkumar Arulappan, Ranesh Naha, Aniket Mahanti, Joarder Kamruzzaman, and In-Ho Ra. 2024 a . Large language models and sentiment analysis in financial markets: A review, datasets and case study. IEEE Access

    2024

  21. [29]

    Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, and Dong Yu. 2023. Mmc: Advancing multimodal chart understanding with large-scale instruction tuning. arXiv preprint arXiv:2311.10774

    2023 arXiv

  22. [30]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. 2024 b . Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216--233. Springer

    2024

  23. [31]

    J. Luo, W. Zhang, Y. Yuan, et al. 2025. Large language model agent: A survey on methodology, applications and challenges. arXiv preprint arXiv:2503.21460

    2025 arXiv

  24. [32]

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244

    2022 arXiv

  25. [33]

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer ...

    2015

  26. [34]

    Ramit Sawhney, Puneet Mathur, Ayush Mangal, Piyush Khanna, Rajiv Ratn Shah, and Roger Zimmermann. 2020. Multimodal multi-task financial risk forecasting. In Proceedings of the 28th ACM international conference on multimedia, pages 456--465

    2020

  27. [35]

    Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. 2020. Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part II 16, pages 742--758. Springer

    2020

  28. [36]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  29. [37]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-vl: Enhancing vision-language mode...

    2024 arXiv

  30. [38]

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. 2023. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079

    2023 arXiv

  31. [39]

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, et al. 2024. https://arxiv.org/abs/2412.10302 Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding . Preprint, arXiv:2412.10302

    2024 arXiv

  32. [40]

    Qianqian Xie, Weiguang Han, Zhengyu Chen, et al. 2024. https://arxiv.org/abs/2402.12659 Finben: A holistic financial benchmark for large language models . Preprint, arXiv:2402.12659

    2024 arXiv

  33. [41]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  34. [42]

    Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Florencio, Lijuan Wang, Cha Zhang, Lei Zhang, and Jiebo Luo. 2021. Tap: Text-aware pre-training for text-vqa and text-caption. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8751--8761

    2021

  35. [43]

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490

    2023 arXiv

  36. [44]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. 2024. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conferenc...

    2024

  37. [45]

    Huaqin Zhao, Zhengliang Liu, Zihao Wu, Yiwei Li, Tianze Yang, Peng Shu, Shaochen Xu, Haixing Dai, Lin Zhao, Gengchen Mai, et al. 2024. Revolutionizing finance with llms: An overview of applications and insights. arXiv preprint arXiv:2401.11641

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.