Pith. sign in

REVIEW 2 major objections 8 minor 51 references

FCMR: Robust Evaluation of Financial Cross-Modal Multi-Hop Reasoning

T0 review · 2 major / 8 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Financial statements that require combining a 10-K report, a balance-sheet table, and a chart in three reasoning hops are answered correctly only 30% of the time by the best current multimodal model, Claude 3.5 Sonnet.

desk verdict A solid, useful new benchmark for cross-modal multi-hop reasoning, held back mainly by under-reported label validation. read the letter →

arxiv 2412.12567 v4 pith:WMAIS7EP submitted 2024-12-17 cs.CL

Seunghee Kim , Changhyeon Kim , Taeuk Kim This is my paper · ORCID
classification cs.CL
keywords cross-modalreasoningmulti-hopQAmultimodalLLMevaluationfinancialbenchmarkconstructiondatacontaminationchartunderstandinginformationretrievalbottleneck
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents FCMR, a benchmark for testing whether multimodal large language models can combine facts from text, tables, and charts to verify financial statements. It argues that existing benchmarks, notably MMQA, are contaminated—models can answer their hardest questions without the images—and contain almost no questions that genuinely need three modalities and three reasoning hops. FCMR's Hard tier is built so that all three modalities must be used, and on that tier the best tested model, Claude 3.5 Sonnet, reaches 30.4% accuracy against a 12.3% random baseline. The authors also trace model failures through four reasoning stages and find the main bottleneck is retrieving the right information from the right modality.

What carries the argument

The load-bearing object is CMRGen, an automated pipeline that aligns real 10-K text with Compustat tables, plots chart columns with varied chart types and styles, and generates statements by combining five base templates (Fact-Checking, Conditional Threshold, Arithmetic, Trend, Ranking) into one-, two-, and three-hop cross-modal statements. A second mechanism is the four-stage decomposition of reasoning—Planning, Modality Identification, Information Retrieval, Information Reasoning—used to locate where models fail; the paper reports that Information Retrieval is the stage where most errors occur.

What would settle it

Re-annotate all 2,142 Hard-level statements with independent human judges and verify a random sample of Easy and Medium labels; if the corrected labels change model rankings or close the gap between GPT-4o with and without charts, the robustness claim would be undermined.

Watch

Extended reading notes

Core claim

The central claim is that current MLLMs cannot yet perform reliable cross-modal three-hop reasoning in a professional domain, and that a carefully constructed benchmark can measure this ability without being cheated by memorized knowledge. The paper supports this by building FCMR from SEC 10-K reports and Compustat financial tables, generating statements at three difficulty levels, and showing that accuracy collapses toward random when charts are withheld: GPT-4o falls from 24.37% to 14.71% on the Hard tier, which the paper takes as evidence against data contamination. The strongest result is that the Hard tier, which forces text–table–chart integration in every item, caps the best model at 30.4% accuracy.

Load-bearing premise

The ground-truth truth values of the generated statements must be correct; since only 10% of samples received initial human review and Hard-level labels were fully checked with 22 of 714 items revised, a few percent of label errors could shift the reported accuracy gaps.

Editorial extensions

If this is right

  • If FCMR is right, prior benchmark scores overstate cross-modal reasoning ability; GPT-4o's 43% exact match on MMQA three-hop questions without images shows contamination.
  • Models that convert charts to tables with DePlot sometimes beat native MLLMs, suggesting that visual interpretation itself is a limiting factor.
  • The identified Information Retrieval bottleneck implies that improving retrieval from a specific modality could raise overall accuracy more than improving reasoning alone.
  • The Hard tier caps state-of-the-art models near 30%, so any future model claiming cross-modal reasoning competence should be measured against this tier.
  • The zero-to-three-correct multiple-choice design forces models to avoid guessing, making the reported accuracy gaps conservative rather than inflated.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the contamination check is specific to charts; the same withholding test applied to text or table modality could reveal whether the benchmark is contaminated along other channels.
  • Editorial inference: because Hard statements are generated from templates, a model could eventually learn template patterns; a dynamic version that regenerates statements from new 10-K filings would keep the benchmark contamination-resistant.
  • Editorial inference: the observed tendency of models to default to 'false' under uncertainty suggests calibration, not just reasoning ability, may account for part of the accuracy gap.
  • Editorial inference: if CMRGen transfers to other domains as claimed, the same contamination test could be applied to law or medicine datasets, where memorized knowledge is even harder to disentangle from reasoning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 8 minor

Summary. This paper introduces FCMR, a financial cross-modal multi-hop reasoning benchmark containing 2,199 instances at three difficulty levels (Easy, Medium, Hard), generated by an automated pipeline called CMRGen from SEC 10-K reports, Compustat financial tables, and synthetic charts. The authors evaluate a wide range of multimodal LLMs and text-only LLMs (with DePlot-based chart translation) under a zero-shot chain-of-thought setting, and report that state-of-the-art models still struggle, with Claude 3.5 Sonnet achieving only 30.39% accuracy on the Hard tier versus a 12.28% random baseline. They also present fine-grained stage-based analysis claiming that the main failure point is the Information Retrieval phase, and report preliminary prompting optimizations. The dataset and code are publicly released.

Significance. If the benchmark construction is valid, FCMR fills a real gap: it targets three-modality cross-modal multi-hop reasoning, whereas MMQA contains only 0.8% such cases and shows signs of visual-modality leakage. The paper ships a public dataset and generation code, a machine-checkable chart-removal ablation (Table 4: GPT-4o drops from 24.37% to 14.71% on Hard without charts), a broad model evaluation, paraphrase-diversity metrics, and several human-error analyses. These are concrete strengths that make the benchmark potentially useful to the community. The main risks are the incomplete verification of gold labels and the small manual sample underlying the central bottleneck claim.

major comments (2)
  1. [Section 3.2 and Appendix C.1] The gold labels used for all headline accuracy numbers are largely produced by the LLM pipeline (GPT-4o-mini generation, GPT-4o paraphrasing, Claude 3.5 Sonnet filtering); only 10% of Easy/Medium instances received human review, and Hard instances were fully reviewed, with 22/714 (3%) revised. Since the revision count itself shows that the automated filter is imperfect, the paper should report a post-revision, human-verified label error rate for the final dataset (at least for a random sample of Easy and Medium), along with inter-annotator agreement, and demonstrate that the Table 5 accuracy values (e.g., Claude 3.5 Sonnet 30.39% on Hard vs. random 12.28%) are stable when a few percent of labels are corrected. Without this, the central numerical claims could shift due to label noise.
  2. [Section 5.2 and Figure 22] The conclusion that the 'critical bottleneck' is the Information Retrieval stage is based on manually monitoring 40 Hard-level samples, with no inter-annotator agreement or release of the traced trajectories. Given that this finding is a major analytical contribution and is used to motivate the optimization in Section 6.4, the authors should either (i) provide the annotated trajectories and agreement statistics, (ii) extend the sample, or (iii) explicitly reframe the claim as an illustrative case study rather than a general bottleneck. As written, the strength of the claim exceeds the evidence.
minor comments (8)
  1. [Section 3.1 heading] The section title contains a typo: 'Datset' should be 'Dataset'.
  2. [Section 6.1] The cross-reference '(see Appendix 6.2 for details)' appears to refer to Section 6.2, not an appendix; please correct the reference.
  3. [Section 5.2 vs. Figure 22 caption] The main text says '40 given samples' while the Figure 22 caption says '40 Hard-level samples'; make the sample description consistent.
  4. [Section 2.2] The phrase 'revisit the drawbacks MMQA' is missing the preposition 'of' before 'MMQA'; grammatically it should be 'the drawbacks of MMQA'.
  5. [Table 3] The comparison of WPD/LD scores with MRPC and PAWS is illustrative, but these are different source corpora and tasks, so the claim of 'superiority' of the paraphrasing method should be phrased as a descriptive comparison rather than a controlled evaluation.
  6. [Section 3.4 and Figure 23] The prompt instructs models to treat any statement as false when there is insufficient information; this design choice may bias toward false predictions and should be discussed in relation to the conservative decision-making behavior reported in Section 5.1.
  7. [Table 2 and Table 4] The 'Random Selection' baselines differ between MMQA (random word from question/text/table, Appendix C.2) and FCMR (uniformly random among the eight answer sets); this difference should be stated in the main text near the tables to avoid misinterpretation.
  8. [Section 4.2 and Table 5] For GPT-4o and Claude 3.5 Sonnet, the DePlot-augmented LLM versions outperform their direct MLLM versions on Hard (32.91 vs. 24.37 for GPT-4o), which is a striking and non-obvious result; the paper should offer at least a brief explanation or acknowledge this explicitly in the main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark is generated once and evaluated against external models, so the central accuracy claims do not reduce to their inputs.

full rationale

FCMR's evaluation chain is self-contained and does not reduce to its inputs. The benchmark instances are produced once by CMRGen (Section 3.2) from SEC EDGAR 10-K reports and WRDS Compustat tables; difficulty levels are assigned by hop count (Table 12), and the gold labels are fixed before any model is evaluated. The headline accuracies (Table 5 and Table 4) are external measurements of held-out models against those fixed labels, with no parameter fitted to the model outputs and no prediction derived from the generation procedure. The no-chart contamination check is an independent control: GPT-4o's accuracy falls from 24.37% to 14.71% when charts are withheld (Table 4), and this drop is an empirical observation, not a consequence of the generation equations. No load-bearing self-citation or imported uniqueness theorem appears; the only self-referential component is using Claude 3.5 Sonnet to filter LLM-generated statements, which affects dataset construction but does not by itself force any model's later accuracy scores. The skeptic's concern about gold-label verification (only 10% of Easy/Medium human-reviewed; 22/714 Hard revisions) is a legitimate data-quality and correctness risk, not circularity: even if some labels are wrong, the error would be a measurement error in the benchmark, not a derivation that presupposes the conclusion. Therefore the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim of benchmark validity rests on generated ground truth, on the non-redundancy of modalities, and on the representativeness of small manual analyses; none of these are externally verified in the paper.

assumptions (4)
  • domain assumption LLM-generated statements and their truth labels are accurate.
    The pipeline uses GPT-4o-mini to extract facts and generate statements, GPT-4o for paraphrasing, Claude 3.5 Sonnet for filtering, and human review only on a subset; any errors propagate into the benchmark. See Section 3.2 and Appendix C.1.
  • domain assumption Hard questions cannot be solved without all three modalities.
    The chart column is removed from the table, and text facts are unique to text, so by construction all three are needed; only chart-ablation is tested, not text- or table-ablation. See Section 3.1 and Table 4.
  • domain assumption Manual stage-based failure analysis is representative.
    Only 40 Hard samples were manually traced to conclude an Information Retrieval bottleneck; no inter-annotator agreement is reported. See Section 5.2 and Figure 22.
  • domain assumption 10-K and Compustat sources are correctly aligned by company and year.
    The two sources are matched via common company entities; misalignment would create false statements. See Section 3.2(1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of FCMR: Robust Evaluation of Financial Cross-Modal Multi-Hop Reasoning." pith.science (2026). https://pith.science/paper/WMAIS7EP

@misc{pith2026241212567,
  author       = {Pith},
  title        = {Pith review of: FCMR: Robust Evaluation of Financial Cross-Modal Multi-Hop Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WMAIS7EP}},
  note         = {Machine review of arXiv:2412.12567}
}
read the original abstract

Real-world decision-making often requires integrating and reasoning over information from multiple modalities. While recent multimodal large language models (MLLMs) have shown promise in such tasks, their ability to perform multi-hop reasoning across diverse sources remains insufficiently evaluated. Existing benchmarks, such as MMQA, face challenges due to (1) data contamination and (2) a lack of complex queries that necessitate operations across more than two modalities, hindering accurate performance assessment. To address this, we present Financial Cross-Modal Multi-Hop Reasoning (FCMR), a benchmark created to analyze the reasoning capabilities of MLLMs by urging them to combine information from textual reports, tables, and charts within the financial domain. FCMR is categorized into three difficulty levels-Easy, Medium, and Hard-facilitating a step-by-step evaluation. In particular, problems at the Hard level require precise cross-modal three-hop reasoning and are designed to prevent the disregard of any modality. Experiments on this new benchmark reveal that even state-of-the-art MLLMs struggle, with the best-performing model (Claude 3.5 Sonnet) achieving only 30.4% accuracy on the most challenging tier. We also conduct analysis to provide insights into the inner workings of the models, including the discovery of a critical bottleneck in the information retrieval phase.

Figures

Figures reproduced from arXiv: 2412.12567 by the authors.

Figure 1
Figure 1. Prior benchmarks for cross-modal multi-hop [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An example from FCMR at the Hard difficulty level, where all statements require cross-modal three-hop [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. CMRGen is an automated and efficient framework for building cross-modal multi-hop reasoning datasets. (1) Modality Data Construction extracts text, table, and chart modalities from sources sharing common entities. (2) Statement Generation produces cross-modal one-, two-, and three-hop statements using LLM and templates. (3) Paraphrasing & Filtering applies two-stage paraphrasing with LLMs, followed by LLM & Human fi… view at source ↗
Figures from the paper (24 more)
Figure 4
Figure 4. Figure 4: Fine-grained stage-based analysis of three advanced MLLMs. This unique strategy discovers several [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Confusion matrices for three advanced MLLMs, with metrics in percentages (%). smaller open-source models. We focus on their per￾formance at the Hard level, as this subset presents the most challenging questions for the models. 5.1 Statement-Level Analysis While each pr…
Figure 6
Figure 6. Figure 6: Error categories and their distributions across 7k 8k dp [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 8
Figure 8. Figure 8: An example from Material-CMR. company executives to discuss operational results, providing insight into trends, risks, and strategies. In addition, Item 7A covers quantitative and qual￾itative disclosures regarding market risks, while Item 8 presents the audited financ…
Figure 9
Figure 9. Figure 9: Fine-grained Stage-based Analysis of Preliminary Task Optimization results applying Modality Integration, [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Example of data contamination in MMQA. Despite the requirement to infer the pink-highlighted [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: An example of decomposing the reasoning process of the Claude 3.5 Sonnet’s response into fine-grained, [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]
Figure 12
Figure 12. Figure 12: An example where the Gemini 1.5 Pro succeeds in (3) Information Retrieval but fails in (4) Information [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: An example where the Gemini 1.5 Pro succeeds in (1) Planning but fails in (2) Modality Identification. [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: In the chart, WBD shows a consistently increasing trend in act values across all years, but Claude 3.5 [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]
Figure 15
Figure 15. Figure 15: An example where the Claude model fails to interpret a cumulative graph: Although the company [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]
Figure 16
Figure 16. Figure 16: The condition that the dpc value is less than 1953.5 for all years is satisfied by both US Foods and [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]
Figure 17
Figure 17. Figure 17: -5487 is not smaller than -5543.5, but Claude failed to correctly compare the two numbers and [PITH_FULL_IMAGE:figures/full_fig_p021_17.png]
Figure 18
Figure 18. Figure 18: Although charts and tables were successfully interpreted to determine that Salesforce had the highest act [PITH_FULL_IMAGE:figures/full_fig_p021_18.png]
Figure 19
Figure 19. Figure 19: To compare the aloloch value, the chart should have been considered. Instead, having relied solely on [PITH_FULL_IMAGE:figures/full_fig_p022_19.png]
Figure 20
Figure 20. Figure 20: Although Claude found that Salesforce satisfied all the conditions, it misunderstood the statement and [PITH_FULL_IMAGE:figures/full_fig_p022_20.png]
Figure 21
Figure 21. Figure 21: Pie charts for answer types, statement types, and library usage categorized by difficulty levels. [PITH_FULL_IMAGE:figures/full_fig_p023_21.png]
Figure 22
Figure 22. Figure 22: Distribution of inference failures for each MLLM across the four reasoning stages (Planning, Modality [PITH_FULL_IMAGE:figures/full_fig_p023_22.png]
Figure 23
Figure 23. Figure 23: MLLMs zero-shot prompt. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_23.png]
Figure 24
Figure 24. Figure 24: (M)LLMs + Deplot zero-shot prompt. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_24.png]
Figure 25
Figure 25. Figure 25: Modality Integration prompt. For each statement, carry out your reasoning using the following strategy: (1) Planning: identifying the required values. (2) Modality Identification: recognizing which modality contains these values. (3) Information Retrieval: extracting …
Figure 26
Figure 26. Figure 26: 4-Stage Reasoning Strategy prompt. Reasoning Agent made the following reasoning about the problem: {Initial Output} You are the Refinement Agent tasked with refining Reasoning Agent's reasoning. If you find any errors in Reasoning Agent's reasoning for each statement,…
Figure 27
Figure 27. Figure 27: Self-Refine prompt. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_27.png]
Figure 28
Figure 28. Figure 28: Description of each column in the Annual Simplified Financial Statement. [PITH_FULL_IMAGE:figures/full_fig_p029_28.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 21 canonical work pages

  1. [1]

    Amirhossein Abaskohi, Spandana Gella, Giuseppe Carenini, and Issam H Laradji. 2024. Fm2ds: Few-shot multimodal multihop data synthesis with knowledge distillation for question answering. arXiv preprint arXiv:2412.07030

    arXiv 2024

  2. [2]

    Mubashara Akhtar, Chenxi Pang, Andreea Marzoca, Yasemin Altun, and Julian Martin Eisenschlos. 2024. Tanq: An open domain dataset of table answered questions. arXiv preprint arXiv:2405.07765

    work page Pith review arXiv 2024

  3. [3]

    Amazon. 2024. https://www.amazon.science/publications/amazon-nova-premier-technical-report-and-model-card Amazon nova premier: Technical report and model card . Amazon Technical Reports

    work page 2024

  4. [4]

    Anthropic . 2024. Claude: Large language model by anthropic. Available online at https://www.anthropic.com. Accessed on 2024-10-15

    work page 2024

  5. [5]

    Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. 2022. Webqa: Multihop and multimodal qa. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16495--16504

    work page 2022

  6. [6]

    Wenhu Chen, Ming - Wei Chang, Eva Schlinger, William Yang Wang, and William W. Cohen. 2021 a . https://openreview.net/forum?id=MmCRswl1UYl Open question answering over tables and text . In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net

    work page 2021

  7. [7]

    Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.91 H ybrid QA : A dataset of multi-hop question answering over tabular and textual data . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1026--1036, Online. Association for Computational...

    doi:10.18653/v1/2020.findings-emnlp.91 2020

  8. [8]

    Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021 b . https://doi.org/10.18653/v1/2021.emnlp-main.300 F in QA : A dataset of numerical reasoning over financial data . In Proceedings of the 2021 Conference on Empirical Methods in Natural L...

    doi:10.18653/v1/2021.emnlp-main.300 2021

Show all 51 references
  1. [9]

    Theodore E Christensen, Karson E Fronk, Joshua A Lee, and Karen K Nelson. 2024. Data visualization in 10-k filings. Journal of Accounting and Economics, 77(2-3):101631

    2024

  2. [10]

    Deepmind. 2025. https://deepmind.google/technologies/gemini/flash-thinking Gemini 2.0 flash thinking

    2025

  3. [11]

    DeepMind. 2025. https://arxiv.org/abs/2503.19786 Gemma 3 technical report . Preprint, arXiv:2503.19786

    2025 arXiv

  4. [12]

    Dolan and Chris Brockett

    William B. Dolan and Chris Brockett. 2005. https://aclanthology.org/I05-5002 Automatically constructing a corpus of sentential paraphrases . In Proceedings of the Third International Workshop on Paraphrasing ( IWP 2005)

    2005

  5. [13]

    Yifan Du, Zikang Liu, Yifan Li, Wayne Xin Zhao, Yuqi Huo, Bingning Wang, Weipeng Chen, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. 2025. https://arxiv.org/abs/2501.01904 Virgo: A preliminary exploration on reproducing o1-like mllm . Preprint, arXiv:2501.01904

    2025 arXiv

  6. [14]

    Ziliang Gan, Yu Lu, Dong Zhang, Haohan Li, Che Liu, Jian Liu, Ji Liu, Haipang Wu, Chaoyou Fu, Zenglin Xu, Rongjunchen Zhang, and Yong Dai. 2024. https://arxiv.org/abs/2411.03314 Mme-finance: A multimodal finance benchmark for expert-level understanding and reasoning . Preprint...

    2024 arXiv

  7. [15]

    Darryl Hannan, Akshay Jain, and Mohit Bansal. 2020. Manymodalqa: Modality disambiguation and qa over diverse inputs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7879--7886

    2020

  8. [16]

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023. https://arxiv.org/abs/2307.16125 Seed-bench: Benchmarking multimodal llms with generative comprehension . Preprint, arXiv:2307.16125

    2023 arXiv

  9. [17]

    Yongqi Li, Wenjie Li, and Liqiang Nie. 2022. Mmcoqa: Conversational question answering over text, tables, and images. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 4220--4231

    2022

  10. [18]

    Fangyu Liu, Julian Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun. 2023. https://doi.org/10.18653/v1/2023.findings-acl.660 D e P lot: One-shot visual language reasoning by plot-to-table tran...

    2023 doi

  11. [19]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024 a . Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296--26306

    2024

  12. [20]

    Timothy Liu and De Wen Soh. 2022. https://doi.org/10.18653/v1/2022.acl-long.588 Towards better characterization of paraphrases . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8592--8601, Dublin, Irelan...

    2022 doi

  13. [21]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. 2024 b . Mmbench: Is your multi-modal model an all-around player? In Computer Vision -- ECCV 2024, pages 216--233, Cham. Springer N...

    2024

  14. [22]

    Haohao Luo, Ying Shen, and Yang Deng. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.626 Unifying text, tables, and images for multimodal question answering . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9355--9367, Singapore. Associat...

    2023 doi

  15. [23]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. https://arxiv.o...

    2023 arXiv

  16. [24]

    Ahmed Masry and Mehrad Shahmohammadi. 2024. https://doi.org/10.18653/v1/2024.findings-acl.619 C hart I nstruct: Instruction tuning for chart comprehension and reasoning . In Findings of the Association for Computational Linguistics: ACL 2024, pages 10387--10409, Bangkok, Thail...

    2024 doi

  17. [25]

    OpenAI. 2024 a . https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    2024 arXiv

  18. [26]

    OpenAI. 2024 b . https://openai.com/index/learning-to-reason-with-llms Learning to reason with llms

    2024

  19. [27]

    Chaoxu Pang, Yixuan Cao, Chunhao Yang, and Ping Luo. 2024. https://doi.org/10.18653/V1/2024.FINDINGS-ACL.82 Uncovering limitations of large language models in information seeking from tables . In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Tha...

    2024 doi

  20. [28]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  21. [29]

    Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. 2024. Vision language models are blind. In Proceedings of the Asian Conference on Computer Vision, pages 18--34

    2024

  22. [30]

    Hossein Rajabzadeh, Suyuchen Wang, Hyock Ju Kwon, and Bang Liu. 2023. https://arxiv.org/abs/2309.08922 Multimodal multi-hop question answering through a conversation between tools and efficiently finetuned large language models . Preprint, arXiv:2309.08922

    2023 arXiv

  23. [31]

    Revant Gangi Reddy, Xilin Rui, Manling Li, Xudong Lin, Haoyang Wen, Jaemin Cho, Lifu Huang, Mohit Bansal, Avirup Sil, Shih-Fu Chang, et al. 2022. Mumuqa: Multimedia multi-hop news question answering via cross-media knowledge extraction and grounding. In Proceedings of the AAAI...

    2022

  24. [32]

    Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, and Jonathan Berant. 2021. https://openreview.net/forum?id=ee6W5UgQLa Multimodalqa: complex question answering over text, tables and images . In 9th International Co...

    2021

  25. [33]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://arxiv.org/abs/2302.13971 Llama:...

    2023 arXiv

  26. [34]

    Haoran Wang, Aman Rangapur, Xiongxiao Xu, Yueqing Liang, Haroon Gharwi, Carl Yang, and Kai Shu. 2024. Piecing it all together: Verifying multi-hop multimodal claims. arXiv preprint arXiv:2411.09547

    2024 arXiv

  27. [35]

    xAI. 2024. https://x.ai/news/grok-1212 Bringing grok to everyone

    2024

  28. [36]

    Suchow, Xiao-Yang Liu, Benyou Wang, Alejandro Lopez-Lira, Jimin Huang, and Sophia Ananiadou

    Qianqian Xie, Dong Li, Mengxi Xiao, Zihao Jiang, Ruoyu Xiang, Xiao Zhang, Zhengyu Chen, Yueru He, Weiguang Han, Yuzhe Yang, Shunian Chen, Yifei Zhang, Lihang Shen, Daniel Kim, Zhiwei Liu, Zheheng Luo, Yangyang Yu, Yupeng Cao, Zhiyang Deng, Zhiyuan Yao, Haohang Li, Duanyu Feng,...

    2024 arXiv

  29. [37]

    An Yang and Baosong Yang. 2024. https://arxiv.org/abs/2407.10671 Qwen2 technical report . Preprint, arXiv:2407.10671

    2024 arXiv

  30. [38]

    Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song, Haocheng Feng, Li Shen, and Dacheng Tao. 2024. https://arxiv.org/abs/2412.18319 Mulberry: Empowering mllm with o1-like reasoning and reflection via collective monte carlo tree...

    2024 arXiv

  31. [39]

    Yuan Yao and Tianyu Yu. 2024. https://arxiv.org/abs/2408.01800 Minicpm-v: A gpt-4v level mllm on your phone . Preprint, arXiv:2408.01800

    2024 arXiv

  32. [40]

    Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, Jiayi Lei, Quanfeng Lu, Runjian Chen, Peng Xu, Renrui Zhang, Haozhe Zhang, Peng Gao, Yali Wang, Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. 2024. https://proc...

    2024

  33. [41]

    Bowen Yu, Cheng Fu, Haiyang Yu, Fei Huang, and Yongbin Li. 2023 a . https://doi.org/10.18653/v1/2023.findings-acl.292 Unified language representation for question answering over text, tables, and images . In Findings of the Association for Computational Linguistics: ACL 2023, ...

    2023 doi

  34. [42]

    Bowen Yu, Cheng Fu, Haiyang Yu, Fei Huang, and Yongbin Li. 2023 b . https://arxiv.org/abs/2306.16762 Unified language representation for question answering over text, tables, and images . Preprint, arXiv:2306.16762

    2023 arXiv

  35. [43]

    Xiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2024. htt...

    2024

  36. [44]

    Qing Zhang, Haocheng Lv, Jie Liu, Zhiyun Chen, Jianyong Duan, Hao Wang, Li He, and Mingying Xu. 2024 a . An entailment tree generation approach for multimodal multi-hop question answering with mixture-of-experts and iterative feedback mechanism. In Proceedings of the 32nd ACM ...

    2024

  37. [45]

    Wenxuan Zhang, Sharifah Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2024 b . M3exam: a multilingual, multimodal, multilevel benchmark for examining large language models. In Proceedings of the 37th International Conference on Neural Information Processing System...

    2024

  38. [46]

    Yuan Zhang, Jason Baldridge, and Luheng He. 2019. https://doi.org/10.18653/v1/N19-1131 PAWS : Paraphrase adversaries from word scrambling . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Tec...

    2019 doi

  39. [47]

    Bowen Zhao, Tianhao Cheng, Yuejie Zhang, Ying Cheng, Rui Feng, and Xiaobo Zhang. 2024. Ct2c-qa: Multimodal question answering over chinese text, table and chart. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 3897--3906

    2024

  40. [48]

    Qi Zhi Lim, Chin Poo Lee, Kian Ming Lim, and Ahmad Kamsani Samingan. 2024. https://doi.org/10.1109/ACCESS.2024.3403101 Unirag: Unification, retrieval, and generation for multimodal question answering with pre-trained language models . IEEE Access, 12:71505--71519

    2024

  41. [49]

    Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. https://doi.org/10.18653/v1/2021.acl-long.254 TAT - QA : A question answering benchmark on a hybrid of tabular and textual content in finance . In Proceedings of...

    2021 doi

  42. [50]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  43. [51]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.