{"id":"076e060d-b274-4975-a5c2-fa2819a87724","arxiv_id":"2412.12567","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"FCMR is a finance-domain benchmark where every question requires combining text, tables, and charts; state-of-the-art MLLMs reach only 30.4% accuracy on the hardest tier.","lead":"This paper introduces FCMR, a new benchmark for testing whether multimodal AI models can answer questions that require combining text, tables, and charts in finance. It shows that even the best models, like Claude 3.5 Sonnet, get only about 30% right on the hardest questions, and that a key failure is in retrieving the right information from the right modality.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold-label verification is the weakest link: the 22/714 Hard revisions show LLM filtering errs, yet Easy/Medium lack full human checks and no label-error rate is reported.","rationale":"The reader's weakest assumption identifies the same risk I would stress. Every headline number is a comparison to gold labels, and those labels are mostly LLM-produced: Easy and Medium rely on GPT-4o-mini generation, GPT-4o paraphrasing, and Claude 3.5 Sonnet filtering, with only a 10% human sample; Hard got full human review but the 22/714 revisions show the upstream filter was not reliable. The paper's own numbers therefore imply a lower bound on pipeline error of about 3% on Hard, and possibly higher on Easy/Medium, yet no corrected-label evaluation is reported. A label error rate of 2 to 5 percent could shift reported accuracies by a similar amount, enough to change the exact '30.39%' figure and the 14.71% versus 12.28% robustness comparison, and to distort the difficulty gradient that motivates the benchmark design. I also considered whether the more fundamental risk is that Hard questions are solvable without all three modalities, because real 10-K text may contain the same financial values as the synthetic charts. The chart-ablation experiment (Table 4) is partial evidence against this, since GPT-4o drops to near random without charts, but it tests only one model and one modality. Still, the paper's design and the ablation make that concern less clearly decisive than the unverified labels. The proposed re-annotation directly measures the missing quantity, and the public release makes it feasible. A conditional acceptance is appropriate: the benchmark is valuable and the qualitative 'models struggle' conclusion is likely robust, but the exact figures and the contamination-robustness claim should be re-derived after a proper label-quality study.","tokens_in":25680,"tokens_out":11472,"duration_ms":106837,"concrete_test":"Draw a stratified random sample of 150 instances from each difficulty tier (450 total). Have two independent annotators, blind to the gold labels, verify each statement against the provided text/table/chart, with a third adjudicator for disagreements. Compute per-tier label error rate and Cohen's kappa. If error exceeds 2% or kappa is below 0.8, correct the labels and re-run the models in Table 5 on the corrected instances; if the Hard accuracy spread or the no-chart gap changes by more than 2 points, revise the headline claims.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central numerical claims—Claude 3.5 Sonnet at 30.39% on Hard versus 12.28% random, the Easy-to-Hard difficulty gradient, and the no-chart GPT-4o drop to 14.71%—are all accuracy measurements against gold labels. Those labels are produced by an automated pipeline: GPT-4o-mini generates, GPT-4o paraphrases, Claude 3.5 Sonnet filters, and only 10% of Easy/Medium instances received any human inspection; Hard instances received full review, which revised 22/714 (approximately 3%) of items (Section 3.2, Appendix C.1). The revision count demonstrates that the LLM filter is imperfect, yet the paper reports neither a post-revision label-error rate nor inter-annotator agreement, and no model is re-scored on corrected labels. If a few percent of labels are wrong—particularly if some 'true' statements are not actually supported by the source text, since LLM extraction can hallucinate—then the headline accuracies are mismeasured by a similar order of magnitude, and the conclusion that models 'struggle' at Hard could be partly an artifact of label noise. The public dataset makes this testable, which is why the conditional verdict is appropriate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces FCMR, a financial cross-modal multi-hop reasoning benchmark containing 2,199 instances at three difficulty levels (Easy, Medium, Hard), generated by an automated pipeline called CMRGen from SEC 10-K reports, Compustat financial tables, and synthetic charts. The authors evaluate a wide range of multimodal LLMs and text-only LLMs (with DePlot-based chart translation) under a zero-shot chain-of-thought setting, and report that state-of-the-art models still struggle, with Claude 3.5 Sonnet achieving only 30.39% accuracy on the Hard tier versus a 12.28% random baseline. They also present fine-grained stage-based analysis claiming that the main failure point is the Information Retrieval phase, and report preliminary prompting optimizations. The dataset and code are publicly released.","tokens_in":25923,"tokens_out":6437,"duration_ms":55023,"significance":"If the benchmark construction is valid, FCMR fills a real gap: it targets three-modality cross-modal multi-hop reasoning, whereas MMQA contains only 0.8% such cases and shows signs of visual-modality leakage. The paper ships a public dataset and generation code, a machine-checkable chart-removal ablation (Table 4: GPT-4o drops from 24.37% to 14.71% on Hard without charts), a broad model evaluation, paraphrase-diversity metrics, and several human-error analyses. These are concrete strengths that make the benchmark potentially useful to the community. The main risks are the incomplete verification of gold labels and the small manual sample underlying the central bottleneck claim.","major_comments":[{"comment":"The gold labels used for all headline accuracy numbers are largely produced by the LLM pipeline (GPT-4o-mini generation, GPT-4o paraphrasing, Claude 3.5 Sonnet filtering); only 10% of Easy/Medium instances received human review, and Hard instances were fully reviewed, with 22/714 (3%) revised. Since the revision count itself shows that the automated filter is imperfect, the paper should report a post-revision, human-verified label error rate for the final dataset (at least for a random sample of Easy and Medium), along with inter-annotator agreement, and demonstrate that the Table 5 accuracy values (e.g., Claude 3.5 Sonnet 30.39% on Hard vs. random 12.28%) are stable when a few percent of labels are corrected. Without this, the central numerical claims could shift due to label noise.","section":"Section 3.2 and Appendix C.1"},{"comment":"The conclusion that the 'critical bottleneck' is the Information Retrieval stage is based on manually monitoring 40 Hard-level samples, with no inter-annotator agreement or release of the traced trajectories. Given that this finding is a major analytical contribution and is used to motivate the optimization in Section 6.4, the authors should either (i) provide the annotated trajectories and agreement statistics, (ii) extend the sample, or (iii) explicitly reframe the claim as an illustrative case study rather than a general bottleneck. As written, the strength of the claim exceeds the evidence.","section":"Section 5.2 and Figure 22"}],"minor_comments":[{"comment":"The section title contains a typo: 'Datset' should be 'Dataset'.","section":"Section 3.1 heading"},{"comment":"The cross-reference '(see Appendix 6.2 for details)' appears to refer to Section 6.2, not an appendix; please correct the reference.","section":"Section 6.1"},{"comment":"The main text says '40 given samples' while the Figure 22 caption says '40 Hard-level samples'; make the sample description consistent.","section":"Section 5.2 vs. Figure 22 caption"},{"comment":"The phrase 'revisit the drawbacks MMQA' is missing the preposition 'of' before 'MMQA'; grammatically it should be 'the drawbacks of MMQA'.","section":"Section 2.2"},{"comment":"The comparison of WPD/LD scores with MRPC and PAWS is illustrative, but these are different source corpora and tasks, so the claim of 'superiority' of the paraphrasing method should be phrased as a descriptive comparison rather than a controlled evaluation.","section":"Table 3"},{"comment":"The prompt instructs models to treat any statement as false when there is insufficient information; this design choice may bias toward false predictions and should be discussed in relation to the conservative decision-making behavior reported in Section 5.1.","section":"Section 3.4 and Figure 23"},{"comment":"The 'Random Selection' baselines differ between MMQA (random word from question/text/table, Appendix C.2) and FCMR (uniformly random among the eight answer sets); this difference should be stated in the main text near the tables to avoid misinterpretation.","section":"Table 2 and Table 4"},{"comment":"For GPT-4o and Claude 3.5 Sonnet, the DePlot-augmented LLM versions outperform their direct MLLM versions on Hard (32.91 vs. 24.37 for GPT-4o), which is a striking and non-obvious result; the paper should offer at least a brief explanation or acknowledge this explicitly in the main text.","section":"Section 4.2 and Table 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a standard benchmark paper in scope for cs.CL. The label-verification gap is the main technical risk, and the authors should be encouraged to address it head-on, since the rest of the empirical contribution is solid. The public release of code and data is a genuine strength."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper does something real: it builds a benchmark where every instance needs text, table, and chart, with a Hard tier that genuinely forces three-hop cross-modal reasoning. That is a genuine gap, and the chart ablation (GPT-4o drops from 24.37% to 14.71% without charts) is clean evidence that the dataset is not trivially solvable from text alone. The public dataset and the CMRGen pipeline, which cuts generation cost from $0.33 to $0.004 per question, are useful contributions in their own right.\n\nWhat is new: FCMR is the first financial benchmark combining three modalities in every instance with supervised difficulty levels and a select-all-true statements format. The evaluation is broad—many MLLMs and LLMs with DePlot—and the analysis of reasoning stages, though small, is thoughtful. I particularly like the attempt to quantify contamination risk rather than just assert it.\n\nThe soft spot is exactly where the stress-test note lands: label verification. Only 10% of Easy and Medium instances got any human look; Hard got full review, but that review revised 22 of 714 items. That revision count tells us the LLM filter is imperfect, yet the paper reports neither a post-revision label-error rate nor inter-annotator agreement. So the headline accuracies carry an unquantified error bar. I would not call this fatal—the gap between random (12%) and the best model (30%) is large enough that a few percent label noise will not overturn the \"models struggle\" conclusion. But it does weaken the precision of the difficulty gradient and the specific model rankings.\n\nThe \"critical bottleneck\" claim rests on manual tracing of 40 Hard samples. That is a small n for such a strong statement, and the paper even admits reliance on manual analysis in its Limitations. I would soften the claim or scale up the sample before publication.\n\nOverall, this is a solid paper that deserves a serious referee. The main revisions I would request: report label-error rates and IAA, ideally re-score a subset on corrected labels, and either expand or temper the manual bottleneck analysis. The dataset being public means these checks are actually doable, which is why I would accept this for review rather than desk-reject.","headline":"A solid, useful new benchmark for cross-modal multi-hop reasoning, held back mainly by under-reported label validation.","tokens_in":26444,"tokens_out":1249,"would_cite":true,"duration_ms":13474,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Financial statements that require combining a 10-K report, a balance-sheet table, and a chart in three reasoning hops are answered correctly only 30% of the time by the best current multimodal model, Claude 3.5 Sonnet.","keywords":["cross-modal reasoning","multi-hop QA","multimodal LLM evaluation","financial QA","benchmark construction","data contamination","chart understanding","information retrieval bottleneck"],"falsifier":"Re-annotate all 2,142 Hard-level statements with independent human judges and verify a random sample of Easy and Medium labels; if the corrected labels change model rankings or close the gap between GPT-4o with and without charts, the robustness claim would be undermined.","tokens_in":25496,"feed_emoji":"📊","tokens_out":4193,"duration_ms":34236,"temperature":0.7,"pith_summary":"The paper presents FCMR, a benchmark for testing whether multimodal large language models can combine facts from text, tables, and charts to verify financial statements. It argues that existing benchmarks, notably MMQA, are contaminated—models can answer their hardest questions without the images—and contain almost no questions that genuinely need three modalities and three reasoning hops. FCMR's Hard tier is built so that all three modalities must be used, and on that tier the best tested model, Claude 3.5 Sonnet, reaches 30.4% accuracy against a 12.3% random baseline. The authors also trace model failures through four reasoning stages and find the main bottleneck is retrieving the right information from the right modality.","feed_headline":"Best AI scores 30% on a new three-modal finance test","feed_subtitle":"A benchmark built from SEC filings shows GPT-4o and Claude 3.5 Sonnet fall to near random when they must combine text, tables, and charts.","key_machinery":"The load-bearing object is CMRGen, an automated pipeline that aligns real 10-K text with Compustat tables, plots chart columns with varied chart types and styles, and generates statements by combining five base templates (Fact-Checking, Conditional Threshold, Arithmetic, Trend, Ranking) into one-, two-, and three-hop cross-modal statements. A second mechanism is the four-stage decomposition of reasoning—Planning, Modality Identification, Information Retrieval, Information Reasoning—used to locate where models fail; the paper reports that Information Retrieval is the stage where most errors occur.","core_discovery":"The central claim is that current MLLMs cannot yet perform reliable cross-modal three-hop reasoning in a professional domain, and that a carefully constructed benchmark can measure this ability without being cheated by memorized knowledge. The paper supports this by building FCMR from SEC 10-K reports and Compustat financial tables, generating statements at three difficulty levels, and showing that accuracy collapses toward random when charts are withheld: GPT-4o falls from 24.37% to 14.71% on the Hard tier, which the paper takes as evidence against data contamination. The strongest result is that the Hard tier, which forces text–table–chart integration in every item, caps the best model at 30.4% accuracy.","pith_inferences":["Editorial inference: the contamination check is specific to charts; the same withholding test applied to text or table modality could reveal whether the benchmark is contaminated along other channels.","Editorial inference: because Hard statements are generated from templates, a model could eventually learn template patterns; a dynamic version that regenerates statements from new 10-K filings would keep the benchmark contamination-resistant.","Editorial inference: the observed tendency of models to default to 'false' under uncertainty suggests calibration, not just reasoning ability, may account for part of the accuracy gap.","Editorial inference: if CMRGen transfers to other domains as claimed, the same contamination test could be applied to law or medicine datasets, where memorized knowledge is even harder to disentangle from reasoning."],"forward_implications":["If FCMR is right, prior benchmark scores overstate cross-modal reasoning ability; GPT-4o's 43% exact match on MMQA three-hop questions without images shows contamination.","Models that convert charts to tables with DePlot sometimes beat native MLLMs, suggesting that visual interpretation itself is a limiting factor.","The identified Information Retrieval bottleneck implies that improving retrieval from a specific modality could raise overall accuracy more than improving reasoning alone.","The Hard tier caps state-of-the-art models near 30%, so any future model claiming cross-modal reasoning competence should be measured against this tier.","The zero-to-three-correct multiple-choice design forces models to avoid guessing, making the reported accuracy gaps conservative rather than inflated."],"supporting_citations":[{"why":"MMQA is the prior benchmark the paper critiques and the direct comparison point for the contamination and three-hop scarcity arguments.","marker":"(Talmor et al., 2021)"},{"why":"DePlot converts charts into tables and is the mechanism behind the (M)LLMs + Deplot experiments that reveal visual interpretation limits.","marker":"(Liu et al., 2023)"},{"why":"Supplies the Word Position Deviation and Lexical Deviation metrics used to validate paraphrase quality.","marker":"(Liu and Soh, 2022)"},{"why":"MRPC is one of two reference paraphrase datasets used to benchmark FCMR's lexical and syntactic diversity.","marker":"(Dolan and Brockett, 2005)"},{"why":"PAWS is the second reference paraphrase dataset used in the WPD and LD comparison.","marker":"(Zhang et al., 2019)"},{"why":"Supports the design choice of multiple-choice statements over free-form answers for reliable model evaluation.","marker":"(Pang et al., 2024)"},{"why":"Provides evidence that MLLMs rely more on textual clues than visual ones, motivating the DePlot comparison.","marker":"(Rahmanzadehgervi et al., 2024)"},{"why":"Justifies the chart-type coverage in FCMR by documenting the frequency of bar, pie, and line charts in real 10-K filings.","marker":"(Christensen et al., 2024)"}],"fun_headline_variants":["FCMR benchmark: Even top AI scores just 30% on hardest tier","AI struggles to merge text, tables, charts in new finance test","Best model hits 30% on FCMR's hardest three-hop reasoning","New benchmark shows AI fails at financial multi-hop reasoning","Claude 3.5 tops FCMR at 30%, still near random on hard tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ground-truth truth values of the generated statements must be correct; since only 10% of samples received initial human review and Hard-level labels were fully checked with 22 of 714 items revised, a few percent of label errors could shift the reported accuracy gaps.","fun_headline_variants_meta":{"raw":{"variants":["FCMR benchmark: Even top AI scores just 30% on hardest tier","AI struggles to merge text, tables, charts in new finance test","Best model hits 30% on FCMR's hardest three-hop reasoning","New benchmark shows AI fails at financial multi-hop reasoning","Claude 3.5 tops FCMR at 30%, still near random on hard tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000375,"raw_usage":{"total_tokens":1995,"prompt_tokens":938,"completion_tokens":1057,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":957}},"tokens_in":554,"tokens_out":1057,"duration_ms":9387,"temperature":1.0,"reasoning_tokens":957,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:56:24.728113+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate all 2,142 Hard-level statements with independent human judges and verify a random sample of Easy and Medium labels; if the corrected labels change model rankings or close the gap between GPT-4o with and without charts, the robustness claim would be undermined.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MMQA is the prior benchmark the paper critiques and the direct comparison point for the contamination and three-hop scarcity arguments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Word Position Deviation and Lexical Deviation metrics used to validate paraphrase quality."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Justifies the chart-type coverage in FCMR by documenting the frequency of bar, pie, and line charts in real 10-K filings."}],"review_version":1}