{"id":"881a429d-102f-4976-9321-a90ea485e436","arxiv_id":"2506.13055","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A 9,356-pair Chinese multimodal financial benchmark reveals that state-of-the-art multimodal LLMs, including GPT-4V, still score below 53% on objective and 39% on subjective financial chart tasks.","lead":"CFBenchmark-MM is a new Chinese multimodal financial benchmark with 9,356 image-question pairs built from real financial reports. It shows that even the best tested model, GPT-4V, reaches only 52% objective accuracy and 38% subjective scores, suggesting current multimodal LLMs are far from reliable for financial chart analysis.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 52%/38% figures rest on an unaudited annotation step: no inter-annotator agreement or release, so the benchmark may not actually measure chart-only reasoning.","rationale":"The reader's weakest assumption correctly identifies the manual examination and GPT-4-generated reference answers as the keystone of the benchmark's validity. I agree because the headline claim is quantitative: 52% objective and 38% subjective accuracy. Those numbers only mean what the paper says if the questions are genuinely solvable from the chart and the reference answers are correct. The paper provides no evidence beyond an undocumented manual pass, and the dataset is withheld, so the claim cannot currently be checked. This is not a mere reproducibility nit; it determines whether the central empirical conclusion is interpretable. I also note the paper's positive aspects: the staged evaluation design is thoughtful, and the limitation section is candid about the delayed release and missing proprietary models. However, those strengths do not supply the missing audit. The concrete test I propose would settle the concern directly: a small independent annotation study with agreement statistics, followed by re-scoring on the validated subset. If the results survive that check, the paper's claims would be substantially strengthened; if not, the benchmark construction needs rework. The reader's CONDITIONAL verdict remains appropriate, so I recommend no change.","tokens_in":38456,"tokens_out":7337,"duration_ms":86291,"concrete_test":"Independently audit a random stratified sample of 200 image-question pairs spanning all five question categories and chart types. Have two finance-trained annotators apply the Section 3.1.3 checklist: (a) answerable from chart alone, (b) reference answer correct and unambiguous, (c) options valid and non-overlapping. Compute Cohen's kappa. Then re-run the GPT-4V and LLaVA-NeXT-34B evaluations on the subset both annotators deem fully valid, using the same scoring protocol. If the mean accuracy/score shifts by more than 5 points absolute, or if any sampled question is judged invalid, the reported figures are not robust and the benchmark should be revised before release.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central finding — GPT-4V reaches only 52% objective and 38% subjective accuracy on CFBenchmark-MM — presupposes that every question is answerable from the chart alone and that the GPT-4-generated reference answers are correct. The only quality gate is the manual examination in Section 3.1.3, but the paper reports no inter-annotator agreement, no count of rejected items, and no independent audit. In particular, step 2 of the manual examination checks only that the designated correct option is included among the choices, not that the chart uniquely determines it. Appendix A further states that the benchmark will be released only after publication, so no external check is currently possible. If even a modest fraction of questions are not answerable from the chart, or if some reference answers are wrong, the 52%/38% numbers are not a clean measure of multimodal financial reasoning: scores could be deflated by unanswerable items or inflated by textual leakage from the source reports. This is the load-bearing assumption behind the paper's headline conclusion that general-purpose MLLMs are not yet reliable for financial chart analysis.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CFBenchmark-MM, a Chinese-language multimodal financial benchmark with 9,356 image-question pairs built from 2,339 financial-report charts. Questions are generated by GPT-4 from the text analyses accompanying the charts, then manually screened. The authors evaluate 13 MLLMs under four input settings (Q, Q+C, Q+I, Q+I+C), using multiple-choice accuracy for objective tasks and GPT-4-based key-point matching for subjective tasks. The headline finding is that GPT-4V reaches 52% objective and 38% subjective accuracy in the Q+I setting, indicating that state-of-the-art MLLMs remain limited for financial chart analysis. The paper also reports instruction and language ablations, failure cases, and self-correction examples.","tokens_in":38659,"tokens_out":6275,"duration_ms":64437,"significance":"If the benchmark is valid, this is a useful resource: it uses real financial reports, covers five question types and five chart types, and proposes a staged evaluation protocol that separates textual and visual contributions. The failure-case analysis (Section 4.4 and Appendix D.2) is a strength and gives concrete evidence of the claimed error types. However, the central quantitative claims rest on unverified assumptions about question answerability and scorer reliability, and the benchmark is not released, so the reported 52%/38% figures cannot currently be independently checked. The significance is therefore conditional on strengthening the validity evidence.","major_comments":[{"comment":"The manual examination does not certify that each question is answerable from the chart alone, and no inter-annotator agreement or rejection statistics are reported. Step 2 only verifies that the correct option is included among the choices; it does not check that the chart uniquely determines that option. Since Appendix A states that the benchmark will be released only after publication, the 52%/38% headline figures in Section 4.2 cannot currently be externally verified. If a nontrivial fraction of items are unanswerable from the chart, or if some reference answers are wrong, the Q+I scores are not a clean measure of chart-based reasoning. I ask the authors to report the number of items rejected at each manual-examination step, provide inter-annotator agreement for the examination, and release a substantial sample of the benchmark for review.","section":"Section 3.1.3 / Appendix B.3"},{"comment":"The subjective scores are computed by GPT-4 as a scorer, but no validation of this scoring is provided. Because the reference answers were themselves generated by GPT-4 (Section 3.1.2), the reported 38% subjective score for GPT-4V may reflect an alignment between the generator and the scorer rather than the quality of the model's answer. The paper should report agreement between GPT-4 scoring and human scoring on a random sample, using at least two human annotators, and should consider an additional independent LLM scorer. Without this, the subjective numbers are not interpretable at face value.","section":"Section 3.2 / Table 8"},{"comment":"The captions used in the Q+C and Q+I+C conditions are generated by GPT-4V, the same model family as the strongest evaluated model, and the paper concludes that captions provide additional useful information to open-source models but not to GPT-4V, attributing this to the captions containing only part of the chart information. This is an uncontrolled comparison: the caption source is confounded with the model being evaluated, and no independent caption generator (e.g., human-written captions or captions from a different MLLM) is used. The claim about caption usefulness should be rephrased or supported by a condition with an independent caption source.","section":"Section 3.2 / Section 4.2"}],"minor_comments":[{"comment":"The headline '52% accuracy and 38% score' should be defined explicitly; Table 8 does not show these aggregate values, and the GPT-4V row average is 46.7. Please report the objective and subjective sub-averages in the table or in the text.","section":"Section 4.2 / Table 8"},{"comment":"The numbers 2,500 collected pairs, 2,421 suitable pairs, and 2,339 valid charts should be reconciled; currently it is unclear how many pairs were rejected at each stage.","section":"Section 3.1.2"},{"comment":"There is a typo: '13 cutting-edged MLLMs' should read '13 cutting-edge MLLMs'.","section":"Section 4.1"},{"comment":"The model name 'mPLUG-0w1' appears to be a typo for 'mPLUG-Owl'.","section":"Table 8 / Table 9"},{"comment":"The screening step says 'Only when both individuals agree', but the paper does not report the level of agreement or the resolution of disagreements.","section":"Section 3.1.1"},{"comment":"The definition of the subjective score ('if three out of five key points are matched, the score is 0.6') implies a fixed key-point count that is not explained; please clarify how the number of key points is determined for each reference answer.","section":"Section 3.2"},{"comment":"The English-translation experiment translates instructions, questions, and answers using GPT-4; this means the translation itself is unvalidated, and the observed 1-3% differences may partly reflect translation quality. Please acknowledge this limitation.","section":"Section 4.3.2 / Table 4"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is the central artifact of the paper and is not released, which makes external verification impossible. Combined with the heavy reliance on GPT-4 for question generation, captioning, and scoring, the paper needs a substantial validity and reproducibility pass before publication. I would want to see at least a public sample, an independent audit of a random subset, and human-machine scorer agreement before recommending acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Straight answer: this is a genuine benchmark contribution that currently cannot be independently checked. The staged evaluation across four input settings is the most useful idea, and the scale (9,356 pairs, five task types, real report charts) exceeds prior multimodal financial benchmarks. The finding that GPT-4V only reaches ~52% objective and ~38% subjective accuracy is plausible and the failure cases are informative. I also want to credit the LLM-vs-MLLM comparison in Appendix C: it shows multimodal training can hurt pure-text financial performance, which is a nice observation.\n\nThe soft spots are concentrated in verification. The benchmark is not released (Appendix A), and the manual examination described in Section 3.1.3 reports no inter-annotator agreement, no rejection counts, and no independent audit. So the 52%/38% numbers are only as good as that unvalidated filter. That is load-bearing but not fatal—if the authors release the data and show some reliability evidence, the headline conclusion holds. Another real issue: reference [5] is listed as FinVQA but is actually FinTral, so the related-work comparison is partly wrong. There is also circularity in using GPT-4 to generate questions/answers, GPT-4V to generate captions, and GPT-4 to score subjective answers, all from the same model family. That is a legitimate concern, though somewhat mitigated by external charts and the manual pass.\n\nFor a benchmark paper, this deserves a serious referee. The right outcome is likely major revision with the dataset released and the annotation validated. I would not desk-reject it.","headline":"Useful staged benchmark for Chinese multimodal finance, but the headline numbers rest on an unvalidated annotation step and an unreleased dataset.","tokens_in":39158,"tokens_out":3160,"would_cite":true,"duration_ms":35445,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new Chinese multimodal financial benchmark shows GPT-4V scoring only 52% on objective chart questions, indicating current MLLMs cannot yet be trusted for financial chart analysis.","keywords":["multimodal large language model evaluation","financial benchmark","Chinese finance","chart understanding","visual question answering","GPT-4V","staged evaluation","financial reasoning"],"falsifier":"Take a random sample of 200 benchmark questions, have two independent financial analysts answer them from the chart alone without the original textual analysis, and compare their answers to the paper's reference answers. If analysts cannot reliably reproduce the reference answers, or if inter-annotator agreement is low, the claim that questions are answerable from the chart alone — and therefore the benchmark's validity — fails.","tokens_in":38289,"feed_emoji":"📊","tokens_out":4586,"duration_ms":44936,"temperature":0.7,"pith_summary":"The paper introduces CFBenchmark-MM, a Chinese multimodal financial benchmark of 9,356 image-question pairs built from real financial research reports. It claims that current multimodal large language models, including GPT-4V, perform only moderately on financial chart analysis: 52% accuracy on objective questions and 38% on subjective questions when charts are provided. The authors argue this shows general-purpose MLLMs are not yet reliable for financial decision-making, and that the primary failure modes are misinterpretation of visual content and misunderstanding of financial concepts. The benchmark's value would be to guide domain-specific optimization of MLLMs for finance.","feed_headline":"GPT-4V tops out at 52% on Chinese financial chart quiz","feed_subtitle":"New 9,356-question benchmark grades 13 multimodal models on real reports; chart misreads drive most errors.","key_machinery":"The load-bearing mechanism is the three-step annotation pipeline followed by a staged evaluation protocol. In the pipeline, chart-analysis pairs are collected from real financial research reports; GPT-4 generates two questions per task category from the accompanying textual analysis; and manual examination filters out questions that cannot be answered from the chart alone. The staged evaluation (question-only, question+caption, question+image, question+image+caption) then isolates whether performance gains come from memory, textual grounding, or visual grounding, making the benchmark a diagnostic for multimodal financial capability.","core_discovery":"The central claim is that recognizing and reasoning over financial charts remains an unsolved problem for state-of-the-art MLLMs. The paper constructs CFBenchmark-MM from 2,339 verified charts and 9,356 question-answer pairs across five task types (arithmetic reasoning, statistical reasoning, structural reasoning, financial explanation, and financial knowledge), and evaluates 13 models in a staged protocol that separates inherent knowledge (question only), visual grounding (question + image), textual description (question + caption), and combined inputs. In the chart-inclusive setting, GPT-4V reaches 52% on objective multiple-choice questions and 38% on subjective short-answer questions; all other models score lower. The paper further identifies, through error analysis, that wrong answers mostly come from misreading charts and confusing financial concepts rather than from lack of domain knowledge.","pith_inferences":["A testable extension the paper does not pursue: replace the charts with equivalent tabular data to separate chart-reading skill from numerical reasoning, since the current design does not isolate those two competences.","The gap between the Q+C and Q+I settings could be formalized as a measure of visual grounding quality, a diagnostic the paper reports but does not quantify.","If the benchmark is released without reference answers, its long-term value as a held-out testbed will depend on an independent expert audit of the ground truth, which the paper does not provide."],"forward_implications":["If the benchmark accurately measures multimodal financial skill, then no current MLLM is reliable enough for automated financial chart analysis: the best model still fails nearly half of objective questions.","The staged evaluation implies that adding GPT-4V-generated captions helps open-source models but not GPT-4V itself, suggesting caption information is a subset of what the image already provides.","The error analysis implies that progress will require better visual alignment for charts and stronger financial-concept understanding, not merely larger language models.","The comparison of LLMs and their multimodal counterparts suggests that multimodal training can degrade pure-text financial knowledge, raising a trade-off for future financial MLLM development."],"supporting_citations":[{"why":"GPT-4 is used both to generate the benchmark's questions and reference answers and to score subjective model responses, so the entire ground-truth and evaluation chain rests on this model.","marker":"[2]"},{"why":"FinQA is a prior financial numerical-reasoning dataset that motivates the arithmetic and statistical task categories in CFBenchmark-MM.","marker":"[10]"},{"why":"MME-Finance is the closest existing bilingual multimodal finance benchmark, and the paper positions CFBenchmark-MM's comprehensiveness and scale against it.","marker":"[17]"},{"why":"FinTral is cited as a multimodal financial LLM that motivates probing MLLMs on non-text financial data.","marker":"[5]"},{"why":"LLaVA is the model family several of the evaluated open-source MLLMs are based on, so its reference defines the open-source performance baseline.","marker":"[28]"},{"why":"FinBen is a holistic financial benchmark for LLMs that CFBenchmark-MM extends by adding chart images and a staged multimodal evaluation.","marker":"[43]"}],"fun_headline_variants":["GPT-4V hits 52% on Chinese financial chart benchmark","New 9,356-question test: best AI gets 52% on finance charts","Why AI fails at Chinese financial charts: misreads dominate","Multimodal AI struggles with Chinese financial charts: 52% ceiling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark is valid only if the manual examination truly removed every question whose answer cannot be derived from the chart alone, and if the GPT-4-generated reference answers are correct; neither is independently verified in the paper.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4V hits 52% on Chinese financial chart benchmark","New 9,356-question test: best AI gets 52% on finance charts","Why AI fails at Chinese financial charts: misreads dominate","Multimodal AI struggles with Chinese financial charts: 52% ceiling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000411,"raw_usage":{"total_tokens":2116,"prompt_tokens":923,"completion_tokens":1193,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":1114}},"tokens_in":539,"tokens_out":1193,"duration_ms":12312,"temperature":1.0,"reasoning_tokens":1114,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:36:22.305137+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of 200 benchmark questions, have two independent financial analysts answer them from the chart alone without the original textual analysis, and compare their answers to the paper's reference answers. If analysts cannot reliably reproduce the reference answers, or if inter-annotator agreement is low, the claim that questions are answerable from the chart alone — and therefore the benchmark's validity — fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LLaVA is the model family several of the evaluated open-source MLLMs are based on, so its reference defines the open-source performance baseline."}],"review_version":1}