{"id":"6c57a275-43f8-47d2-9216-f92d8aa61caf","arxiv_id":"2505.23242","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ChartMind is a new bilingual chart QA benchmark, and ChartLLM's structured context extraction yields higher scores than three existing prompting paradigms in the paper's evaluations.","lead":"ChartMind introduces a multilingual, seven-task benchmark for chart question answering with 757 human-validated instances. The paper also proposes ChartLLM, a context-extraction framework, and reports that it improves accuracy over instruction-following, OCR-enhanced, and chain-of-thought baselines across 14 vision-language models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Primary metric's judge never sees the chart: the GPT-4o scoring prompt (Figure 8) inputs only question, human reference, and model answer, so ChartMind scores may reflect reference-text similarity rather than chart-grounded correctness.","rationale":"The reader's weakest assumption identified the GPT-4o scoring metric as the key vulnerability, citing the small undisclosed human-validation sample. My independent reading confirms the metric is the load-bearing link, but locates a sharper and more specific problem: the scoring prompt never gives the judge the chart. This makes the ChartMind primary metric an ungrounded reference-similarity score, which directly threatens the paper's main comparative claim. I agree with the reader's CONDITIONAL verdict, since the structured-output benchmarks (ChartQA, Chart-to-Text, OpenCQA) use ACC/BLEU and show consistent ChartLLM improvements, so the empirical claim is not entirely unsupported. However, the paper's distinctive contributions—multilingual, open-ended, real-world chart QA on ChartMind—are evaluated only by the potentially biased judge. The concern is concrete and testable: adding the chart image to the judge's input is a minimal change that would either confirm or refute the published ordering. The sample-size issue raised by the reader is real but secondary; even a large human sample would not fix the fact that the automated judge cannot access the chart. I therefore set verdict_should_be to CONDITIONAL rather than REJECT, because the proposed check can settle the issue and the underlying framework may still be valid.","tokens_in":22771,"tokens_out":3774,"duration_ms":42357,"concrete_test":"Randomly sample 100 ChartMind items and re-run GPT-4o scoring under two conditions: (1) the published prompt without the chart, and (2) the same prompt with the chart image attached. Compare per-item score distributions and per-paradigm averages. If ChartLLM's advantage over OCR-enhanced and CoT shrinks, reverses, or shifts by a non-negligible margin (e.g., >5 points in mean score), the published ChartMind results are artifacts of the judge's lack of chart access. A complementary check is to have human annotators score the same 100 items with chart access and report item-level agreement with the no-chart GPT-4o scores.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's headline claim—that ChartLLM-based methods consistently outperform instruction-following, OCR-enhanced, and CoT paradigms on ChartMind—rests on the GPT-4o automated score. The scoring prompt in Appendix C (Figure 8) takes a JSON object containing only question, human_reference, and model_answer; no chart image or chart-derived data is provided to the judge. Yet the scoring rubric for \"Output Correctness\" tells the judge to assess consistency with \"the reference answer or the chart content,\" and the chart content is absent from the input. Consequently, correctness can only be judged by textual resemblance to the human reference and the question, not by actual chart-grounded factuality. A fluent but chart-contradicting answer that echoes the reference could score highly, while a correct answer using different wording could score poorly. The validation in Table 4 does not resolve this: PCC=93.09 is computed on average scores across 14 models, not on item-level judgments, and the human evaluation sample size is not disclosed. Such aggregate correlation cannot detect a systematic bias in the judge (e.g., preference for reference-like phrasing) or establish that the score measures chart-grounded correctness. Since Section 5.4's cross-lingual superiority claim is based on these scores, the central comparison on the new benchmark is potentially confounded by the missing chart in the evaluation input.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ChartMind, a 757-item bilingual (English/Chinese) chart question-answering benchmark spanning seven task categories and seven chart types, with questions and reference answers generated by GPT-4o and then human-validated. It also proposes ChartLLM, a model-agnostic prompting framework that extracts structured chart context (title, legend, axis labels) and feeds it to a multimodal LLM before answering. The authors compare four paradigms—instruction-following, OCR-enhanced, chain-of-thought, and ChartLLM—across 14 MLLMs on ChartMind, ChartQA, Chart-to-Text, and OpenCQA, and claim that ChartLLM-based methods consistently outperform the other three paradigms, particularly on ChartMind's open-ended tasks.","tokens_in":23066,"tokens_out":5378,"duration_ms":59894,"significance":"If the empirical claims held, ChartMind could be a valuable resource for evaluating open-ended, multilingual chart reasoning, and ChartLLM is a simple, model-agnostic context-extraction recipe that could transfer to other MLLMs. The benchmark construction pipeline is described in sufficient detail to be reproduced, and the inclusion of both English and Chinese material across multiple chart and task types fills a gap in existing chart QA benchmarks. The paper also explicitly acknowledges limitations around automated evaluation and data bias. However, the validity of the headline results currently hinges on an automated judge that never receives the chart and on correlation evidence that is too weak to rule out judge bias, so the central quantitative conclusions are not yet established.","major_comments":[{"comment":"The GPT-4o scoring prompt takes as input only a JSON object containing \"question\", \"human_reference\", and \"model_answer\"; no chart image or chart-derived data is provided to the judge. The rubric nevertheless asks the judge to assess consistency with \"the reference answer or the chart content\", and the scoring criteria state that data should be \"consistent with the reference answer or the chart content\". Since the chart content is absent from the judge's inputs, the correctness score can only measure textual similarity to the reference answer. A fluent answer that contradicts the chart but paraphrases the reference would score high, while a correct answer with different wording could score low. Because the ChartMind open-ended results in Table 3 and the cross-lingual claims in Section 5.4 rest on this score, the central comparison is confounded.","section":"Appendix C, Figure 8"},{"comment":"The reported validation of the automated score is insufficient. The Pearson correlation coefficient of 93.09 is computed on average GPT-4o and human scores across 14 models, not on item-level judgments; the number of human-annotated items is not disclosed, and no inter-annotator agreement is reported. An aggregate correlation across model means can be high even if the judge systematically prefers reference-like phrasing, because that bias would affect all models to different degrees. To validate the metric, the authors need item-level human scores with agreement statistics, a description of how human annotators accessed the chart, and either per-item correlation or a bias analysis that includes adversarial answers that are fluent but chart-contradicting or correct but differently worded.","section":"Table 4, Section 5.3"},{"comment":"The abstract and Section 5.2 claim that ChartLLM-based methods \"significantly outperform\" the other paradigms, but no error bars, repeated runs, significance tests, or effect sizes are reported. Several deltas in Table 3 are small—for example, TinyChart gains of +1.35 on ChartMind ACC and +1.62 on GPT-4o score, and ChartLlama loses 0.27 on Chart-to-Text Avg. BLEU under OCR enhancement—while some OCR-enhanced results degrade substantially. The claim of consistent superiority therefore requires at least bootstrap or permutation tests over chart/task samples, or confidence intervals for the reported differences.","section":"Table 3, Section 5.2"},{"comment":"There is a residual circularity risk: the gold answers are generated by GPT-4o, and GPT-4o is also used as the automated judge. The high GPT-4o scores on ChartMind (e.g., 73.89 under ChartLLM in Table 3) may partly reflect stylistic similarity to GPT-4o-generated references rather than reasoning quality. The human validation and answer rewriting in Stage III reduce but do not eliminate this risk, especially because the judge does not see the chart. A concrete test would be to score the same model answers with a judge that receives only the chart and question (not the human reference), or to include reference-paraphrase and chart-contradicting answers in the human evaluation sample to measure whether the judge is biased toward reference-like wording.","section":"Sections 3.2, 3.3, and Appendix C"}],"minor_comments":[{"comment":"The text contains a LaTeX artifact \"textcolorblue\" that should be removed or rendered properly.","section":"Section 3.3"},{"comment":"The scoring prompt says scores are 0 or 1, while Table 3 reports GPT-4o scores on a 0-100 scale; the scaling should be stated explicitly.","section":"Appendix C, Figure 8"},{"comment":"The paper does not provide a download link, repository URL, or dataset license for ChartMind; a data availability statement should be added.","section":"Appendix B / Data Availability"},{"comment":"There are formatting errors in Table 3, such as \"23.04 (+0.72 19.73 (-8.75)\", where a closing parenthesis is missing; the table should be proofread.","section":"Table 3"},{"comment":"The claim that ChartMind is the \"first dual-language evaluation setting\" for chart QA is stronger than necessary and should be qualified with a comparison to existing multilingual chart benchmarks.","section":"Section 1"},{"comment":"The prompt text says the judge should evaluate based on \"charts\", but the input JSON has no chart field; this internal inconsistency should be fixed or explained.","section":"Appendix C, Figure 8"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the dataset itself is a genuine contribution: 757 human-validated English/Chinese QA pairs across seven task types and seven chart formats, including open-ended summarization and suggestion questions. That combination doesn't exist in the cited prior benchmarks. Second, the paper's central comparison on ChartMind is weaker than it looks, because the GPT-4o judge receives only the question, the human reference, and the model answer—no chart, no extracted chart data. The scoring rubric asks about \"chart content,\" but the judge cannot see it. So a fluent answer that matches the reference's wording scores well even if it contradicts the chart, and a correct but differently worded answer can score poorly.\n\nWhat's actually new: the benchmark and the cross-lingual results. ChartLLM itself is just context extraction—title, legend, axes—fed into the model. It's a small, sensible intervention, and Table 3 shows it helps on the public benchmarks (ChartQA, Chart-to-Text, OpenCQA) where scores are standard accuracy/BLEU and not GPT-judged. That part of the evidence holds up.\n\nWhere it's soft, in proportion: (1) The GPT-4o judge problem above is real and load-bearing for the ChartMind-specific rank. Table 4's PCC of 93.09 doesn't fix it: it's computed over 14 model averages, not item-level judgments, and the human sample size isn't disclosed. Aggregate correlation can hide systematic reference-phrasing bias. (2) The dataset is small—757 instances—and no data or code is released, so nothing is checkable yet. (3) No error bars or significance tests on any comparison. Some deltas in Table 3 are a few points; several OCR/CoT deltas are negative, which suggests noise. (4) The \"first benchmark\" claim needs qualification: OpenCQA already does open-ended chart QA, and the authors' own mChartQA is cited but not compared in Table 1. (5) The appendix lists limitations candidly, including possible model bias in generation and evaluation, which is good to see.\n\nVerdict: worth a serious referee. This paper is for people building chart QA benchmarks and methods; the benchmark idea fills a real gap and the public-benchmark results give ChartLLM some genuine support. But the ChartMind headline comparison needs item-level human agreement, a chart-visible judge, released artifacts, and significance testing before it is convincing.","headline":"ChartMind is a useful bilingual open-ended chart QA benchmark, but its headline comparison on that benchmark rests on a judge that never sees the chart.","tokens_in":23571,"tokens_out":2090,"would_cite":false,"duration_ms":22534,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ChartMind and ChartLLM claim that feeding only a chart's title, legend, and axes to vision-language models outperforms OCR, chain-of-thought, and plain instruction following.","keywords":["chart question answering","multimodal large language models","benchmark construction","open-ended evaluation","multilingual charts","context extraction","chain-of-thought comparison","OCR enhancement"],"falsifier":"Score the same 757 ChartMind answers with human raters on a disclosed sample (for example, 200 answers per model) and compare the resulting model ranking to the GPT-4o ranking; if the ChartLLM advantage reverses or the GPT-4o-human correlation falls well below 93, the central claim fails. A second check is to swap the context extractor (the paper uses Qwen2-VL) for another model and see whether ChartLLM's edge survives.","tokens_in":22596,"feed_emoji":"📊","tokens_out":7995,"duration_ms":73169,"temperature":0.7,"pith_summary":"This paper claims that the standard ways of prompting vision-language models for chart questions—plain instructions, OCR text dumps, and chain-of-thought decomposition—are not the most effective route, and that extracting a small structured context first is better. To make the case it builds ChartMind, a 757-question benchmark with seven task types, seven chart formats, and English and Chinese charts, with open-ended answers instead of single labels. It then proposes ChartLLM, a model-agnostic wrapper that pulls out a chart's title, legend, and axis labels and gives them to the model before answering. Across 14 models and four benchmarks, the wrapper consistently outscores the three competing paradigms, especially on Chinese charts and on open-ended tasks. If the claim holds, simple input restructuring is a cheap and portable way to improve real-world chart reasoning.","feed_headline":"Chart context beats OCR and chain-of-thought on chart QA","feed_subtitle":"A new benchmark and wrapper that feed titles, legends, and axis labels to 14 models, winning on four chart QA datasets.","key_machinery":"The mechanism is ChartLLM's context extraction: given a chart $C$, it defines $C_{\\text{context}} = \\{T, L, X, Y\\}$, the chart's title, legend, X-axis label, and Y-axis label, and asks the model to answer using these cues rather than the raw image alone. The paper contrasts this with chain-of-thought reasoning, which decomposes the answer into steps $r_1,\\dots,r_k$; ChartLLM instead reduces the perceptual burden before reasoning begins. The objective is to choose the answer $a$ that maximizes the expected log-likelihood $\\sum_i E_{C_{\\text{context}},Q}[\\log P(a_i | C_{\\text{context}}, Q; \\Theta)]$. The benchmark itself is built through chart collection, GPT-4o-based QA generation, and human validation with answer rewriting, which supplies the evaluation setting where the method's advantage is measured.","core_discovery":"The central discovery is that a context-aware input representation, not a more elaborate reasoning procedure, is what most improves current multimodal chart question answering. ChartMind provides a real-world evaluation setting where ChartLLM-based methods achieve the best average GPT-4o scores across seven task categories in both English and Chinese, and the same pattern holds on ChartQA, Chart-to-Text, and OpenCQA. The paper interprets this as evidence that pre-structuring the semantically relevant parts of a chart reduces perceptual noise and lets the underlying model reason from the components that matter, a strategy that generalizes across model families and chart formats. The paper also reports that GPT-4o scoring correlates with human scoring at 93.09, which it uses to justify measuring open-ended answers automatically.","pith_inferences":["The paper leaves implicit that ChartLLM's context extraction depends on a strong reader (it uses Qwen2-VL); a natural extension is to test whether weaker extractors erode the advantage, which would make the method's value contingent on extraction quality.","A cross-lingual implication: since Chinese tasks are harder for every paradigm, bilingual benchmarks could use the English-Chinese gap as a diagnostic for how much of a model's chart reasoning is language-grounded versus vision-grounded.","A testable extension is to feed ChartLLM's extracted context into the OCR and CoT paradigms as well; if those also improve, the three 'competing' paradigms may converge on the same best practice rather than being fundamentally different.","If the advantage replicates on unseen chart types and languages, chart QA practice could shift from prompt engineering toward explicit semantic parsing of chart structure as a standard preprocessing step."],"forward_implications":["ChartLLM-based methods outperform instruction-following, OCR-enhanced, and chain-of-thought paradigms on ChartMind and on ChartQA, Chart-to-Text, and OpenCQA, across most of the 14 models tested.","Adding raw OCR text is not always helpful: it improves structured-output tasks but lowers accuracy on open-ended reasoning, so input noise matters as much as reasoning style.","Chinese chart questions are consistently harder than English ones, establishing a measurable cross-lingual gap that monolingual benchmarks cannot expose.","GPT-4o's automatic open-ended scoring tracks human scores closely enough (PCC = 93.09) that future CQA work can use it as a cheaper stand-in for human evaluation.","Because ChartLLM is model-agnostic, the same extracted context can be plugged into any multimodal model, making the gain a property of the input format rather than of one architecture."],"supporting_citations":[{"why":"Supplies ChartQA, one of the three structured-output benchmarks used to compare the four CQA paradigms.","marker":"Masry et al., 2022"},{"why":"Supplies OpenCQA, the open-ended benchmark whose structured-output constraint ChartMind is designed to loosen.","marker":"Kantharaj et al., 2022a"},{"why":"Supplies Chart-to-Text, the chart captioning dataset used as a public baseline.","marker":"Kantharaj et al., 2022b"},{"why":"Describes the OCR-to-text approach that defines the OCR-enhanced baseline paradigm and is the main point of contrast.","marker":"Liu et al., 2023"},{"why":"Defines chain-of-thought prompting, the step-by-step reasoning paradigm that ChartLLM is compared against.","marker":"Wei et al., 2022"},{"why":"The GPT-4o model used to generate ChartMind QA pairs, produce the human-rewritten references, and score open-ended answers in Table 3.","marker":"Achiam et al., 2023"},{"why":"Provides the Pearson correlation coefficient used to validate GPT-4o scoring against human scores (PCC = 93.09).","marker":"Cohen et al., 2009"},{"why":"Supplies Qwen2-VL, the model used to extract the title, legend, and axis context in the ChartLLM implementation.","marker":"Bai et al., 2023"}],"fun_headline_variants":["ChartMind: Context beats OCR and chain-of-thought in chart QA","Context-aware input wins chart QA over OCR and CoT","For chart QA, context beats OCR and CoT, says ChartMind","ChartLLM shows context-aware cues trump OCR and CoT in QA","New chart QA benchmark: context beats OCR and chain-of-thought"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the GPT-4o-based automatic scorer measures open-ended answer quality the same way a human would; the paper supports this with a single Pearson correlation of 93.09 and does not report the human-evaluation sample size.","fun_headline_variants_meta":{"raw":{"variants":["ChartMind: Context beats OCR and chain-of-thought in chart QA","Context-aware input wins chart QA over OCR and CoT","For chart QA, context beats OCR and CoT, says ChartMind","ChartLLM shows context-aware cues trump OCR and CoT in QA","New chart QA benchmark: context beats OCR and chain-of-thought"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1514,"prompt_tokens":912,"completion_tokens":602,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":510}},"tokens_in":528,"tokens_out":602,"duration_ms":6141,"temperature":1.0,"reasoning_tokens":510,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:49:03.048385+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Score the same 757 ChartMind answers with human raters on a disclosed sample (for example, 200 answers per model) and compare the resulting model ranking to the GPT-4o ranking; if the ChartLLM advantage reverses or the GPT-4o-human correlation falls well below 93, the central claim fails. A second check is to swap the context extractor (the paper uses Qwen2-VL) for another model and see whether ChartLLM's edge survives.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Pearson correlation coefficient used to validate GPT-4o scoring against human scores (PCC = 93.09)."}],"review_version":1}