{"id":"a2821d54-15aa-4fe4-9e32-6703e9f4e7a5","arxiv_id":"2412.12150","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new benchmark built from real scientific paper charts, including flowcharts and context-dependent questions, shows large multimodal models perform far below human level on chart understanding.","lead":"This paper introduces SCI-CQA, a benchmark of 37,607 charts drawn from computer science papers, with 5,629 expert-reviewed questions for testing multimodal models. It reports that even the best proprietary models score around 72 on flowcharts while human teams score about 90, suggesting that existing chart benchmarks overstate model abilities.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GPT-4o-generated ground truth and GPT-4o-based open-ended scoring are unvalidated; if the scorer is biased, the headline model-human gap and the \"overestimate\" claim are not yet supported.","rationale":"The paper is a good-faith effort to build a harder, more realistic chart-understanding benchmark, and the objective MC/TF results are less vulnerable to scoring bias. But the headline conclusions—models are far from human performance and existing benchmarks overestimate ability—depend on the composite scores, and the weakest link is the unvalidated GPT-4o-as-scorer and GPT-4o-as-generator pipeline. The reader independently flagged this same assumption, and the evidence in the paper (47% pre-review correctness, no human-scoring validation, no dataset release) supports treating it as load-bearing. The concern does not disprove the paper, but it means the central numbers should not be accepted until the scoring and curation are independently validated. Keeping the reader's CONDITIONAL verdict is appropriate; no change is needed.","tokens_in":20509,"tokens_out":11638,"duration_ms":111602,"concrete_test":"Sample 100 open-ended answers from GPT-4o, InternVL2, LLaVA-1.6, and human evaluators on SCI-CQA flowcharts/data charts; have two independent chart-annotation experts score them with the published rubric, blind to model identity. Compute inter-rater reliability (Cohen's kappa / ICC) and the mean signed difference between GPT-4o and human scores, splitting by whether the answer was written by GPT-4o. If GPT-4o scores its own outputs more than 0.5 points higher than expert judges, the composite scores in Table 2 are biased. Separately, re-check a random 200-question subset of the curated QA bank; if expert agreement on correct answers is below about 90%, the ground truth is not stable enough to support the 'overestimate' claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that existing benchmarks overestimate chart understanding and SCI-CQA measures it accurately—rests on the validity of SCI-CQA's question/answer ground truth and its scoring procedure. The paper reports that only about 47% of GPT-4o-generated QA pairs were correct before manual review (Related Work), meaning the evaluation set depends on human curation to repair a generator that fails more than half the time. Yet no inter-annotator agreement, independent audit, or released dataset is provided to confirm that curation removed systematic biases. More critically, open-ended answers (worth 20 of the 100 composite points in the exam format) are scored by GPT-4o itself (Experiment Setting; Fig. 8). Since GPT-4o also generated the reference answers and is one of the evaluated models, the scorer and the scored share the same output distribution. The paper provides no validation that GPT-4o's rubric scores correlate with human judges, no blind evaluation, and no analysis of self-preference. A systematic self-preference of even 1 point on the 0-5 scale translates to 4 points per test paper and can reorder models near the top of Table 2. The human-model gap in Table 5 is also affected if human open-ended answers were scored by the same GPT-4o procedure. Thus the benchmark's core quantitative findings are not yet robustly supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SCI-CQA, a chart question-answering benchmark built from figures, captions, and surrounding context extracted from 75,506 papers at 15 computer science conferences. After filtering, 37,607 chart-context-caption triplets remain, and 5,629 multiple-choice, true/false, and open-ended questions are curated, with about half of the test items targeting flowcharts. The benchmark uses an exam-inspired evaluation protocol (40 MC, 40 TF, 4 open-ended per paper) and reports results for 14 multimodal models. The authors also propose an automated annotation pipeline based on a fine-tuned LLaVA variant (Chart2A/ChartAS) and study context-based reasoning with previously unanswerable questions. The main empirical claims are that existing benchmarks overestimate chart understanding, that flowcharts remain a hard problem for all models, and that proprietary models still lag humans by a large margin on SCI-CQA.","tokens_in":20772,"tokens_out":10235,"duration_ms":93651,"significance":"If the benchmark and its scoring procedure are validated, SCI-CQA would be a useful community resource: it draws on authentic scientific literature, includes flowcharts as a distinct and previously underevaluated chart type, combines objective and open-ended evaluation formats, and provides a human baseline. The automated annotation pipeline is a practical contribution with clear cost-savings potential, and the paper is transparent about its data collection prompts and scoring rubrics. The context-based experiment showing the value of surrounding text for answering chart questions is a valuable direction. However, the load-bearing quantitative findings depend on the validity of the ground truth, the scoring of open-ended answers, and the comparability of the human-model evaluation; these are not yet established with sufficient evidence.","major_comments":[{"comment":"Open-ended answers are scored by GPT-4o, which also generated the reference answers and is one of the evaluated models. The paper provides no validation that GPT-4o's rubric scores match human judgments, no blind evaluation, and no analysis of self-preference. Because open-ended questions contribute 20 of the 100 composite points, a systematic bias of 1 point on the 0-5 scale changes the composite by 4 points per test paper, which is large enough to reorder models near the top of Table 2. Please add a human-scored subset with inter-rater agreement, compare GPT-4o scores against human scores, and analyze whether GPT-4o favors answers in its own style.","section":"Experiment Setting; Fig. 8"},{"comment":"The paper reports that only about 47% of GPT-4o-generated QA pairs were correct before manual review, yet it does not provide inter-annotator agreement, annotation guidelines, or an independent audit of the final 5,629 questions. If the human curation process has systematic blind spots—for example, accepting GPT-4o's phrasing or missing subtle errors—the ground truth would inherit generator bias. The dataset is not yet released, so the curation quality cannot be independently assessed. Please release the QA bank and provide detailed annotation statistics, including agreement measures and examples of rejected and corrected items.","section":"Related Work; SCI-CQA data processing pipeline"},{"comment":"The human-model comparison in Table 5 only includes Chartllama, CogVLM, and Qwen-VL-chat, with the highest model score reported as 41/100. This is not the same set of models whose scores appear in Tables 2-3, where GPT-4o reaches 72.60 on flowcharts and 66.17 on data charts. The sentence in the text stating that the highest model score was 41 is therefore misleading in context, and the claimed human-model gap of roughly 50 points does not apply to the best models evaluated in the paper. Please report human performance on the same test papers taken by the models in Tables 2-3, or clearly restrict the human comparison to the three models actually tested.","section":"Human-Inspired evaluation method; Table 5"},{"comment":"The claim that existing benchmarks 'often overestimate' chart understanding is supported only by Chartllama's score drop from 69.66 on ChartQA to 21.12 on SCI-CQA flowcharts. A lower score on a new, more difficult benchmark does not by itself demonstrate that the old benchmark's scores are inflated; it may simply reflect task difficulty, distribution shift, or the model's training data. To support the inflation claim, the paper should either compare multiple models on both benchmarks with a human baseline on both, or show that prior benchmarks contain questions answerable without consulting the chart. Otherwise, the conclusion should be weakened to 'SCI-CQA is more challenging.'","section":"Performance analysis of flowchart [1]"},{"comment":"The composite score definition—scaling open-ended scores to 0-100 and averaging with Avg MC and Avg TF—does not reproduce the reported composite scores under the natural interpretation. For example, GPT-4o on flowcharts would score (41.48 + 78.75 + 4.45*20)/3 = 69.7, not the reported 72.60. The column layout in Table 2 is also ambiguous. Please provide the exact composite formula and the underlying per-paper raw scores so that the headline numbers are reproducible.","section":"Human-Inspired evaluation method; Table 2"},{"comment":"Table 1 reports the number of evaluation samples as 1.5k for FigureQA, 1.5k for ChartQA, and 2.3k for CharXiv, but these numbers are inconsistent with the published versions of those benchmarks (e.g., ChartQA contains 32k questions, and FigureQA contains well over a million QA pairs). This makes the statement that 'SCI-CQA stands out with the highest number of evaluation samples (5.6k)' incorrect and undermines the comprehensiveness comparison. Please correct the table and revise the associated claim.","section":"Analysis of the Comparison Across Datasets; Table 1"}],"minor_comments":[{"comment":"The caption contains a typo: 'whichs led' should be 'which led'.","section":"Figure 2 caption"},{"comment":"The phrase 'a lager training dataset' should be 'a larger training dataset'.","section":"Automated labeling tools"},{"comment":"The model name 'Qwen-VL-Chart' in Table 2 appears to be 'Qwen-VL-Chat' elsewhere in the paper; please standardize the naming.","section":"Tables 2 and 3"},{"comment":"The paper lists 'Unanswerable Questions' as a question type, but the dataset statistics and evaluation description only cover multiple-choice, true/false, and open-ended questions. Please clarify how unanswerable questions and the 'unable to answer' option are scored.","section":"Human-Inspired evaluation method"},{"comment":"The appendix uses inconsistent casing for 'GPT-4O' and 'GPT-4o'; please choose one convention throughout.","section":"Appendix, Q&A Data generation"},{"comment":"The dataset and code are only promised for future release ('will be released'); for a benchmark paper, availability at submission time is important for verification and adoption.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper has clear potential value as a benchmark resource, but the referee concerns about GPT-4o as both generator and scorer, the lack of inter-annotator agreement data, and the non-comparable human evaluation are central to the paper's main claims. These issues can likely be addressed with additional experiments and clarifications, so major revision seems appropriate rather than rejection. I would also flag that the incorrect evaluation-sample counts in Table 1 should be fixed before any final acceptance, as they currently support an inaccurate comprehensiveness claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the benchmark is real and mostly new: 37,607 chart-context-caption triples from 15 CS conferences, 5,629 human-reviewed QA pairs, and the first large-scale natural flowchart set for chart QA. The exam-style format (multiple choice, true/false, open-ended) plus context-dependent 'unanswerable' questions go beyond CharXiv and ChartX in scope. Second, the headline finding—models score 21-73 on flowcharts while humans score 88-91—survives the main methodological concern about GPT-4o scoring, because the objective questions alone show the same large gap.\n\nNow the soft spots. The stress-test note is correct that open-ended answers (20 of 100 composite points) are scored by GPT-4o, which also generated the references and is under test. There's no validation of that scorer against human judges, no self-preference check, and no error bars anywhere. A 1-point bias in open-ended scores moves the composite by up to 4 points, which can reorder models near the top of Table 2. But this does not break the paper's central conclusion, since MC and TF results are objective and the model-human gap is enormous there too.\n\nThe other issue: the 'existing benchmarks overestimate' claim is not directly tested. The paper compares Chartllama's flowchart score to its ChartQA score, which are different tasks and datasets, not a head-to-head run on CharXiv or ChartX. So that claim is plausible but unsupported as stated.\n\nTwo smaller notes. The dataset and code are not available yet (only a placeholder repo), so the resource cannot be independently audited. And the automated annotation pipeline section (fine-tuning LLaVA to generate pseudo-labels) is interesting but tangential to the benchmark.\n\nWho this is for: anyone building or evaluating chart-understanding models, especially for scientific literature. It deserves a serious referee, because the resource is substantial and novel, and the objective results are clean. The referee should require dataset release, human validation of the open-ended scoring, and direct comparisons with CharXiv/ChartX. If those are addressed, this becomes a valuable field asset.\n\nMy recommendation: engage with it, but treat the open-ended scores as provisional until the scoring is validated.","headline":"A substantial new chart QA benchmark with a real flowchart contribution, whose objective results are solid but whose open-ended scoring needs independent validation before the numbers are taken at face value.","tokens_in":21283,"tokens_out":4555,"would_cite":false,"duration_ms":39560,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Real scientific charts, especially flowcharts, show current multimodal models are far from human-level chart understanding, with the best proprietary model scoring 72.60 to humans' 90 on flowcharts.","keywords":["scientific chart QA","flowchart understanding","multimodal large language models","benchmark evaluation","context-based reasoning","chart question answering","exam-inspired evaluation","hallucination measurement"],"falsifier":"Have independent computer-science graduate students re-grade the open-ended answers from the three exam papers with no access to the model-generated scores; if the human scores differ systematically—say by more than one point in the direction of favoring GPT-style phrasing—then the reported model-versus-human gap and the inflation claim would be weakened.","tokens_in":20323,"feed_emoji":"📊","tokens_out":7203,"duration_ms":64401,"temperature":0.7,"pith_summary":"SCI-CQA is a new benchmark for evaluating multimodal models on charts drawn from real scientific papers, with a deliberate focus on flowcharts—diagrams of model architectures, algorithm pipelines, and training processes—that most prior benchmarks ignore. The paper argues that existing chart benchmarks overestimate model abilities because they rely on narrow chart types, simple template-based questions, and one-note evaluation, so high scores do not survive contact with authentic scientific literature. The benchmark is built from 202,760 image-caption-context triplets mined from top computer science conferences, filtered to 37,607 charts, and paired with 5,629 expert-reviewed questions in multiple-choice, true/false, and open-ended formats. On the flowcharts, the best proprietary model scores 72.60 while human evaluators score around 90, and open-source models lag further, which the paper presents as evidence that real chart understanding remains unsolved. It also shows that adding the paper's surrounding textual context lets models answer questions that the image alone cannot answer.","feed_headline":"Best chart AI scores 72.6 on real flowcharts; humans score 90","feed_subtitle":"A benchmark from CS papers shows template chart tests inflate AI scores, and context drives real reasoning.","key_machinery":"The load-bearing machinery is the SCI-CQA dataset itself, organized as (chart, caption, context) triplets extracted from LaTeX sources of 15 computer-science conferences, filtered by model-based classification and manual review into 37,607 charts. On top of this sits a 5,629-question bank with a 5:4:1 ratio of multiple-choice, true/false, and open-ended questions, assembled into 100-point exam papers that mirror human testing, with open-ended answers scored by a rubric-based model grader. The design that carries the argument is the addition of a sixth question type, scientific reasoning, and the inclusion of questions that cannot be answered from the image alone: these force the model either to use the surrounding caption and context or to say 'unable to answer', turning hallucination into a measurable failure. An automated annotation pipeline, built by fine-tuning an open-source vision-language model under supervision from free proprietary models, is used to generate pseudo-labels at reduced cost.","core_discovery":"The central discovery is that existing benchmarks often overestimate chart understanding: when models are tested on authentic scientific charts, including flowcharts, their performance drops markedly relative to human performance. On SCI-CQA's flowchart test, the strongest proprietary model reaches 72.60 composite points while human graduate students in computer science score about 90; on data charts the same models score in the 60s, and open-source models trail by a wider margin. The paper attributes the gap to three flaws in prior benchmarks—limited chart variety, template-based questions that can be answered without reading the chart, and evaluation methods that do not separate perception from reasoning—and argues that a human-exam-inspired mix of multiple-choice, true/false, and open-ended questions gives a more accurate picture. A key supporting finding is that about 47% of machine-generated question-answer pairs were wrong before manual review, which the paper uses to argue that datasets without human review can mislead performance estimates.","pith_inferences":["If the inflation claim generalizes, then public leaderboards built on synthetic/template chart QA should be re-read as measuring benchmark-fit rather than chart competence, and new benchmarks should include adversarial 'unanswerable' questions to expose hallucination.","A testable extension: because the paper reports that only about 47% of unaudited generated QA pairs were correct, other large multimodal benchmarks that skip human review may carry systematic label noise; re-auditing a sample of those datasets would reveal whether the inflation problem is widespread.","Since context mattered far more than captions for unanswerable questions, a natural next step is to measure how much context length or retrieval precision changes model scores, which could turn this benchmark into a tool for studying grounded scientific reasoning.","The automated annotation pipeline's gains were larger for objective questions than for open-ended ones, suggesting that genuinely hard open-ended chart reasoning may still require human annotation for the foreseeable future."],"forward_implications":["High scores on template/synthetic chart benchmarks should not be read as evidence of real chart understanding; SCI-CQA shows the same models score far lower on authentic scientific charts.","Flowcharts should be treated as a first-class evaluation category, not an afterthought, because they stress structured reasoning and currently separate proprietary from open-source models.","Evaluations that mix multiple-choice, true/false, and open-ended questions reveal differences that a single question type masks, such as the 10.73-point composite gap between two models that differ by only 1.96 points on multiple-choice.","Supplying contextual text from the paper alongside the chart substantially improves performance on questions that are unanswerable from the image alone, while captions alone help much less.","Perception and reasoning are partially independent: once perception crosses a threshold, a model's reasoning score is not determined by its perception score."],"supporting_citations":[{"why":"Supplies the real-chart data and five question categories that SCI-CQA extends with a sixth reasoning type; also the strongest real-chart baseline.","marker":"(Wang et al. 2024)"},{"why":"Provides the GPT-4-family models whose performance on SCI-CQA is the main proprietary comparison.","marker":"(Achiam et al. 2023)"},{"why":"FigureQA, a template-based synthetic benchmark that SCI-CQA contrasts its question complexity against.","marker":"(Kahou et al. 2018)"},{"why":"DVQA, a synthetic chart QA benchmark whose template questions the paper identifies as allowing answers without chart understanding.","marker":"(Kafle et al. 2018)"},{"why":"PlotQA, a synthetic plot benchmark where simple visual elements limit the validity of high scores.","marker":"(Methani et al. 2020)"},{"why":"ChartLLaMA, an open-source chart model evaluated on SCI-CQA and an example of code-generated charts with only 4-8 chart types.","marker":"(Han et al. 2023)"},{"why":"ChartX, the 18-category chart benchmark that still lacks flowcharts and visual-element diversity, used as a comparison point.","marker":"(Xia et al. 2024)"},{"why":"FlowVQA, which recognizes flowcharts as a modality but uses synthetic diagrams that differ in complexity from SCI-CQA's natural flowcharts.","marker":"(Singh et al. 2024)"}],"fun_headline_variants":["Real chart benchmark shows AI scores 72.6, humans 90","Template chart tests overrate AI: new benchmark reveals gap","Flowcharts trip up AI: humans score 90, best model 72.6","Scientific chart benchmark exposes AI understanding gap","Context is key: new chart QA benchmark for AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the human-curated, GPT-4o-generated questions are unbiased ground truth and that the GPT-4o-based grader scores open-ended answers fairly, without systematically favoring answers written in a GPT-4o-like style.","fun_headline_variants_meta":{"raw":{"variants":["Real chart benchmark shows AI scores 72.6, humans 90","Template chart tests overrate AI: new benchmark reveals gap","Flowcharts trip up AI: humans score 90, best model 72.6","Scientific chart benchmark exposes AI understanding gap","Context is key: new chart QA benchmark for AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000772,"raw_usage":{"total_tokens":3423,"prompt_tokens":955,"completion_tokens":2468,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":2383}},"tokens_in":571,"tokens_out":2468,"duration_ms":17386,"temperature":1.0,"reasoning_tokens":2383,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:12:32.761062+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent computer-science graduate students re-grade the open-ended answers from the three exam papers with no access to the model-generated scores; if the human scores differ systematically—say by more than one point in the direction of favoring GPT-style phrasing—then the reported model-versus-human gap and the inflation claim would be weakened.","supporting_citations":[{"cited_title":"E.; Michalski, V.; Atkinson, A.; K \\' a d \\' a r, \\' A .; Trischler, A.; and Bengio, Y","cited_arxiv_id":null,"evidence_quote":"FigureQA, a template-based synthetic benchmark that SCI-CQA contrasts its question complexity against."},{"cited_title":"L.; Cohen, S.; and Kanan, C","cited_arxiv_id":null,"evidence_quote":"DVQA, a synthetic chart QA benchmark whose template questions the paper identifies as allowing answers without chart understanding."},{"cited_title":"M.; and Kumar, P","cited_arxiv_id":null,"evidence_quote":"PlotQA, a synthetic plot benchmark where simple visual elements limit the validity of high scores."}],"review_version":1}