{"id":"f333ff7d-4fad-4bdf-b9dd-1c6a7337dbe0","arxiv_id":"2507.06183","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On the SciVQA 2025 benchmark, an InternVL3 model with optimized prompts and chain-of-thought instructions reaches ROUGE-1 and ROUGE-L F1 of 0.740, and a figure-type-aware ensemble ranks 5th.","lead":"This paper describes a shared-task system for answering questions about scientific charts and figures. The authors show that careful prompts, step-by-step reasoning, and choosing models per chart type improve scores on the SciVQA benchmark, with their best single model reaching a ROUGE-L F1 of 0.740.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 0.740 ROUGE-F1 is not attributable to the model alone because Section 4.4 canonicalizes every 'insufficient information' output to the gold unanswerable phrase and no raw-output or unanswerable-subset scores are reported.","rationale":"The paper is a credible shared-task system description: the code is public, the leaderboard rank is consistent with Table 2's ensemble row (0.735), and the headline numbers are internally consistent across Table 2 and the abstract. The reader's weakest-assumption analysis correctly identifies the Section 4.4 canonicalization as the point where the central claim is least secure. Because the rule is disclosed and is a standard way to score semantically equivalent unanswerable answers, I would not reject the paper; but the missing raw-score ablation means the 0.740 figure cannot yet be read as a measure of InternVL3's reasoning alone. The paper's own Appendix C shows gold-format inconsistencies, which makes the canonicalization's effect genuinely uncertain. The only other notable issue is the '+0.30' typo in Section 4.5, which should read '+0.03'; this is a presentation error, not a challenge to the main results. A single ablation experiment with and without post-processing, plus an unanswerable-subset breakdown, would settle whether the concern lands. If the gap is negligible, the verdict should be upgraded to accept; until then CONDITIONAL is appropriate, so no change to the reader's verdict. The arithmetic typo should be corrected regardless.","tokens_in":10228,"tokens_out":5967,"duration_ms":63662,"concrete_test":"Re-run the evaluation on the same SciVQA test split for InternVL3 with CoT twice: once with the Section 4.4 standardization enabled and once with it disabled (raw model outputs), and report ROUGE-1/ROUGE-L/BERTScore overall and separately for the unanswerable and answerable subsets. If the raw-output overall score is materially below 0.740 (e.g., more than about 0.02 lower) or the unanswerable subset shows near-perfect scores while the answerable subset is much lower, the headline result is substantially a post-processing artifact rather than a model-only capability. Also count how many gold unanswerable answers equal the canonical phrase to verify the rule's matching assumption.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.4 states that, after removing |end| tags, \"for questions where the reasoning process determined insufficient information to give a valid response, outputs were standardized to 'It is not possible to answer this question based only on the provided data.' regardless of the initial model output.\" This rule is applied before the ROUGE/BERTScore numbers in Tables 1 and 2, but the paper reports no evaluation with the rule disabled and no breakdown of the unanswerable subset. Unanswerable items are substantial: Table 4 lists 2,160 of 15,120 train samples (~14%). If SciVQA gold unanswerable answers use exactly that canonical sentence, then any model output classified as unanswerable is converted into an exact match, contributing near-perfect ROUGE and BERTScore on those items irrespective of the model's visual reasoning. The abstract's attribution of 0.740 to InternVL3 with prompt optimization and CoT is therefore conditional on this unablated post-processing; the number may describe the system including canonicalization rather than the model's question-answering ability. This is a missing-ablation concern, not an internal contradiction, but it is load-bearing for the central empirical claim. (A secondary arithmetic slip in Section 4.5 says '+0.30' where the table shows 0.740 vs 0.709, about +0.03.)","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper describes a system submitted to the SciVQA 2025 shared task for answering questions about scientific figures. The authors evaluate four vision-language models in the 5–8B parameter range (InternVL3-8B, Qwen2.5-VL-7B, Bespoke MiniChart 7B, Phi-4 Multimodal), combine task-specific prompt templates, chain-of-thought with a rethink mechanism, and a figure-type-aware ensemble, followed by a post-processing step that standardizes outputs classified as unanswerable. The headline result is that InternVL3 reaches ROUGE-1 and ROUGE-L F1 of 0.740 and BERTScore of 0.983 on the SciVQA test split, while the ensemble reaches 0.735 and ranked 5th on the leaderboard. The paper also reports validation error analysis and identifies several dataset quality issues.","tokens_in":10504,"tokens_out":2076,"duration_ms":24754,"significance":"If the reported numbers are reproducible, this is a useful shared-task system description with a clear, checkable empirical claim: prompt optimization and CoT make InternVL3 a strong baseline on SciVQA. The paper's positive features include public code release, full prompt templates in the appendix, per-figure-type validation tables, and explicit documentation of annotation inconsistencies in the gold data. The contribution is not a new mechanism or theory, but a carefully engineered system; its value lies in the reproducibility of the benchmark result and the practical lessons about prompt design and post-processing. However, the central performance claim depends on an unablated post-processing rule and on single-run evaluations, so the significance is conditional on those issues being resolved.","major_comments":[{"comment":"The central 0.740 ROUGE claim is not attributable to the model alone because the post-processing rule in §4.4 rewrites every output judged to contain insufficient information into the exact sentence \"It is not possible to answer this question based only on the provided data.\" This canonicalization is applied before the reported ROUGE/BERTScore numbers, but no evaluation with the rule disabled and no breakdown of the unanswerable subset is given. Since Table 4 lists 2,160 of 15,120 train samples (about 14%) as unanswerable, the rule can mechanically convert a large fraction of outputs into exact matches irrespective of visual reasoning. Please report scores with post-processing disabled, and also report metrics separately on the answerable and unanswerable subsets, so the headline number can be attributed correctly.","section":"§4.4, Tables 1–2"},{"comment":"The claim in §4.5 that InternVL3 outperforms other individual models \"by at least +0.30 ROUGE-1 F1 score on the test split\" is arithmetically inconsistent with the reported numbers: the gap between InternVL3 (0.740) and the next-best individual model, Bespoke (0.709), is about +0.031, not +0.30. This should be corrected, and the comparison should also state whether the delta is statistically meaningful.","section":"§4.5, Tables 1–2"},{"comment":"All headline metrics appear to be single-run evaluations with no error bars, confidence intervals, or significance tests. The claims that \"CoT prompting achieved consistent improvements across all VLMs\" and that InternVL3 is the strongest standalone model are based on differences that may be within run-to-run noise (e.g., InternVL3 0.740 vs. ensemble 0.735, or CoT vs. no-CoT differences of 0.01–0.06). Please provide at least bootstrap confidence intervals over the test items, or multiple inference runs, for the central comparisons.","section":"§4.5, Tables 1–2"},{"comment":"The figure-type-aware ensemble assignment is based on validation-set accuracy per chart type, but many rows in Table 9 contain very small per-type sample sizes (for example, line_chart,table, vector_plot, and several compound types have roughly 7–14 validation items), and the reported standard deviations are accordingly large. The resulting assignment of models to figure types may not be robust. Please report the number of validation items per figure type and assess whether the ensemble's 0.735 test score is reliably better than simply running InternVL3 on all items, especially given that the ensemble underperforms InternVL3 by 0.005 in ROUGE-1 F1.","section":"§4.3, Table 9"}],"minor_comments":[{"comment":"The abstract and §4.5 credit InternVL3 with the 0.740 test score, while Table 10 reports the submitted system \"Infyn\" at 0.735 ROUGE-1 F1. Please clarify explicitly that the leaderboard submission was the ensemble, not the single InternVL3 model, so the reader can reconcile the numbers.","section":"Abstract and Table 10"},{"comment":"The description of post-processing should state the exact criterion used to determine that \"the reasoning process determined insufficient information,\" since this judgment is presumably made by the model itself; without that criterion, the rule is not fully reproducible.","section":"§4.4"},{"comment":"The sentence \"which our models corrects predict\" contains a grammatical error and should be rewritten as \"which our models predict correctly\" or similar.","section":"Appendix C"},{"comment":"For readability, Table 9 should include a column with the number of validation examples per figure type, since the accuracy means are otherwise hard to interpret when sample sizes vary from below 10 to over 1,000.","section":"Table 9"},{"comment":"The relation between the baselines and the final prompts could be clearer: Baseline 1 achieves very low ROUGE F1 (0.180) with high recall (0.498), which suggests a formatting mismatch; please state whether the evaluation script strips extra text and whether baselines were subject to the same post-processing as the main system.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a shared-task system description with a modest but potentially reproducible contribution. For a general journal venue, the main interest lies in the prompt and post-processing recipe rather than in any new scientific finding, so the editor may wish to consider whether the scope fits. The missing post-processing ablation is the key technical gap; if the authors supply it and the numbers still stand, a revision would be defensible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: treat this as a shared-task system description, not a research contribution. The paper is honest, well-scoped, and includes code, but the headline 0.740 is partly an artifact of a post-processing rule that rewrites every \"insufficient information\" output to the gold unanswerable sentence. That rule is described openly, but never ablated, so the number does not purely measure the models' reasoning ability.\n\nWhat's good: the authors pick four 5-8B VLMs, do careful prompt engineering, CoT with a rethink step, and a figure-type-aware ensemble. The ensemble choices are grounded in validation accuracy per chart type. They include a useful error analysis and flag real dataset quality issues (inconsistent gold formatting, annotation errors). They also release the code. For a shared task, this is exactly the right kind of write-up: transparent, reproducible, and appropriately modest in its claims.\n\nThe soft spots: the missing ablation of Section 4.4 is the main one. Unanswerable items are about 14% of the train split; if the gold answer uses that exact sentence, then any model output classified as unanswerable becomes an exact match. That can inflate absolute ROUGE/BERTScore substantially and also skew model comparisons, since a model that over-predicts \"insufficient information\" would get free wins. The paper does not report how often the rule fired, nor raw numbers without it. That is a one-section fix. Second, there are no error bars or significance tests; with a test set of 4,200, the difference between InternVL3 (0.740) and the ensemble (0.735) is likely noise, and the paper wisely does not over-interpret it. Third, there is an arithmetic slip in Section 4.5: \"at least +0.30 ROUGE-1\" should be +0.03. Minor, but it should be corrected.\n\nThe central empirical report is likely accurate as a system description, but the attribution to the model-plus-prompt-plus-CoT is too strong given the unablated post-processing. With an ablation and a corrected abstract, this would be a sound record of a fifth-place finish. I would send it to peer review as a shared-task system description, with a request for the ablation and some modest re-framing. For a reading group, it is a decent case study in how evaluation metrics can be gamed by post-processing. I would not cite it in my own work unless I needed a SciVQA baseline number.","headline":"Solid shared-task system description; the 0.740 is partly a post-processing artifact and needs an ablation before it supports the claims.","tokens_in":11033,"tokens_out":2835,"would_cite":false,"duration_ms":29923,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prompt optimization plus chain-of-thought pushes InternVL3 to ROUGE and BERTScore of 0.740/0.983 on the SciVQA test set, and a figure-type ensemble improves most models but not the best single one.","keywords":["scientific visual question answering","SciVQA 2025 shared task","chain-of-thought prompting","vision-language models","ensemble modeling","prompt optimization","figure-type selection","answer post-processing"],"falsifier":"Re-run the InternVL3 evaluation on the SciVQA test split with the answer-standardization rule disabled, keeping only the end-tag removal, and recompute ROUGE-1 and BERTScore; if the scores fall noticeably below 0.740 and 0.983, part of the reported performance comes from the rewriting rule. Independently, inspect the gold answers for the unanswerable questions in the training or validation splits and count how many literally match the standardized 'It is not possible...' sentence, since the rule can only inflate scores on questions whose gold phrasing is identical.","tokens_in":10034,"feed_emoji":"📊","tokens_out":8377,"duration_ms":77109,"temperature":0.7,"pith_summary":"SciVQA 2025 asks systems to answer closed-ended questions about charts and figures in scientific papers. This paper shows that InternVL3, an 8-billion-parameter vision-language model, becomes a strong answerer when it is steered by a carefully composed prompt, a two-step chain-of-thought procedure, and a post-processing rule that standardizes 'cannot answer' responses. The reported test results are ROUGE-1 and ROUGE-L F1 of 0.740 and a BERTScore of 0.983. The authors also report that chain-of-thought prompting improved every tested model, and that a figure-type-aware ensemble of four models beat most individual models while still trailing InternVL3 alone. The practical point is that prompt engineering, ensembling, and answer standardization can push mid-size models near the top of a scientific VQA leaderboard without changing the underlying architecture.","feed_headline":"Prompt-plus-CoT recipe drives InternVL3 to 0.74 ROUGE-F1","feed_subtitle":"Mid-size vision-language models near the top of the 2025 SciVQA board via smart prompts and ensembling.","key_machinery":"The system is carried by three components working together. The first is a composite prompt that concatenates a base formatting instruction with figure-type-specific cues (colors, axes, scale, trends for line charts; segments and percentages for pie charts; row and column labels for confusion matrices), a binary answer constraint, a choice-letters constraint, and the figure caption. The second is a two-step chain-of-thought prompt with a rethink stage: step one asks the model to reason aloud with self-questioning, self-verification, and self-correction, and step two demands a bare numeric or single-sentence answer, with answer letters only for choice questions. The third is a post-processing rule that removes XML end tags and rewrites any output classified as having insufficient information into the exact sentence 'It is not possible to answer this question based only on the provided data.' Around these, a figure-type-aware ensemble assigns Qwen2.5-VL to scatter plots and trees, Bespoke MiniChart to pie and bar charts, and Phi-4 to line charts and tables, based on per-type validation accuracy.","core_discovery":"The central claim is that InternVL3-8B with an optimized composite prompt and a two-step chain-of-thought 'rethink' procedure achieves ROUGE-1 and ROUGE-L F1 of 0.740 and BERTScore of 0.983 on the SciVQA test split, establishing a reproducible baseline for the task. A supporting claim is that chain-of-thought prompting yields consistent gains across all four tested vision-language models, and that an ensemble assigning each model to its strongest figure types reaches an ensemble ROUGE-1 F1 of 0.735, which improves on most individual systems but not on InternVL3 alone. The paper further argues that inconsistencies in the gold annotations, such as multiple-answer formatting and a verifiable labeling error, mean the metric scores likely underestimate how well the models actually answer.","pith_inferences":["A clean ablation that disables the answer-standardization rule would quantify how much of the 0.740 ROUGE-F1 comes from the models themselves versus from rewriting insufficient-information outputs; the paper does not report this comparison.","A learned router that selects models by figure type, rather than hand-picked assignments, could plausibly beat InternVL3 alone, since the validation table shows different models dominate different chart types.","Comparing the two-step rethink prompt to sampling multiple chain-of-thought answers and voting would separate the benefit of self-correction from the benefit of simply generating more tokens.","Reporting exact-match accuracy as well as ROUGE and BERTScore on a human-audited subset would make future leaderboard movements easier to interpret, since the paper shows annotation noise can mask real model improvements."],"forward_implications":["Future SciVQA entries can treat InternVL3 with the reported prompt recipe as a strong baseline, since a 0.740 ROUGE-1 F1 result is now documented and reproducible.","Chain-of-thought prompting is a cheap gain: every model in the study improved on the test set with CoT, so teams with limited compute can extract improvements without fine-tuning or larger models.","Figure-type ensembling is a bounded strategy: it beat three of the four individual models on test (0.735 versus 0.709, 0.695, and 0.562) but lost to InternVL3 alone, suggesting vision-encoder quality outweighs model diversity.","The answer-standardization rule means reproducing the reported numbers requires also reproducing the exact 'It is not possible...' sentence rewriting, because the rule converts many outputs into the reference phrasing before scoring.","The documented gold-annotation inconsistencies, such as '52,3%' versus '3%', imply ROUGE and BERTScore estimates are lower bounds on true agreement and that a cleaned evaluation set could shift rankings."],"supporting_citations":[{"why":"Supplies the InternVL3-8B model and reports its strong OCR, chart, and document understanding, making it the best standalone model.","marker":"(Zhu et al., 2025)"},{"why":"Supplies the Qwen2.5-VL-7B Instruct model, selected for its mathematical reasoning and used in the ensemble.","marker":"(Bai et al., 2025)"},{"why":"Supplies the Phi-4 Multimodal Instruct model, assigned to line charts, tables, and diagrams in the ensemble.","marker":"(Microsoft et al., 2025)"},{"why":"Defines the SciVQA task, dataset splits, and ROUGE/BERTScore evaluation that all reported numbers are measured against.","marker":"(Borisova et al., 2025)"},{"why":"Provides the chain-of-thought prompting method that the two-step CoT prompts are built on.","marker":"(Wei et al., 2023)"},{"why":"Provides the rethink/self-correction mechanism used in the first CoT step.","marker":"(Wang et al., 2025)"},{"why":"ChartQA benchmark used to justify the choice of 5-8B vision-language models for the task.","marker":"(Masry et al., 2022)"},{"why":"MathVista benchmark, emphasizing precise numerical reasoning in visual contexts, used to motivate model selection.","marker":"(Lu et al., 2024)"}],"fun_headline_variants":["InternVL3 hits 0.74 ROUGE-F1 on SciVQA via CoT and prompts","CoT-prompted InternVL3 tops SciVQA with 0.74 ROUGE-F1","Prompt engineering and CoT push InternVL3 to 0.74 ROUGE-F1","InternVL3's CoT recipe: 0.74 ROUGE-F1 on SciVQA","Ensemble of VLMs with CoT: InternVL3 leads at 0.74 ROUGE-F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline scores assume that every unanswerable question in the SciVQA gold set uses exactly the standardized phrase 'It is not possible to answer this question based only on the provided data,' because the post-processing step rewrites all insufficient-information outputs into that string before scoring.","fun_headline_variants_meta":{"raw":{"variants":["InternVL3 hits 0.74 ROUGE-F1 on SciVQA via CoT and prompts","CoT-prompted InternVL3 tops SciVQA with 0.74 ROUGE-F1","Prompt engineering and CoT push InternVL3 to 0.74 ROUGE-F1","InternVL3's CoT recipe: 0.74 ROUGE-F1 on SciVQA","Ensemble of VLMs with CoT: InternVL3 leads at 0.74 ROUGE-F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001527,"raw_usage":{"total_tokens":6119,"prompt_tokens":952,"completion_tokens":5167,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":5034}},"tokens_in":568,"tokens_out":5167,"duration_ms":36675,"temperature":1.0,"reasoning_tokens":5034,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:08:18.063356+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the InternVL3 evaluation on the SciVQA test split with the answer-standardization rule disabled, keeping only the end-tag removal, and recompute ROUGE-1 and BERTScore; if the scores fall noticeably below 0.740 and 0.983, part of the reported performance comes from the rewriting rule. Independently, inspect the gold answers for the unanswerable questions in the training or validation splits and count how many literally match the standardized 'It is not possible...' sentence, since the rule can only inflate scores on questions whose gold phrasing is identical.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the SciVQA task, dataset splits, and ROUGE/BERTScore evaluation that all reported numbers are measured against."}],"review_version":1}