{"id":"577380e1-2ba7-4784-be81-043e36531125","arxiv_id":"2507.02357","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"An ensemble of InternVL3-78B and Pixtral-Large with retrievable few-shot examples and a confidence threshold beats individual models on scientific figure question answering.","lead":"This paper describes a visual question answering system that combines two open-weight multimodal language models and retrieves similar few-shot examples from the training set. The system ranked third of seven in the SciVQA 2025 shared task with an average F1 of 85.12.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set comparison to 'single best individual configuration' is not established because only one individual configuration is reported on test.","rationale":"The reader's verdict of ACCEPT is reasonable for a shared-task system paper: the system is described transparently, dev-set ablations are thorough, code is public, and the official rank is third. My concern targets one specific load-bearing formulation of the central claim rather than the overall contribution. The reader's weakest assumption was calibration transfer from dev to test. That is a legitimate concern, but the test result actually provides some evidence that the threshold behaved reasonably, since the confidence-informed ensemble improved on the test set. The more concrete gap I see is the test-set baseline: the paper demonstrates the ensemble beats one individual configuration on test, not the best individual configuration across all tested settings. This matters because the strongest claim explicitly says 'single best individual configuration on both dev and test.' That exact claim is not established by the reported table. The check is straightforward and does not require re-running expensive models if the organizers or authors can release per-configuration test scores; otherwise, a precise rewording resolves the issue. I therefore recommend a conditional acceptance: the paper should either provide test scores for all individual configurations or explicitly limit the test-set comparison to the development-best individual configuration. This does not imply any methodological dishonesty; it is a scope-of-claim clarification.","tokens_in":12795,"tokens_out":6637,"duration_ms":79420,"concrete_test":"Obtain or compute test-set ROUGE-1 F1 for every individual configuration listed in Table 2 (or at least all configurations used in the ensembles), using the same official evaluation protocol. If any individual configuration reaches or exceeds 78.6 R1-F1, the claim that the confidence-informed ensemble outperforms the single best individual configuration on the test set does not hold as stated. If the test set is not available with gold labels, the claim should be reworded to 'outperforms the development-best individual configuration on the test set.'","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central empirical claim, as stated in Section 4 and in the reader's summary, is that the confidence-informed ensemble outperforms the single best individual configuration on both development and test sets. The development-side comparison is solid: Table 2 reports all individual configurations on the development set, and InternVL3-78B (1s_q_img_f, BLIP2) is the best at 75.0 R1-F1. The test-side comparison, however, is not supported by the data shown. Table 2's test section reports only three rows: InternVL (1s_q_img_f, BLIP2) at 77.2, the question/figure-type ensemble at 77.7, and the confidence-informed ensemble at 78.6. No other individual configurations are reported on test, so we do not know whether 77.2 is actually the best individual test configuration. The test rows appear to be official or otherwise limited evaluations, not a full sweep. Therefore '78.6 vs 77.2' compares the ensemble to the development-best individual, not necessarily the test-best individual. If another individual configuration (e.g., Pixtral 2s_q_img_f) scored at least 78.6 on test, the headline claim would be false. This is not an accusation of selective reporting; it is an unstated scope limitation in the comparison. The shared-task rank and the development-set ablation remain valid, but the test-set superlative needs qualification or additional data.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a system submitted to the SciVQA 2025 shared task on scientific visual question answering. The system combines two open-weight multimodal LLMs (InternVL3-78B and Pixtral-Large-2411) in a zero/few-shot setting, with several few-shot example retrieval strategies based on SBERT question similarity, CLIP question-image similarity, and BLIP-2 embeddings. A question/figure-type ensemble selects the best configuration per instance group on the development set, and a confidence-informed ensemble first accepts answers from InternVL3-78B (1s_q_img_f, BLIP-2) when the mean token-log-probability confidence is at least 0.9, then routes remaining instances to per-question-type configurations. The paper reports extensive development-set ablations (Table 2) and blind test results where the system ranks third out of seven with an average ROUGE-1/ROUGE-L/BERTScore F1 of 85.12.","tokens_in":13042,"tokens_out":11364,"duration_ms":126766,"significance":"If the results are taken at face value, the paper makes a modest but useful contribution: a reproducible, compute-constrained recipe for applying MLLMs to scientific VQA without fine-tuning. The strengths are the public code, the transparent ablation over individual configurations on the development set, and the external validation from the official leaderboard. The confidence-informed ensembling idea is simple and effective, showing a consistent gain over its component model on both dev (76.9 vs 75.0 R1-F1) and test (78.6 vs 77.2) for the evaluated baseline. The main limitation is that the test-set comparison is reported for only one individual configuration, so the claim that the ensemble outperforms 'using only one approach' on test is scoped to that baseline rather than to the best individual test configuration; this is a reporting issue that can be fixed by qualification.","major_comments":[],"minor_comments":[{"comment":"The test-set section reports only three rows (one individual configuration and two ensembles), so the statement that ensembling 'improve[s] the performance compared to using only one approach' should be scoped to the InternVL (1s_q_img_f, BLIP2) baseline actually evaluated. Please either report additional individual configurations on the test set (if available) or replace 'work best' with 'work best among the configurations evaluated on the test set' and clarify whether test runs were limited by submission constraints.","section":"Section 4, Table 2"},{"comment":"The 0.9 confidence threshold is selected on the development set without a sensitivity analysis; a small threshold sweep (e.g., 0.8–0.95) on the development set would help readers judge how robust the choice is and whether the calibration-based justification is stable.","section":"Section 3.4"},{"comment":"There are several typos: 'configruations' in Section 3.4; 'InvernVL' in Table 2's test row header; 'InternvL3-78B' in Table 7's note; and a stray colon in the key 'answer_options:' in the prompt template shown in Appendix A.2.","section":"Section 3.4, Table 2, Table 7, Appendix A.2"},{"comment":"The caption describes the plot as a calibration plot but does not specify what is plotted; adding axis labels and a diagonal reference would clarify whether it shows binned empirical R1-F1 versus mean confidence and what the 'calibration gap' represents.","section":"Figure 6"},{"comment":"No significance tests are reported, and the test set is a single blind sample; the 1.4-point test gain (78.6 vs 77.2) should be presented as a point estimate rather than a definitive improvement.","section":"Section 4"}],"recommendation":"minor_revision","confidential_remarks":"I believe the paper is acceptable for a shared-task system-description venue. The stress-test concern about the test-set comparison is real but limited to a wording/scope issue; the development ablation and leaderboard rank are solid. The authors should qualify the test-side claim in a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, transparent shared-task system description. The methods are all off-the-shelf—SBERT/CLIP/BLIP-2 retrieval, log-probability confidence, ensembling—but the combination is sensible, the ablations are thorough, and the honesty is unusual. It deserves a serious referee, though the test-set comparison is a bit narrower than the text implies.\n\nWhat it does well: every individual configuration is reported on the development set, broken down by question type and figure type. The dev ablation shows the confidence-informed ensemble at 76.9 R1-F1 beats the best single model (75.0) and the question/figure type ensemble (76.6). Test results are consistent: 78.6 vs 77.7 vs 77.2. The code is public, the models are open-weights, and the limitations section is candid about possible contamination and about the unanswerable questions being an artifact of the dataset.\n\nThe soft spot the stress-test flagged is real but minor. On the test set, Table 2 reports only one individual configuration (InternVL 1s_q_img_f, BLIP2) at 77.2. So the claim \"confidence ensemble outperforms the best individual\" is only established on the development set; on test, it outperforms that one individual, not necessarily the best of all individuals. The paper's own wording in Section 4 is careful—it says \"compared to using only one approach\"—so the reader's summary oversells it slightly. This doesn't undermine the main result, because configuration selection is properly done on dev, and the test set is a single blind sample anyway.\n\nOther soft spots: no significance tests, and the gain over the question/figure ensemble is small (0.3 dev, 0.9 test). The 0.9 confidence threshold is chosen on dev without sensitivity analysis, though the calibration plot is helpful. The 3600 GPU hours is just a fact about cost.\n\nWho it's for: practitioners building MLLM systems for scientific VQA, and anyone curious whether confidence-based ensembling beats per-type routing. It is not a methods paper and does not claim to be.\n\nRecommendation: send it to peer review. It is a legitimate system description with reproducible experiments and honest reporting. I'd ask the authors to qualify the test-set comparison—state explicitly that only one individual configuration was evaluated on test, or give the full sweep—and to add a sentence about the lack of significance tests. That's a revision, not a rejection.","headline":"A clean, honest shared-task system paper whose test-set superlative is slightly oversold, but the dev-side ablation and public code make it worth refereeing.","tokens_in":13576,"tokens_out":3484,"would_cite":false,"duration_ms":35096,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A confidence-informed ensemble of two open-weights multimodal large language models, using few-shot example retrieval and a 90% confidence gate, outperforms every single configuration on the SciVQA 2025 scientific visual question…","keywords":["scientific visual question answering","multimodal large language models","few-shot example retrieval","confidence calibration","confidence-informed ensembling","question-type routing","SciVQA 2025"],"falsifier":"Compute the calibration curve of InternVL3-78B (1s_q_img_f, BLIP-2) on a held-out split of new scientific figures from a different distribution (e.g., different journals or figure types than the 65% line-chart-dominated training data). If the fraction of correct answers among instances with confidence >= 0.9 drops meaningfully below the 0.9 development value, or if the threshold selects far fewer or far more than half the instances, the ensemble's first-stage gating is broken and the overall score should fall back toward the single-model baseline.","tokens_in":12601,"feed_emoji":"📊","tokens_out":5915,"duration_ms":54402,"temperature":0.7,"pith_summary":"Can off-the-shelf multimodal large language models, with no fine-tuning, answer questions about scientific figures competitively? This paper argues yes, and it identifies where the gains come from: not from a better model, but from a confidence-informed ensemble that routes questions to different model-and-example configurations. The system accepts an answer whenever InternVL3-78B, prompted with one few-shot example retrieved by image similarity, reports a confidence of at least 0.9, which covers roughly half of all instances with about 91% accuracy on development data. All remaining questions are sent to the configuration that performed best for that question type, and the resulting ensemble beats every single configuration on both development and test sets. The paper's claim is that on this task, answer confidence—computed as the mean token log-probability—is calibrated enough to serve as a reliable gate.","feed_headline":"Trusting model confidence beats its best single model","feed_subtitle":"At 90% confidence, InternVL3-78B is right 91% of the time; remaining questions go to the best per-type model.","key_machinery":"The two-stage ensemble is carried by the confidence gate. The first stage computes answer confidence as the exponentiated mean log-probability of all generated answer tokens, and trusts InternVL3-78B (with one few-shot example retrieved by BLIP-2 embedding similarity, filtered by figure type) whenever that score is at least 0.9. The second stage, for the remaining instances, routes by question type to Pixtral-Large-2411 or InternVL3-78B with one or two few-shot examples retrieved by SBERT question similarity or by CLIP/BLIP-2 image-question similarity. The routing table is chosen by repeated 5-fold cross-validation on development data grouped by figure type and question type.","core_discovery":"The paper's central empirical discovery is that a two-stage ensemble built from two unmodified MLLMs outperforms any of its component configurations on the SciVQA 2025 task. The first stage trusts InternVL3-78B, prompted with one few-shot example retrieved by BLIP-2 image-question similarity and filtered by figure type, whenever its answer confidence—computed as the exponentiated mean log-probability of the generated answer tokens—is at least 0.9. This captures about half of all instances with roughly 91% ROUGE-1 F1 accuracy on the development set. The second stage sends every remaining instance to a configuration chosen per question type: Pixtral-Large-2411 with two few-shot examples for binary questions, Pixtral-Large-2411 with two examples retrieved by question similarity plus image similarity for infinite-answer-set questions, and InternVL3-78B with one question-similarity example for everything else. The ensemble reaches 76.9 ROUGE-1 F1 on development and 78.6 on the test set, beating the single best individual run (75.0 and 77.2) and placing the system third in the official ranking with an average F1 of 85.12 across ROUGE-1, ROUGE-L, and BERTScore.","pith_inferences":["The confidence-gating recipe could transfer to other VQA benchmarks with the same two models; a cheap test is to run the same two-stage ensemble on ChartQA or PlotQA and compare the gate's selected fraction and accuracy against development-calibrated expectations.","Because 65% of the development figures are line charts, the per-figure-type routing is effectively tuned for line charts; on a balanced figure distribution the chosen configurations would likely differ, and the ensemble's margin over the single model might shrink.","The shared task provides oracle metadata (caption, figure type, subfigure count) as input; a real-world deployment would need to predict these, so a useful extension is to measure the score drop when the metadata is replaced by an automatic figure-type classifier.","The paper compares BLIP-2 retrieval against question-similarity retrieval but does not isolate pure image-only retrieval; since BLIP-2 embeddings mostly reflect image content, a direct image-only baseline would test whether the question component adds any signal."],"forward_implications":["The confidence-informed ensemble reaches 78.6 ROUGE-1 F1 on the test set, improving on the best individual configuration by 1.4 points and on the shared-task baseline by about 4 points.","Providing one answerable and one unanswerable few-shot example improves the model's precision at identifying unanswerable questions, compared with using a single example.","Few-shot example retrieval by question similarity and by question-image similarity yield similar overall performance, so the simpler question-only retrieval is usually sufficient.","Performance varies strongly by question type, with binary and multiple-choice questions answered far better than infinite-answer-set questions about visual features, which remain the main weakness.","InternVL3-78B's confidence estimates are better calibrated than Pixtral-Large-2411's on this task, which is why the high-confidence gate is assigned to InternVL3-78B."],"supporting_citations":[{"why":"Defines the SciVQA shared task, its dataset, and the official metrics against which the system is scored.","marker":"(Borisova et al., 2025)"},{"why":"Provides InternVL3-78B, the model whose confidence scores form the first-stage gate and which handles most non-binary questions.","marker":"(Zhu et al., 2025)"},{"why":"Supplies BLIP-2 embeddings, used to retrieve the one few-shot example that yields the well-calibrated high-confidence configuration.","marker":"(Li et al., 2023)"},{"why":"Supplies SBERT question embeddings, used for the question-similarity few-shot retrieval in the second stage.","marker":"(Reimers and Gurevych, 2019)"},{"why":"Defines ROUGE-1 and ROUGE-L, the primary evaluation metrics for the system's development and test scores.","marker":"(Lin, 2004)"},{"why":"Defines BERTScore, the third metric in the official average F1.","marker":"(Zhang et al., 2020)"},{"why":"Provides one of the two source datasets (ACL-Fig) from which the shared task figures are drawn.","marker":"(Karishma et al., 2023)"},{"why":"Provides the other source dataset (SciGraphQA) from which the shared task figures are drawn.","marker":"(Li and Tajbakhsh, 2023)"}],"fun_headline_variants":["Confidence gate boosts ensemble past solo models","Trusting confident model wins SciVQA 2025","Ensemble with confidence check beats single models","Two-model ensemble outperforms best individual run","Confidence-based ensemble ranks third on SciVQA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The confidence scores of InternVL3-78B with BLIP-2-based example retrieval are calibrated well enough on the blind test set that a 0.9 threshold, tuned on development data, still selects about half of the instances with high accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Confidence gate boosts ensemble past solo models","Trusting confident model wins SciVQA 2025","Ensemble with confidence check beats single models","Two-model ensemble outperforms best individual run","Confidence-based ensemble ranks third on SciVQA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00015,"raw_usage":{"total_tokens":1189,"prompt_tokens":932,"completion_tokens":257,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":186}},"tokens_in":548,"tokens_out":257,"duration_ms":3648,"temperature":1.0,"reasoning_tokens":186,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:30:44.886816+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the calibration curve of InternVL3-78B (1s_q_img_f, BLIP-2) on a held-out split of new scientific figures from a different distribution (e.g., different journals or figure types than the 65% line-chart-dominated training data). If the fraction of correct answers among instances with confidence >= 0.9 drops meaningfully below the 0.9 development value, or if the threshold selects far fewer or far more than half the instances, the ensemble's first-stage gating is broken and the overall score should fall back toward the single-model baseline.","supporting_citations":[{"cited_title":"Lee Giles","cited_arxiv_id":null,"evidence_quote":"Provides one of the two source datasets (ACL-Fig) from which the shared task figures are drawn."}],"review_version":1}