{"id":"f2e456e9-e6b7-4fdc-a85a-a9b4797b69a2","arxiv_id":"2608.06931","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new expert-curated multimodal benchmark, SEE, shows the strongest AI models answer fewer than half of real-lab science questions correctly, and tool access brings only small gains.","lead":"The authors built a new benchmark, SEE, of 1,116 expert-written questions that test AI models on reading real experimental images and data from chemistry, biology, and materials science. The best model scores 48.7 percent, and giving models web search and code tools lifts it only to 52.7 percent, suggesting the hard part is reasoning within the evidence, not finding information.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No human-expert baseline or inter-annotator agreement is reported for SEE gold answers; if many questions are ambiguous or not answerable from the provided evidence, the 48.7% headline reflects benchmark flaws rather than a model reasoning deficit.","rationale":"The reader's weakest assumption—gold answer correctness and answerability from the provided evidence—is the same load-bearing concern I identify. The paper is otherwise careful: the LLM-as-a-judge pipeline is cross-validated with an alternative judge and manual inspection (Section B.4, Cohen's kappa ≥ 0.989), the public-subset reproducibility check rules out withheld-question artifacts (Tables 5–6), and the benchmark construction pipeline is described in unusual detail. These strengths make the benchmark credible, but they do not substitute for a human baseline or inter-annotator agreement on the gold answers. The central interpretive claim ('cannot reliably make justified and evidence-bounded inferences') depends on the benchmark being a fair test of expert reasoning from the given evidence. Without a human calibration, the low model scores could equally indicate that the questions are ambiguous, under-specified, or based on expert judgment that is not recoverable from the provided materials. The concrete test I propose—independent experts answering a sample from the public subset using only the model inputs—would directly settle this. If expert accuracy is high and agreement is strong, the conditional acceptance is justified and the benchmark's conclusions stand. If not, the headline number loses its interpretive force. Since this is an addressable empirical check and not a demonstrated internal error, the verdict remains CONDITIONAL; my analysis does not move it.","tokens_in":25572,"tokens_out":4968,"duration_ms":61703,"concrete_test":"Recruit three independent PhD-level experts per question for a stratified random sample of 100 public SEE questions (balanced across the 17 sub-discipline labels and question formats). Give each expert only the exact question text and image files that the models received (not the source DOI). Have each expert answer each question and also rate whether the provided evidence is sufficient and the answer is unique. Compute expert accuracy against the gold answers, Fleiss' kappa across experts, and the fraction of gold answers disputed by at least two experts. If expert accuracy is not substantially above the 48.7% model ceiling (e.g., <75%) or if >10% of gold answers are disputed, the headline result does not establish a model reasoning deficit.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that current MLLMs cannot reliably make justified and evidence-bounded inferences—rests entirely on the validity of the SEE gold answers and on each question being answerable from the provided text and images. The paper asserts in Section 3.2 and Figure 1 that questions pass 'information sufficiency' and 'answer uniqueness' checks by expert review, but it reports no quantitative support: no human expert accuracy on SEE, no inter-annotator agreement (e.g., Cohen's kappa or Fleiss' kappa), and no count of questions rejected or revised during expert QA. Without this, the headline 48.7% is uncalibrated: if experts also score low on the same evidence, or if experts disagree with the gold answers on a nontrivial fraction of questions, the benchmark is not a valid instrument for measuring scientific reasoning. The problem propagates to the secondary claims: the modality ablation ('models rarely acknowledge missing images') and the tool-augmented comparisons ('tools add only 4.6 pp') are both conditioned on the gold answers being correct and the images being truly indispensable. The public-subset reproducibility check (Table 5) confirms that the 67 withheld questions do not drive the results, but it does not validate the gold answers themselves. A benchmark with ambiguous or unsupported gold answers would produce exactly the observed pattern—low and tightly clustered model scores—without implying any deficiency in evidence-bounded reasoning. Thus the missing human calibration is the load-bearing weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces Science Edge Evaluation (SEE), a multimodal benchmark of 1,116 expert-curated questions in chemistry, biology, and materials science, grounded in peer-reviewed literature and real experimental practices. The authors evaluate 19 MLLMs under a strict binary scoring protocol and report that the best model reaches 48.7% accuracy, that general-purpose models outperform science-specialized ones on average, that models rarely acknowledge missing visual evidence in a text-only ablation (explicit acknowledgment rate 4.6%), and that a tool-augmented visual-agent setting improves the best accuracy to 52.7% while also introducing new error modes. The paper concludes that current MLLMs cannot reliably make justified, evidence-bounded inferences from experimental results, and that the main barrier is the management of multimodal evidence rather than knowledge access.","tokens_in":25777,"tokens_out":7663,"duration_ms":79234,"significance":"If the benchmark is valid, SEE would be a valuable complement to existing multimodal scientific benchmarks by focusing on experimental evidence, interdisciplinary reasoning, and evidence-bounded inference rather than exam-style knowledge recall. The evaluation pipeline has several strengths: strict binary scoring with exact-match multiple-choice and tolerance-based numerical answers; an LLM-judge sensitivity analysis reporting Cohen's kappa >= 0.99 and explicitly checking the primary judge against self-favoring bias; public-subset reproducibility tables showing the withheld 67 questions do not drive the results; and a text-only ablation in which every instance remains in the denominator. The tool-augmented visual-agent evaluation with human-in-the-loop trajectory attribution also goes beyond aggregate accuracy by diagnosing action selection, observation interpretation, evidence integration, and termination failures. However, the headline 'reliability' claim rests on the assumption that the gold answers are unambiguously correct and answerable from the provided evidence, which the paper has not yet established quantitatively.","major_comments":[{"comment":"The central claim that the best model reaches only 48.7% and therefore that MLLMs 'cannot reliably make justified and evidence-bounded inferences' is uncalibrated without a human-expert baseline and inter-annotator agreement on the gold answers. Section 3.2 and Figure 1 state that questions pass 'information sufficiency' and 'answer uniqueness' checks by expert review, but the paper reports no human expert accuracy on SEE, no Cohen's or Fleiss' kappa for the gold-answer construction, and no count of questions rejected or revised during the expert QA stages. Table 5 demonstrates that the 67 withheld questions do not drive the public-subset results, but it does not validate the gold answers themselves. If human experts also score near 50% on the same evidence, or if experts disagree with the gold answers on a nontrivial fraction of items, the observed clustering of model scores could reflect benchmark ambiguity rather than a model reasoning deficit. A human baseline, and preferably per-question ambiguity or difficulty annotations, is needed to support the 'reliability' interpretation.","section":"Section 3.2 / Figure 1 / Table 5"},{"comment":"The dataset description (Section 3.1) says the associated visual files 'may include both visual evidence presented with the question and images used for expert verification' and that 'reported image counts are not the exact number of figures displayed in the question itself.' Supplementary B.1 states that models are given 'the image inputs retained in the evaluation payload,' but the paper never specifies which images were actually shown to the evaluated models. If the evaluation payload includes the expert-verification images, then the information-sufficiency check (Figure 1) and the text-only ablation (Section 4.4) are not well defined; if it excludes them, the released dataset metadata is misleading. This ambiguity directly affects the multimodal evidence claim, because the models' inputs and the benchmark's evidence-bounding rationale depend on exactly which images constitute the question. Please clarify the relationship between released entries and evaluation payloads, and audit the released files so that each question's required visual evidence is unambiguous and reproducible.","section":"Section 3.1 / Supplementary B.1 / Figure 1"},{"comment":"The claim that 'general-purpose models outperform science-specialized models on average' is confounded by model scale and recency: the general-purpose set includes frontier proprietary models (e.g., GPT-5.6-Sol, Gemini 3.1 Pro) while the science-specialized set includes much smaller open models (e.g., S1-VL-32B-RL) and only one model near the top tier (Intern-S2 Preview-397B). The observed gap may be an artifact of model selection rather than evidence that domain specialization does not help. The paper's further conclusion that 'simply specializing a model through post-training or augmenting it with RAG does not necessarily lead to improved performance' goes beyond what an uncontrolled comparison of these particular models can support. Please add a matched or at least scale- and recency-constrained comparison, or explicitly reframe the claim as a property of the tested model set rather than a general property of domain specialization.","section":"Section 4.2 / Figure 4"}],"minor_comments":[{"comment":"With 1,116 questions, the 95% confidence interval around the 48.7% point estimate is roughly +/- 2.9 percentage points, so the statement 'no model exceeds 50%' would be more precise if accompanied by confidence intervals or a note about sampling uncertainty.","section":"Section 4.1 / Table 5"},{"comment":"Several sub-disciplines have very small denominators (e.g., Physiology has 19 questions, Immunology 32, Genetics 36), so the per-label accuracy differences may not be meaningful; please report uncertainty intervals or aggregate these tail labels.","section":"Table 8 / Supplementary B.9"},{"comment":"The 'missing-image acknowledgment' metric depends on how the text-only prompt is phrased and how acknowledgments are classified; please provide the exact text-only prompt and a few concrete examples of responses counted as acknowledgments versus generic refusals.","section":"Section 4.4 / Supplementary B.7"},{"comment":"The manual inspection of the judge disagreement cases is described qualitatively but no sample size or human-verification agreement rate is given; a small human-verified set of judge decisions would further strengthen the LLM-judge validation.","section":"Supplementary B.4"},{"comment":"The trajectory attribution procedure reports 411 improvements and 145 regressions attributed to tool use, but no inter-annotator agreement is provided for the human-in-the-loop categorization; please report agreement or a detailed resolution protocol for the expert-validated labels.","section":"Section 4.5 / Table 4"},{"comment":"The tool-augmented visual-agent evaluation uses each model's own official web-search and code-interpreter implementations, so the tool environments are not standardized across models; please state whether this was the case and discuss how it affects cross-model comparisons of tool gains.","section":"Supplementary B.5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a benchmark paper with a strong negative result about frontier MLLMs. The missing human-expert baseline is the main correctness risk; if the authors can supply it, the paper could become publishable. The authors are from Alibaba and Qwen Team and evaluate their own Qwen models as well as Seed models; the evaluation protocol is transparent, but an explicit conflict-of-interest statement or independent evaluation would help. The science-specialized comparison should be reframed or matched to avoid overclaiming."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thanks for the pass at the SEE paper. My take: this is a solid piece of benchmark construction, and the missing-image diagnostic is the part I'd steal. The 1,116 questions are drawn from real experimental practice, the scoring is strict, the judge-sensitivity analysis is careful (kappa over 0.99, self-favoring bias checked), and the public-subset reproducibility check on 1,049 questions shows the withheld 67 don't drive anything. The trajectory attribution for the tool-agent runs is a nice extra: they actually have experts verify whether a tool call caused a correction or a regression. That's more than most agent evaluations do.\n\nThe headline result, best model at 48.7%, is credible as a measurement of these models on this benchmark. But the stress-test note is right: there is no human-expert accuracy on the same 1,116 questions, and no inter-annotator agreement on the gold answers. Section 3.2 says experts check for 'information sufficiency' and 'answer uniqueness,' but no numbers come out of it. Without knowing how well domain experts do on the same evidence, the 48.7% could reflect benchmark ambiguity rather than a model reasoning deficit. That's the load-bearing weakness, and it's fixable. They should run a small expert panel on a sample (or the whole set), report agreement, and report human accuracy per discipline. That would also calibrate the text-only ablation: if experts also can't answer without images, the 4.6% acknowledgment rate is even more damning; if experts are split, the metric is muddier.\n\nMinor soft spots: the tool-agent comparison gives models web search, and since the sources are DOIs, the models might retrieve the original figures or text. The paper doesn't check for that. And the general-purpose-vs-science-specialized comparison rests on four specialized models, which is thin, though the paper at least flags it.\n\nNo fatal flaws. The math is simple accuracy, no fitted parameters, no circularity beyond the usual LLM-judge setup, which they handle well. I'd bring this to a reading group and would cite the missing-image diagnostic. It deserves serious peer review. The referee should ask for the human baseline and the retrieval-contamination check, but the core benchmark is worth engaging.","headline":"A solid, carefully built benchmark with a genuinely new diagnostic, but the headline 48.7% needs a human-expert baseline before the interpretation lands.","tokens_in":26490,"tokens_out":2526,"would_cite":true,"duration_ms":27871,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new benchmark of 1,116 expert questions from real laboratory practice shows that the strongest multimodal AI still answers fewer than half correctly, and the paper argues the bottleneck is managing visual evidence rather than accessing…","keywords":["Science Edge Evaluation","multimodal large language models","scientific benchmarking","evidence-bounded reasoning","experimental data interpretation","visual-agent evaluation","tool-augmented reasoning","hallucination diagnosis"],"falsifier":"Recruit a panel of expert chemists, biologists, and materials scientists, give them the same 1,116 SEE questions with the same images and strict binary scoring, and compare their accuracy with the models' 48.7%; expert accuracy at or below that level would indicate the benchmark itself is the problem, while expert accuracy near 80% or higher would confirm a genuine multimodal reasoning gap.","tokens_in":25313,"feed_emoji":"🔬","tokens_out":8722,"duration_ms":87288,"temperature":0.7,"pith_summary":"The paper introduces Science Edge Evaluation (SEE), a benchmark of 1,116 questions written by active researchers and grounded in peer-reviewed literature and real laboratory practice across chemistry, biology, and materials science. It sets out to test whether current multimodal language models can make justified, evidence-bounded inferences from experimental data rather than recalling scientific facts. On all 19 models evaluated, the best score is 48.7% and no model exceeds 50%; adding web search and a code interpreter raises the best score to 52.7% but leaves large errors intact. A sympathetic reader should care because the claim is diagnostic: the bottleneck is not knowledge access but the ability to manage multimodal evidence, which is exactly the capability AI would need to contribute new experimental insights rather than only explain established ones.","feed_headline":"Best AI scores 48.7% on real-lab science questions","feed_subtitle":"On 1,116 expert questions, the strongest model reaches 48.7%; tools add only 4.0 points.","key_machinery":"The central machinery is SEE itself: a multimodal benchmark whose questions combine text with the visual evidence of real laboratory work, including spectra, microscopy, gel images, diffraction patterns, and thermal curves, and whose scoring requires exact matches to expert answers. It carries the argument by measuring whether a model stays within the evidence, and by adding two controlled perturbations: a text-only ablation that removes the image to test whether a model notices missing evidence, and a tool-augmented visual-agent protocol that adds web search and code execution to test whether more information fixes the gap. The benchmark also records discipline labels and task types so that failures can be attributed to perception, inference, or evidence-management stages.","core_discovery":"On its own terms, the paper establishes that no current multimodal large language model reliably performs evidence-bounded scientific reasoning. SEE's headline result is that the strongest of 19 models reaches 48.7% accuracy, with a general-purpose average of 34.9% compared with 22.2% for science-specialized models, so domain specialization does not close the gap. Removing the image from questions makes every model worse, by 12.2 points on average, yet models explicitly acknowledge the missing image in only 4.6% of 20,088 text-only instances, indicating that they prefer to answer from linguistic priors rather than recognize insufficient evidence. In the tool-augmented visual-agent setting, web search and a code interpreter raise the best accuracy to 52.7%, but trajectory analysis attributes 145 new errors to tool use, including evidence-integration and action-selection failures. The paper concludes that the central limitation is managerial: models can gather more information, but they cannot yet keep tool-derived information within the boundaries of the original experimental evidence.","pith_inferences":["The paper does not report a human-expert baseline; running the same 1,116 questions through trained scientists under identical binary scoring would calibrate whether 48.7% is a model deficit or a benchmark difficulty.","A direct testable extension is to allow models to abstain: if evidence-boundedness is the bottleneck, a properly calibrated model should keep high accuracy while refusing questions whose evidence is insufficient, whereas current models would need to learn when to stop.","Because the benchmark shows more information does not reliably improve reasoning, a follow-up could deliberately inject conflicting or irrelevant retrieved evidence and measure whether models hold the original experimental evidence as the authority.","The authors point toward agentic discovery; extending SEE from fixed questions to open-ended tasks where a model proposes a conclusion from a new dataset and experts judge whether each step is justified would test the transition from recognition to derivation."],"forward_implications":["No current multimodal model can be trusted to draw conclusions from laboratory data without human oversight.","Science-specialized training and retrieval augmentation are not sufficient; general multimodal flexibility and evidence handling matter more.","Tool access helps only if the model can select actions, judge tool outputs, integrate them with the original observation, and stop when evidence is sufficient.","Scientific agent evaluation must score trajectory quality, including justification, uncertainty recognition, and revision, not just final-answer accuracy.","The low missing-image awareness rate implies a concrete risk in autonomous lab workflows: models will confidently answer from priors when data are absent."],"supporting_citations":[{"why":"The paper follows its image-ablation practice when checking that questions cannot be solved from text alone.","marker":"[7]"},{"why":"It supplies the nearest existing capability, scientific figure interpretation, that SEE claims to go beyond.","marker":"[11]"},{"why":"It represents a current chemistry-domain evaluation focused on knowledge and reasoning, providing the contrast for SEE's experimental-evidence focus.","marker":"[16]"},{"why":"It represents workflow-oriented biology evaluation that SEE extends to multimodal evidence.","marker":"[19]"},{"why":"It provides the closest prior multimodal chemistry-and-materials benchmark for comparison on experimental realism.","marker":"[20]"},{"why":"It supplies the standard scoring protocols that SEE adopts for exact-match and numerical-tolerance grading.","marker":"[47]"},{"why":"It supplies the LLM-as-a-judge method used for strict binary correctness scoring and judge-sensitivity analysis.","marker":"[63]"},{"why":"It reports general-purpose models outperforming specialized clinical systems, supporting the paper's analogous finding.","marker":"[64]"}],"fun_headline_variants":["Best AI hits only 48.7% on lab science benchmark","AI lab reasoning: best score 48.7%, tools add 4","General AI outperforms science-specialized models on lab questions","AI ignores missing images 95% of the time in text-only queries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the expert gold answers are correct and every one of the 1,116 questions is genuinely answerable from the supplied text and images, yet the paper reports no human-expert accuracy or inter-annotator agreement, so ambiguous or unsupported gold answers would make the low model scores reflect benchmark flaws rather than a reasoning deficit.","fun_headline_variants_meta":{"raw":{"variants":["Best AI hits only 48.7% on lab science benchmark","AI lab reasoning: best score 48.7%, tools add 4","General AI outperforms science-specialized models on lab questions","AI ignores missing images 95% of the time in text-only queries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000404,"raw_usage":{"total_tokens":2108,"prompt_tokens":957,"completion_tokens":1151,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":1074}},"tokens_in":573,"tokens_out":1151,"duration_ms":10061,"temperature":1.0,"reasoning_tokens":1074,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:07:23.393091+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recruit a panel of expert chemists, biologists, and materials scientists, give them the same 1,116 SEE questions with the same images and strict binary scoring, and compare their accuracy with the models' 48.7%; expert accuracy at or below that level would indicate the benchmark itself is the problem, while expert accuracy near 80% or higher would confirm a genuine multimodal reasoning gap.","supporting_citations":[{"cited_title":"SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific 19 Research","cited_arxiv_id":null,"evidence_quote":"It represents workflow-oriented biology evaluation that SEE extends to multimodal evidence."},{"cited_title":"Laurent, Joseph D","cited_arxiv_id":null,"evidence_quote":"It provides the closest prior multimodal chemistry-and-materials benchmark for comparison on experimental realism."},{"cited_title":"Official model page","cited_arxiv_id":null,"evidence_quote":"It reports general-purpose models outperforming specialized clinical systems, supporting the paper's analogous finding."}],"review_version":1}