Pith. sign in

REVIEW 4 major objections 4 minor 4 references

MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MME-SCI, a 1,019-question benchmark spanning four sciences, five languages, and three input modes, breaks the saturation of existing science benchmarks and exposes fine-grained model weaknesses.

desk verdict MME-SCI is a genuinely useful and hard multilingual science benchmark with a transparent pipeline, but its numbers rest on an unvalidated LLM-as-a-judge; worthwhile to review if that gets fixed. read the letter →

arxiv 2508.13938 v1 pith:RRMKZ7OL submitted 2025-08-19 cs.CL cs.CV

Jiacheng Ruan , Dan Jiang , Xian Gao , Ting Liu , Yuzhuo Fu , Yangyang Kang This is my paper
classification cs.CLcs.CV
keywords multimodallargelanguagemodelssciencereasoningbenchmarkmultilingualevaluationfine-grainedknowledgepointscross-lingualconsistencyimage-onlyvisualquestionansweringMLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MME-SCI is a new evaluation benchmark for multimodal large language models, built from 1,019 manually selected Chinese high-school science exam questions covering mathematics, physics, chemistry, and biology. Each question exists in five languages (Chinese, English, French, Spanish, and Japanese) and can be administered in three modes: text-only, image-only, or image-text hybrid, with every item tagged by a fine-grained knowledge point. The paper's central claim is that current science benchmarks have become saturated—strong models score in the 70–85% range—whereas on MME-SCI the best evaluated model answers only about 41% of image-only questions correctly, so the benchmark separates models that older tests cannot tell apart. Because the same question stem appears across languages and modalities, the benchmark also yields fine-grained diagnostics: cross-lingual consistency, modality-specific drops, and subject- and knowledge-point-level failure patterns. The authors argue this makes MME-SCI a more demanding and more informative yardstick for scientific reasoning in multimodal models.

What carries the argument

The load-bearing design is the triple-axis structure: the same 1,019 question stems are kept constant while (a) the language changes across Chinese, English, French, Spanish, and Japanese, (b) the input mode changes between text-only, image-only (a screenshot of the original problem), and image-text hybrid, and (c) each item carries a fine-grained knowledge-point tag (63 tags total) from high-school textbook systems. Constant stems make language and modality effects directly attributable, while the knowledge-point tags turn the benchmark into a diagnostic instrument rather than just a scoreboard. Scoring uses an LLM-as-a-judge protocol with language-specific templates, temperature zero, and

What would settle it

Have two independent human experts score a random sample of 100 MME-SCI items across all five languages; if their correctness labels disagree with the automatic judge on more than a small fraction (for example, more than 5%) of unambiguous items, the reported accuracy gaps and language-consistency findings are at least partly artifacts of the scoring method.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that a carefully curated set of high-school science questions can remain extremely hard for current multimodal models when it is rendered in non-English languages and in screenshot (image-only) form. MME-SCI contains 1,019 question-answer pairs, 83.3% drawn from 2025 mock exams to reduce overlap with training data, annotated with 63 fine-grained knowledge points across mathematics, physics, chemistry, and biology, and translated into Chinese, English, French, Spanish, and Japanese. Across 20 models, the best open-source model reaches 19.43% accuracy on Chinese image-only questions and the best closed-source model 41.32%; o4-mini scores 52.11%, 24.73%,

Load-bearing premise

The benchmark's difficulty and diagnostic findings rest on the assumption that each question has one correct, unambiguous answer, that the five translations preserve it, and that the automatic judge scores a correct response in any language about as reliably as a human expert would.

Editorial extensions

If this is right

  • If MME-SCI is taken as the yardstick, current multimodal models have a large headroom: even the best closed-source model answers under half of the image-only questions, so progress in scientific reasoning remains measurable for several generations of models.
  • The five-language versions imply that cross-lingual consistency is a separable capability: models that reason from meaning rather than surface language should answer the same stem consistently across languages, and current models mostly do not.
  • The three modality modes make visual perception and OCR a distinct bottleneck, especially for open-source models, whose image-only accuracy falls below their text-based accuracy while some closed-source models improve.
  • Reasoning-specialized model variants materially outperform their base counterparts, suggesting that chain-of-thought and reasoning training is the direction the benchmark rewards.
  • The fine-grained knowledge-point tags reveal concrete content areas where models fail, such as magnetic field questions, which can be used to target data collection and training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: translation quality is the main threat to the multilingual results; a useful follow-up is to have the five versions independently back-translated by native speakers and check whether the language-consistency gaps persist.
  • My inference: because 83.3% of the questions come from 2025 exam papers and are public, the benchmark's difficulty is time-sensitive; models trained on future instances of similar exam content could saturate it within a few years, so the construction pipeline is more durable than the specific item set.
  • My inference: the triple-axis structure could be extended to other disciplines and other modalities, such as audio, video, and diagrams, to test whether the observed cross-lingual and cross-modal gaps generalize beyond science exam questions.
  • My inference: the knowledge-point error profiles could support adaptive testing, where a model's weakest tagged concept is probed with more items, producing a capability map rather than a single score.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces MME-SCI, a multimodal science benchmark composed of 1,019 manually selected high-school-level question-answer pairs spanning mathematics, physics, chemistry, and biology, with 63 fine-grained knowledge-point labels. The benchmark supports five languages (Chinese, English, French, Spanish, Japanese) and three evaluation modes (text-only, image-only, image-text). The construction pipeline is described as four stages: expert-based difficulty filtering, GPT-4o/OCR digitization, translation into four additional languages, and a human post-audit. The authors evaluate 16 open-source and 4 closed-source MLLMs, reporting low accuracies (e.g., 35.62% for o4-mini on image-only Chinese questions), cross-lingual consistency analyses, knowledge-point-level accuracy, and an error taxonomy. The central claim is that MME-SCI breaks the performance saturation of existing scientific benchmarks and effectively distinguishes model capabilities.

Significance. If the reported accuracy numbers are trustworthy, the benchmark fills genuine gaps: it is multilingual, covers three input modalities, and provides fine-grained knowledge-point annotations, enabling diagnostic analyses that existing benchmarks do not offer. The manuscript is transparent about the curation pipeline and evaluates a broad set of models, and the data and evaluation code are publicly released. However, the headline conclusions are conditional on the correctness of an LLM-as-a-judge protocol that is not validated against a human gold set, and on translation/digitization quality that is asserted but not quantified. With those validations added, the benchmark would be a useful contribution; as submitted, the central claims are not yet fully supported.

major comments (4)
  1. [Evaluation details] The correctness labels underlying all of Table 2, Figure 4, and Figure 5 are produced by an 'LLM-as-a-Judge' protocol with 'different evaluation templates for various languages,' yet no validation of the judge is reported. No human-judge agreement, judge accuracy on a gold subset, or error-rate audit is given. Since most items are single/multiple-choice or fill-in-the-blank, a judge that is strict about formatting, cannot parse CoT outputs, or is biased by a language template can systematically distort accuracies. This is load-bearing for the claim that MME-SCI 'effectively distinguishes' models. Please report a judge-validation study on a random sample with human ground truth and report confidence intervals for model accuracies.
  2. [Data Curation Process (Post-Audit)] The multilingual versions D_en, D_fr, D_es, D_ja are produced by translating D_zh, and the Post-Audit step is described only as human cross-validation. No numbers are reported: how many OCR or translation errors were found, how many were fixed, or what inter-annotator agreement was. Translation errors directly affect the non-Chinese columns of Table 2 and the cross-lingual consistency analysis in Figure 4. Please provide a quantitative audit (e.g., error counts by language, agreement rates, or a back-translation/spot-check statistic) and make the audit results available.
  3. [Sample Filtering] The difficulty of MME-SCI is partly guaranteed by construction: the three volunteers were asked to 'filter out the questions they answered incorrectly or found confusing.' The low model accuracies therefore reflect an inclusion criterion, not a neutral measurement of benchmark difficulty. The claim that MME-SCI 'breaks the performance saturation' of existing benchmarks needs a comparison of the same models on existing benchmarks under the same evaluation/judge protocol, rather than raw accuracy differences across different datasets. Please add such a controlled comparison or soften the saturation claim.
  4. [Fine-grained knowledge points (Figure 5)] The 63 knowledge-point bins are derived from only 1,019 questions, so many per-KP sample sizes are small; for example, the abstract highlights 'Magnetic Field' with 33 questions, and several bins in Figure 5 are likely far smaller. No confidence intervals are reported for KP-level accuracies, so the fine-grained weaknesses may be noise. Please report per-KP sample sizes and confidence intervals, or aggregate KPs with fewer than a minimum number of questions.
minor comments (4)
  1. [Related works] There is a typo: 'LLaV A' should be 'LLaVA'. Also, the notation D_zh, D_img, D_en, D_fr, D_es, D_ja is used in Figure 3 and Table 2 but not formally defined in one place; please add a notation table or define all terms at first use.
  2. [Further Analysis] In 'Impacts of in-context learning setting', 'pre-knowledge descriptions' should be 'prior-knowledge descriptions', and 'Qwen2.5-VL-7/72B' is ambiguous; write 'Qwen2.5-VL-7B/72B'.
  3. [Figure 4] The caption describes green, red, and yellow segments, but the figure itself uses shades that may be difficult to distinguish in grayscale. Please increase contrast or use patterns, and also report the exact counts for each segment in the text.
  4. [Conclusion] The claim that MME-SCI is 'capable of effectively distinguishing the performance differences among various models' would benefit from a statistical test (e.g., bootstrap or McNemar) showing that pairwise model differences are not attributable to chance given the 1,019-item sample size.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark's challenge claim is an empirical measurement; the hard-question filtering is disclosed curation, not a derivation.

full rationale

The paper contains no equation-level derivation and no fitted parameters whose outputs are relabeled as predictions. The central claim that MME-SCI is challenging is supported by direct accuracy measurements on 20 MLLMs. The only element that superficially resembles circularity is the data curation: volunteers were asked to 'filter out the questions they answered incorrectly or found confusing' (Data Curation Process). Low model scores are therefore partly anticipated. However, this is a disclosed property of benchmark construction, not a logical reduction: model accuracies are not defined in terms of the filtering criterion, and the empirical results could in principle have been high if the models exceeded the volunteers. The multilingual versions, modality modes, and knowledge-point annotations are independent contributions, not derived from the measured accuracies. The LLM-as-a-judge protocol ('We adopt the ‘LLM-as-a-Judge’ paradigm ... and introduce different evaluation templates for various languages,' Evaluation details) is not validated against a human gold set, and the Post-Audit step does not report quantitative agreement; that is a reliability/validity concern about the scores, not a circularity in which the conclusion is forced by construction. No self-citation chain is load-bearing, and no specific reduction of the form 'output X is equivalent to input Y by definition' can be exhibited. Thus the paper is self-contained as an empirical benchmark study and receives a circularity score of 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The benchmark rests on data quality assumptions rather than mathematical axioms. There are no fitted parameters or newly postulated physical entities; the only 'new thing' is the dataset itself, which is the intended deliverable.

assumptions (4)
  • domain assumption Reference answers are correct
    Ground truth comes from manually filtered mock exams with human verification, but no inter-annotator agreement or independent validation is reported.
  • domain assumption Translations preserve semantic content and difficulty
    Five language versions were produced via GPT-4o and OCR with human review; fidelity across languages is not quantitatively evaluated.
  • domain assumption LLM-as-a-judge scoring is reliable
    The evaluation relies on an automated judge; the paper reports no comparison of judge decisions with human judgments.
  • domain assumption Mock exams are not in model training data
    The authors chose mock exams over Gaokao papers to reduce data leakage, but no contamination tests are provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/RRMKZ7OL

@misc{pith2026250813938,
  author       = {Pith},
  title        = {Pith review of: MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RRMKZ7OL}},
  note         = {Machine review of arXiv:2508.13938}
}
read the original abstract

Recently, multimodal large language models (MLLMs) have achieved significant advancements across various domains, and corresponding evaluation benchmarks have been continuously refined and improved. In this process, benchmarks in the scientific domain have played an important role in assessing the reasoning capabilities of MLLMs. However, existing benchmarks still face three key challenges: 1) Insufficient evaluation of models' reasoning abilities in multilingual scenarios; 2) Inadequate assessment of MLLMs' comprehensive modality coverage; 3) Lack of fine-grained annotation of scientific knowledge points. To address these gaps, we propose MME-SCI, a comprehensive and challenging benchmark. We carefully collected 1,019 high-quality question-answer pairs, which involve 3 distinct evaluation modes. These pairs cover four subjects, namely mathematics, physics, chemistry, and biology, and support five languages: Chinese, English, French, Spanish, and Japanese. We conducted extensive experiments on 16 open-source models and 4 closed-source models, and the results demonstrate that MME-SCI is widely challenging for existing MLLMs. For instance, under the Image-only evaluation mode, o4-mini achieved accuracy of only 52.11%, 24.73%, 36.57%, and 29.80% in mathematics, physics, chemistry, and biology, respectively, indicating a significantly higher difficulty level compared to existing benchmarks. More importantly, using MME-SCI's multilingual and fine-grained knowledge attributes, we analyzed existing models' performance in depth and identified their weaknesses in specific domains. The Data and Evaluation Code are available at https://github.com/JCruan519/MME-SCI.

Figures

Figures reproduced from arXiv: 2508.13938 by the authors.

Figure 1
Figure 1. Comparison between prevalent benchmarks (e.g., [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of MME-SCI. This benchmark consists of 1,019 manually and carefully selected questions, covering four [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Construction pipeline of MME-SCI, which consists of three stages. Overall, a total of approximately 300 person-days [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Consistency of responses across five languages. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Performance of o4-mini on Dzh with fine-grained knowledge points. We present the top-2 and bottom-2 knowledge points in terms of accuracy across 4 subjects. model capabilities improve, the number of samples with consistent correct responses shows an increasing trend. T…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

4 extracted references · 4 linked inside Pith

  1. [3]

    arXiv preprint arXiv:2501.03124

    PRMBench: A fine-grained and challenging bench- mark for process-level reward models. arXiv preprint arXiv:2501.03124. Team, K.; Du, A.; Yin, B.; Xing, B.; Qu, B.; Wang, B.; Chen, C.; Zhang, C.; Du, C.; Wei, C.; et al. 2025. Kimi-vl technical report. arXiv preprint arXiv:2504.07491. Wang, F.; Wang, H.; Guo, Z.; Wang, D.; Wang, Y .; Chen, M.; Ma, Q.; Lan, ...

    arXiv 2025

  2. [4]

    arXiv preprint arXiv:2305.12474

    Evaluating the performance of large language models on gaokao benchmark. arXiv preprint arXiv:2305.12474. Zheng, L.; Chiang, W.-L.; Sheng, Y .; Zhuang, S.; Wu, Z.; Zhuang, Y .; Lin, Z.; Li, Z.; Li, D.; Xing, E.; et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems , 36: 46595–46623. Zhu, J....

    arXiv 2023

  3. [2023]

    See https://vicuna

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) , 2(3): 6. Gu, J.; Jiang, X.; Shi, Z.; Tan, H.; Zhai, X.; Xu, C.; Li, W.; Shen, Y .; Ma, S.; Liu, H.; et al. 2024. A survey on llm-as-a- judge. arXiv preprint arXiv:2411.15594. Guo, M.-H.; Xu, J.; Zhang, Y .; Song, J.; Peng, H...

    arXiv 2023

  4. [2025]

    arXiv preprint arXiv:2505.02018

    R-bench: Graduate-level multi-disciplinary bench- marks for llm & mllm complex reasoning evaluation. arXiv preprint arXiv:2505.02018. Guo, X.; Zhang, R.; Duan, Y .; He, Y .; Zhang, C.; Liu, S.; and Chen, L. 2024. Drivemllm: A benchmark for spatial un- derstanding with multimodal large language models in au- tonomous driving. arXiv e-prints, arXiv–2411. Ha...

    arXiv 2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.