Pith. sign in

REVIEW 4 major objections 7 minor 41 references

Current vision-language models remain far behind expert understanding of materials phase diagrams, topping out at 0.407 BERTScore Recall on open-ended, literature-aligned description.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 05:59 UTC pith:BAHCDIZZ

load-bearing objection Careful phase-diagram VLM benchmark with real construction value; the “far behind expert” headline is only as strong as ROUGE/BERTScore on selective literature GT. the 4 major comments →

arxiv 2607.02934 v1 pith:BAHCDIZZ submitted 2026-07-03 cs.CV cond-mat.mtrl-scics.AI

MatPhaseBench: A Semantics-Guided Benchmark for Materials Phase Diagrams Understanding

Hanwen Wang , Sihan Liang , Zhiwei Liu , Yangang Wang , Wei Yan , Yuqin Liu , Zongguo Wang This is my paper
classification cs.CV cond-mat.mtrl-scics.AI
keywords materials phase diagramsvision-language modelsscientific image understandingmultimodal benchmarksthermodynamic reasoningphase equilibriaopen-ended evaluationimage-text alignment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Materials phase diagrams encode temperature, composition, phase stability, and transformation pathways; reading them well requires thermodynamic reasoning, not just visual recognition. This paper introduces MatPhaseBench, a 200-sample benchmark built from thousands of classical phase-equilibria papers, pairing diagrams with carefully matched and human-validated expert descriptions across 189 material systems. Models must produce free-form scientific descriptions guided by five semantic dimensions that experts actually use: materials system, diagram type, coverage, phase regions and boundaries, and invariant reactions. Across thirteen leading vision-language models, scores stay low: even the strongest model only modestly covers the information experts emphasize, and failures cluster around surface-level reading, missing domain focus, and weak comparison of multi-panel or composite diagrams. The result matters because if multimodal systems cannot interpret these core materials-science images at expert depth, they cannot yet be trusted for phase analysis or materials discovery workflows.

Core claim

On MatPhaseBench, current vision-language models remain substantially behind expert-level materials phase diagram understanding: they are largely limited to surface visual perception, lack deep reasoning grounded in thermodynamic mechanisms, show limited domain problem insight and expert analytical focus, and perform poorly at distinguishing fine-grained differences in composite or multi-diagram settings, with the best evaluated model reaching only 0.407 BERTScore Recall.

What carries the argument

MatPhaseBench: a publication-derived, open-ended phase-diagram understanding task built by two-stage image–text matching (direct plus associative), LLM-assisted description integration, multi-annotator human scoring on completeness, accuracy, and factuality, and five predefined semantic dimensions that both prompt the model and structure evaluation against literature ground truth.

Load-bearing premise

The central premise is that selectively written literature descriptions, after two-stage matching and human filtering, plus ROUGE and BERTScore overlap, are adequate proxies for expert-level phase diagram understanding.

What would settle it

Have materials-science experts independently score whether the best model outputs cover the same thermodynamic claims as the ground-truth texts; if expert agreement is high while automatic Recall stays near 0.4, or if a model reaches near-expert human ratings on the five semantic dimensions without lifting BERTScore/ROUGE, the claimed gap or the metrics’ link to it would be falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The manuscript introduces MatPhaseBench, a 200-sample open-ended VLM benchmark for materials phase-diagram understanding, built from 3,681 classical phase-equilibrium papers (Bulletin of Alloy Phase Diagrams / Journal of Phase Equilibria). Construction uses MinerU parsing, a two-stage direct+associative image–text match, Qwen-based phase-diagram screening and system parsing, GLM-assisted description integration, and three-annotator human QC on completeness/accuracy/factuality (IAA in Table III). Descriptions are organized into five semantic dimensions (materials system, diagram type, coverage, phase regions/boundaries, invariant reactions) that are also injected into zero-shot prompts. Thirteen closed- and open-source VLMs are evaluated with ROUGE-1/L and BERTScore (Recall/F1); the best model (Claude Opus 4.8) reaches only 0.407 BERTScore Recall. Qualitative cases argue that models stay at surface visual perception, lack thermodynamic and domain-focused reasoning, and struggle on composite diagrams. The authors position the resource as a research-grade platform for complex scientific image understanding and trustworthy multimodal AI in materials science.

Significance. If the construction and evaluation hold, MatPhaseBench fills a clear gap: existing scientific VLM benchmarks under-represent logically dense, mechanism-laden diagrams such as phase diagrams, and materials-specific resources rarely combine publication-derived multi-paragraph evidence with fully human-supervised open-ended targets. Strengths that should be credited include the end-to-end literature pipeline (two-stage matching that goes beyond captions), multi-annotator QC with reported IAA, explicit five-dimension taxonomy, public code/data, and an honest qualitative discussion of metric failure modes (Table V Case 4). The resource is therefore useful for the materials-informatics and multimodal-evaluation communities even if absolute “expert-level gap” numbers need tighter support. Significance is high for a specialized benchmark paper, contingent on firmer construct validity of the automatic scores used for the headline claim.

major comments (4)
  1. §IV.B, §V.B, and Table V Case 4: The headline claim that current VLMs remain “substantially behind expert-level understanding” (Abstract, §V.C, Conclusion) is load-bearing and rests almost entirely on ROUGE/BERTScore against a single literature-derived reference T*_i. The paper itself shows that scientifically valid model content can score low when narrative order, terminology, or granularity differ from the selective expert passage. Without a complementary human evaluation of model outputs (e.g., expert pairwise preference, dimension-level scoring of ˆT_i, or expert rewrites under the same dimension-guided prompt), the numeric ranking and the “behind expert” framing over-claim what lexical/semantic overlap can establish. Please add such human assessment on at least a substantial subset, or substantially temper the claim to “low overlap with literature-derived expert emphases under autom
  2. §IV.A and Fig. 3: Semantic dimensions are assigned from the ground-truth text and then embedded in the test prompt that produces ˆT_i, which is scored against that same T*_i. This design reduces free-form viewpoint drift but also steers models toward the reference’s information set, so high/low scores partly reflect instruction-following under a GT-derived checklist rather than unaided diagram understanding. Report an ablation with dimension-free prompts (and, if feasible, dimensions derived only from the image/caption without full GT) so that the contribution of dimension guidance versus pure visual–scientific reasoning can be separated.
  3. §III.E / Table IV and §V: Dimension coverage is uneven (e.g., phase-diagram coverage n=73, invariant reactions n=116 vs. materials system n=199), yet main results (Table VI) are only aggregate. Because the scientific claim emphasizes thermodynamic mechanism and invariant-reaction reasoning, please report per-dimension automatic scores (and any human scores) and, where possible, error breakdowns for multi-panel/composite samples called out in Case 3. Without this, the assertion of specifically weak mechanistic and composite-diagram understanding remains qualitative.
  4. §III.B–D and Table II: The final evaluation set is N=200 selected from 4,763 matched samples. Selection criteria (completeness/accuracy/factuality on a 0–3 scale) and the concentration of high scores that depresses Cohen’s κ (Table III) should be accompanied by a clearer statement of inclusion/exclusion rates, inter-annotator resolution procedure for the remaining 180 samples, and whether multi-panel vs. single-panel and binary vs. multicomponent systems are stratified. As written, it is hard to judge selection bias relative to the broader literature phase-diagram distribution the authors claim to represent.
minor comments (7)
  1. Table I: The dual-check notation (“Two ✓ symbols indicate…”) is easy to miss; make the two-stage matching column explicit (e.g., “2-stage”) so the comparison to MATRIX/SEM-VLM/Cephalo is immediately readable.
  2. Eq. (1)–(2): The arg max formulation is standard captioning notation but does not reflect the actual decoding settings (temperature=0, top-p=0.8, top-k=20). Align the formal task statement with the experimental protocol in §V.A.
  3. Table III: Briefly explain in the main text why Simple Agreement/Gwet’s AC1 are preferred over κ when score mass is concentrated at high quality, so readers do not misread the moderate κ as poor reliability.
  4. Table V Case 1–4: Phase-diagram images are described only in text; ensure the camera-ready version includes the actual figures (or high-resolution crops) so readers can verify the surface-vs-mechanism contrast.
  5. Related work / Table I: A short paragraph situating MatPhaseBench against chart/document VLM benchmarks (beyond materials) would help non-materials readers see what is unique about invariant reactions and thermodynamic constraints versus generic plot reading.
  6. Minor wording: Abstract and §I use “high-reliability” / “trustworthy” repeatedly; reserve stronger reliability language for claims backed by the human QC and any added human evaluation of model outputs.
  7. APPENDIX Table VII: Publish the full scoring rubrics and example scored items with the code release so the completeness/accuracy/factuality protocol is reproducible.

Circularity Check

0 steps flagged

No circular derivation: MatPhaseBench is an external VLM evaluation on a literature-built dataset; reported scores are not forced by self-definition or fitted identity.

full rationale

MatPhaseBench is a dataset-and-benchmark paper, not a first-principles derivation. The load-bearing claim—that current VLMs lag expert-level phase-diagram understanding (best BERTScore Recall 0.407 for Claude Opus 4.8)—is obtained by (i) mining 3681 papers, two-stage image–text matching, LLM-assisted description integration, and multi-annotator human QC to form 200 GT pairs, then (ii) zero-shot prompting 13 third-party VLMs with dimension-guided prompts and scoring outputs against those GTs with ROUGE-1/L and BERTScore. None of these steps equates a claimed prediction to its own fitted inputs by construction: model outputs are not parameters of the GT construction; semantic dimensions are labels of the reference text used only as prompt guidance and multi-label structure, not as a definition of the numeric scores; and no uniqueness theorem, self-cited ansatz, or renamed empirical law is used to force the ranking. Use of Qwen/GLM models in preprocessing does not circularly determine evaluation scores of the same families under human-validated GT. Construct-validity concerns about selective literature GT and lexical/semantic metrics (the paper’s own Case 4) are evaluation-proxy issues, not circularity of the derivation chain. Score 0 is therefore appropriate; steps empty.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

The paper is an empirical benchmark, not a first-principles theory paper. Load-bearing commitments are methodological: that literature text after matching/cleaning is the right supervision target; that ROUGE/BERTScore (especially Recall) measure coverage of expert-emphasized content; that the five predefined semantic dimensions capture the main axes of phase-diagram understanding; and that zero-shot dimension-conditioned generation is a fair probe of VLM capability. Free parameters are mostly decoding and annotation-scale choices. No new physical entities are postulated.

free parameters (4)
  • Final evaluation set size N=200 (from 4763 matched samples)
    Manual selection threshold that defines the official benchmark; different selection would change measured difficulty and coverage.
  • Annotation discrete scores {3,2,1,0} on completeness/accuracy/factuality
    Human quality gate for inclusion; scale and guidelines shape which samples enter MatPhaseBench.
  • VLM decoding: temperature=0, top-p=0.8, top-k=20
    Fixed inference hyperparameters that can affect open-ended description length and thus Recall vs F1 tradeoffs reported in Table VI.
  • Five predefined semantic dimensions used in prompts and annotation
    Hand-designed taxonomy that steers both ground-truth labeling and model generation; not derived from a uniqueness theorem.
axioms (4)
  • domain assumption Literature-derived, human-validated descriptions are a valid reference for expert-level phase diagram understanding in open-ended generation.
    Core evaluation premise in Task Formulation and Evaluation Methodology; authors note expert texts are selective, which weakens but does not remove the assumption.
  • domain assumption BERTScore and ROUGE (especially Recall) adequately quantify how much expert-emphasized scientific content a model covers.
    Used as primary automatic metrics in Eqs. (1)–(2) and Table VI; partially undermined by the paper’s own Case 4 critique.
  • domain assumption Two-stage direct + associative matching preserves relevant semantic context without introducing unsupported claims after LLM cleaning.
    Dataset Construction pipeline; quality depends on MinerU parsing, Qwen classification, and GLM integration fidelity.
  • standard math Standard text-overlap and embedding-similarity evaluation practices from NLP transfer to materials phase-diagram description.
    ROUGE and BERTScore are applied as conventional generation metrics without new theoretical derivation.
invented entities (2)
  • MatPhaseBench (200-sample evaluation set + construction pipeline) independent evidence
    purpose: Provide a publication-derived, human-supervised open-ended benchmark for VLM phase-diagram understanding.
    Primary contribution; independent evidence is the released dataset/code claim and documented selection from classical journals.
  • Five semantic dimensions taxonomy for phase-diagram description no independent evidence
    purpose: Guide prompts and fine-grained evaluation beyond free-form captioning.
    Predefined by authors from literature content patterns (Table IV); useful but not an externally measured physical quantity.

pith-pipeline@v1.1.0-grok45 · 18066 in / 3417 out tokens · 33131 ms · 2026-07-12T05:59:07.517666+00:00 · methodology

0 comments
read the original abstract

Materials phase diagrams are a core knowledge representation in materials science, encoding temperature,composition, phase stability, and phase transformation pathways, with their full understanding requiring thermodynamic mechanism analysis and scientific reasoning. Although VLMs have shown promise in scientific image understanding, their systematic evaluation on such logically complex images demanding deep mechanistic interpretation remains limited, and phase diagrams provide a challenging testbed for this purpose. We introduce MatPhaseBench, a high-quality, high-reliability benchmark for complex scientific image understanding, focused on materials phase diagrams. MatPhaseBench is constructed from 3681 papers in classical materials science journals, from which 200 high-quality diagram-text pairs were selected, covering 189 material systems and 70 elements. The benchmark has three key features: (1)targeting complex scientific image understanding-it moves beyond simple objective tests to open-ended tasks requiring deep comprehension; (2)comprehensive image-text alignment-semantic information associated with images is fully preserved during literature mining and matching; (3) high-quality human-supervised text acquisition-all descriptions undergo strict manual validation. Experimental results show that current VLMs remain substantially behind expert-level understanding: they are largely limited to surface visual perception, lack deep reasoning grounded in thermodynamic mechanisms, have limited domain awareness and expert analytical experience, and perform poorly in distinguishing fine-grained differences in composite or multi-diagram settings. Overall, MatPhaseBench constitutes a challenging research-grade benchmark, providing a foundational platform for complex scientific image understanding, phase diagram analysis, and trustworthy multi-modal AI in science.

Figures

Figures reproduced from arXiv: 2607.02934 by Hanwen Wang, Sihan Liang, Wei Yan, Yangang Wang, Yuqin Liu, Zhiwei Liu, Zongguo Wang.

Figure 1
Figure 1. Figure 1: An example of material phase diagram. I. INTRODUCTION Materials phase diagrams, as shown in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Data processing pipeline. and labels—with materials-science semantics such as phase stability, thermodynamic constraints, and invariant reactions. To address this challenge, we propose MatPhaseBench, a high￾quality, trustworthy, and open-ended benchmark for evaluat￾ing VLM understanding of materials phase diagrams. Mat￾PhaseBench is constructed from classic phase-diagram liter￾ature published between 1980 … view at source ↗
Figure 3
Figure 3. Figure 3: Schematic diagram of phase diagram understanding task. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 6 linked inside Pith

  1. [1]

    Al-Sc (aluminum-scandium),

    H. Okamoto, “Al-Sc (aluminum-scandium),”Journal of Phase Equilib- ria, vol. 12, no. 5, pp. 612–613, 1991

    1991

  2. [2]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and F.-F. Li, “Imagenet: A large-scale hierarchical image database,” in2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255, 2009

    2009

  3. [3]

    GLUE: A multi-task benchmark and analysis platform for natural language understanding,

    A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” inProceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP, pp. 353–355, 2018

    2018

  4. [4]

    Vqa: Visual question answering,

    S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh, “Vqa: Visual question answering,” inProceedings of the IEEE International Conference on Computer Vision, pp. 2425–2433, 2015

    2015

  5. [6]

    Micro-bench: A microscopy benchmark for vision- language understanding,

    A. Lozano, J. Nirschl, J. Burgess, S. R. Gupte, Y . Zhang, A. Unell, and S. Yeung-Levy, “Micro-bench: A microscopy benchmark for vision- language understanding,”Advances in Neural Information Processing Systems, vol. 37, pp. 30670–30685, 2024

    2024

  6. [7]

    Defining the boundaries: Challenges and advances in identifying cells in microscopy images,

    N. Gogoberidze and B. A. Cimini, “Defining the boundaries: Challenges and advances in identifying cells in microscopy images,”Current Opinion in Biotechnology, vol. 85, p. 103055, 2024

    2024

  7. [8]

    Microvqa: A multimodal reasoning benchmark for microscopy-based scientific research,

    J. Burgess, J. J. Nirschl, L. Bravo-S ´anchez, A. Lozano, S. R. Gupte, J. G. Galaz-Montoya, Y . Zhang, Y . Su, D. Bhowmik, Z. Coman, et al., “Microvqa: A multimodal reasoning benchmark for microscopy-based scientific research,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19552–19564, 2025

    2025

  8. [9]

    Evochart: A benchmark and a self-training approach towards real- world chart understanding,

    M. Huang, H. Lai, X. Zhang, W. Wu, J. Ma, L. Zhang, and J. Liu, “Evochart: A benchmark and a self-training approach towards real- world chart understanding,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 4, pp. 3680–3688, 2025

    2025

  9. [10]

    Encqa: Benchmarking vision- language models on visual encodings for charts,

    K. Mukherjee, D. Ren, and D. Moritz, “Encqa: Benchmarking vision- language models on visual encodings for charts,”IEEE Transactions on Visualization and Computer Graphics, 2025

    2025

  10. [11]

    Docvlm: Make your VLM an efficient reader,

    M. S. Nacson, A. Aberdam, R. Ganz, E. B. Avraham, A. Golts, Y . Kittenplon, S. Mazor, and R. Litman, “Docvlm: Make your VLM an efficient reader,” inProceedings of the Computer Vision and Pattern Recognition Conference, pp. 29005–29015, 2025

    2025

  11. [12]

    Towards natural language-based document image retrieval: New dataset and benchmark,

    H. Guo, X. Qin, J. Y . O. Yang, P. Zhang, G. Zeng, Y . Li, and H. Lin, “Towards natural language-based document image retrieval: New dataset and benchmark,” inProceedings of the Computer Vision and Pattern Recognition Conference, pp. 29722–29732, 2025

    2025

  12. [13]

    OpenAI GPT- 5 system card,

    A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. J. Ostrow, A. Ananthram, et al., “OpenAI GPT- 5 system card,”arXiv preprint arXiv:2601.03267, 2025

    Pith/arXiv arXiv 2025

  13. [14]

    Claude Opus 4.8,

    Anthropic, “Claude Opus 4.8,” 2025. [Online]. Available: https://www. anthropic.com/news/claude-opus-4-8. Accessed: Jun. 6, 2026

    2025

  14. [15]

    Gemini 3.1 Pro model card,

    Google DeepMind, “Gemini 3.1 Pro model card,” 2025. [Online]. Avail- able: https://deepmind.google/models/model-cards/gemini-3-1-pro/. Ac- cessed: Jun. 6, 2026

    2025

  15. [17]

    Mmmu-pro: A more robust multi- discipline multimodal understanding benchmark,

    X. Yue, T. Zheng, Y . Ni, Y . Wang, K. Zhang, S. Tong, Y . Sun, B. Yu, G. Zhang, H. Sun, et al., “Mmmu-pro: A more robust multi- discipline multimodal understanding benchmark,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15134–15186, 2025

    2025

  16. [18]

    Gpqa: A graduate-level Google-proof Q&A benchmark,

    D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y . Pang, J. Dirani, J. Michael, and S. R. Bowman, “Gpqa: A graduate-level Google-proof Q&A benchmark,”arXiv preprint arXiv:2311.12022, 2023

    Pith/arXiv arXiv 2023

  17. [19]

    FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI,

    E. Glazer, E. Erdil, T. Besiroglu, D. Chicharro, E. Chen, A. Gunning, C. F. Olsson, J.-S. Denain, A. Ho, E. de Oliveira Santos, et al., “FrontierMath: A benchmark for evaluating advanced mathematical reasoning in AI,”arXiv preprint arXiv:2411.04872, 2024

    Pith/arXiv arXiv 2024

  18. [20]

    GeneBench: Assessing AI agents for multi-stage inference problems in genomics and quantitative biology,

    J. Li and A. Ho, “GeneBench: Assessing AI agents for multi-stage inference problems in genomics and quantitative biology,”bioRxiv, pp. 2026–04, 2026

    2026

  19. [21]

    SciCode: A research coding benchmark curated by scientists,

    M. Tian, L. Gao, S. D. Zhang, X. Chen, C. Fan, X. Guo, R. Haas, P. Ji, K. Krongchon, Y . Li, et al., “SciCode: A research coding benchmark curated by scientists,”Advances in Neural Information Processing Systems, vol. 37, pp. 30624–30650, 2024

    2024

  20. [22]

    Qwen3.6,

    “Qwen3.6,” GitHub. [Online]. Available: https://github.com/QwenLM/ Qwen3.6

  21. [23]

    GLM-4.6V ,

    “GLM-4.6V ,” Z AI Blog. [Online]. Available: https://z.ai/blog/glm-4.6v

  22. [24]

    LLaMA-4,

    “LLaMA-4,” Meta AI. [Online]. Available: https://www.llama.com/docs/ model-cards-and-prompt-formats/llama4/

  23. [25]

    MA- TRIX: A Multimodal Benchmark and Post-Training Framework for Materials Science,

    D. McGrath, C. Chong, R. Kulkarni, G. Ceder, and A. Kolluru, “MA- TRIX: A Multimodal Benchmark and Post-Training Framework for Materials Science,”arXiv preprint arXiv:2602.00376, 2026

    arXiv 2026

  24. [26]

    MicroscopyGPT: Generating atomic-structure captions from microscopy images of 2D materials with vision-language trans- formers,

    K. Choudhary, “MicroscopyGPT: Generating atomic-structure captions from microscopy images of 2D materials with vision-language trans- formers,”The Journal of Physical Chemistry Letters, vol. 16, no. 27, pp. 7028–7035, 2025

    2025

  25. [27]

    A visual language model enabling intelligent nanomaterial scanning electron micrograph annotation,

    Y . Cai and H. Wang, “A visual language model enabling intelligent nanomaterial scanning electron micrograph annotation,”Nanoscale, vol. 17, no. 43, pp. 25136–25151, 2025

    2025

  26. [28]

    Cephalo: Multi-Modal Vision-Language Models for Bio-Inspired Materials Analysis and Design,

    M. J. Buehler, “Cephalo: Multi-Modal Vision-Language Models for Bio-Inspired Materials Analysis and Design,”Advanced Functional Materials, vol. 34, no. 49, p. 2409531, 2024

    2024

  27. [29]

    Mmsci: A multimodal multi-discipline dataset for PhD-level scientific comprehension,

    Z. Li, X. Yang, K. Choi, W. Zhu, R. Hsieh, H. Kim, J. H. Lim, S. Ji, B. Lee, X. Yan, et al., “Mmsci: A multimodal multi-discipline dataset for PhD-level scientific comprehension,” inAI for Accelerated Materials Design-Vienna 2024, 2024

    2024

  28. [30]

    Mac: A live benchmark for multimodal large language models in scientific understanding,

    M. Jiang, J. Gao, J. Zhan, and D. Wang, “Mac: A live benchmark for multimodal large language models in scientific understanding,”arXiv preprint arXiv:2508.15802, 2025

    Pith/arXiv arXiv 2025

  29. [31]

    Mme-sci: A comprehensive and challenging science benchmark for multimodal large language models,

    J. Ruan, D. Jiang, X. Gao, T. Liu, Y . Fu, and Y . Kang, “Mme-sci: A comprehensive and challenging science benchmark for multimodal large language models,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 11, pp. 8760–8768, 2026

    2026

  30. [32]

    Projudge: A multi-modal multi-discipline benchmark and instruction-tuning dataset for MLLM-based process judges,

    J. Ai, P. Zhou, Z. Xu, M. Li, F. Zhang, Z. Li, J. Sun, Y . Feng, B. Huang, Z. Wang, et al., “Projudge: A multi-modal multi-discipline benchmark and instruction-tuning dataset for MLLM-based process judges,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4681–4690, 2025

    2025

  31. [33]

    mmJEE-Eval: A bilingual multimodal benchmark for evaluating scientific reasoning in vision-language mod- els,

    A. Mukherjee and S. Ghosh, “mmJEE-Eval: A bilingual multimodal benchmark for evaluating scientific reasoning in vision-language mod- els,” inProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pp. 2268– 2290, 2025

    2025

  32. [34]

    Mdk12-bench: A multi-discipline benchmark for evaluating reasoning in multimodal large language models,

    P. Zhou, X. Peng, F. Zhang, Z. Xu, J. Ai, Y . Qiu, W. Zhao, J. Song, C. Li, W. Tang, et al., “Mdk12-bench: A multi-discipline benchmark for evaluating reasoning in multimodal large language models,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 34, pp. 28982–28990, 2026

    2026

  33. [35]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI,

    X. Yue, Y . Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y . Sun, et al., “Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556–9567, 2024

    2024

  34. [36]

    V olumes and issues,

    Springer, “V olumes and issues,”Journal of Phase Equilibria, Springer Nature Link. [Online]. Available: https://link.springer.com/journal/12385/volumes-and-issues. [Accessed: Jun. 6, 2026]

    2026

  35. [37]

    Mineru: An open-source solution for precise document content extraction,

    B. Wang, C. Xu, X. Zhao, L. Ouyang, F. Wu, Z. Zhao, R. Xu, K. Liu, Y . Qu, F. Shang, et al., “Mineru: An open-source solution for precise document content extraction,”arXiv preprint arXiv:2409.18839, 2024. [Online]. Available: https://arxiv.org/abs/2409.18839

    Pith/arXiv arXiv 2024

  36. [38]

    GLM-5.1,

    Z.ai, “GLM-5.1,” Z.ai Blog, Apr. 2026. [Online]. Available: https://z.ai/blog/glm-5.1

    2026

  37. [39]

    Computing inter-rater reliability and its variance in the presence of high agreement,

    K. L. Gwet, “Computing inter-rater reliability and its variance in the presence of high agreement,”British Journal of Mathematical and Statistical Psychology, vol. 61, no. 1, pp. 29–48, 2008

    2008

  38. [40]

    A coefficient of agreement for nominal scales,

    J. Cohen, “A coefficient of agreement for nominal scales,”Educational and Psychological Measurement, vol. 20, no. 1, pp. 37–46, 1960

    1960

  39. [41]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” inText Summarization Branches Out, pp. 74–81, 2004

    2004

  40. [42]

    Bertscore: Evaluating text generation with BERT,

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, “Bertscore: Evaluating text generation with BERT,”arXiv preprint arXiv:1904.09675, 2019

    Pith/arXiv arXiv 1904

  41. [43]

    GLM-5V-Turbo,

    Z.AI, “GLM-5V-Turbo,”Z.AI Developer Documentation, [Online]. Available: https://docs.z.ai/guides/vlm/glm-5v-turbo. [Accessed: Jun. 6, 2026]. APPENDIX TABLE VII ANNOTATIONGUIDELINES OFMATPHASEBENCH Criterion Annotation Dimension Completeness Completeness of basic image information Completeness of expert empirical description Completeness of expert reasonin...

    2026