Pith. sign in

REVIEW 5 major objections 4 minor 41 references

This paper claims to resolve the Scientific Benchmarking Trilemma with SPM-Bench, a fully automated PhD-level multimodal benchmark for microscopy, and SIP-F1, a metric that penalizes speculative guessing.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 20:31 UTC pith:5DXYWQPW

load-bearing objection A credible first SPM-specific multimodal benchmark with a detailed automated pipeline, but the central claim of PhD-level rigor collapses on inspection: the labels are VLM-generated with no human verification, the generator is also an evaluated model, and the headline bimodal effect is built into the metric. the 5 major comments →

arxiv 2602.22971 v2 pith:5DXYWQPW submitted 2026-02-26 cs.AI

SPM-Bench: Benchmarking Large Language Models for Scanning Probe Microscopy

Peiyao Xiao , Xiaogang Li , Xinyi Gao , Chengliang Xu , Ben Wang , Zichao Chen , Zeyu Wang , Lin Qu
show 2 more authors
Bing Zhao Hu Wei
This is my paper
classification cs.AI
keywords SPM-Benchscanning probe microscopymultimodal benchmarkautomated data synthesisSIP-F1large language modelsscientific reasoningdata contamination
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

SPM-Bench is a proposed benchmark of 2,703 image-question pairs for scanning probe and electron microscopy, built almost entirely by automated extraction from recent peer-reviewed papers and preprints. The authors claim this pipeline solves the Scientific Benchmarking Trilemma: it reaches PhD-level difficulty without human expert labels, avoids contamination by using recent literature, and evaluates models with SIP-F1, a metric that harshly penalizes wrong extra choices. If true, the benchmark would give the field a low-cost, contamination-resistant way to measure whether AI models can reason about atomic-scale and nanoscale physics, not just recognize images. The paper also claims that the gap between permissive and penalty-based scores exposes model 'personalities' — conservative, aggressive, gambler, or wise — and that current frontier models still struggle in defect and quantum-material domains.

Core claim

The authors claim that a fully automated, near-zero-human pipeline can distill scanning probe microscopy knowledge from published figures, captions, and discussion snippets into valid expert-level question-answer pairs. The pipeline uses an Anchor-Gated Sieve to locate pages containing both images and figure captions, a cloud vision-language model that returns only an 'llbox' bounding box so local high-resolution cropping stays cheap, and a question generator that must first write a chain-of-thought and scoring rubric before producing multi-select questions. An advisory model audits each pair. The evaluation metric SIP-F1 assigns 1.0 only for exact matches, otherwise λ times an F1 score with

What carries the argument

The load-bearing components are: (1) Anchor-Gated Sieve (AGS), a local page-layout filter that triggers extraction only when image blocks and figure-caption anchors co-occur, filtering out most text-heavy pages; (2) llbox-guided cropping, where a cloud vision-language model outputs only normalized spatial coordinates that the local engine maps back to 300-DPI crops, decoupling visual evidence from surrounding text and cutting token cost; (3) a question generator that is forced to articulate a hidden reasoning path and scoring rubric before producing questions, followed by an advisory-model audit; and (4) SIP-F1, defined as 1.0 on exact match and otherwise λ times an F1 score with precision d

Load-bearing premise

The benchmark's 'correct answers' were written by an AI model that reads images and text and were checked only by a second such model, never by a human expert; if those answers are wrong or ambiguous, everything measured by the benchmark is untrustworthy.

What would settle it

Have several human scanning probe microscopy experts independently answer a random sample of 100 SPM-Bench questions using the same figure-plus-caption inputs, then measure agreement with the AI-generated gold answers; if agreement falls well below the range expected for expert benchmarks, or if a text-only model given only the caption matches a multimodal model given the figure, the benchmark's visual-reasoning claim is falsified. Separately, check whether any 2023–2025 source papers appear in the evaluated models' training corpora; if they do, the contamination-free claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the pipeline works as claimed, expert-level scientific benchmarks can be produced automatically from the literature at a fraction of manual curation cost.
  • SIP-F1 makes 'select all options' and other recall-gaming strategies score worse than a conservative partial answer, so evaluations are more robust to speculative behavior.
  • The reported hierarchy — exact match ≤ SIP-F1 ≤ partial credit — provides a diagnostic window: models with a wide gap between partial credit and SIP-F1 are the ones trading precision for recall.
  • The benchmark's temporal window (2023–2025) is intended to reduce data contamination; if the separation holds, scores reflect reasoning rather than memorization.
  • The uniform bottlenecks in defect engineering and topological states identify concrete weaknesses for future model training and benchmark design.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Putting the same automated pipeline on other image-rich disciplines (cryo-EM, astronomy, materials failure analysis) would be the most direct test of generality; the VLM-label validity risk would travel with it.
  • A cheap diagnostic implied by the paper's own data: if instructing a model to abstain on low-confidence options raises SIP-F1 without changing exact-match accuracy, the 'personality' gap is largely a calibration problem rather than a knowledge gap.
  • The confidence and token analysis suggests the next stress test is not harder questions but adversarial ones — figures from papers outside the 2023–2025 window or with deliberately misleading captions — to see whether 'wise' behavior is stable under distribution shift.
  • Because the benchmark's labels come from the same family of models being evaluated, a future version with a small human-verified core would let the field distinguish benchmark quality from model quality more cleanly.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper presents SPM-Bench, a claimed PhD-level multimodal benchmark for scanning probe microscopy, built by an automated pipeline that extracts figures and captions from arXiv and journal papers (2023–2025), crops sub-figures via a VLM's coordinate outputs, and generates multi-select QA pairs with a VLM generator plus a second VLM advisory audit. The authors introduce a new scoring rule, SIP-F1, that gives 1.0 only to exact matches and otherwise multiplies a precision-recall F1 by λ=0.6 with a false-positive penalty Γ=6. They report rankings of nine frontier models under EM, SPC, and SIP-F1, and interpret the resulting gaps as 'model personalities' (Conservative, Aggressive, Gambler, Wise). The paper also claims temporal separation to avoid contamination and generalizable automatic scientific data synthesis.

Significance. If the benchmark were valid and the labels truly PhD-level, SPM-Bench could fill a real gap: there is currently no careful multimodal evaluation for SPM/AFM/STM reasoning, and an automated pipeline with a strict anti-gaming metric would be useful. The engineering details of the pipeline (AGS filtering, llbox-guided cropping, local high-resolution execution) are concrete and plausible, and the idea of penalizing over-selection in scientific QA is reasonable. However, the paper's central claims are not supported by the evidence. The ground-truth labels are generated entirely by an evaluated VLM with no human verification; the SIP-F1 metric's bimodal gap is built into its definition rather than discovered; temporal separation is asserted but not demonstrated; and no dataset or code is released. These issues affect every reported score, ranking, and 'personality' conclusion, so the contribution as presented cannot be independently checked or used.

major comments (5)
  1. [§2.3–2.4, Appendix D] The benchmark's ground truth is generated entirely by gemini-3-flash-preview from figure crops, captions, and discussion snippets, and then 'audited' by another VLM. There is no human expert verification, no inter-annotator agreement, no error-rate estimate, and no release of the questions/answers beyond four Appendix D examples. Since gemini-3-flash-preview is itself one of the evaluated models (Figs. 7–8), construction and evaluation are not independent. Every EM/SPC/SIP-F1 score and every 'personality' classification is computed against these unvalidated labels. Unless the authors supply a human expert audit (e.g., 100+ items with agreement statistics) and open the dataset, the claimed 'PhD-level' authority is unsupported.
  2. [§3.3, Eq. (3)] Equation (3) assigns 1.0 only to exact matches and caps every non-perfect answer at λ×F1-Gamma with λ=0.6. Hence the score distribution must have a gap between 1.0 and ≤0.6 regardless of model behavior. The 'bimodal distribution' and the tier structure described in §3.3 and interpreted in §4.2 are built into the metric, not discovered from the data. The paper should instead report continuous metrics and treat SIP-F1 as a deliberately harsh scoring rule; as written, the 'scientific reasoning barrier' and the 'personality taxonomy' are artifacts of the score transformation.
  3. [§2.1, Table 1] The abstract and introduction claim 'rigorous temporal separation to ensure data purity,' but the paper only states that sources were published 2023–2025. There is no list of exact source papers, no per-question timestamp, no check against the training data of the evaluated models, and no contamination analysis. A model trained on the same papers can trivially reproduce the 'correct' answers from the caption text, which is especially concerning because the generator is also an evaluated model. Without demonstrating temporal separation or releasing the source list, the 'uncontaminated' claim is not established.
  4. [§4.1, Fig. 7 vs text] The text states 'The EM scores (ranging from 0.43 to 0.68)' but Figure 7 reports qwen3.5-plus EM=0.832, well outside this range. Similarly, the same model's SPC/SIP-F1 values in Figure 7 (0.865/0.881) are not consistent with the narrative that all models fall below 0.68. This internal numerical inconsistency in the headline results undermines confidence in the reported rankings. All tables and figures need careful reconciliation.
  5. [Entire manuscript] No dataset or code is released, and no URL is provided. For a paper whose central artifact is a benchmark (with a new metric), readers cannot inspect the questions, answers, rubrics, prompts, or scoring code. The Appendix D examples are illustrative only. Without release, the benchmark cannot be used or independently verified. This is not a presentation issue; it precludes the claimed contribution.
minor comments (4)
  1. [§2.3] Typo: 'gemini-3-flash-previrew' should be 'gemini-3-flash-preview'. Also, §2.2 contains 'w pping labels', and Table 2 formats 'SPCSIP' without a separator.
  2. [§4.3] The paragraph beginning 'Universal Bottlenecks in Quantum Domains' is nearly duplicated later in the section. One copy should be removed.
  3. [Model names] Model names are inconsistent across figures/text: 'qwen3-vl-235b-a22b-thinking' appears in Figure 7 but 'qwen3-vl-235b-thinking' appears elsewhere; 'gemini-2.5-flash-image' and 'gemini-2.5-flash-image-preview' are used interchangeably. Please standardize.
  4. [§3.3 / Table 4] The hyperparameters λ=0.6 and Γ=6 are said to be chosen from a pilot study, but no pilot-study protocol or criterion is described. The sensitivity table shows the behavior under different settings, but a reader cannot assess how the values were selected.

Circularity Check

1 steps flagged

SIP-F1's bimodal gap and personality tiers are built into Eq. (3), not discovered.

specific steps
  1. self definitional [Section 3.3, Eq. (3); 'High Discrimination and Bimodal Effect'; Section 4.1]
    "Score=(1.0 if S_model = S_correct; λ×Score F1-Gamma otherwise) (3). The parameters are set as λ=0.6 and Γ=6 based on a pilot study, as shown in Table 4, to ensure that any hallucinated or incorrect option leads to a final score lower than the most conservative, correct partial answer. ... By introducing SIP-F1, we found that the model score distribution presents a significant Bimodal Distribution ... This draws an insurmountable gap between perfect models and flawed models."

    By construction, Eq. (3) returns 1.0 only for exact match and λ×(Asym-F1) ≤ λ = 0.6 for every other answer. Hence no score in (0.6,1.0) exists; the 'significant Bimodal Distribution' and 'insurmountable gap' are arithmetic properties of the metric, not empirical findings about models. The 'Conservative (~0.40)' and 'Aggressive (<0.25)' tiers (Sec. 3.3) are likewise computed from Eq. (3)-(5) with λ=0.6, Γ=6; the paper then reports these tiers as discovered 'personalities' and a validated 'rigorous capability hierarchy.' The construction is tuned (pilot study) specifically to enforce the conservative-over-aggressive ordering, so the subsequent confirmation of that ordering in Tables 2/4 and Section 4 is not independent evidence.

full rationale

The only concrete circularity I can exhibit is in the evaluation-metric claim. Eq. (3) defines the score as 1.0 on exact match and λ×F1-Gamma otherwise, with λ=0.6; therefore the score interval (0.6, 1.0) is empty by definition. The paper's claim that SIP-F1 'found' a significant Bimodal Distribution and an 'insurmountable gap' between perfect and flawed models is exactly this definitional fact, not an empirical discovery. Similarly, the Conservative (~0.40) and Aggressive (<0.25) tiers follow from the same formula and from the hand-set Γ=6, and the pilot-study tuning explicitly aims to put false-positive answers below conservative partial answers; reporting the resulting ordering as a validated 'capability hierarchy' or 'Scientific Reasoning Barrier' is therefore circular. I do not see equivalent circularity in the data-synthesis side: questions/answers are generated by gemini-3-flash-preview from paper snippets, which raises validity and independence concerns, but the pipeline does not fit model predictions to labels, so it is not an input-output identity. There are no load-bearing self-citations: the cited works are external benchmarks and technique papers, not the present authors' prior theorems or ansatzes. Because the model-specific rankings (e.g., qwen3.5-plus > GPT-5) and domain breakdowns are not forced by Eq. (3) alone, the circularity is partial, not total.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 3 invented entities

The benchmark's central claims rest on several unverified assumptions: that VLM-generated QA is correct, that recent papers are uncontaminated, that model self-reports are meaningful, and that the SIP-F1 metric validly measures scientific integrity. The three free parameters are design choices that directly shape the reported 'bimodal' and 'personality' results.

free parameters (3)
  • lambda (SIP-F1 cutoff coefficient) = 0.6
    Chosen from a pilot study to cap all imperfect scores at 0.6, which mechanically creates the 'bimodal' gap between perfect and imperfect answers (Section 3.3, Table 4).
  • Gamma (SIP-F1 false-positive penalty) = 6
    Chosen from a pilot study to ensure that any false positive scores below a conservative partial answer, thereby enforcing the 'aggressive vs conservative' ranking by construction (Section 3.3, Table 4).
  • Structural richness threshold = 4 sub-panels
    The pipeline selects only figures with at least four sub-panels, a hand-set choice that determines which figures enter the benchmark and affects dataset composition (Section 2.2).
axioms (5)
  • domain assumption Figure captions and discussion snippets from published papers are expert-verified annotations and can serve as reliable ground truth for generated questions.
    Section 2.1 claims that peer-reviewed captions are 'gold-standard, expert-verified annotations' and uses them as the source of correct answers, without human verification of the derived QA pairs.
  • ad hoc to paper The VLM question generator (gemini-3-flash-preview) produces correct PhD-level questions and answers without human validation.
    Section 2.3 and 2.4 rely on the generator and an advisory VLM to create and audit all 2,703 questions; no human expert checks the final labels.
  • domain assumption Papers published between 2023 and 2025 are temporally separated from model training data, ensuring no contamination.
    The abstract claims 'rigorous temporal separation' and data purity, but no training-cutoff analysis or contamination check is provided.
  • domain assumption Model-reported confidence and perceived difficulty are meaningful metacognitive signals that can be compared across models.
    Section 4.2 interprets confidence and difficulty scores as indicators of 'epistemic humility' without validating what these self-reports measure.
  • ad hoc to paper The SIP-F1 score is a valid proxy for 'scientific integrity' and 'reasoning personality'.
    Section 3.3 defines SIP-F1 and then uses it to label models as Wise, Conservative, Aggressive, or Gambler; there is no external validation linking the metric to actual scientific behavior.
invented entities (3)
  • SPM-Bench dataset no independent evidence
    purpose: A benchmark of 2,703 multimodal questions for evaluating LLMs on scanning probe microscopy.
    The dataset is not released and its labels are unverified by humans, so there is no external handle to check its quality.
  • SIP-F1 metric no independent evidence
    purpose: A scoring rule intended to penalize false-positive answers and quantify model behavior.
    The metric is defined in the paper and its hyperparameters are chosen to enforce a desired ranking, but it is not validated against human grading or external benchmarks.
  • Model personality taxonomy (Conservative, Aggressive, Gambler, Wise) no independent evidence
    purpose: Categorizes LLM behavior based on the gap between SPC and SIP-F1 scores.
    The categories are constructed from the metric's score tiers and have no independent behavioral or experimental validation.

pith-pipeline@v1.3.0-alltime-deepseek · 19691 in / 12967 out tokens · 119411 ms · 2026-08-02T20:31:17.701834+00:00 · methodology

0 comments
read the original abstract

As LLMs achieved breakthroughs in general reasoning, their proficiency in specialized scientific domains reveals pronounced gaps in existing benchmarks due to data contamination, insufficient complexity, and prohibitive human labor costs. Here we present SPM-Bench, an original, PhD-level multimodal benchmark specifically designed for scanning probe microscopy (SPM). We propose a fully automated data synthesis pipeline that ensures both high authority and low-cost. By employing Anchor-Gated Sieve (AGS) technology, we efficiently extract high-value image-text pairs from arXiv and journal papers published between 2023 and 2025. Through a hybrid cloud-local architecture where VLMs return only spatial coordinates "llbox" for local high-fidelity cropping, our pipeline achieves extreme token savings while maintaining high dataset purity. To accurately and objectively evaluate the performance of the LLMs, we introduce the Strict Imperfection Penalty F1 (SIP-F1) score. This metric not only establishes a rigorous capability hierarchy but also, for the first time, quantifies model "personalities" (Conservative, Aggressive, Gambler, or Wise). By correlating these results with model-reported confidence and perceived difficulty, we expose the true reasoning boundaries of current AI in complex physical scenarios. These insights establish SPM-Bench as a generalizable paradigm for automated scientific data synthesis.

Figures

Figures reproduced from arXiv: 2602.22971 by Ben Wang, Bing Zhao, Chengliang Xu, Hu Wei, Lin Qu, Peiyao Xiao, Xiaogang Li, Xinyi Gao, Zeyu Wang, Zichao Chen.

Figure 1
Figure 1. Figure 1: Distribution of SPM-Bench [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Statistical analysis of question complexity. Upper: [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overview of the data synthesis pipeline in SPM-Bench. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Parametric sensitivity of the SIP-F1 metric. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Analysis of accuracy under EM, SPC, and SIP-F1 [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Comparison of model confidence, difficulty and [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Domain-specific competence envelopes across [PITH_FULL_IMAGE:figures/full_fig_p013_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 2 canonical work pages

  1. [1]

    Shuai Bai, Keqin Chen, et al. 2025. Qwen2.5-VL Technical Report.arXiv preprint arXiv:2502.13923(2025)

    Pith/arXiv arXiv 2025

  2. [2]

    John Canny. 1986. A computational approach to edge detection.IEEE Transactions on Pattern Analysis and Machine Intelligence6 (1986), 679–698

    1986

  3. [3]

    Jiaze Chen, Tiantian Fan, et al. 2025. Seed1.5-thinking: Advancing Superb Rea- soning Models with Reinforcement Learning.arXiv preprint arXiv:2504.13914 (2025)

    arXiv 2025

  4. [4]

    Dai et al

    X. Dai et al. 2025. PHYSICSARENA: A Multimodal Physics Reasoning Bench- mark Exploring Variable, Process, and Solution Dimensions.arXiv preprint arXiv:2505.15472(2025)

    Pith/arXiv arXiv 2025

  5. [5]

    Xinrun Du, Yifan Yao, et al. 2025. SuperGPQA: Scaling LLM Evaluation Across 285 Graduate Disciplines.arXiv preprint arXiv:2502.14739(2025)

    Pith/arXiv arXiv 2025

  6. [6]

    Richard O Duda and Peter E Hart. 1972. Use of the Hough transformation to detect lines and curves in pictures.Commun. ACM15, 1 (1972), 11–15

    1972

  7. [7]

    Jingxuan Fan et al . 2024. Hardmath: A Benchmark Dataset for Challenging Problems in Applied Mathematics.arXiv preprint arXiv:2410.09988(2024)

    Pith/arXiv arXiv 2024

  8. [8]

    Lloyd Alan Fletcher and Rangachar Kasturi. 1988. A robust algorithm for text string separation from mixed text/graphics images.IEEE Transactions on Pattern Analysis and Machine Intelligence10, 6 (1988), 910–918

    1988

  9. [9]

    Bofei Gao, Feifan Song, et al. 2025. Omni-math: A Universal Olympiad Level Mathematic Benchmark for Large Language Models. InThe Thirteenth Interna- tional Conference on Learning Representations (ICLR)

    2025

  10. [10]

    Yuchao Gu et al . 2024. AstroMMBench: A Benchmark for Evaluating Mul- timodal Large Language Models Capabilities in Astronomy.arXiv preprint arXiv:2410.00063(2024)

    Pith/arXiv arXiv 2024

  11. [11]

    Daya Guo, Dejian Yang, et al . 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.arXiv preprint arXiv:2501.12948 (2025)

    Pith/arXiv arXiv 2025

  12. [12]

    Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. 2025. Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark. arXiv:2501.05444 [cs.CV] https://arxiv.org/abs/2501.05444

    Pith/arXiv arXiv 2025

  13. [13]

    Chaoqun He, Renjie Luo, et al . 2024. OlympiadBench: A Challenging Bench- mark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems. InProceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (ACL). 3828–3850

    2024

  14. [14]

    Dan Hendrycks, Collin Burns, et al . 2021. Measuring Massive Multitask Lan- guage Understanding. InProceedings of the International Conference on Learning Representations (ICLR)

    2021

  15. [15]

    Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. 2023. Evaluating and inducing personality in pre-trained language models. InProceedings of the 37th International Conference on Neural Information Processing Systems(New Orleans, LA, USA)(NIPS ’23). Curran Associates Inc., Article 466, 22 pages

    2023

  16. [16]

    Saurav Kadavath, Tom Conerly, Amanda Askell, et al. 2022. Language Models (Mostly) Know What They Know. arXiv:2207.05221 [cs.CL] https://arxiv.org/ abs/2207.05221

    Pith/arXiv arXiv 2022

  17. [17]

    Manyu Li et al. 2025. MicroVQA++: High-Quality Microscopy Reasoning Dataset with Weakly Supervised Graphs for Multimodal Large Language Model.arXiv preprint arXiv:2511.11407(2025)

    arXiv 2025

  18. [18]

    Percy Liang, Rishi Bommasani, Tony Lee, et al . 2023. Holistic Evaluation of Language Models.Transactions on Machine Learning Research(2023). https: //openreview.net/forum?id=iO4LZibEqW 9

    2023

  19. [19]

    Junqi Liu, Jonas Bayer, Xiaodan Liang, et al. 2025. CombiBench: Benchmarking LLM Capability for Combinatorial Mathematics.arXiv preprint arXiv:2501.11284 (2025)

    Pith/arXiv arXiv 2025

  20. [20]

    Yang Liu, Dan Iter, Shuo Xu, et al. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment.arXiv preprint arXiv:2303.16634(2023)

    Pith/arXiv arXiv 2023

  21. [21]

    LINHAO LUO, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. 2024. Rea- soning on Graphs: Faithful and Interpretable Large Language Model Reasoning. (2024). https://openreview.net/forum?id=ZGNWW7xZ6Q

    2024

  22. [22]

    O’Gorman

    L. O’Gorman. 1993. The document spectrum for page layout analysis.IEEE Transactions on Pattern Analysis and Machine Intelligence15, 11 (1993), 1162–1173. doi:10.1109/34.244677

    doi:10.1109/34.244677 1993

  23. [23]

    Long Phan et al. 2025. Humanity’s Last Exam.arXiv preprint arXiv:2501.14249 (2025). https://arxiv.org/pdf/2501.14249

    Pith/arXiv arXiv 2025

  24. [24]

    Hui Shen et al . 2025. PHYX: Does Your Model Have the “Wits” for Physical Reasoning?arXiv preprint arXiv:2502.15929(2025)

    Pith/arXiv arXiv 2025

  25. [25]

    2025.From Chart to QA Pairs: A Context-A ware Generation Framework for Chart-Containing Documents

    Qianwen Shen, Zeyu Ma, and Mingang Chen. 2025.From Chart to QA Pairs: A Context-A ware Generation Framework for Chart-Containing Documents. Asso- ciation for Computing Machinery, 101–107. https://doi.org/10.1145/3728725. 3728741

    work page doi:10.1145/3728725 2025

  26. [26]

    Aarush Sinha, OmKumar Chandra Umakanthan, and Sudhakaran Gajendran

  27. [27]

    Karl Tombre, Christian Ah-Soon, Philippe Dosch, Gerald Masini, and Salvatore Tabbone. 2002. Text/graphics separation revisited. InWorkshop on Document Analysis Systems. Springer, 1–12

    2002

  28. [28]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. arXiv:2203.11171 [cs.CL] https://arxiv.org/abs/2203.11171

    Pith/arXiv arXiv 2023

  29. [29]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, et al. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models.Advances in Neural Information Processing Systems35 (2022), 24824–24837

    2022

  30. [30]

    Yonghao Weng et al. 2025. MatQnA: A Benchmark Dataset for Multi-modal Large Language Models in Materials Characterization and Analysis.arXiv preprint arXiv:2509.11335(2025)

    arXiv 2025

  31. [31]

    Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, Elena Ericheva, Katharyn Garcia, Brian Goodrich, Nikola Jurkovic, Holden Karnofsky, Megan Kinniment, Aron Lajko, Seraphina Nix, Lucas Sato, William Saunders, Maksym Taran, Ben West, and Elizabeth Barnes. 2025. RE-Bench: Eval...

    Pith/arXiv arXiv 2025

  32. [32]

    Kun Xiang, Heng Li, Xiaodan Liang, et al. 2025. SEEPHYS: Does Seeing Help Thinking? – Benchmarking Vision-Based Physics Reasoning.arXiv preprint arXiv:2505.19099(2025)

    arXiv 2025

  33. [33]

    Xin Xu et al. 2025. UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models.arXiv preprint arXiv:2502.00334 (2025)

    Pith/arXiv arXiv 2025

  34. [34]

    An Yang et al. 2024. Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-improvement.arXiv preprint arXiv:2409.12122(2024)

    Pith/arXiv arXiv 2024

  35. [35]

    Junchi Yu, Ran He, and Zhitao Ying. 2024. Thought propagation: An analogical approach to complex reasoning with large language models. (2024). https: //openreview.net/forum?id=SBoRhRCzM3

    2024

  36. [36]

    Xiang Yue, Yuansheng Ni, et al. 2024. MMMU: A Massive Multi-discipline Multi- modal Understanding and Reasoning Benchmark for Expert AGI. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2024

  37. [37]

    Xinyu Zhang et al. 2025. PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning.arXiv preprint arXiv:2502.12054(2025)

    Pith/arXiv arXiv 2025

  38. [38]

    Yaowei Zheng et al. 2024. LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models.arXiv preprint arXiv:2403.13372(2024)

    Pith/arXiv arXiv 2024

  39. [39]

    Wanjun Zhong et al. 2023. AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models.arXiv preprint arXiv:2304.06364(2023)

    Pith/arXiv arXiv 2023

  40. [40]

    PhD-level

    Minjun Zhu, Yixuan Weng, Linyi Yang, and Yue Zhang. 2025. Personality Align- ment of Large Language Models. (2025). https://openreview.net/forum?id= 0DZEs8NpUH 10 A Details in SPM-Bench As illustrated in Figure 1 and detailed in Table 1, SPM-Bench com- prises 2,703 high-quality multimodal questions, each paired with a unique, high-resolution microscopy im...

    2025

  41. [2025]

    https://doi.org/ 10.1038/s41598-025-18622-6

    DR-CoT: dynamic recursive chain of thought with meta reasoning for parameter efficient models.Scientific Reports15, 1 (2025), 4125. https://doi.org/ 10.1038/s41598-025-18622-6

    work page doi:10.1038/s41598-025-18622-6 2025