Pith. sign in

REVIEW 2 major objections 6 minor 20 references

Multimodal AI models classify and summarize ALD/E scientific figures well but still fail at data extraction and visual scientific reasoning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 19:17 UTC pith:6GRSNMC3

load-bearing objection Solid ICDAR competition report: real ALD/E figure corpus, four end-to-end tasks, and a clear empirical gap that SOTA VLMs still fail data extraction and domain VQA. the 2 major comments →

arxiv 2607.26848 v1 pith:6GRSNMC3 submitted 2026-07-29 cs.CV

ICDAR 2026 Competition on Information Extraction from Atomic Layer Deposition/Etching (ALD/E) Scientific Figures

Fahad Ahmed , S\"oren Auer , Jennifer D'Souza This is my paper
classification cs.CV
keywords Scientific Figure ComprehensionScientific ReasoningClassificationData Table ExtractionSummarizationVisual Question AnsweringALD/ALEMultimodal Benchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces Sci-ImageMiner, an expert-annotated benchmark of nearly two thousand Atomic Layer Deposition and Etching figures drawn from real materials-science papers, together with an ICDAR 2026 community competition built around four linked tasks: figure classification, chart data extraction, summarization, and visual question answering. Sixty-eight teams submitted more than twelve hundred entries. The results show that current vision-language models already reach usable accuracy on classification and summarization, yet remain weak when they must recover precise numerical tables or answer domain-grounded reasoning questions. The work therefore supplies both a realistic test bed and clear evidence of where domain-aware multimodal systems still fall short, giving researchers a concrete target for the next generation of scientific figure understanding.

Core claim

State-of-the-art multimodal models perform adequately on classification and summarization of authentic ALD/E scientific figures, but struggle markedly with structured data extraction and scientific visual question-answering; the Sci-ImageMiner benchmark and its four-task competition establish a rigorous end-to-end platform that makes this performance gap measurable and actionable.

What carries the argument

Sci-ImageMiner: a 1,951-figure, expert-annotated ALD/E dataset organized into four progressive tasks (49-class taxonomy classification, chart-to-Markdown table extraction, caption-level summarization, and Bloom-taxonomy VQA) that together test end-to-end scientific figure comprehension.

Load-bearing premise

The expert annotations, especially the manually written VQA pairs, are reliable enough ground truth even though inter-annotator agreement was only moderate on a small shared sample.

What would settle it

A new independent multi-annotator adjudication of the full VQA and table-extraction labels that either raises agreement substantially above the reported 0.46 kappa or shows that top-team rankings reverse when labels are corrected.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Future multimodal systems for materials science can be ranked on a public, domain-specific leaderboard rather than on generic chart benchmarks.
  • Data-extraction and VQA remain the primary bottlenecks that must be solved before automated scientific knowledge capture from figures becomes practical.
  • Hierarchical fine-tuning, synthetic chart data, retrieval-augmented context, and cross-task chaining are the concrete techniques that currently move the needle.
  • The same four-task template can be reused for other specialized scientific domains that lack end-to-end figure benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because table extraction and VQA scores remain low even for the strongest entrants, production pipelines that claim to ‘read’ process papers will still need human verification for any quantitative claim taken from a figure.
  • The moderate inter-annotator agreement on VQA suggests that part of the measured model gap may actually be label noise; tightening the annotation protocol could shrink the apparent reasoning deficit.
  • Success of context-injection and multi-agent designs implies that pure vision backbones are insufficient; future gains will likely come from tighter coupling to the surrounding paper text rather than larger image encoders alone.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This manuscript reports the ICDAR 2026 Sci-ImageMiner competition and its accompanying expert-annotated benchmark of 1,951 ALD/ALE scientific figures from 205 papers. It defines four complementary end-to-end tasks—49-way figure classification, chart-to-Markdown data extraction, summarization, and domain-grounded VQA with Bloom-inspired question types and four answer formats—together with train/dev/test splits, evaluation metrics (F1; RMS/TEDS; ROUGE/BERTScore; answer-type-specific VQA scores), baselines from several LVLMs, and leaderboards from 68 participants and 1,263 submissions. The central empirical claim is that current multimodal models perform relatively well on classification and summarization but remain weak on data extraction and especially VQA, and that the benchmark provides a rigorous platform for domain-aware scientific figure understanding.

Significance. If the reported difficulty ordering and platform value hold, Sci-ImageMiner fills a genuine gap relative to prior ICDAR figure challenges and general chart/materials benchmarks by offering authentic, expert-curated ALD/E figures with multi-task annotations and a large community evaluation. Strengths include a clear multi-task design, public dataset and evaluation scripts under CC BY 4.0, substantial participation, transparent top-team method summaries (Table 6), and consistent gaps across independent baselines and many submissions (Tables 4–8). These make the work a useful reference resource for multimodal scientific document AI, even though absolute VQA scores should be interpreted cautiously.

major comments (2)
  1. [§2.3; Table 8; §4.4] §2.3 reports Fleiss’ κ = 0.46 (moderate) on only a 5-paper shared sample, with VQA reserved for fully manual expert labeling and bi-weekly automated checks rather than full multi-annotator adjudication. Because Task 4 rankings and the paper’s strongest claim about scientific-reasoning failure rest on these labels (Abstract; §4.3–4.4; Table 8), the manuscript should quantify how annotation noise affects answer-type scores (especially Paragraph/List/Factoid) and leaderboard stability—e.g., via bootstrap over the overlap set, disagreement analysis by question type, or a second-pass adjudication subsample—and state more explicitly which comparative conclusions are robust to moderate IAA.
  2. [§3.3; Tables 5, 7, 8] Task ranking scores use fixed equal weights (Task 2: ½ RMS + ½ TEDS; Task 3: ½ mean ROUGE + ½ BERTScore-F1; Task 4: equal mix of four answer-type aggregates; §3.3). Tables 5, 7, and 8 show that component metrics can disagree in magnitude and ordering (e.g., low RMS vs. higher TEDS; very low Exact Match / set-F1 vs. higher ROUGE/BERTScore). Without sensitivity analysis or justification that equal weights match scientific utility, small leaderboard gaps may be metric artifacts. A short ablation of weightings or primary/secondary metric reporting would strengthen the claim that extraction and VQA are categorically harder.
minor comments (6)
  1. [Table 8] Table 8 caption incorrectly reads “Task 3 VQA scores”; it should be Task 4.
  2. [Table 3; §4.1] Table 3 participant counts (81/53/43/47) exceed the stated 68 active participants; clarify unique vs. per-track registration.
  3. [§2.3] IAA is reported as a single κ without breakdown by task or question/answer type; even a brief per-task note would help readers weight VQA vs. classification labels.
  4. [§4.2] Baseline rows are helpful but prompts and decoding settings are only linked externally; a one-sentence note on zero/few-shot vs. fine-tuned status per baseline would aid reproducibility in the PDF.
  5. [Table 1; §3.3; §1] Minor typos/grammar: “T otal” spacing in Table 1; “Evaluationisconductedusing” spacing in §3.3; “and next generation nanoelectronics” fragment in §1.
  6. [Fig. 3] Figure 3 examples are strong; ensuring axis/legend readability at print scale would help non-ALD readers.

Circularity Check

0 steps flagged

No significant circularity: empirical competition benchmark with held-out rankings, not a self-defining derivation.

full rationale

Sci-ImageMiner is an ICDAR competition report that curates an expert-annotated ALD/E figure dataset and four evaluation tasks, then ranks external participant systems and open LVLMs on a blind test split via fixed CodaBench scripts (Accuracy/F1, RMS/TEDS, ROUGE/BERTScore, answer-type-specific VQA metrics). The central claim—that SOTA multimodal models do relatively well on classification and summarization but struggle on data extraction and VQA—is an empirical ordering read off Tables 4–8 and 1,263 public/private submissions from 68 participants, not a quantity derived from fitted parameters or a uniqueness theorem. Organizers define the 49-class taxonomy, annotation schema, metrics, and baselines (normal for a competition paper); those choices do not make the leaderboard scores equal the inputs by construction. There are no load-bearing self-citations that force the result, no ansatz smuggled in as a theorem, and no renaming of a known pattern presented as a first-principles prediction. Annotation IAA (κ=0.46 on a 5-paper overlap) is a ground-truth reliability concern, not circularity. Score 0 is the appropriate honest finding.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 2 invented entities

As a benchmark paper, load-bearing commitments are definitional and evaluative rather than physical laws: the figure taxonomy, task formulations, metric weightings, train/dev/test splits, and the premise that expert (plus model-pre) annotations constitute ground truth for scientific figure understanding in ALD/E.

free parameters (3)
  • Task ranking metric weights = equal 1/2 or 1/4 mixtures as stated in §3.3
    Composite scores use fixed equal weights (e.g., Task2 ½ RMS + ½ TEDS; Task3 ½ mean ROUGE + ½ BERTScore-F1; Task4 equal weight over four answer types). These weights are design choices that determine leaderboard order.
  • 49-figure taxonomy categories = 49 predefined categories
    Class inventory and boundaries (e.g., multiple line chart vs line chart vs multi-axis) are curator-defined and directly control Task 1 difficulty and scores.
  • Train/dev/test partition sizes = Table 1 splits
    Split counts (128/20/57 papers; 1180/201/570 figures) are chosen by organizers and fix generalization claims.
axioms (5)
  • domain assumption Standard classification and NLG metrics (Accuracy/F1, ROUGE, BERTScore, Exact Match, set-F1) plus RMS and TEDS adequately rank scientific figure understanding systems.
    Evaluation protocols in §3.3 treat these weighted composites as primary ranking truth without human preference validation of the composites.
  • domain assumption Expert annotations (with Qwen2.5-VL pre-annotation for Tasks 1–3 and manual VQA) are sufficiently correct ground truth despite moderate IAA.
    §2.3 reports Fleiss’ κ=0.46 on 5 shared papers and automated bi-weekly checks; competition conclusions rest on these labels.
  • domain assumption ALD/E experimental and simulation figures are a representative, high-value test domain for broader scientific figure comprehension.
    Introduction frames semiconductor ALD/ALE as foundational and the gap as general to scientific figures (§1).
  • domain assumption MinerU extraction preserves semantic structure and visual fidelity enough for downstream multimodal tasks.
    §2.1 states MinerU yields structured JSON and high-res JPEGs used as the dataset substrate.
  • ad hoc to paper Bloom-inspired question types and four answer formats cover progressive scientific reasoning over figures.
    §2.3 defines Process, Comparative/Trend, Structure–Property, Application/Performance plus Yes/No, Factoid, List, Paragraph as the VQA schema.
invented entities (2)
  • Sci-ImageMiner benchmark independent evidence
    purpose: Provide the expert-annotated multi-task dataset and competition substrate for ALD/E figure comprehension.
    Newly curated resource (205 papers, 1951 figures, four tasks); existence is evidenced by release claims and competition, not a physical postulate.
  • 49-type scientific figure taxonomy for ALD/E no independent evidence
    purpose: Define Task 1 label space and organize figure diversity.
    Curator-constructed category system; utility is internal to the benchmark unless reused/validated elsewhere.

pith-pipeline@v1.2.0-daily-grok45 · 17101 in / 3521 out tokens · 75342 ms · 2026-07-30T19:17:14.296521+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of ICDAR 2026 Competition on Information Extraction from Atomic Layer Deposition/Etching (ALD/E) Scientific Figures." pith.science (2026). https://pith.science/paper/6GRSNMC3

@misc{pith2026260726848,
  author       = {Pith},
  title        = {Pith review of: ICDAR 2026 Competition on Information Extraction from Atomic Layer Deposition/Etching (ALD/E) Scientific Figures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6GRSNMC3}},
  note         = {Machine review of arXiv:2607.26848}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Scientific figure comprehension and reasoning using multimodal AI requires integrating visual perception with domain-specific reasoning to extract meaningful knowledge, often not presented in the text of a research publication. The Sci-ImageMiner benchmark dataset, accompanied by a community-driven competition, raises the bar over prior scientific competitions by curating a comprehensive, expert-annotated dataset across four end-to-end complementary tasks. The competition attracted 68 active participants and 1,263 public/private submissions from 9th January 2026 to 8th April 2026. Our results show that state-of-the-art multimodal models perform well on classification and summarization tasks but struggle with data extraction and scientific reasoning, particularly in visual question-answering. These findings reveal key limitations and highlight challenges and opportunities for improving domain-aware multimodal AI systems. Overall, the Sci-ImageMiner benchmark and competition establish a rigorous platform for advancing research in scientific figure comprehension and reasoning and demonstrate the potential of state-of-the-art approaches for a challenging and complex research area.

Figures

Figures reproduced from arXiv: 2607.26848 by Fahad Ahmed, Jennifer D'Souza, S\"oren Auer.

Figure 1
Figure 1. Figure 1: Overview of the Sci-ImageMiner benchmark end-to-end curation work [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Illustration describing the Sci-ImageMiner dataset organization hierarchy. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Examples of some challenging scientific figures with domain-grounded VQA in Sci-ImageMiner. Task 1: Classification Task Description: To classify each figure into one of the 49 predefined categories defined in the taxonomy [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 5 linked inside Pith

  1. [1]

    In: Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS 2024) (2024)

    Alampara, N., Mandal, I., Khetarpal, P., Grover, H.S., Schilling-Wilhelmi, M., Krishnan, N.A., Jablonka, K.M.: Macbench: a multimodal chemistry and materials science benchmark. In: Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS 2024) (2024)

    2024

  2. [2]

    In: International Conference on Document Analysis and Recognition

    Chen, M., Wu, H., Chang, Q., Cheng, H., Ma, J., Hu, P., Zhang, Z., Liu, C., Pi, C., Hu, J., et al.: Icdar 2024 competition on recognition of chemical structures. In: International Conference on Document Analysis and Recognition. pp. 397–409. Springer (2024)

    2024

  3. [3]

    Clark, C., Zhang, J., Ma, Z., Park, J.S., Tripathi, R., Lee, S., Salehi, M., Ren, J., Kim, C.D., Yang, Y., et al.: Molmo2: Open weights and data for vision-language modelswithvideounderstandingandgrounding.In:ProceedingsoftheIEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 28652–28668 (2026)

    2026

  4. [4]

    Advances in neural information processing systems36, 10088–10115 (2023)

    Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: Qlora: Efficient fine- tuning of quantized llms. Advances in neural information processing systems36, 10088–10115 (2023)

    2023

  5. [5]

    Chemical reviews110(1), 111–131 (2010)

    George, S.M.: Atomic layer deposition: an overview. Chemical reviews110(1), 111–131 (2010)

    2010

  6. [6]

    Jablonka, K., Krishnan, N.: Vision language models excel at perception but strug- gles with scientific reasoning (2025)

    2025

  7. [7]

    Journal of Vacuum Science & Technology A33(2) (2015)

    Kanarik, K.J., Lill, T., Hudson, E.A., Sriraman, S., Tan, S., Marks, J., Vahedi, V., Gottscho, R.A.: Overview of atomic layer etching in the semiconductor industry. Journal of Vacuum Science & Technology A33(2) (2015)

    2015

  8. [8]

    Theory Into Practice41(4), 212–218 (2002).https://doi.org/10.1207/s15430421tip4104_2, https://doi.org/10.1207/s15430421tip4104_2 ICDAR 2026 Sci-ImageMiner Competition 17

    Krathwohl, D.R.: A revision of bloom’s taxonomy: An overview. Theory Into Practice41(4), 212–218 (2002).https://doi.org/10.1207/s15430421tip4104_2, https://doi.org/10.1207/s15430421tip4104_2 ICDAR 2026 Sci-ImageMiner Competition 17

    doi:10.1207/s15430421tip4104_2 2002

  9. [9]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Li, L., Wang, Y., Xu, R., Wang, P., Feng, X., Kong, L., Liu, Q.: Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 14369–14387 (2024)

    2024

  10. [10]

    In: Mure- san, S., Nakov, P., Villavicencio, A

    Masry, A., Long, D.X., Tan, J.Q., Joty, S., Hoque, E.: ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In: Mure- san, S., Nakov, P., Villavicencio, A. (eds.) Findings of the Association for Com- putational Linguistics: ACL 2022. pp. 2263–2279. Association for Computational Linguistics, Dublin, Ireland (May 2022)...

    doi:10.18653/v1/2022 2022

  11. [11]

    In: Proceedings of the ieee/cvf winter conference on applications of computer vision

    Methani, N., Ganguly, P., Khapra, M.M., Kumar, P.: Plotqa: Reasoning over sci- entific plots. In: Proceedings of the ieee/cvf winter conference on applications of computer vision. pp. 1527–1536 (2020)

    2020

  12. [12]

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C.D., Ermon, S., Finn, C.: Direct Preference Optimization: Your Language Model is Secretly a Re- ward Model. Advances in Neural Information Processing Systems36, 53728– 53741 (Dec 2023),https://papers.nips.cc/paper_files/paper/2023/hash/ a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html

    2023

  13. [13]

    Team, Q.: Qwen2.5-vl (January 2025),https://qwenlm.github.io/blog/qwen2. 5-vl/

    2025

  14. [14]

    Team, Q.: Qwen3 technical report (2025),https://arxiv.org/abs/2505.09388

    Pith/arXiv arXiv 2025

  15. [15]

    Team, V., Hong, W., Yu, W., Gu, X., Wang, G., Gan, G., Tang, H., Cheng, J., Qi, J., Ji, J., Pan, L., Duan, S., Wang, W., Wang, Y., Cheng, Y., He, Z., Su, Z., Yang, Z., Pan, Z., Zeng, A., Wang, B., Chen, B., Shi, B., Pang, C., Zhang, C., Yin, D., Yang, F., Chen, G., Xu, J., Zhu, J., Chen, J., Chen, J., Chen, J., Lin, J., Wang, J., Chen, J., Lei, L., Gong, ...

    Pith/arXiv arXiv 2025

  16. [16]

    arXiv preprint arXiv:2409.18839 (2024)

    Wang, B., Xu, C., Zhao, X., Ouyang, L., Wu, F., Zhao, Z., Xu, R., Liu, K., Qu, Y., Shang, F., et al.: Mineru: An open-source solution for precise document content extraction. arXiv preprint arXiv:2409.18839 (2024)

    Pith/arXiv arXiv 2024

  17. [17]

    arXiv preprint arXiv:2508.18265 (2025)

    Wang, W., Gao, Z., Gu, L., Pu, H., Cui, L., Wei, X., Liu, Z., Jing, L., Ye, S., Shao, J., et al.: Internvl3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265 (2025)

    Pith/arXiv arXiv 2025

  18. [18]

    Research in science education52(3), 891–907 (2022)

    Wanselin, H., Danielsson, K., Wikman, S.: Analysing multimodal texts in sci- ence—a social semiotic perspective. Research in science education52(3), 891–907 (2022)

    2022

  19. [19]

    In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR)

    Yang, C., Yin, X.C., Yu, H., Karatzas, D., Cao, Y.: Icdar2017 robust reading chal- lenge on text extraction from biomedical literature figures (detext). In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR). vol. 1, pp. 1444–1447. IEEE (2017)

    2017

  20. [20]

    arXiv preprint arXiv:2510.12171 (2025)

    Zhang, J., Gan, J., Wang, X., Jia, Z., Gu, C., Chen, J., Zhu, Y., Ma, M.D., Zhou, D.,Li,L.,etal.:Matscibench:Benchmarkingthereasoningabilityoflargelanguage models in materials science. arXiv preprint arXiv:2510.12171 (2025)

    Pith/arXiv arXiv 2025