Pith. sign in

REVIEW 3 major objections 5 minor 75 references

Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that current multimodal models answer scientific-diagram questions with up to 86% accuracy yet reconstruct their TikZ code at only 31-57% object-level F1, so diagram-to-code generation, not diagram understanding, is the…

desk verdict A serious TikZ diagram benchmark worth a real review: the coding-vs-reasoning asymmetry is probably real, but the object-level F1 pipeline quantifying it is unvalidated and its bbox numbers look like an artifact. read the letter →

arxiv 2608.12262 v1 pith:SIFGW64S submitted 2026-08-12 cs.CV cs.AI

Weihao Bo , Shan Zhang , Yanpeng Sun , Jie Liu , Yongke Yao , Jinhao Du , Wei He , Kai Zou
show 2 more authors
Zechao Li Jingdong Wang
This is my paper · ORCID
classification cs.CVcs.AI
keywords diagram-to-codegenerationTikZmultimodallargelanguagemodelsbenchmarkscientificdiagramsdiagramquestionansweringagenticevaluationobject-levelF1
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Diagram-MMU is a benchmark of 3,744 curated scientific diagrams and 18,305 human-validated questions spanning charts, planar geometry, 3D shapes, graphs, chemistry, and circuits. The paper's central claim is that today's multimodal language models can reason over diagrams, reaching up to 86% accuracy on diagram question answering, but struggle to reproduce them as LaTeX TikZ code, with object-level F1 between 31% and 57% across twelve models. The benchmark adds a diagram-to-code editing task and, for each task, sixteen settings that test foundational ability plus agentic capabilities such as tool use, context utilization, state management, and planning. On the agentic settings, most models improve at editing, degrade at question answering, and Claude-4.6 Opus is the only model that improves on all three tasks. If this asymmetry holds, then improving diagram-to-code generation, not visual reasoning, is the key to AI-assisted scientific writing.

What carries the argument

The central machinery is the Semantic Object Model (SOM) pipeline: TikZ code is compiled to DVI with injected semantic tags, converted to SVG via dvisvgm, and parsed into typed, attributed elements such as nodes, paths, text labels, data series, and circuit components, whose type, text, color, and bounding box are scored by four F1 measures with greedy one-to-one matching, where text uses exact string match, color uses permutation-based CIEDE2000 assignment, and bbox uses IoU at least 0.3. These object-level F1 scores sit alongside code-level CrystalBLEU and image-level SSIM, CLIP, LPIPS, and FID, and for editing the object and code metrics are split into preserve-only and edit-only partitions. Around this machinery, the benchmark builds 16 evaluation settings: three foundational settings for direct parsing, editing, and answering, plus thirteen agentic variants that add perception objects as context, a TikZ search tool served over the Model Context Protocol, required or optional intermediate code generation, and planning combinations of these. The SOM metrics are what turn the statement that generated code looks plausible into a measurable claim about whether the model actually perceived the right objects at the right places.

What would settle it

Take a random sample of generated and ground-truth TikZ pairs from Diagram-MMU, have two independent annotators manually list every object with its type, text, color, and bounding box, and compute annotator-versus-SOM agreement on the same sample; if per-dimension F1 agreement falls below roughly 0.85, the reported 31-57% parsing F1 range would not be a trustworthy measure of code quality.

Watch

Extended reading notes

Core claim

The discovery is an asymmetry in current MLLMs: perception and coding lag reasoning. Across six scientific diagram domains, models answer descriptive and reasoning questions about diagrams accurately, up to 86.46% on the best model, yet when asked to parse a diagram into compilable TikZ code they reach only 31-57% object-level F1, and when asked to edit diagrams they score even lower on the edit-only partition. The paper attributes this to weak fine-grained spatial grounding: the F1bbox dimension is consistently the lowest, with strong models scoring 62-71 on type, text, and color but 8.0-12.7 on bounding boxes. In agentic settings, providing object perception data and a TikZ documentation search tool helps editing more than parsing, while most models degrade on question answering when asked to plan multi-step workflows; Claude-4.6 Opus is the exception that improves on all three tasks. The paper also introduces the first benchmark to cover chemistry and circuit diagrams for diagram-to-code tasks, and the first to evaluate both foundational and agentic abilities in one controlled setup.

Load-bearing premise

The central asymmetry claim rests on the Semantic Object Model pipeline's automated extraction of object types, text, colors, and bounding boxes from compiled TikZ code, which is never checked against human-annotated object ground truth; if that extraction mis-matches elements, the reported F1 values misstate true code quality.

Editorial extensions

If this is right

  • Diagram-to-code generation, not diagram understanding, is the binding constraint for MLLM-assisted scientific writing, so training and fine-tuning should target object-level spatial grounding in code output.
  • Object-level spatial grounding (F1bbox) is the weakest perception dimension, so methods that improve coordinate and layout encoding should lift both parsing and editing scores.
  • Textual editing instructions partially compensate for coding weakness, implying that routing edits through text-to-code abilities is a practical way to improve diagram editing today.
  • Agentic tool use and perception context help editing but hurt question answering for most models, and planning is the weakest agentic capability, so tool access should be gated per task.
  • A benchmark for vibe-writing workspaces must report parsing, editing, and answering separately, since a single accuracy number hides the coding gap.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: because most models degrade on DQA when forced to generate TikZ code first in settings S14 and S15, code-first reasoning pipelines are likely to underperform direct visual answering until diagram-to-code fidelity improves; this is testable by comparing S12 with S14 on the released benchmark.
  • My inference: the near-zero edit-only CrystalBLEU values (0.51-2.98) together with moderate object F1 suggest the two metric families measure different failure modes, so a model can name the right objects yet write non-idiomatic code, and future work should analyze disagreement cases to decide which metric better predicts human edit quality.
  • My inference: the SOM pipeline, once validated against human object annotations, could serve as a reusable automatic metric for TikZ code quality beyond this benchmark, enabling larger-scale training and evaluation of diagram-to-code models.
  • My inference: for practitioners, the cheapest immediate gain is to supply models with explicit object perception data for editing tasks and to disable tool use for question answering, where it currently costs most models accuracy.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Diagram-MMU, a benchmark of 3,744 curated TikZ-sourced diagrams with 18,305 human-validated evaluation instances spanning three tasks: diagram-to-code parsing (D2C-P), diagram-to-code editing (D2C-E), and diagram question answering (DQA), together with agentic variants of each task. Twelve MLLMs are evaluated. The headline finding is an asymmetry: models reason well over diagrams (DQA accuracy up to 86%) but perform poorly at diagram-to-code parsing (reported object-level F1avg of 31-57%), suggesting that diagram-to-code generation, rather than diagram understanding, is the binding constraint for AI-assisted scientific writing. The paper also reports that agentic settings improve parsing and editing for most models but degrade DQA, with Claude-4.6 Opus improving across all three tasks. The central quantitative claim rests on a custom object-level evaluation pipeline and on comparing DQA accuracy with D2C-P F1avg, both of which are examined critically below.

Significance. If the measurements are reliable, Diagram-MMU is a valuable contribution: it is the first TikZ-based benchmark covering six diagram domains with three tasks plus agentic settings, the data curation is careful (13 graduate students, cross-validation, source filtering), the DQA judge is validated against human agreement with kappa 0.937 on 200 samples, and the MCP-based TikZ documentation server is a useful reusable infrastructure piece. The paper also provides falsifiable predictions about model weaknesses (spatial grounding, planning, tool-use loops). However, the central asymmetry claim is currently supported mainly by an unvalidated object-extraction pipeline and by a cross-task comparison of non-equivalent metrics; the circularity in ground-truth generation is also a real concern. These issues are addressable, but they are load-bearing for the paper's main conclusion.

major comments (3)
  1. [3.4 / Appendix C.1-C.2 / Table G.1] The Semantic Object Model (SOM) pipeline that produces the headline D2C-P F1avg values is never validated against human-annotated object ground truth. The reported numbers are internally suspicious: in Table G.1, F1type/F1text/F1color are roughly 60-71 for most models while F1bbox is only 7.8-17.1 for all six representative models, a uniform collapse that is more consistent with a systematic extraction or matching artifact than with a universal spatial-grounding failure. The greedy IoU>=0.3 matching iterates ground-truth boxes in document order and removes the first matching predicted box (Appendix C.2), so semantically correct boxes in a different order can be missed, and the pipeline has no stated tolerance for the many equivalent TikZ encodings of the same diagram (e.g., different coordinate origins, alias names, or path decompositions). Because the paper's central claim that coding is harder than understanding is based on this F1avg, the authors need to validate the SOM pipeline on a human-annotated sample (reporting extraction precision/recall per object dimension), replace the order-dependent greedy matching with an order-invariant matching procedure, and run a sensitivity analysis of the IoU threshold and color tolerance. Without this, the reported D2C-P numbers cannot be taken at face value.
  2. [Appendix B.2 / Table 5] There is a mild circularity in the benchmark construction: the D2C-E and DQA ground-truth answers were generated by Gemini-3 Flash and verified by GPT-5.2 and Gemini-3 Pro (Appendix B.2), and all three of these models are later evaluated in Table 5. This can systematically favor exactly the models the paper highlights, for example Gemini-3.0 Pro's highest DQA accuracy and its characterization as the most balanced profile. The paper does not analyze how the rankings change as a function of ground-truth provenance. The authors should construct a model-independent validation subset or report per-model scores split by whether the ground-truth item was generated or verified by that model, and ideally show that the main conclusions survive when the evaluated models play no role in creating the gold labels.
  3. [Section 1 / Table 5] The headline asymmetry compares DQA accuracy (a semantic correctness rate) with D2C-P F1avg (an average of four exact-reconstruction dimensions, one of which requires IoU>=0.3 bounding-box overlap). These metrics operate at different strictness levels: answering 'what is the degree of node A?' correctly is not the same as reconstructing every object, label, color, and spatial position of a diagram. A low object-level F1avg therefore does not by itself establish that models 'struggle to code' in a practically meaningful sense; it may simply reflect that exact reconstruction is a much stricter criterion than semantic question answering. To support the claim that coding is the binding constraint, the authors should supplement the object-level comparison with human judgments of whether generated diagrams are visually or semantically faithful to the target, or report the fraction of generated diagrams that a human would accept as faithful reproductions. As written, the evidence is consistent with the stated asymmetry but does not uniquely establish it.
minor comments (5)
  1. [3.2 heading] The heading 'Diagram-to-Code Pasing' contains a typo; it should read 'Diagram-to-Code Parsing'.
  2. [4.1, item (3)] The text refers to 'D2C-E and DU' where the intended abbreviation appears to be DQA; please correct this for clarity.
  3. [Figure 6] The radar panels use per-panel axis scales (e.g., F1bbox ranges differently from F1type), which makes the bbox collapse harder to compare across panels; consider using common axis ranges or adding explicit numerical labels on each axis.
  4. [Table 5 / text after Eq. (5)] The paper says D2C-P object-level F1 'ranges 31-57%' but the specialist TikZero+10B scores 15.43 in Table 5; please clarify whether the range intentionally excludes the specialist model and state the inclusion/exclusion criterion.
  5. [Appendix C.2] The edge-case rule that F1=1.0 when both predicted and ground-truth sets are empty is reasonable, but it should also be mentioned in the main text where F1avg is first defined, since it can inflate scores for diagrams with few objects of a given type.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; the benchmark's headline asymmetry is an independent empirical comparison, with model-generated ground-truth noted as a non-circular contamination caveat.

full rationale

Diagram-MMU is a benchmark and evaluation study, not a derivation. The central claim—that models answer diagram questions well (DQA up to 86%) but parse diagrams into TikZ poorly (D2C-P object-level F1 31–57%)—is an empirical measurement against fixed ground-truth TikZ sources and human-validated QA pairs, not a quantity fitted from the reported metrics. D2C-P ground truth is the original TikZ/PGF, ChemFig, CircuiTikZ, etc. source code from official manuals and community resources (Section 3.1, Figure 2), so the coding difficulty result does not reduce to the evaluation pipeline's own outputs. The object-level F1, CrystalBLEU, and image-level metrics are computed by independent, well-known procedures (greedy IoU matching, CrystalBLEU, SSIM/CLIP/LPIPS/FID) and are not defined in terms of the model outputs being scored. The only near-circular element is that Appendix B.2 reports D2C-E and DQA ground-truth instances were initially generated by Gemini-3 Flash and verified by GPT-5.2 and Gemini-3 Pro, models later evaluated in Table 5; however, the same appendix states all instances were then manually reviewed and cross-validated by 13 graduate students, and the paper's key coding-vs-answering asymmetry is driven by D2C-P, whose ground truth is not model-generated. This is a potential fairness/contamination caveat, not a definitional reduction. The unvalidated Semantic Object Model pipeline and the near-universal low F1bbox are measurement-validity risks belonging to correctness assessment, not circularity. No self-citation chain or uniqueness theorem is load-bearing for the conclusions.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The benchmark itself introduces no new physical or mathematical entities. Its central measurements rest on hand-chosen metric thresholds (IoU, color similarity, numeric tolerance) and on domain assumptions about the compilation pipeline, LLM-judge reliability, and the quality of LLM-generated ground truth.

free parameters (3)
  • IoU threshold for bbox matching = 0.3
    Chosen by hand in Appendix C.2 for F1bbox matching (IoU >= 0.3). This threshold directly affects F1bbox, the weakest metric, and thus influences F1avg and the central finding that spatial grounding is poor.
  • Color similarity threshold = Delta E00 / 100
    In Appendix C.2, color F1 uses sim = max(0, 1 - DeltaE00/100). This mapping is hand-chosen and affects F1color.
  • DQA numeric tolerance = 1e-6 absolute or 1% relative
    Appendix C.3: numeric answers are judged correct within abs 1e-6 or rel 1%. This grading threshold influences reported DQA accuracy.
assumptions (4)
  • domain assumption The TikZ compilation pipeline (pdflatex/lualatex/xelatex fallback) renders faithful images and DVI/SVG for metric computation.
    Section D.2 and C.1: The entire evaluation assumes generated code can be compiled and rendered without introducing artifacts that distort metrics.
  • domain assumption The 200-sample human agreement study generalizes to the full 7,146 DQA grading set.
    Appendix C.4 validates the LLM judge on 200 stratified cases; the paper assumes the same agreement holds across the full set.
  • domain assumption The three annotator models (Gemini-3 Flash, GPT-5.2, Gemini-3 Pro) produce correct ground-truth edits and QA pairs after human cross-validation.
    Appendix B.2: Ground truth for D2C-E and DQA was generated and verified by these LLMs, then checked by 13 graduate students. The paper assumes the human review fully corrects any model errors.
  • standard math Standard similarity metrics (SSIM, CLIP, LPIPS, FID, CrystalBLEU) are valid for comparing diagrams and code.
    Section 3.4 uses established metrics without modification; treated as trusted background.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams." pith.science (2026). https://pith.science/paper/SIFGW64S

@misc{pith2026260812262,
  author       = {Pith},
  title        = {Pith review of: Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagrams},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SIFGW64S}},
  note         = {Machine review of arXiv:2608.12262}
}
read the original abstract

Multimodal Large Language Models (MLLMs) have been growing the capability for scientific writing and collaboration. For example, OpenAI Prism is a free workspace for scientific writing and collaboration. One important feature in Prism is turning scientific diagrams directly into LaTeX TikZ code. In this paper, we build a benchmark, Diagram-MMU, a multi-modal benchmark designed to assess MLLMs' ability for scientific diagram parsing and understanding. Diagram-MMU features 3.7k curated diagrams and 18.3k human-validated questions across six domains. It evaluates MLLMs on three tasks common in vibe writing workspaces: diagram-to-code parsing, diagram-to-code editing, and diagram question answering, alongside agentic settings per task. The evaluation of 12 MLLMs reveals that diagram-to-code tasks are more challenging than diagram question answering: models can reason well over diagrams but struggle to parse and edit them, underscoring the need for methods to enhance MLLMs' capability in diagram-to-code generation. Under agentic settings, most models improve parsing and editing performance but degrade on question answering, while Claude-4.6 Opus consistently improves across all three tasks. Project Page: https://vi-ocean.github.io/projects/diagram-mmu.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 42 canonical work pages

  1. [1]

    Minimax m2.5: Built for real-world productivity.https://www.minimax.io/news/minimax-m25, 2026

    Minimax. Minimax m2.5: Built for real-world productivity.https://www.minimax.io/news/minimax-m25, 2026

    work page 2026

  2. [2]

    Kimi k2: Open agentic intelligence.arXiv preprint arXiv:2507.20534, 2025

    Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence.arXiv preprint arXiv:2507.20534, 2025

    arXiv 2025

  3. [3]

    Glm-5: from vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763, 2026

    Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chengxing Xie, Cunxiang Wang, et al. Glm-5: from vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763, 2026

    arXiv 2026

  4. [4]

    Gemini 3.1 pro model card

    Google DeepMind. Gemini 3.1 pro model card. https://storage.googleapis.com/deepmind-media/ Model-Cards/Gemini-3-1-Pro-Model-Card.pdf, 2026

    work page 2026

  5. [5]

    Qwen3.5: Towards native multimodal agents.https://qwen.ai/blog?id=qwen3.5, 2026

    Qwen Team. Qwen3.5: Towards native multimodal agents.https://qwen.ai/blog?id=qwen3.5, 2026

    2026

  6. [6]

    Gpt-5.2.https://developers.openai.com/api/docs/models/gpt-5.2, 2025

    OpenAI. Gpt-5.2.https://developers.openai.com/api/docs/models/gpt-5.2, 2025

    work page 2025

  7. [7]

    Seed2.0 model card: Towards intelligence frontier for real-world complex- ity

    Bytedance Seed. Seed2.0 model card: Towards intelligence frontier for real-world complex- ity. https://lf3-static.bytednsdoc.com/obj/eden-cn/lapzild-tss/ljhwZthlaukjlkulzlp/seed2/0214/ Seed2.0%20Model%20Card.pdf, 2026

    work page 2026

  8. [8]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmarkforexpertagi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmarkforexpertagi. In Proceedings oftheIEEE/CVFconferenceoncomputervisionandpatternrecognition, pages 9556–9567, 2024

    2024

Show all 75 references
  1. [9]

    Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark

    Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, et al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. In Proceedings of the 63rd Annual Meeting of the Association for Computati...

    2025

  2. [10]

    Descriptive caption enhancement with visual specialists for multimodal perception.arXiv preprint arXiv:2412.14233, 2024

    Yanpeng Sun, Jing Hao, Ke Zhu, Jiang-Jiang Liu, Yuxiang Zhao, Xiaofan Li, Gang Zhang, Zechao Li, and Jingdong Wang. Descriptive caption enhancement with visual specialists for multimodal perception.arXiv preprint arXiv:2412.14233, 2024

    2024

  3. [11]

    Agentic learner with grow-and-refine multimodal semantic memory.arXiv preprint arXiv:2511.21678, 2025

    Weihao Bo, Shan Zhang, Yanpeng Sun, Jingjing Wu, Qunyi Xie, Xiao Tan, Kunbin Chen, Wei He, Xiao- fan Li, Na Zhao, et al. Agentic learner with grow-and-refine multimodal semantic memory.arXiv preprint arXiv:2511.21678, 2025

    2025 arXiv

  4. [12]

    Prism.https://openai.com/prism/, 2026

    OpenAI. Prism.https://openai.com/prism/, 2026

    2026

  5. [13]

    Charxiv: Charting gaps in realistic chart understanding in multimodal llms.Advances in Neural Information Processing Systems, 37:113569–113697, 2024

    Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, et al. Charxiv: Charting gaps in realistic chart understanding in multimodal llms.Advances in Neural Information Processing Systems, 37:113569–113697, 2024

    2024

  6. [14]

    Chartmimic: Evaluating lmm’s cross-modal reasoning capability via chart-to-code generation

    Cheng Yang, Chufan Shi, Yaxin Liu, Bo Shui, Junjie Wang, Mohan Jing, Linran XU, Xinyu Zhu, Siheng Li, Yuxiang Zhang, et al. Chartmimic: Evaluating lmm’s cross-modal reasoning capability via chart-to-code generation. In The Thirteenth International Conference on Learning Repres...

    2025

  7. [15]

    Math blind: Failures in diagram understanding undermine reasoning in mllms.arXiv preprint arXiv:2503.20745, 2025

    Yanpeng Sun, Shan Zhang, Wei Tang, Aotian Chen, Piotr Koniusz, Kai Zou, Yuan Xue, and Anton van den Hengel. Math blind: Failures in diagram understanding undermine reasoning in mllms.arXiv preprint arXiv:2503.20745, 2025

    2025

  8. [16]

    Viscodex: Unified multimodal code generation via merging vision and coding models.arXiv preprint arXiv:2508.09945, 2025

    Lingjie Jiang, Shaohan Huang, Xun Wu, Yixia Li, Dongdong Zhang, and Furu Wei. Viscodex: Unified multimodal code generation via merging vision and coding models.arXiv preprint arXiv:2508.09945, 2025

    2025 arXiv

  9. [17]

    Plot2code: A comprehensive benchmark for evaluating multi-modal large language models in code generation from scientific plots

    Chengyue Wu, Zhixuan Liang, Yixiao Ge, Qiushan Guo, Zeyu Lu, Jiahao Wang, Ying Shan, and Ping Luo. Plot2code: A comprehensive benchmark for evaluating multi-modal large language models in code generation from scientific plots. InFindings of the Association for Computational Li...

    2025

  10. [18]

    Starvector: Generating scalable vector graphics code from images and text

    Juan A Rodriguez, Abhay Puri, Shubham Agarwal, Issam H Laradji, Pau Rodriguez, Sai Rajeswar, David Vazquez, Christopher Pal, and Marco Pedersoli. Starvector: Generating scalable vector graphics code from images and text. In Proceedings of the Computer Vision and Pattern Recogn...

    2025

  11. [19]

    From words to structured visuals: A benchmark and framework for text-to-diagram generation and editing

    Jingxuan Wei, Cheng Tan, Qi Chen, Gaowei Wu, Siyuan Li, Zhangyang Gao, Linzhuang Sun, Bihui Yu, and Ruifeng Guo. From words to structured visuals: A benchmark and framework for text-to-diagram generation and editing. In Proceedings of the Computer Vision and Pattern Recognitio...

    2025

  12. [20]

    Automatikz: Text-guided synthesis of scientific vector graphics with tikz

    Jonas Belouadi, Anne Lauscher, and Steffen Eger. Automatikz: Text-guided synthesis of scientific vector graphics with tikz. InThe TwelfthInternational Conference on Learning Representations

  13. [21]

    Detikzify: Synthesizing graphics programs for scientific figures and sketches with tikz.Advances in Neural Information Processing Systems, 37:85074–85108, 2024

    Jonas Belouadi, Simone Ponzetto, and Steffen Eger. Detikzify: Synthesizing graphics programs for scientific figures and sketches with tikz.Advances in Neural Information Processing Systems, 37:85074–85108, 2024

    2024

  14. [22]

    Image2struct: Benchmarking structure extraction for vision-language models

    Josselin S Roberts, Tony Lee, Chi H Wong, Michihiro Yasunaga, Yifan Mai, and Percy Liang. Image2struct: Benchmarking structure extraction for vision-language models. Advances in Neural Information Processing Systems, 37:115058–115097, 2024

    2024

  15. [23]

    ChartE3: A comprehensive benchmark for end-to-end chart editing.arXiv preprint arXiv:2601.21694, 2026

    Shuo Li, Jiajun Sun, Zhekai Wang, Xiaoran Fan, Hui Li, Dingwen Yang, Zhiheng Xi, Yijun Wang, Zifei Shan, Tao Gui, et al. ChartE3: A comprehensive benchmark for end-to-end chart editing.arXiv preprint arXiv:2601.21694, 2026

    2026

  16. [24]

    Chartm3: Benchmarking chart editing with multimodal instructions

    Donglu Yang, Liang Zhang, Zihao Yue, Liangyu Chen, Yichen Xu, Wenxuan Wang, and Qin Jin. Chartm3: Benchmarking chart editing with multimodal instructions. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 5001–5009, 2025

    2025

  17. [25]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. InFindings of the association for computational linguistics: ACL 2022, pages 2263–2279, 2022

    2022

  18. [26]

    Can large language models understand symbolic graphics programs? In The Thirteenth International Conference on Learning Representations, 2025

    Zeju Qiu, Weiyang Liu, Haiwen Feng, Zhen Liu, Tim Z Xiao, Katherine M Collins, Joshua B Tenenbaum, Adrian Weller, Michael J Black, and Bernhard Schölkopf. Can large language models understand symbolic graphics programs? In The Thirteenth International Conference on Learning Re...

    2025

  19. [27]

    Model context protocol.https://modelcontextprotocol.io, 2024

    Anthropic. Model context protocol.https://modelcontextprotocol.io, 2024

    2024

  20. [28]

    Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233. Springer, 2024

    2024

  21. [29]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. InProceedings of the IEEE/CVF conference on ...

    2025

  22. [30]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In The TwelfthInternational Conference on Learning ...

  23. [31]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186. Sprin...

    2024

  24. [32]

    Image quality assessment: from error visibility to structural similarity.IEEE transactions on image processing, 13(4):600–612, 2004

    Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity.IEEE transactions on image processing, 13(4):600–612, 2004

    2004

  25. [33]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  26. [34]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018

    2018

  27. [35]

    Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems, 30, 2017

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems, 30, 2017. 16

    2017

  28. [36]

    Crystalbleu: precisely and efficiently measuring the similarity of code

    Aryaz Eghbali and Michael Pradel. Crystalbleu: precisely and efficiently measuring the similarity of code. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pages 1–12, 2022

    2022

  29. [37]

    Worldvqa: Measuring atomic world knowledge in multimodal large language models.arXiv preprint arXiv:2602.02537, 2026

    Runjie Zhou, Youbo Shao, Haoyu Lu, Bowei Xing, Tongtong Bai, Yujie Chen, Jie Zhao, Lin Sui, Haotian Yao, Zijia Zhao, et al. Worldvqa: Measuring atomic world knowledge in multimodal large language models.arXiv preprint arXiv:2602.02537, 2026

    2026

  30. [38]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, 2022

    2022

  31. [39]

    Agentic artificial intelligence (ai): Architectures, taxonomies, and evaluation of large language model agents.arXiv preprint arXiv:2601.12560, 2026

    Rajkumar Buyya et al. Agentic artificial intelligence (ai): Architectures, taxonomies, and evaluation of large language model agents.arXiv preprint arXiv:2601.12560, 2026

    2026

  32. [40]

    Ai, agentic models and lab automation for scientific discovery—the beginning of scaince

    Thomas Hartung. Ai, agentic models and lab automation for scientific discovery—the beginning of scaince. Frontiersin Artificial Intelligence, 8:1649155, 2025

    2025

  33. [41]

    Human-centric foundation models: Perception, generation and agentic modeling.arXiv preprint arXiv:2502.08556, 2025

    Shixiang Tang, Yizhou Wang, Lu Chen, Yuan Wang, Sida Peng, Dan Xu, and Wanli Ouyang. Human-centric foundation models: Perception, generation and agentic modeling.arXiv preprint arXiv:2502.08556, 2025

    2025 arXiv

  34. [42]

    Babyvision: Visual reasoning beyond language.arXiv preprint arXiv:2601.06521, 2026

    Liang Chen, Weichu Xie, Yiyan Liang, Hongfeng He, Hans Zhao, Zhibo Yang, Zhiqi Huang, Haoning Wu, Haoyu Lu, Yiping Bao, et al. Babyvision: Visual reasoning beyond language.arXiv preprint arXiv:2601.06521, 2026

    2026 arXiv

  35. [43]

    Gemini 3 pro model card.https://storage.googleapis.com/deepmind-media/Model-Cards/ Gemini-3-Pro-Model-Card.pdf, 2025

    Google DeepMind. Gemini 3 pro model card.https://storage.googleapis.com/deepmind-media/Model-Cards/ Gemini-3-Pro-Model-Card.pdf, 2025

    2025

  36. [44]

    Gemini 3 flash model card

    Google DeepMind. Gemini 3 flash model card. https://storage.googleapis.com/deepmind-media/ Model-Cards/Gemini-3-Flash-Model-Card.pdf, 2025

    2025

  37. [45]

    Claude opus 4.6.https://www.anthropic.com/claude/opus, 2026

    Anthropic. Claude opus 4.6.https://www.anthropic.com/claude/opus, 2026

    2026

  38. [46]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    2025 arXiv

  39. [47]

    Kimi k2.5: Visual agentic intelligence.arXiv preprint arXiv:2602.02276, 2026

    Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao, SH Cai, Yuan Cao, Y Charles, HS Che, Cheng Chen, Guanduo Chen, et al. Kimi k2.5: Visual agentic intelligence.arXiv preprint arXiv:2602.02276, 2026

    2026 arXiv

  40. [48]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479, 2025

    2025 arXiv

  41. [49]

    Tikzero: Zero-shot text-guided graphics program synthesis

    Jonas Belouadi, Eddy Ilg, Margret Keuper, Hideki Tanaka, Masao Utiyama, Raj Dabre, Steffen Eger, and Simone Ponzetto. Tikzero: Zero-shot text-guided graphics program synthesis. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 17793–17806, 2025

    2025

  42. [50]

    Nodes are early, edges are late: Probing diagram representations in large vision-language models.arXiv preprint arXiv:2603.02865, 2026

    Haruto Yoshida, Keito Kudo, Yoichi Aoki, Ryota Tanaka, Itsumi Saito, Keisuke Sakaguchi, and Kentaro Inui. Nodes are early, edges are late: Probing diagram representations in large vision-language models.arXiv preprint arXiv:2603.02865, 2026

    2026

  43. [51]

    Chain-of-region: Visual language models need details for diagram analysis

    Xue Li, Yiyou Sun, Wei Cheng, Yinglun Zhu, and Haifeng Chen. Chain-of-region: Visual language models need details for diagram analysis. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  44. [52]

    Challenges and feasibility of multimodal llms in er diagram evaluation

    Mojdeh Rahmanian, Ashkan Sami, and Yanchao Yu. Challenges and feasibility of multimodal llms in er diagram evaluation. Cogent Education, 12(1):2590901, 2025

    2025

  45. [53]

    Ontology-driven multimodal framework for automated interpretation and description of architecture diagrams

    Sakthivel Thangaraj, Neelesh Kumar Shukla, and Viji Krishnamurthy. Ontology-driven multimodal framework for automated interpretation and description of architecture diagrams. In2025 IEEE International Conference on Big Data (BigData), pages 2493–2502. IEEE, 2025

    2025

  46. [54]

    Overcoming vision language model challenges in diagram understanding: A proof-of-concept with xml-driven large language models solutions.arXiv preprint arXiv:2502.04389, 2025

    Shue Shiinoki, Ryo Koshihara, Hayato Motegi, and Masumi Morishige. Overcoming vision language model challenges in diagram understanding: A proof-of-concept with xml-driven large language models solutions.arXiv preprint arXiv:2502.04389, 2025

    2025 arXiv

  47. [55]

    Bowen Yu and Cláudio T. Silva. Flowsense: A natural language interface for visual data exploration within a dataflow system. IEEE Transactions on Visualization and Computer Graphics, 26(1):1–11, 2020. 17

    2020

  48. [56]

    Editclip: Representation learning for image editing

    Qian Wang, Aleksandar Cvejić, Abdelrahman Eldesokey, and Peter Wonka. Editclip: Representation learning for image editing. pages 15960–15970, 2025

    2025

  49. [57]

    Sketchagent: Generating structured diagrams from hand-drawn sketches

    Cheng Tan, Qi Chen, Jingxuan Wei, Gaowei Wu, Zhangyang Gao, Siyuan Li, Bihui Yu, Ruifeng Guo, and Stan Z Li. Sketchagent: Generating structured diagrams from hand-drawn sketches. 2025

    2025

  50. [58]

    Davinci: Reinforcing visual-structural syntax in mllms for generalized scientific diagram parsing

    ZENG Xingchen, Zhewei Su, Hengming Zhang, Juyong Jiang, Jiazhi Xia, and Wei Zeng. Davinci: Reinforcing visual-structural syntax in mllms for generalized scientific diagram parsing. InThe Fourteenth International Conference on Learning Representations, 2026

    2026

  51. [59]

    Visual autoregressive modeling for instruction-guided image editing.arXiv preprint arXiv:2508.15772, 2025

    Qingyang Mao, Qi Cai, Yehao Li, Yingwei Pan, Mingyue Cheng, Ting Yao, Qi Liu, and Tao Mei. Visual autoregressive modeling for instruction-guided image editing.arXiv preprint arXiv:2508.15772, 2025

    2025

  52. [60]

    Vibe coding as a reconfiguration of intent mediation in software development: Definition, implications, and research agenda

    Christian Meske, Tobias Hermanns, Esther Von der Weiden, Kai-Uwe Loser, and Thorsten Berger. Vibe coding as a reconfiguration of intent mediation in software development: Definition, implications, and research agenda. IEEE Access, 13:213242–213259, 2025

    2025

  53. [61]

    Tikzilla: Scaling text-to-tikz with high-quality data and reinforcement learning

    Christian Greisinger and Steffen Eger. Tikzilla: Scaling text-to-tikz with high-quality data and reinforcement learning. arXiv preprint arXiv:2603.03072, 2026

    2026

  54. [62]

    Mathcoder-vl: Bridging vision and code for enhanced multimodal mathematical reasoning

    Ke Wang, Junting Pan, Linda Wei, Aojun Zhou, Weikang Shi, Zimu Lu, Han Xiao, Yunqiao Yang, Houxing Ren, Mingjie Zhan, et al. Mathcoder-vl: Bridging vision and code for enhanced multimodal mathematical reasoning. In Findings of the Association for Computational Linguistics: ACL...

    2025

  55. [63]

    model-in-the-loop

    Sher Badshah, Moamen Moustafa, and Hassan Sajjad. Clev: Llm-based evaluation through lightweight efficient voting for free-form question-answering. InProceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacifi...

    2025

  56. [64]

    Output the COMPLETE LaTeX code inside a single“‘latexcode block

  57. [65]

    Start from\documentclassand end with\end{document}

  58. [66]

    Do NOT output any text, explanation, or reasoning outside the code block. S1 and S6 share the same system prompt; the task is distinguished by the user message: S1 sends only the diagram image, while S6 additionally includes an editing textual instruction. S12: DQA Direct Answ...

  59. [68]

    If you need specific TikZ syntax or package usage, use theSearchLaTeXKnowledgeBasetool

  60. [69]

    Generate the COMPLETE LaTeX code inside a single“‘latexcode block 47

  61. [70]

    Before writing code, analyze the image systematically

    Start from\documentclassand end with\end{document} S4: + Model Generated Objects (State Management) You are a LaTeX expert. Before writing code, analyze the image systematically. Step 1 — Perception(inside<perception>tags):

  62. [71]

    Start from\documentclassand end with\end{document}

    Spatial layout Step 2 — Code Generation:Based on your perception, output the COMPLETE LaTeX code inside a single“‘latex code block. Start from\documentclassand end with\end{document}. S5: + Objects & TikZ Search T ool (Planning) You are a LaTeX expert with access to a TikZ doc...

  63. [72]

    Observe the diagram image carefully

  64. [73]

    If you need specific TikZ syntax or package usage for the edit task, use theSearchLaTeXKnowledgeBasetool

  65. [74]

    Apply the user’s modification instruction

  66. [75]

    Output the final modified code inside a single“‘latexcode block

  67. [76]

    addplot bar chart stacked

    Start from\documentclassand end with\end{document} S9: + Required TikZ Codes (State Management) You are a LaTeX expert. Complete this task in two steps within a single response. Step 1 — Reconstruct Original Code:Look at the diagram image and generate complete LaTeX code for t...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.