{"id":"98019ff4-efd3-42c7-8543-a7bc5534177c","arxiv_id":"2608.12262","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A benchmark of 3,744 TikZ scientific diagrams and 18.3k human-validated questions shows models answer diagram questions well (up to 86% accuracy) but parse diagrams into code poorly (object-level F1 31-57%), with agentic tools helping coding but hurting question answering.","lead":"Diagram-MMU is a new benchmark that tests how well multimodal AI models can turn scientific diagrams into LaTeX code, edit them, and answer questions about them. It finds that current models are much better at answering questions about diagrams than at generating or editing the code that draws them.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The D2C-P object-level F1 metric is unvalidated and F1bbox is implausibly low across all models, so the headline coding-vs-understanding asymmetry may be partly a metric artifact.","rationale":"The reader's weakest assumption correctly identifies the unvalidated SOM pipeline as the main load-bearing risk. The paper reports high DQA accuracy and relatively high F1type/F1text/F1color but near-zero F1bbox across all models; that pattern is exactly what one would expect if the bbox extraction or matching stage is broken, and it is what drives F1avg down to the 31–57% range. Because the central asymmetry claim is quantified with this F1avg, a robustness check on the metric is the single most decisive experiment: it separates genuine diagram-to-code failure from artifacts of non-unique TikZ code and order-sensitive IoU matching. I agree with the reader's conditional verdict; no verdict change is needed, but the condition should explicitly include validation of the SOM pipeline and release of the evaluation code so that Table 5 can be reproduced and re-examined. The paper does have independent strengths: the benchmark is large, human cross-validation is documented, DQA judge agreement with humans is measured (Appendix C.4), and the agentic mini-split results are internally consistent; however, those strengths do not remove the need to validate the metric that carries the headline finding.","tokens_in":49271,"tokens_out":5767,"duration_ms":61265,"concrete_test":"Choose 100 D2C-P diagrams. For each, create three semantically equivalent TikZ variants by non-semantic refactorings (renaming node aliases, reordering draw commands, splitting one path into two, translating all coordinates by a constant while preserving relative layout, and using equivalent syntax such as -- vs to). Run the SOM pipeline on original-vs-variant pairs and record F1avg and F1bbox. If F1avg falls materially below ~90 or F1bbox below ~50 on these perfect-match pairs, the metric conflates valid code variation with error; then fix the matching/extraction (e.g., Hungarian bbox assignment and human-validated SVG extraction on 50 diagrams) and recompute Table 5 to see whether the 31–57% D2C-P range and the DQA-vs-D2C-P gap survive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on comparing DQA accuracy (up to 86%) with D2C-P object-level F1 (31–57%). The object-level comparison depends entirely on the Semantic Object Model (SOM) pipeline in Appendix C.1: TikZ is compiled to DVI, converted to SVG by dvisvgm, and parsed into type, text, color, and bbox attributes, including glyph reconstruction from character codes and greedy IoU>=0.3 bbox matching. This pipeline is never validated against human-annotated object ground truth, and its output is suspicious: F1bbox is 7.8–17.1 for all six representative models, while F1type/F1text/F1color are 60–71 (Table G.1). A universal near-zero bbox score suggests a systematic extraction or matching problem rather than a universal spatial-grounding failure. The greedy matching iterates ground-truth boxes in document order and removes the first predicted box with IoU>=0.3, so correct boxes in a different order can be mismatched; dvisvgm glyph reconstruction and SVG transform handling are also unverified. TikZ is highly non-unique: equivalent diagrams can be drawn with different command order, alias names, path decompositions, or coordinate systems, and the metric has no stated tolerance for such variation. If the pipeline penalizes valid code variants or mis-extracts bboxes, the F1avg gap between DQA and D2C-P is inflated, and the paper's central asymmetry—that diagram understanding is strong while diagram-to-code is the binding constraint—is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Diagram-MMU, a benchmark of 3,744 curated TikZ-sourced diagrams with 18,305 human-validated evaluation instances spanning three tasks: diagram-to-code parsing (D2C-P), diagram-to-code editing (D2C-E), and diagram question answering (DQA), together with agentic variants of each task. Twelve MLLMs are evaluated. The headline finding is an asymmetry: models reason well over diagrams (DQA accuracy up to 86%) but perform poorly at diagram-to-code parsing (reported object-level F1avg of 31-57%), suggesting that diagram-to-code generation, rather than diagram understanding, is the binding constraint for AI-assisted scientific writing. The paper also reports that agentic settings improve parsing and editing for most models but degrade DQA, with Claude-4.6 Opus improving across all three tasks. The central quantitative claim rests on a custom object-level evaluation pipeline and on comparing DQA accuracy with D2C-P F1avg, both of which are examined critically below.","tokens_in":49623,"tokens_out":5787,"duration_ms":54653,"significance":"If the measurements are reliable, Diagram-MMU is a valuable contribution: it is the first TikZ-based benchmark covering six diagram domains with three tasks plus agentic settings, the data curation is careful (13 graduate students, cross-validation, source filtering), the DQA judge is validated against human agreement with kappa 0.937 on 200 samples, and the MCP-based TikZ documentation server is a useful reusable infrastructure piece. The paper also provides falsifiable predictions about model weaknesses (spatial grounding, planning, tool-use loops). However, the central asymmetry claim is currently supported mainly by an unvalidated object-extraction pipeline and by a cross-task comparison of non-equivalent metrics; the circularity in ground-truth generation is also a real concern. These issues are addressable, but they are load-bearing for the paper's main conclusion.","major_comments":[{"comment":"The Semantic Object Model (SOM) pipeline that produces the headline D2C-P F1avg values is never validated against human-annotated object ground truth. The reported numbers are internally suspicious: in Table G.1, F1type/F1text/F1color are roughly 60-71 for most models while F1bbox is only 7.8-17.1 for all six representative models, a uniform collapse that is more consistent with a systematic extraction or matching artifact than with a universal spatial-grounding failure. The greedy IoU>=0.3 matching iterates ground-truth boxes in document order and removes the first matching predicted box (Appendix C.2), so semantically correct boxes in a different order can be missed, and the pipeline has no stated tolerance for the many equivalent TikZ encodings of the same diagram (e.g., different coordinate origins, alias names, or path decompositions). Because the paper's central claim that coding is harder than understanding is based on this F1avg, the authors need to validate the SOM pipeline on a human-annotated sample (reporting extraction precision/recall per object dimension), replace the order-dependent greedy matching with an order-invariant matching procedure, and run a sensitivity analysis of the IoU threshold and color tolerance. Without this, the reported D2C-P numbers cannot be taken at face value.","section":"3.4 / Appendix C.1-C.2 / Table G.1"},{"comment":"There is a mild circularity in the benchmark construction: the D2C-E and DQA ground-truth answers were generated by Gemini-3 Flash and verified by GPT-5.2 and Gemini-3 Pro (Appendix B.2), and all three of these models are later evaluated in Table 5. This can systematically favor exactly the models the paper highlights, for example Gemini-3.0 Pro's highest DQA accuracy and its characterization as the most balanced profile. The paper does not analyze how the rankings change as a function of ground-truth provenance. The authors should construct a model-independent validation subset or report per-model scores split by whether the ground-truth item was generated or verified by that model, and ideally show that the main conclusions survive when the evaluated models play no role in creating the gold labels.","section":"Appendix B.2 / Table 5"},{"comment":"The headline asymmetry compares DQA accuracy (a semantic correctness rate) with D2C-P F1avg (an average of four exact-reconstruction dimensions, one of which requires IoU>=0.3 bounding-box overlap). These metrics operate at different strictness levels: answering 'what is the degree of node A?' correctly is not the same as reconstructing every object, label, color, and spatial position of a diagram. A low object-level F1avg therefore does not by itself establish that models 'struggle to code' in a practically meaningful sense; it may simply reflect that exact reconstruction is a much stricter criterion than semantic question answering. To support the claim that coding is the binding constraint, the authors should supplement the object-level comparison with human judgments of whether generated diagrams are visually or semantically faithful to the target, or report the fraction of generated diagrams that a human would accept as faithful reproductions. As written, the evidence is consistent with the stated asymmetry but does not uniquely establish it.","section":"Section 1 / Table 5"}],"minor_comments":[{"comment":"The heading 'Diagram-to-Code Pasing' contains a typo; it should read 'Diagram-to-Code Parsing'.","section":"3.2 heading"},{"comment":"The text refers to 'D2C-E and DU' where the intended abbreviation appears to be DQA; please correct this for clarity.","section":"4.1, item (3)"},{"comment":"The radar panels use per-panel axis scales (e.g., F1bbox ranges differently from F1type), which makes the bbox collapse harder to compare across panels; consider using common axis ranges or adding explicit numerical labels on each axis.","section":"Figure 6"},{"comment":"The paper says D2C-P object-level F1 'ranges 31-57%' but the specialist TikZero+10B scores 15.43 in Table 5; please clarify whether the range intentionally excludes the specialist model and state the inclusion/exclusion criterion.","section":"Table 5 / text after Eq. (5)"},{"comment":"The edge-case rule that F1=1.0 when both predicted and ground-truth sets are empty is reasonable, but it should also be mentioned in the main text where F1avg is first defined, since it can inflate scores for diagrams with few objects of a given type.","section":"Appendix C.2"}],"recommendation":"major_revision","confidential_remarks":"This is a well-executed benchmark paper with strong data curation and a careful DQA judge validation. The main risk is that the central asymmetry claim depends on an unvalidated object-extraction pipeline and on comparing non-equivalent metrics; the circularity in ground-truth generation is a secondary but real concern. I would not reject the paper, but the authors should be required to validate the SOM pipeline, re-run the headline numbers with order-invariant and human-validated matching, and address the provenance sensitivity before the main conclusion can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this one. First, it's a genuine contribution: it's the first TikZ-based benchmark to combine diagram-to-code parsing, editing, and QA across six diagram types, with a 16-setting design that cleanly separates foundational from agentic ability, plus an MCP-based TikZ documentation search tool. The data work is the strongest part — 3,744 curated diagrams, 18.3k human-cross-validated instances, and the DQA judge is validated against human agreement (kappa 0.937 on 200 samples). The headline finding, models answer diagram questions at 68–86% accuracy while parsing diagrams into code at only 31–57% object F1, is plausible and visible even in the qualitative examples.\n\nSecond, the soft spot is exactly where the stress-test puts it: the object-level F1 pipeline (Appendix C.1) is never validated against human-annotated object ground truth. The F1bbox numbers are the tell. Every one of the six models scores between 7.8 and 17.1 on bbox while scoring 60–71 on type, text, and color. A universal near-zero bbox smells like a systematic extraction or matching artifact, not universal spatial-grounding failure. Order-dependent greedy matching at IoU>=0.3, dvisvgm glyph reconstruction, and zero tolerance for TikZ's non-uniqueness (equivalent diagrams with different command order, aliases, or coordinate decompositions) will all punish valid code. Until the pipeline is checked against human-annotated objects, the object-level magnitudes shouldn't be read literally.\n\nThat caveat doesn't destroy the central asymmetry, though. The same models get 15–38 CrystalBLEU and 62–77 image-level similarity, so code generation is clearly harder than answering regardless of the bbox artifact; the magnitude of the gap, especially the bbox component, is what's overstated.\n\nOther notes, in proportion: the D2C-E and DQA ground truths were generated by Gemini-3 Flash and verified by GPT-5.2 and Gemini-3 Pro, two of the evaluated models — mild circularity, disclosed, diluted by the later human cross-validation, and it doesn't touch the D2C-P ground truth that drives the headline gap. No error bars on pass@1 at temperature 1.0 is a minor omission. The dataset and code aren't released yet, which is the one thing that actually determines whether a benchmark gets adopted.\n\nWho this is for: MLLM evaluation people and anyone building diagram-to-code or scientific-writing agents. The agentic results (tools help editing more than parsing; planning is the weakest capability) are useful data. It deserves a serious referee — the main asks should be SOM pipeline validation, release of data and code, and error bars.","headline":"A serious TikZ diagram benchmark worth a real review: the coding-vs-reasoning asymmetry is probably real, but the object-level F1 pipeline quantifying it is unvalidated and its bbox numbers look like an artifact.","tokens_in":50172,"tokens_out":4954,"would_cite":true,"duration_ms":43442,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that current multimodal models answer scientific-diagram questions with up to 86% accuracy yet reconstruct their TikZ code at only 31-57% object-level F1, so diagram-to-code generation, not diagram understanding, is the…","keywords":["diagram-to-code generation","TikZ","multimodal large language models","benchmark","scientific diagrams","diagram question answering","agentic evaluation","object-level F1"],"falsifier":"Take a random sample of generated and ground-truth TikZ pairs from Diagram-MMU, have two independent annotators manually list every object with its type, text, color, and bounding box, and compute annotator-versus-SOM agreement on the same sample; if per-dimension F1 agreement falls below roughly 0.85, the reported 31-57% parsing F1 range would not be a trustworthy measure of code quality.","tokens_in":49102,"feed_emoji":"📐","tokens_out":7218,"duration_ms":60314,"temperature":0.7,"pith_summary":"Diagram-MMU is a benchmark of 3,744 curated scientific diagrams and 18,305 human-validated questions spanning charts, planar geometry, 3D shapes, graphs, chemistry, and circuits. The paper's central claim is that today's multimodal language models can reason over diagrams, reaching up to 86% accuracy on diagram question answering, but struggle to reproduce them as LaTeX TikZ code, with object-level F1 between 31% and 57% across twelve models. The benchmark adds a diagram-to-code editing task and, for each task, sixteen settings that test foundational ability plus agentic capabilities such as tool use, context utilization, state management, and planning. On the agentic settings, most models improve at editing, degrade at question answering, and Claude-4.6 Opus is the only model that improves on all three tasks. If this asymmetry holds, then improving diagram-to-code generation, not visual reasoning, is the key to AI-assisted scientific writing.","feed_headline":"AI models ace diagram questions, stumble on diagram code","feed_subtitle":"A 3.7k-diagram benchmark shows up to 86% accuracy on diagram Q&A but 31-57% object-level F1 on TikZ code.","key_machinery":"The central machinery is the Semantic Object Model (SOM) pipeline: TikZ code is compiled to DVI with injected semantic tags, converted to SVG via dvisvgm, and parsed into typed, attributed elements such as nodes, paths, text labels, data series, and circuit components, whose type, text, color, and bounding box are scored by four F1 measures with greedy one-to-one matching, where text uses exact string match, color uses permutation-based CIEDE2000 assignment, and bbox uses IoU at least 0.3. These object-level F1 scores sit alongside code-level CrystalBLEU and image-level SSIM, CLIP, LPIPS, and FID, and for editing the object and code metrics are split into preserve-only and edit-only partitions. Around this machinery, the benchmark builds 16 evaluation settings: three foundational settings for direct parsing, editing, and answering, plus thirteen agentic variants that add perception objects as context, a TikZ search tool served over the Model Context Protocol, required or optional intermediate code generation, and planning combinations of these. The SOM metrics are what turn the statement that generated code looks plausible into a measurable claim about whether the model actually perceived the right objects at the right places.","core_discovery":"The discovery is an asymmetry in current MLLMs: perception and coding lag reasoning. Across six scientific diagram domains, models answer descriptive and reasoning questions about diagrams accurately, up to 86.46% on the best model, yet when asked to parse a diagram into compilable TikZ code they reach only 31-57% object-level F1, and when asked to edit diagrams they score even lower on the edit-only partition. The paper attributes this to weak fine-grained spatial grounding: the F1bbox dimension is consistently the lowest, with strong models scoring 62-71 on type, text, and color but 8.0-12.7 on bounding boxes. In agentic settings, providing object perception data and a TikZ documentation search tool helps editing more than parsing, while most models degrade on question answering when asked to plan multi-step workflows; Claude-4.6 Opus is the exception that improves on all three tasks. The paper also introduces the first benchmark to cover chemistry and circuit diagrams for diagram-to-code tasks, and the first to evaluate both foundational and agentic abilities in one controlled setup.","pith_inferences":["My inference: because most models degrade on DQA when forced to generate TikZ code first in settings S14 and S15, code-first reasoning pipelines are likely to underperform direct visual answering until diagram-to-code fidelity improves; this is testable by comparing S12 with S14 on the released benchmark.","My inference: the near-zero edit-only CrystalBLEU values (0.51-2.98) together with moderate object F1 suggest the two metric families measure different failure modes, so a model can name the right objects yet write non-idiomatic code, and future work should analyze disagreement cases to decide which metric better predicts human edit quality.","My inference: the SOM pipeline, once validated against human object annotations, could serve as a reusable automatic metric for TikZ code quality beyond this benchmark, enabling larger-scale training and evaluation of diagram-to-code models.","My inference: for practitioners, the cheapest immediate gain is to supply models with explicit object perception data for editing tasks and to disable tool use for question answering, where it currently costs most models accuracy."],"forward_implications":["Diagram-to-code generation, not diagram understanding, is the binding constraint for MLLM-assisted scientific writing, so training and fine-tuning should target object-level spatial grounding in code output.","Object-level spatial grounding (F1bbox) is the weakest perception dimension, so methods that improve coordinate and layout encoding should lift both parsing and editing scores.","Textual editing instructions partially compensate for coding weakness, implying that routing edits through text-to-code abilities is a practical way to improve diagram editing today.","Agentic tool use and perception context help editing but hurt question answering for most models, and planning is the weakest agentic capability, so tool access should be gated per task.","A benchmark for vibe-writing workspaces must report parsing, editing, and answering separately, since a single accuracy number hides the coding gap."],"supporting_citations":[{"why":"CharXiv defines the chart-question-answering benchmark that Diagram-MMU extends from charts to six diagram domains.","marker":"[13]"},{"why":"ChartMimic supplies the permutation-based CIEDE2000 color matching and chart-to-code task lineage used in the object-level F1.","marker":"[14]"},{"why":"AutomaTikZ is the prior TikZ diagram-to-code benchmark whose CrystalBLEU evaluation is adopted here.","marker":"[20]"},{"why":"DeTikZify provides TikZ synthesis methods and a benchmark that D2C-P builds on and extends to new domains.","marker":"[21]"},{"why":"Image2Struct contributes the four image-level metrics (SSIM, CLIP, LPIPS, FID) used to compare rendered output.","marker":"[22]"},{"why":"The Model Context Protocol is the standard on which the TikZ search tool server is implemented for agentic tool use.","marker":"[27]"},{"why":"CrystalBLEU defines the code-level syntactic similarity metric, including its filtering of trivial n-grams.","marker":"[36]"},{"why":"TikZero is the specialist diagram-to-TikZ baseline whose much lower F1 of 15.43 sharpens the finding that coding is hard.","marker":"[49]"}],"fun_headline_variants":["AI aces diagram Q&A, flops at TikZ code","Diagram-MMU: AI answers, but can't code diagrams","AI's diagram code lags reasoning - spatial grounding gap","From diagrams to TikZ: AI's weak spot exposed","MLLMs reason diagrams well, code them poorly"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central asymmetry claim rests on the Semantic Object Model pipeline's automated extraction of object types, text, colors, and bounding boxes from compiled TikZ code, which is never checked against human-annotated object ground truth; if that extraction mis-matches elements, the reported F1 values misstate true code quality.","fun_headline_variants_meta":{"raw":{"variants":["AI aces diagram Q&A, flops at TikZ code","Diagram-MMU: AI answers, but can't code diagrams","AI's diagram code lags reasoning - spatial grounding gap","From diagrams to TikZ: AI's weak spot exposed","MLLMs reason diagrams well, code them poorly"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00056,"raw_usage":{"total_tokens":2691,"prompt_tokens":1004,"completion_tokens":1687,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":1604}},"tokens_in":620,"tokens_out":1687,"duration_ms":12566,"temperature":1.0,"reasoning_tokens":1604,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:10:24.381800+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of generated and ground-truth TikZ pairs from Diagram-MMU, have two independent annotators manually list every object with its type, text, color, and bounding box, and compute annotator-versus-SOM agreement on the same sample; if per-dimension F1 agreement falls below roughly 0.85, the reported 31-57% parsing F1 range would not be a trustworthy measure of code quality.","supporting_citations":[{"cited_title":"Chartmimic: Evaluating lmm’s cross-modal reasoning capability via chart-to-code generation","cited_arxiv_id":null,"evidence_quote":"ChartMimic supplies the permutation-based CIEDE2000 color matching and chart-to-code task lineage used in the object-level F1."},{"cited_title":"Automatikz: Text-guided synthesis of scientific vector graphics with tikz","cited_arxiv_id":null,"evidence_quote":"AutomaTikZ is the prior TikZ diagram-to-code benchmark whose CrystalBLEU evaluation is adopted here."},{"cited_title":"Model context protocol.https://modelcontextprotocol.io, 2024","cited_arxiv_id":null,"evidence_quote":"The Model Context Protocol is the standard on which the TikZ search tool server is implemented for agentic tool use."},{"cited_title":"Crystalbleu: precisely and efficiently measuring the similarity of code","cited_arxiv_id":null,"evidence_quote":"CrystalBLEU defines the code-level syntactic similarity metric, including its filtering of trivial n-grams."}],"review_version":1}