{"id":"cd70bf0e-417d-4071-8801-cf728a7a0094","arxiv_id":"2411.11916","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"DiagramAgent, a four-module agent system, claims state-of-the-art results on DiagramGenBenchmark for generating, coding, and editing structured diagrams from text and images.","lead":"This paper introduces a benchmark and an agent framework for generating and editing structured diagrams, such as flowcharts and model architectures, from text. It reports that the framework outperforms existing text-to-image and text-to-code baselines on accuracy and editability.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fine-tuning asymmetry confounds the headline outperformance claim: DiagramAgent's Code Agent is trained on the benchmark while all baselines are zero-shot, so the reported gaps do not demonstrate a framework advantage.","rationale":"The reader's weakest_assumption is the validity of the auto-generated benchmark ground truth; that is a real concern for the benchmark's meaning. However, the single most load-bearing issue for the paper's central claim is the unfair comparison between DiagramAgent and the baselines. The fine-tuned Code Agent is trained on the same distribution as the test set, while all baselines are zero-shot, so the reported Pass@1, ROUGE-L, and CodeBLEU advantages do not isolate the contribution of the multi-agent framework. Even if the ground truth were perfectly valid, the headline 'DiagramAgent significantly outperforms existing baselines' would not follow from these experiments. The reader's rationale does mention the fine-tuning asymmetry, but the stated weakest_assumption focuses on ground truth validity, so there is partial agreement. The paper's benchmark construction and agent workflow are valuable contributions, and the ablations show the Check Agent helps, but the performance claim needs a controlled comparison before it can be accepted at face value. Hence the reader's CONDITIONAL verdict remains appropriate: the concern is substantial but addressable by adding fine-tuned baselines and releasing code and data.","tokens_in":23894,"tokens_out":5674,"duration_ms":52730,"concrete_test":"Take the strongest baseline, Qwen2.5-Coder-7B, and fine-tune it on the same 6,713-sample generation split for four epochs with the same 8192-token limit; evaluate on the 270 test instances. If the fine-tuned baseline reaches Pass@1 within, say, 3 points of DiagramAgent's 58.15 (or similarly for editing with the 1,400-sample split), the framework-specific gain is not established. Additionally, run DiagramAgent's Code Agent without any fine-tuning (base Qwen2.5-Coder-7B + Plan/Check) on the same test set; if it falls far below the fine-tuned version while still beating zero-shot baselines, the reported superiority is attributable to fine-tuning rather than the multi-agent design.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—DiagramAgent significantly outperforms existing baselines—is not supported by the reported experiments because the comparison is not controlled. In Sec. 6 (Setup), the Code Agent is 'fine-tuned for four epochs' on DiagramGenBenchmark's training split, while the 16 baseline models listed in Table 2 (and all editing baselines in Table 6) are evaluated without any fine-tuning. Thus Table 2's gap (e.g., Pass@1 58.15 vs. DeepSeek-Coder 55.56, GPT-4o 49.81) can be explained entirely by in-distribution fine-tuning rather than by the Plan/Check/Code agent design. The same asymmetry afflicts the editing task (Table 6), where the Code Agent is trained on 1,400 editing samples but the baselines are not. For diagram coding, the Diagram-to-Code Agent is a default Qwen2-VL-7B, yet DiagramAgent augments it with compiler debugging plus GPT-4o verification (Sec. 4.4), so the comparison to GPT-4o as a single-pass baseline again conflates the framework with privileged extra inference-time feedback. To support the claim, the paper must either fine-tune the baselines on the same data or evaluate DiagramAgent's fine-tuned Code Agent with the agent loop disabled; without such a controlled comparison, the headline numbers do not demonstrate that the framework, rather than the training regime, is responsible for the improvement.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DiagramGenBenchmark, a dataset and evaluation suite for text-to-diagram generation, diagram coding, and diagram editing across eight diagram types (flowchart, model architecture, mind map, bar chart, etc.), and presents DiagramAgent, a four-module framework (Plan Agent, Code Agent, Check Agent, Diagram-to-Code Agent) that generates, codes, and edits structured diagrams via LaTeX/TikZ or DOT code. The experiments compare DiagramAgent against 16 code-LLM baselines and 11 vision-language-model baselines, reporting large gains (e.g., Pass@1 58.15 for generation, 68.89 for coding, 98.00 for editing) together with ablations of the compiler-debugging and GPT-4o-verification components and a human evaluation.","tokens_in":24249,"tokens_out":4490,"duration_ms":43613,"significance":"If the reported gains are genuine, this is a valuable contribution: it defines a new task family, provides a reusable benchmark with three subtasks, and offers a modular agentic baseline that is reproducible from open models plus a closed verifier. The paper ships explicit prompts, data statistics, and ablations, and it makes falsifiable metric claims. However, the central comparative claim—that DiagramAgent significantly outperforms existing baselines—is not yet supported because the head-to-head comparisons are confounded by fine-tuning asymmetry and by GPT-4o involvement in both the benchmark construction and the evaluation pipeline. The benchmark itself may still be useful as a resource, but its validity as a measure of general user intent requires additional human-written references or a calibration study.","major_comments":[{"comment":"The DiagramAgent Code Agent is fine-tuned for four epochs on the DiagramGenBenchmark training split, whereas all 16 baselines in Table 2 and all editing baselines in Table 6 are evaluated off-the-shelf. Consequently, the reported gaps (e.g., Pass@1 58.15 vs. 55.56 for DeepSeek-Coder in Table 2; 98.00 vs. 95.00 for DeepSeek V2.5 in Table 6) conflate in-distribution fine-tuning with the multi-agent framework. The paper must provide a controlled comparison—either fine-tuning the baselines on the same training data, or evaluating the fine-tuned Code Agent without the Plan/Check agent loop—before the claim of significant outperformance can be accepted.","section":"Sec. 6 (Setup), Sec. 4.2, Eq. (6), Tables 2 and 6"},{"comment":"The benchmark's natural-language instructions, expanded queries, and edit suggestions were all generated by GPT-4o, and the Check Agent uses GPT-4o for verification. This creates a systematic bias: the reference outputs and the verifier are drawn from the same model family, so the reported metrics may measure alignment with GPT-4o's output distribution rather than general diagram-generation ability. The paper does not discuss this risk. Please add a human-written test subset (or at least a manually verified subset) and report whether the relative rankings and absolute scores are stable when the references are not GPT-4o-generated.","section":"Appendix A.2, Sec. 4.4, Figs. 8–13"},{"comment":"The Diagram-to-Code Agent is described as 'Qwen2-VL-7B in its default configuration,' yet the full DiagramAgent pipeline augments it with compiler debugging and GPT-4o verification, while the coding baselines are single-pass. Moreover, the ablation 'w/o GPT-4o & Compiler' in Table 10 still achieves Pass@1 of 52.59 versus 28.89 for the off-the-shelf Qwen2-VL baseline; this large unexplained gap indicates that the comparison isolates more than just the removed modules (e.g., prompt design, Plan Agent orchestration, or evaluation harness differences). The paper needs a clean ablation that varies only the framework components on the same base model and same inference-time budget.","section":"Sec. 4.3–4.4, Table 4, Table 10 (Appendix E)"}],"minor_comments":[{"comment":"All objective metrics are reported as point estimates without confidence intervals or significance tests; given the small test sets (270 generation/coding and 200 editing samples) and gaps of only a few points, please provide bootstrap intervals or significance tests.","section":"Tables 2–7"},{"comment":"The human evaluation reports mean scores from three raters but no inter-annotator agreement statistic (e.g., Krippendorff's alpha) and no per-model variance; please add these to support the claim of alignment between human and automated metrics.","section":"Fig. 4 and Appendix D"},{"comment":"The 'Complete Query Length (tokens)' row for the editing column reports an average of 98 while the minimum is 115, which is impossible; the value likely belongs to a different column or needs correction.","section":"Table 1"},{"comment":"The Check Agent operation is written as f_check(c) = f_debug(...) + f_verify(...), but f_debug and f_verify return errors and completeness judgments, not scalars; the '+' notation is formally unexplained and should be replaced with a clearer composition description.","section":"Eq. (9)"},{"comment":"References [41] and [42] cite the same paper; please disambiguate. The Figure 3 caption contains 'inlcuding' for 'including,' and the Figure 4 caption mentions a 'Modify diagram generation' task that does not match the task names used elsewhere in the paper.","section":"References and typos"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and framework are useful, but the main empirical claim currently rests on an uncontrolled comparison. The fix (fine-tuned baselines or no-fine-tuning evaluation of DiagramAgent, plus a human-written reference subset) is within the scope of a major revision. The paper may also be somewhat outside the core cs.DB scope, as its contribution is primarily an NLP/agentic benchmark; the editor may wish to consider fit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the DiagramGenBenchmark itself is the contribution worth taking seriously—eight diagram types, three tasks, reasonably sized, compiled-filtered. DiagramAgent is a plausible agent composition, and the ablation shows the debugging/verification loop adds value. But the paper's central claim of significant outperformance is not supported by the experiments as run. The Code Agent is fine-tuned on the benchmark train split (Sec. 6), while all 16 baselines in Table 2 and the editing baselines in Table 6 are evaluated off-the-shelf. That asymmetry alone can explain most of the Pass@1 gap (58.15 vs 55.56 for DeepSeek-Coder, for instance). The same issue hits diagram coding: the Diagram-to-Code Agent is a default Qwen2-VL-7B, but the full DiagramAgent adds compiler feedback and GPT-4o verification, so the comparison to GPT-4o as a single pass conflates framework with extra inference-time compute.\n\nThere is also a contamination risk the paper doesn't discuss. The natural-language queries, expanded queries, and edit instructions were all generated by GPT-4o (Appendix A.2), and the Check Agent itself uses GPT-4o for verification. So the benchmark partially measures agreement with GPT-4o's output distribution. That's not fatal for a benchmark, but it should be stated, and the fine-tuned Code Agent is effectively being tuned to that distribution.\n\nThe test sets are small (270/200), no error bars are reported, and the editing task filters for compilability, which can inflate Pass@1. It's not clear whether baselines receive the same filtering.\n\nWhat would make this solid: fine-tune the strongest baselines on the same data (or ablate the fine-tuned Code Agent with the agent loop disabled), add error bars, release the benchmark and code, and discuss the GPT-4o circularity explicitly.\n\nThe paper is clearly written, and the error analysis in Appendix F is genuinely useful. The benchmark, once released, could become a standard evaluation suite for text-to-diagram work. The current experimental evidence just doesn't support the framework claim.\n\nRecommendation: send to peer review with a major-revision decision. The benchmark deserves referee time; the comparison needs to be redone.","headline":"The benchmark is a genuine contribution, but the headline outperformance claim is not supported by the experiments as run because of fine-tuning asymmetry and GPT-4o contamination; send to peer review, but expect a major revision.","tokens_in":24727,"tokens_out":2958,"would_cite":false,"duration_ms":28629,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces DiagramGenBenchmark, a dataset of nearly 7,000 diagram code–image pairs across eight diagram types, and DiagramAgent, a four-agent pipeline that generates, codes, and edits structured diagrams from text, reporting…","keywords":["text-to-diagram generation","diagram editing","benchmark dataset","multi-agent framework","structured diagrams","TikZ","DOT","code generation"],"falsifier":"Collect a held-out set of diagram requests written independently by human users (not derived from the target code), run DiagramAgent on them, and have those same users rate whether the output matches their intended diagram. If Pass@1 and human scores drop substantially on human-authored requests while staying high on the benchmark's auto-generated queries, the central claim of superior general text-to-diagram ability would be falsified.","tokens_in":23735,"feed_emoji":"📊","tokens_out":4035,"duration_ms":38659,"temperature":0.7,"pith_summary":"The paper argues that generating structured diagrams—flowcharts, model architectures, mind maps, and similar—should be treated as a code-generation problem with an explicit edit loop, not as a text-to-image problem. To support this, it contributes a new benchmark with human-like instructions, code, and images for eight diagram categories, plus a four-agent framework that plans, writes diagram code, checks it, and converts diagrams back into code. The authors claim that this framework outperforms 16 existing text-to-code and vision-language baselines on accuracy, structural coherence, and modifiability, with the largest gains in the editing task. If correct, the work gives the research community a standardized task definition and a strong baseline for building editable, logically coherent diagram generators.","feed_headline":"Text-to-diagram agent tops 16 baselines on new benchmark","feed_subtitle":"Eight-type dataset and four-agent framework report Pass@1 of 58.15 for generation and 98.00 for editing.","key_machinery":"The load-bearing mechanism is the four-agent loop: Plan Agent (query expansion via LLM), Code Agent (a fine-tuned open-weight code model generating TikZ or DOT), Check Agent (compiler-based debugging plus LLM-based verification), and Diagram-to-Code Agent (a vision-language model turning diagram images back into code). The Check Agent's compiler feedback loop is what converts raw code generation into a self-correcting process, and the inverse Diagram-to-Code Agent is what makes editing possible by recovering editable source from a rendered diagram.","core_discovery":"The central claim is that diagram generation reduces to two inverse code mappings: generation as $D_{\\mathrm{gen}} = f_{\\mathrm{img}}(f_{\\mathrm{code}}(x_{\\mathrm{ins}}))$ and editing as $D_{\\mathrm{mod}} = f_{\\mathrm{img}}(f_{\\mathrm{mod}}(f_{\\mathrm{code}}^{-1}(D_{\\mathrm{ori}}), x_{\\mathrm{edit}}))$. The paper's DiagramAgent implements these mappings with four cooperating modules: a Plan Agent that expands incomplete instructions, a Code Agent fine-tuned to emit compilable LaTeX/TikZ or DOT code, a Check Agent that compiles code and uses a large language model to verify completeness, and a Diagram-to-Code Agent that converts an existing diagram image back into code. On the introduced DiagramGenBenchmark, the authors report that this pipeline beats all compared baselines on the generation, coding, and editing tasks, with human evaluation aligning with the objective metrics.","pith_inferences":["An unstated consequence is that the benchmark's auto-generated queries may make the generation task easier than real user requests, since those queries were reverse-engineered from the target code; a human-requested test set would clarify whether the reported gains hold outside the benchmark distribution.","The same four-agent loop could plausibly transfer to other structured visual outputs such as SVG, UML, or circuit schematics, but the paper only demonstrates TikZ and DOT.","The reliance on a commercial LLM for data annotation and verification implies a reproducibility ceiling: the dataset and agent behavior are partly tied to a model the authors do not control.","A testable extension would be to use the Diagram-to-Code Agent as a self-training signal, letting the system generate new diagrams, convert them back to code, and use mismatches as synthetic training data."],"forward_implications":["If the reported results hold, fine-tuned code-generation models with a compile-and-verify loop become the practical route to producing editable, logically coherent diagrams from text.","The benchmark provides a standardized set of 8 diagram types and three tasks (generation, coding, editing), enabling direct comparison of future text-to-diagram systems.","The editing results being far stronger than generation results suggests that interactive refinement over generated drafts is a more tractable path than one-shot perfect generation.","The inverse diagram-to-code capability opens the door to editing existing diagrams found in documents and slide decks, not only diagrams the system itself produced."],"supporting_citations":[{"why":"Supplies vector-graphics code and image data that the benchmark draws from.","marker":"[48]"},{"why":"Supplies the datikz and datikz-v2 TikZ code collections used as raw benchmark sources.","marker":"[6]"},{"why":"The large language model used to auto-generate queries and editing instructions and to verify code completeness in the Check Agent.","marker":"[3]"},{"why":"The base open-weight code model that is fine-tuned to become DiagramAgent's Code Agent.","marker":"[17]"},{"why":"The vision-language model that serves as the basis for the Diagram-to-Code Agent.","marker":"[35]"},{"why":"The code model used to generate edited code from modification suggestions when constructing the diagram editing task.","marker":"[46]"},{"why":"Provides the self-planning method that motivates the Plan Agent's instruction-analysis and expansion behavior.","marker":"[29]"}],"fun_headline_variants":["New benchmark and agent for text-to-diagram generation","DiagramAgent: beating 16 baselines on new diagram benchmark","Text-to-diagram via inverse code mappings: agent wins","Benchmark for editable diagrams: agent excels in generation and editing","From text to structured diagrams: agent beats baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's ground truth is valid: the natural-language queries, expanded complete queries, and edit instructions were auto-generated by a large language model, and the edited codes were produced by another model and then filtered for compilability, so if these references do not match real user intent, every reported score measures alignment with an automatic pipeline's output distribution rather than general diagram-generation ability.","fun_headline_variants_meta":{"raw":{"variants":["New benchmark and agent for text-to-diagram generation","DiagramAgent: beating 16 baselines on new diagram benchmark","Text-to-diagram via inverse code mappings: agent wins","Benchmark for editable diagrams: agent excels in generation and editing","From text to structured diagrams: agent beats baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000701,"raw_usage":{"total_tokens":3159,"prompt_tokens":932,"completion_tokens":2227,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":2146}},"tokens_in":548,"tokens_out":2227,"duration_ms":14885,"temperature":1.0,"reasoning_tokens":2146,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:44:19.950645+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a held-out set of diagram requests written independently by human users (not derived from the target code), run DiagramAgent on them, and have those same users rate whether the output matches their intended diagram. If Pass@1 and human scores drop substantially on human-authored requests while staying high on the benchmark's auto-generated queries, the central claim of superior general text-to-diagram ability would be falsified.","supporting_citations":[{"cited_title":"Au- tomatikz: Text-guided synthesis of scientific vector graphics with tikz","cited_arxiv_id":null,"evidence_quote":"Supplies the datikz and datikz-v2 TikZ code collections used as raw benchmark sources."},{"cited_title":"Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x","cited_arxiv_id":null,"evidence_quote":"The code model used to generate edited code from modification suggestions when constructing the diagram editing task."},{"cited_title":"Autoact: Automatic agent learning from scratch for qa via self-planning","cited_arxiv_id":null,"evidence_quote":"Provides the self-planning method that motivates the Plan Agent's instruction-analysis and expansion behavior."}],"review_version":1}