SciFlow-Bench: Evaluating Structure-Aware Scientific Diagram Generation via Inverse Parsing

Chong Chen; Honglin Lin; Tong Zhang; Wentao Zhang; Zhou Liu

REVIEW 4 major objections 5 minor 3 cited by

New benchmark shows AI diagrams look right but parse wrong

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 02:39 UTC pith:A3DNKJKK

load-bearing objection Good benchmark idea, but the parser is the unvalidated load-bearing piece — Graphviz scoring 0.09 says the metric may be measuring HMAS preferences, not structural correctness. the 4 major comments →

arxiv 2602.09809 v3 pith:A3DNKJKK submitted 2026-02-10 cs.CV

SciFlow-Bench: Evaluating Structure-Aware Scientific Diagram Generation via Inverse Parsing

Tong Zhang , Honglin Lin , Zhou Liu , Chong Chen , Wentao Zhang This is my paper

classification cs.CV

keywords scientific diagram generationstructure-first evaluationinverse parsingtext-to-imagebenchmarkstructural recoverabilitymulti-agent systemround-trip protocol

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that scientific diagram generation should be judged by whether the diagram's structure can be recovered as a labeled graph, not by how visually plausible it looks. It introduces SciFlow-Bench, which pairs real paper figures with canonical ground-truth graphs, asks text-to-image models to generate diagrams from method descriptions, and then inverse-parses the generated pixels back into a predicted graph for comparison. Across code-driven, diffusion, and autoregressive models, structural recoverability turns out to be consistently low—especially for diagrams with complex topology—while visual-similarity scores remain moderate to high, exposing a 'visual illusion' in current evaluation. If this holds, existing image-centric metrics systematically hide structural failures, and structure-first evaluation is necessary for trustworthy diagram generation.

Core claim

On the paper's own terms, the central discovery is that when generated scientific diagram images are inverse-parsed into graphs and compared against canonical graphs derived from the source figures, all evaluated models achieve only low graph-level scores (edge-level F1 roughly between 0.01 and 0.14), despite moderate to high image-level scores. This dissociation between visual fidelity and structural recoverability appears across model families, and for the strongest autoregressive model the graph-level score actually improves with structural difficulty while weaker models degrade. The paper argues that this establishes structural recoverability as a distinct, necessary, and currently unmet

What carries the argument

The round-trip inverse-parsing protocol, enabled by a hierarchical multi-agent system (HMAS), which has three layers: cognitive planning (methodologist, visual translator), fine-grained perception (environment curator, shape hunter, text spotter, fusion arbiter), and structural reasoning (topology coder, graph architect). The same HMAS pipeline builds canonical ground-truth graphs from source framework figures and parses generated diagram images into predicted graphs, so that evaluation happens in structured graph space rather than pixel space, using graph-level metrics (node and edge F1), text-level metrics, and image-level metrics aggregated with fixed weights.

Load-bearing premise

The inverse parser (HMAS) recovers the same structure from a diagram that a human expert would, so that low recovery scores reflect the generator's failure rather than the parser's blind spots.

What would settle it

Take a set of synthetic diagrams rendered deterministically from known graphs with clear arrows and labels, run the HMAS parser on them, and compare recovered graphs to the known originals; if edge F1 on clean synthetic diagrams is far below 1.0, then the benchmark's low model scores are partly an artifact of parsing error rather than generation failure.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

If the central claim is correct, image-centric metrics like CLIP and LPIPS are insufficient for diagram generation and should be supplemented by recoverability-based metrics.
Code-driven layout baselines show near-zero graph-level scores under inverse parsing, indicating that even explicit topology can fail to be recovered from a rendered image.
A pronounced dissociation between image-level and graph-level scores persists across all model families, so visual-similarity metrics alone cannot certify diagram correctness.
Structural scores vary with difficulty: as topology becomes more complex, some models degrade while the strongest autoregressive model improves, suggesting that richer text descriptions help disambiguate relations.
The benchmark provides a fixed, reproducible protocol with identical prompts, parser, evaluators, and aggregation weights, so performance differences between models come from generation behavior alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Because the parser's edge-level agreement with human experts is only 0.71, a portion of the reported structural failure may reside in the parser itself; a synthetic-diagram calibration test would separate parser error from generation error.
The same pipeline that constructs ground-truth graphs also parses generated images, so any systematic parser bias (for example, conservatism about dashed or implicit arrows) could compress or exaggerate the measured gap between models.
The protocol currently targets directed framework diagrams; extending it to cyclic, undirected, or hypergraph structures would test whether structural recoverability is a general failure mode of text-to-image models.
The paper's limitations section notes that aesthetics and non-framework diagram types are out of scope, so the claimed dissociation between visual fidelity and structure may not hold for other diagram genres such as charts or schematics.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Good benchmark idea, but the parser is the unvalidated load-bearing piece — Graphviz scoring 0.09 says the metric may be measuring HMAS preferences, not structural correctness.

read the letter

Colleague — you should know this paper is a solid idea with a validation gap. SciFlow-Bench evaluates text-to-image models on scientific diagrams by inverse-parsing pixels back into graphs and comparing them to ground truth. That is a genuinely missing axis: existing benchmarks either check code, not pixels, or pixel similarity without topology. The dataset of 500 real arXiv framework figures, the round-trip protocol, and the finding that visual scores far outpace graph scores are all worth having.

The core problem is the parser. The same HMAS pipeline builds the ground-truth graphs (Sec. 4.1) and parses the generated images (Sec. 3.2). Its edge-F1 against human verification on the source figures is only 0.71. So a third of edges are wrong even on perfect input. And the Graphviz reference, which has explicit topology, scores 0.091 on S_graph — near the bottom, below Qwen-Image. A topology-explicit renderer should be the easiest case for structure recovery. Near-zero recovery suggests the parser is sensitive to rendering style and may be the bottleneck, not the generators. That is a real red flag.

The paper is transparent about some of this — Table 1 and Table 4 report the parser's accuracy, and there is a discrepancy analysis. But the Limitations section only mentions aesthetics and coverage, not parser bias. It also claims 'deterministic metrics' while using GPT-4o for alignment and visual flow, and reports no error bars. No code or dataset is released, though the appendix gives implementation details.

What is good: the protocol is a new combination, the ablation shows Shape Hunter and Text Spotter both matter, and the qualitative examples show real differences between model families. The direction is likely right, but the quantitative leaderboard is not trustworthy until the parser is validated on inputs where ground truth is known (e.g., Graphviz output should give near-perfect scores) and the pipeline is released for scrutiny.

Verdict: deserves peer review, not desk rejection. It is a serious benchmark contribution with a load-bearing instrument problem that is fixable. I would send it to a good venue and ask for those validation experiments plus code release. The current version is a strong idea with unproven measurements.

Referee Report

4 major / 5 minor

Summary. The paper introduces SciFlow-Bench, a benchmark for evaluating text-to-image scientific diagram generation by structural recoverability. A hierarchical multi-agent system (HMAS) constructs canonical ground-truth graphs from source framework figures and inverse-parses generated diagram images back into graphs; the predicted graph is compared to the canonical graph via graph-, text-, and image-level metrics. The paper reports experiments with code-driven (Graphviz) and pixel-based (diffusion and autoregressive) models, finding very low graph-level scores for most models and a pronounced gap between visual fidelity and structural recoverability.

Significance. The inverse-parsing evaluation idea is a genuinely useful contribution to a real gap: structural correctness of generated scientific diagrams is largely ignored by existing benchmarks. The 500-instance dataset with five domains and the human-verified subset are valuable resources, and the paper's willingness to report parser ablation and human agreement is commendable. However, the central claim that current text-to-image models systematically fail to preserve recoverable structure is conditional on the parser being a valid and unbiased measurement instrument, which the paper's own evidence does not establish.

major comments (4)

[Secs. 3.2/4.1 and Tables 1/4] The HMAS pipeline both constructs the canonical ground truth (Sec. 4.1) and inverse-parses generated images (Sec. 3.2). Its edge-level agreement with human verification is only 0.71 F1 (Table 1), and Table 4's full pipeline reports 0.64 on the same subset — an unexplained inconsistency. Thus a generator that perfectly reproduces the source figure would score at most ~0.7 on edge metrics, and the Graphviz reference, which has explicit DOT topology, achieves S_graph=0.091 (Table 3), close to weak diffusion models. The benchmark therefore partially measures parser insensitivity, not generator structure. The paper's headline conclusion is load-bearing on parser validity, which is not established.
[Sec. 5.2, Table 3] All results are single-run point estimates with no error bars, confidence intervals, or significance tests. The claimed performance regimes and the 'monotonic improvement' of Gemini 3 Pro Image (0.079 to 0.135 on Hard) rest on differences as small as 0.01–0.03 in aggregated scores, which could be within run-to-run or sample-to-sample noise. Provide multiple seeds/prompt samples and statistical tests, or temper the claims.
[Sec. 4.1/4.2] The evaluation uses automatically constructed HMAS graphs as the canonical ground truth, while human-verified graphs are used only to measure agreement. Given the moderate edge-level agreement (F1 0.71), the reference itself is noisy. The authors should at minimum report benchmark scores on the 100-instance human-verified subset against the human-verified graphs, and ideally calibrate graph-level scores against the parser ceiling (e.g., by inverse-parsing source figures and comparing to G*).
[Appendix A.1] The edge-matching rule permits a predicted edge to count as correct if the matched endpoints are connected by a path in the canonical graph, effectively rewarding the omission of intermediate nodes and inflating edge-level scores. The semantic node-matching threshold is not specified, and the embedding model used to compute semantic similarity is not named. Both are free parameters that affect all reported results; their influence should be analyzed in an ablation.

minor comments (5)

[Sec. 3.2] The pipeline is described as 'deterministic' but relies on LLM agents that are stochastic unless temperature is set to 0. Specify sampling parameters or justify the determinism claim.
[Tables 1 and 4] The same 'full pipeline' is reported with Edge R=0.77 and Edge F1=0.71 in Table 1 but Edge R=0.64 and Edge F1=0.64 in Table 4, both on the human-verified subset. Clarify the difference in protocol or correct the numbers.
[Benchmark release] No URL or release plan for code and data is provided. For a benchmark paper, public availability of the dataset and evaluation scripts is important for adoption and reproducibility.
[Figure 6] The caption does not identify which image corresponds to which model. Add labels or a legend so the qualitative comparison is unambiguous.
[Sec. 5.2.1] The explanation that Gemini's improvement on the Hard subset is due to richer textual descriptions is speculative; provide evidence, e.g., measure correlation of graph-level score with prompt length or description detail.

Circularity Check

1 steps flagged

Partial circularity: the reference graph and the predicted graph are both produced by the same HMAS inverse parser, making the core metric a measure of HMAS self-consistency rather than independent structural correctness.

specific steps

self definitional [Sec. 3.2 and Sec. 4.1 (also Sec. 5.1)]
"The same pipeline is applied consistently during dataset construction and benchmark evaluation, ensuring strict consistency and reproducibility across all experimental stages throughout the framework. ... Canonical ground-truth graphs are automatically constructed from source framework figures by the hierarchical multi-agent system."

The canonical graph G* and the predicted graph Ghat are both outputs of the same HMAS transformation: G*=HMAS(source figure) and Ghat=HMAS(generated image). The benchmark thus scores the agreement between two outputs of one parser, not between a generated image and an independent structural ground truth. Systematic parser biases (e.g., missing dashed arrows, sensitivity to rendering style, OCR limitations) enter both sides: edges invisible to HMAS are absent from G* and are not penalized when absent from Ghat, so low graph scores can reflect parser insensitivity rather than generator failure. Human verification is anchored to the HMAS output (annotators edit the automatically extracted graph rather than constructing one from scratch), so it only partially breaks the loop. The near-zero Gra

full rationale

The central claim that pixel-based models fail to preserve recoverable structure depends on HMAS being a valid and unbiased instrument. The paper's own Table 1 (Edge-F1=0.71) and Table 4 (Edge-F1=0.64 on source figures) show the parser already misses or mislabels a third of edges on perfect inputs, and the Graphviz reference with explicit DOT topology scores only S_graph=0.091, lower than most pixel models. These facts do not by themselves prove circularity, but they demonstrate that the shared parser is not a neutral instrument. Because the same HMAS constructs the ground-truth graphs and inverse-parses the generated images, the graph-level metric is partly self-definitional: it measures 'recoverability by HMAS under HMAS's own priors.' The benchmark is not fully closed: aggregation weights are fixed a priori, the 100-instance human verification provides partial independent grounding, and the comparison is between two different images. However, the load-bearing conclusion about structural correctness is conditional on parser validity, which the shared-pipeline design does not independently establish. This is a partial circularity, not a complete one, so score 4.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The benchmark's conclusions rest on hand-chosen evaluation weights, an unreported semantic threshold, and the assumption that one multi-agent pipeline can both write and read the graphs it scores. No new theoretical or physical entities are introduced.

free parameters (3)

Semantic matching threshold
A.1 says a reference node is recovered if similarity exceeds 'a fixed threshold'; the value is not reported and directly controls node/edge precision-recall.
Evaluation aggregation weights = node 0.4/edge 0.6 (graph); graph 0.4/text 0.3/image 0.3 (overall); alignment 0.4/coverage 0.3/faithfulness 0.3 (text); C
Appendix A sets these a priori; they determine the leaderboard and are not justified by external data.
Semantic embedding backbone and VLM judge = unspecified
The sentence-embedding model used for node matching and the GPT-4o prompt/version used for alignment and visual flow are not specified, but scores depend on them.

axioms (3)

domain assumption The HMAS pipeline's automatically constructed graphs are accurate enough to serve as canonical ground truth for all 500 instances.
Only 100 figures are human-verified (Sec. 4.2), with Edge-F1=0.71; the rest inherit unmeasured parser errors.
domain assumption Structural recoverability as measured by the same HMAS parser is a valid proxy for diagram structural correctness.
Sec. 3.3 defines quality as recoverability; the Graphviz reference score of 0.091 indicates the parser may miss structure even when it is explicitly drawn.
domain assumption The Methodologist's extracted method description contains enough information to reconstruct the source figure's graph.
Prompts come from text only (Sec. 3.2); if the text under-specifies the figure, low graph scores may reflect prompt insufficiency, not generation failure.

pith-pipeline@v1.3.0-alltime-deepseek · 16158 in / 11830 out tokens · 111926 ms · 2026-08-03T02:39:05.721284+00:00 · methodology

0 comments

read the original abstract

Scientific diagrams convey explicit structural information, yet modern text-to-image models often produce visually plausible but structurally incorrect results. Existing benchmarks either rely on image-centric or subjective metrics insensitive to structure, or evaluate intermediate symbolic representations rather than final rendered images, leaving pixel-based diagram generation underexplored. We introduce SciFlow-Bench, a structure-first benchmark for evaluating scientific diagram generation directly from pixel-level outputs. Built from real scientific PDFs, SciFlow-Bench pairs each source framework figure with a canonical ground-truth graph and evaluates models as black-box image generators under a closed-loop, round-trip protocol that inverse-parses generated diagram images back into structured graphs for comparison. This design enforces evaluation by structural recoverability rather than visual similarity alone, and is enabled by a hierarchical multi-agent system that coordinates planning, perception, and structural reasoning. Experiments show that preserving structural correctness remains a fundamental challenge, particularly for diagrams with complex topology, underscoring the need for structure-aware evaluation.

Figures

Figures reproduced from arXiv: 2602.09809 by Chong Chen, Honglin Lin, Tong Zhang, Wentao Zhang, Zhou Liu.

**Figure 1.** Figure 1: Motivation and comparison between existing diagram benchmarks and SciFlow-Bench. Left: Image-centric evaluation may assign high scores to visually plausible diagrams that contain structural errors. Right: SciFlow-Bench adopts a structure-first evaluation by inverse-parsing generated diagrams into graphs and measuring structural recoverability. progress in text-to-image generation (Rombach et al., 2022; … view at source ↗

**Figure 2.** Figure 2: Overview of the SciFlow-Bench framework. A unified round-trip evaluation pipeline based on a hierarchical multi-agent system that constructs canonical ground-truth graphs from source framework figures and recovers predicted graphs from generated diagram images. By inverse-parsing pixel-level outputs into structured graphs, the same pipeline supports both dataset construction and structure-first evaluation… view at source ↗

**Figure 3.** Figure 3: Benchmark statistics of SciFlow-Bench. (a) Domain distribution across research areas. (b) Distribution of structural difficulty levels defined by graph size and relational complexity. (c) Long-tailed distributions of node and edge counts, reflecting substantial structural complexity in real-world scientific diagrams. rather than a separate preprocessing stage. No manual filtering, redrawing, or diagram si… view at source ↗

**Figure 4.** Figure 4: Human verification interface and representative annotation outcomes. Annotators refine the automatically extracted graph by selectively excluding unsupported components and adding missing nodes or relations under a minimal-intervention, identity-consistent editing protocol. recall can be computed deterministically without post-hoc semantic matching or heuristic alignment. This identity-consistent editi… view at source ↗

**Figure 5.** Figure 5: Representative discrepancy cases observed during human verification. Examples illustrate common sources of mismatch between automatic extraction and human-verified ground-truth graphs, including ambiguous or implicit connections, text-related perception errors, and structurally ambiguous relations inherent in real scientific diagrams. Figures are reproduced from prior work (Qi et al., 2025; Chen, 2025; Z… view at source ↗

**Figure 6.** Figure 6: Qualitative comparison of scientific diagram generation under identical structured visual prompts. For each case, the top diagram is a layoutdriven Graphviz reference, while the bottom-left and bottom-right diagrams are generated by Qwen-Image and Gemini 3 Pro Image, respectively. All images are rendered at the same resolution and evaluated without manual tuning. These examples are provided to illustrat… view at source ↗

discussion (0)

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

SciForma: Structure-Faithful Generation of Scientific Diagrams
cs.CV 2026-07 conditional novelty 6.0

A 9B open-weights model trained with axis-decomposed conjunctive preference optimization (M-DPO) and a structural inventory beats GPT-Image-1.5 on scientific-diagram structural-fidelity benchmarks.
AI for Auto-Research: Roadmap & User Guide
cs.AI 2026-05 unverdicted novelty 4.0

The paper delivers a stage-by-stage roadmap for AI in research, showing reliable assistance in retrieval and tool tasks but fragility in novelty and judgment, advocating human-governed collaboration.
AI for Auto-Research: Roadmap & User Guide
cs.AI 2026-05 conditional novelty 4.0

AI can generate research artifacts faster than it can verify them, so across all eight lifecycle stages the credible deployment mode is human-governed collaboration rather than full autonomy.

Reference graph

Works this paper leans on

2 extracted references · 1 linked inside Pith · cited by 2 Pith papers

[2023]

Notably, such discrepancies tend to be sparse and localized rather than struc- tural, indicating that the majority of diagram con- tent is correctly grounded

and remain an open challenge for multimodal perception systems. Notably, such discrepancies tend to be sparse and localized rather than struc- tural, indicating that the majority of diagram con- tent is correctly grounded. C.3 Structurally Underspecified or Symbolic Nodes The third discrepancy category involves nodes or relations that are intentionally ab...
[2025]

Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei

Sridbench: Benchmark of scientific research il- lustration drawing of image generation model.arXiv preprint arXiv:2505.22126. Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei. 2023. Textdiffuser: Dif- fusion models as text painters.Advances in Neural Information Processing Systems, 36:9353–9387. Junsong Chen, Chongjian Ge, Enze Xi...

Pith/arXiv arXiv 2023

[1] [2023]

Notably, such discrepancies tend to be sparse and localized rather than struc- tural, indicating that the majority of diagram con- tent is correctly grounded

and remain an open challenge for multimodal perception systems. Notably, such discrepancies tend to be sparse and localized rather than struc- tural, indicating that the majority of diagram con- tent is correctly grounded. C.3 Structurally Underspecified or Symbolic Nodes The third discrepancy category involves nodes or relations that are intentionally ab...

[2] [2025]

Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei

Sridbench: Benchmark of scientific research il- lustration drawing of image generation model.arXiv preprint arXiv:2505.22126. Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei. 2023. Textdiffuser: Dif- fusion models as text painters.Advances in Neural Information Processing Systems, 36:9353–9387. Junsong Chen, Chongjian Ge, Enze Xi...

Pith/arXiv arXiv 2023