{"id":"bd541364-988b-441f-b94e-ad4d31433bd9","arxiv_id":"2607.24707","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"VLMs recover common ERD elements at F1>0.74 but drop to 0.07–0.28 on N-ary relationships, multivalued attributes, and weak entities; reasoning models gain 15–25% yet stay prior- and complexity-sensitive.","lead":"ERUnderstand is the first large-scale benchmark (2,960 diagrams) for recovering machine-readable database schemas from ER diagram images with Vision-Language Models. It shows VLMs handle basic entities well but collapse on weak entities, multivalued attributes, and N-ary relationships, exposing structural-reasoning gaps relevant to AI-assisted database engineering.","discovery_kind":"new_method","skeptic_critique":{"model":"moonshotai/kimi-k3","headline":"The headline \"reasoning models improve Macro-F1 by 15–25%\" is tier-dependent: it holds on the LLM/Graphviz-generated synthetic tier but collapses to ~2% on the real production Schema tier (0.735→0.748, Table 2), suggesting part of the measured reasoning advantage may reflect synthetic-pipeline reg","rationale":"The reader identified metric/prompt sensitivity and the missing classical-parser baseline as the weakest assumption. I agree those matter, but the sharper soft spot is the data-dependence of the paper's own headline quantitative claim: Table 2 shows the 15–25% reasoning improvement is driven by the synthetic tier and essentially disappears on the production Schema tier (+0.013 Macro-F1), and §D.8 independently demonstrates that synthetic-tier scores are inflated by generator naming regularities that models exploit. The reader's concern is about the measurement instrument; mine is about the measured quantity's generalizability across the benchmark's own tiers — related but distinct, hence partial agreement. The rest of the central claim survives scrutiny: the EER weakness profile (weak entities, multivalued, N-ary) is corroborated by multiple controlled probes (IS-A-only diagrams, CF/LP perturbations, PK ablations, attention analysis), the metric-choice sensitivity is addressed in Appendix B.3 (f varied 0→1 shifts Macro-F1 but not conclusions), and inter-annotator agreement is honestly reported with an appropriate caveat that the released ground truth is adjudicated. I recommend CONDITIONAL rather than REJECT because the fix is presentational and analytical — qualify the reasoning-gain claim by tier and run the ablation test — rather than a flaw that invalidates the benchmark or its failure-mode findings. UNCHANGED would understate a genuine gap between the abstract's unqualified 15–25% figure and the ~2% gap on the most ecologically valid tier.","tokens_in":24120,"tokens_out":3024,"duration_ms":118960,"concrete_test":"Two-step check. (1) Recompute the reasoning vs non-reasoning Macro-F1 gap stratified by tier from the released per-diagram scores; confirm the Schema-tier gap is ~0.01–0.02 while synthetic is 0.15–0.29. (2) Take a random 200-diagram sample of Medium/Hard synthetic ERDs, regenerate them through the released pipeline with the generator's regularities ablated (PK names not following _id, varied renderers/layouts beyond default Graphviz style, shuffled construct frequencies), re-run two reasoning and two non-reasoning models, and compare the reasoning gap to the original synthetic gap. If the ablated gap drops toward the Schema-tier ~2%, the 15–25% claim must be scoped to generator-styled diagrams; if it persists, the concern does not land.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim's quantitative centerpiece — reasoning-augmented models raise overall Macro-F1 by 15–25% — is not uniform across the benchmark's own tiers. From Table 2 averages: synthetic Easy/Med/Hard gains are +0.15/+0.29/+0.28 (reasoning 0.88/0.84/0.80 vs non-reasoning 0.73/0.55/0.52), curated gains are +0.08 to +0.12, and on the Schema tier of 179 real production schemas the gap is +0.013 (0.735→0.748, ~1.8% relative). The synthetic tier dominates the dataset (2,041 of 2,960 diagrams) and is produced by a single LLM-assisted Graphviz pipeline whose outputs have strong stylistic regularities — the paper itself documents one in §D.8: synthetic PKs follow the _id convention, and PK F1 of 0.95–0.99 there collapses when naming conventions are perturbed (missed PKs 40.75→105.50). The same tier also shows the largest reasoning-vs-non-reasoning gaps on exactly the constructs the generation prompts mandate (EER elements, Table 3: e.g., synthetic N-ary 0.07→0.31, IS-A 0.74→0.91). This raises the possibility that the aggregate 15–25% figure partially measures how well reasoning models exploit the generator's regularities (canonical layouts, naming patterns, prompt-mandated construct distributions) rather than a general ERD-understanding advantage — and on the only tier with real-world schemas and no generator regularities, the advantage nearly vanishes. This does not threaten the failure-mode findings (spatial bias, CF/LP degradation, IS-A-only collapse are independently probed) or the EER-weakness profile, but it does mean the headline reasoning-gain number, as stated in the abstract without tier qualification, is the least secure part of the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"moonshotai/kimi-k3","summary":"The paper introduces ERUnderstand, a benchmark of 2,960 ER diagrams from three sources (740 curated educational renderings, 179 production database schemas, 2,041 LLM-assisted synthetic diagrams), each paired with a standardized JSON representation, plus a structure-aware fractional-F1 scoring toolkit. Ten VLMs are evaluated under a fixed zero-temperature prompt. The main findings: models recover entities/attributes/relationships well (F1 often 0.74–1.00) but fail on weak entities (0.28), multivalued attributes (0.14), N-ary relationships (0.07), and IS-A hierarchies; a set of controlled probes (context-free labels, label permutation, IS-A-only diagrams, PK naming vs. visual cues, 20+-entity diagrams, interactive prompting) diagnoses spatial-proximity bias, reliance on linguistic priors over diagram structure, and complexity collapse. The abstract and conclusion headline that reasoning-augmented models improve Macro-F1 by 15–25%.","tokens_in":24650,"tokens_out":6199,"duration_ms":217140,"significance":"If the results hold, this is a useful and well-resourced contribution. ERDs are a natural testbed for visual-structural reasoning, and ERUnderstand is, to my knowledge, the first benchmark to pair ERD images with executable-schema JSON ground truth at this scale (2,960 diagrams, ~17k entities), with public release of data, evaluation toolkit, and generation code. The controlled probes (context-free labels, label permutation, IS-A-only diagrams, PK naming vs. underline, high-density stress set) are genuinely diagnostic and largely independent of the main scoring pipeline, and the failure-mode findings — spatial proximity bias, reliance on naming conventions over visual notation, collapse on 20+-entity diagrams — are valuable to both the VLM and database communities. Honest touches strengthen credibility: the sensitivity analysis over the partial-credit weight (§B.3), the valid-JSON coverage audit (§B.2), and the explicit note that the released annotations are adjudicated rather than double-annotated (§A.2). The failure-mode conclusions would survive even if the headline aggregate numbers were qualified, which is the right sign of a benchmark paper whose durable contribution is the资源","major_comments":[{"comment":"Abstract/§7 vs. Table 2: the headline claim that reasoning-augmented models improve Macro-F1 by 15–25% is tier-dependent. From Table 2's own averages the gain is +0.15/+0.29/+0.28 on the synthetic Easy/Med/Hard tiers and roughly +0.08–0.12 on curated tiers, but only +0.013 (0.735→0.748) on the Schema tier — the only tier of real production schemas. The synthetic tier dominates the benchmark (2,041 of 2,960 diagrams) and is produced by a single LLM-assisted Graphviz pipeline whose regularities the paper itself documents: §D.8 shows synthetic PKs uniformly follow the _id convention, and Appendix A.3 shows EER constructs are prompt-mandated — precisely the categories where reasoning gains are largest in Table 3 (e.g., N-ary 0.07→0.31, IS-A 0.74→0.91). The aggregate 15–25% figure may therefore partly measure exploitation of generator regularities rather than a general ERD-understanding advan","section":"Abstract, §6, Table 2"},{"comment":"The related-work section highlights OmniSch as close in spirit precisely because it compares VLMs against a classical OCR/detector/heuristic baseline, yet ERUnderstand ships no non-VLM baseline at all. Without one, the reader cannot tell whether low scores such as 0.07 F1 on N-ary relationships reflect a VLM-specific structural-reasoning deficit or intrinsic difficulty of the notation/target representation, and the §1 contribution claim of 'strong baselines' is overstated — all baselines are commercial or open VLMs. A simple classical pipeline (OCR plus shape/line detection plus heuristic graph assembly) is feasible at least on the standardized Graphviz-rendered synthetic and Schema tiers and would materially strengthen the failure-mode attributions in §5. The Limitations section concedes this gap, but given the paper's central interpretive claims about 'visual-structural reasoning,' a b","section":"§2, §4, Limitations"},{"comment":"Model identifiers are inconsistent across the manuscript in ways that affect reproducibility of the rankings. §5's CF/LP analysis reports 'Claude-4.5: −0.35', a model absent from Table 2; the complexity analysis in §5 cites 'Gemini-2.0' (0.428) while Table 9 attributes 0.428 to Gemini-3.0-flash; Table 7 and §D.7 include GPT-4o, GPT-5, Claude-3.5, Qwen-3, and Gemini-2.5, while Table 2 uses GPT-5.4-*, Claude-4.6-S, and Qwen-3.5; §6's GED paragraph cites 'GPT-5' but Table 5 lists both GPT-5 and GPT-5.4-pro. Please add a single table mapping every reported name to an exact model version and evaluation date, state which experiments used which versions, and give the protocol behind the §6 robustness claim ('repeated evaluations over different versions, <4%'), which currently has no supporting detail.","section":"§5, §6, Tables 2/5/7/9"},{"comment":"The text states that structural errors are assigned 0.5 weight 'as both False Positive and False Negative', but FN_rel in Eq. (3) includes only Rel_missed + Rel_miscon/2 — cardinality and N-ary errors are penalized on the FP side only, while TP_rel in Eq. (1) subtracts N-ary in full. The worked example in §B.2 describes one missed, one extra, one misconnected ternary, and one wrong cardinality, yet computes FP = 1 + (1+1+1)/2, which appears to count the single misconnected ternary as both Rel_miscon and N-ary. Either the equations, the prose, or the example needs correction, and the exact counting rules (what increments the N-ary term versus Rel_miscon) should be defined. This is load-bearing because the category-level F1 values in Table 3 are a central result and the inter-annotator agreement statistic (0.897) is computed with the same scorer.","section":"§4, Eqs. (1)–(3), §B.2"}],"minor_comments":[{"comment":"§D.10 states N-ary performance remains 'below 0.01 F1 for non-reasoning models', but Table 3 reports 0.07 (and §6 says 'below 0.1'). Please reconcile.","section":"§D.10, Table 3"},{"comment":"The final line of Appendix D reads 'Sections 5 through 5' — presumably a different range was intended. Also 'Label Perumutation' in Figure 5's legend and 'random random 4-letter strings' in §5.","section":"Appendix D, Figure 5, §5"},{"comment":"Several controlled probes use small samples (19 high-density diagrams, 32 IS-A-only, 30 for interactive prompting, 200 for CF/LP) with no variance or confidence intervals reported. Given that per-diagram F1 is the unit of analysis, reporting standard errors (or bootstrap CIs) for the headline deltas in Figure 5 and Tables 7–9 would strengthen the claims.","section":"§5, Tables 7–9"},{"comment":"Appendix A.3 states 'each generated JSON was manually validated' for 2,041 synthetic diagrams. Please clarify the validation procedure — full manual inspection of all 2,041 seems implausible; if a sample was audited, give the sample size and acceptance rate.","section":"Appendix A.3"},{"comment":"The double-annotation pilot is commendably honest, but 21 of 30 diagrams requiring correction is substantial. Please discuss what this implies for residual noise in the released (split-annotation plus adjudication) ground truth, and report per-category agreement, not only overall Macro-F1.","section":"§A.2, Table 4"},{"comment":"References [38] and [39] are the same FlowLearn paper listed twice; [47] and [48] are the same Thalheim book listed twice. Also [8] is titled 'Claude 3.7 Sonnet' while the text evaluates Claude-4.6.","section":"References"},{"comment":"The reasoning/non-reasoning split (GLM-4.5 open-source counted as reasoning; GPT-5.4-chat vs GPT-5.4-pro) rests on the vague criterion 'explicit reasoning or reinforcement-based inference mechanisms'. Please give an operational definition, since the 15–25% claim depends entirely on this grouping.","section":"§6"},{"comment":"Figure 3(b)'s attention-map evidence is suggestive but drawn from a single query on a single model (Qwen). Either temper the causal language ('The model fails to attend...') or note explicitly that this is illustrative rather than systematic evidence.","section":"§5, Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper's main vulnerability is framing rather than execution: the most quotable numbers (15–25% reasoning gains, 0.95–0.99 synthetic PK F1) come from the tier the authors themselves show is convention-laden (§D.8), while the real-world tier shows near-parity between model classes. I would ask the editor to insist on per-tier headline reporting in the abstract before acceptance. The reference list has duplicate entries ([38]/[39], [47]/[48]) worth a copy-edit pass. Otherwise the dataset release and evaluation toolkit appear to be genuine community contributions."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The thing worth knowing is that this is the first large, element-level benchmark that actually targets ER/EER semantics (weak entities, identifying relationships, N-ary, IS-A, multivalued/composite/derived) with machine-readable JSON ground truth, not another chart/flowchart VQA set. That gap was real; they filled it with 2,960 diagrams, a public toolkit, and controlled probes that go beyond a leaderboard.\n\nWhat they do well: the multi-source design (curated educational, 179 production schemas, synthetic Graphviz), the category-level F1 breakdown, and the failure-mode work. Spatial proximity bias, context-free/label-permutation drops, IS-A-only collapse, and PK naming-vs-underline experiments are the useful part of the paper. Inter-annotator Macro-F1 ~0.90 on the pilot is honest. OCR is not the bottleneck; structural recovery is. Code and generation pipeline are released.\n\nSoft spots, in proportion. The abstract’s “reasoning models improve 15–25%” is the weakest headline. Table 2 shows large gains on synthetic Easy/Med/Hard, moderate on curated, and ~1–2% on the Schema tier of real production schemas (0.735 → 0.748). Synthetic is ~2/3 of the set and carries generator regularities (including the _id PK pattern they themselves document). So the aggregate reasoning claim needs tier qualification; it does not kill the EER-weakness or prior-sensitivity findings. Second, no classical OCR/shape/heuristic parser baseline on the same JSON target—mentioned in limitations, still a hole for ranking claims. Third, fractional F1 with fixed 0.5 partial credit and greedy matching is a design choice; they show some sensitivity, which is fine if stated. Curated images are reference-only for copyright; expected, but it limits full visual reproducibility.\n\nMath/data/citations look like a normal empirical systems paper: no circular scoring, related work correctly separates IconQA/flowcharts/OmniSch-style schematic work from ER/EER schema recovery. Who it’s for: people building multimodal DB tools, VLM structural-reasoning eval, or schema-from-image pipelines. Not field-redefining, but subfield-useful and referee-ready.\n\nI’d send it to peer review. Engage with the benchmark and the failure probes; quote the reasoning gain with the tier split.","headline":"Solid first ER/EER schema-recovery benchmark; the EER failure profile holds, but the abstract’s 15–25% reasoning gain is mostly a synthetic-tier story.","tokens_in":25587,"tokens_out":609,"would_cite":true,"duration_ms":20595,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Vision-language models can read common ER diagram parts but collapse on weak entities, multivalued attributes, and N-ary relationships.","keywords":["Entity-Relationship Diagrams","Vision-Language Models","structured schema recovery","EER constructs","multimodal benchmarks","database design","fractional F1"],"falsifier":"Run the released evaluation toolkit on the same 2,960 images with a classical detector-plus-heuristic ER parser (or a different matching/partial-credit scheme) and check whether the same element-wise gaps and model orderings still appear.","tokens_in":25263,"feed_emoji":"📐","tokens_out":903,"duration_ms":18493,"temperature":0.7,"pith_summary":"Entity-relationship diagrams are the usual way people design databases conceptually, yet they almost always live as pictures, not as machine-readable schemas. That blocks AI tools that would otherwise help with documentation, migration, and design assistance. This paper introduces ERUnderstand, a benchmark of 2,960 ER diagrams drawn from textbooks, real production schemas, and controlled synthetic generators, each paired with a standardized JSON schema so recovery can be scored element by element. When current vision-language models are asked to reconstruct those schemas from images alone, they recover ordinary entities, attributes, and binary relationships fairly well, but accuracy falls sharply on the visually subtle Extended ER constructs that actually carry much of the modeling load. Reasoning-augmented models close some of the gap, yet they still lean on linguistic name priors and spatial nearness rather than true diagram connectivity, and most models fail once diagrams grow large. The result is a public yardstick that makes those structural failures measurable and comparable.","feed_headline":"VLMs falter on weak entities and N-ary links in ER diagrams","feed_subtitle":"A 2,960-diagram benchmark shows common parts are easy; the structural notation that matters is not","key_machinery":"ERUnderstand: a three-source corpus (curated educational diagrams, production schemas, and LLM-assisted Graphviz synthetics) paired with a unified JSON schema and a structure-aware fractional F1 that scores entities, relationships, attributes, keys, weak constructs, inheritance, and higher-arity links with partial credit for misconnections.","core_discovery":"On a 2,960-diagram benchmark with machine-readable JSON ground truth, state-of-the-art vision-language models recover common ER elements at high F1 but drop to roughly 0.28 on weak entities, 0.14 on multivalued attributes, and 0.07 on N-ary relationships; reasoning models lift overall Macro-F1 by about 15–25 percent yet remain sensitive to linguistic priors, spatial layout bias, and rising diagram complexity.","pith_inferences":["Closing the gap likely needs architectures that explicitly model edge connectivity and line-style notation, not only larger multimodal pretraining.","Interactive multi-turn prompting can recover omissions but will not fix systematic miswiring, so product pipelines still need verification layers.","Because production relational dumps lack EER constructs by construction, benchmarks built only from DDL will overstate readiness for conceptual modeling tasks."],"forward_implications":["AI database assistants cannot yet treat textbook or slide ER images as reliable executable schemas without human checking.","Progress on multimodal schema recovery will be measured by gains on weak entities, multivalued/derived attributes, and N-ary and IS-A structure, not only entity/attribute OCR.","Synthetic controllable generators become a standard stress test for spatial connectivity and notation sensitivity in VLMs.","Reasoning-time methods help but do not remove dependence on name priors or collapse under dense layouts.","The same JSON-plus-fractional-F1 protocol can be extended to UML class diagrams and property-graph schemas."],"fun_headline_variants":["VLMs hit 0.07 F1 on N-ary links in 2,960 ER diagrams","Weak entities and multivalued attributes stump top VLMs","ERUnderstand: VLMs excel on common ER parts, fail on hard ones","Reasoning lifts VLM ER scores 15-25% but complexity still hurts","Benchmark finds VLMs unreliable on EER constructs that matter"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The ranking and failure diagnosis rest on a single fixed partial-credit F1 with greedy label matching and one zero-temperature prompt template, without a classical shape-and-OCR parser baseline on the same target.","fun_headline_variants_meta":{"raw":{"variants":["VLMs hit 0.07 F1 on N-ary links in 2,960 ER diagrams","Weak entities and multivalued attributes stump top VLMs","ERUnderstand: VLMs excel on common ER parts, fail on hard ones","Reasoning lifts VLM ER scores 15-25% but complexity still hurts","Benchmark finds VLMs unreliable on EER constructs that matter"]},"model":"grok-4.5","effort":"low","cost_usd":0.002702,"raw_usage":{"total_tokens":1030,"prompt_tokens":815,"num_sources_used":0,"completion_tokens":102,"cost_in_usd_ticks":27024000,"prompt_tokens_details":{"text_tokens":815,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":113,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":815,"tokens_out":102,"duration_ms":3989,"temperature":1.0,"reasoning_tokens":113,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T07:12:42.593778+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the released evaluation toolkit on the same 2,960 images with a classical detector-plus-heuristic ER parser (or a different matching/partial-credit scheme) and check whether the same element-wise gaps and model orderings still appear.","supporting_citations":[],"review_version":1}