{"id":"11f25289-3919-4e47-b75a-e4cd789e5e2a","arxiv_id":"2502.04389","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A proof-of-concept showing that parsing diagram metadata from Office XML files into text lets LLMs answer diagram questions more reliably than feeding the rendered image to a VLM.","lead":"This paper tests a text-based alternative to vision-language models for reading diagrams: instead of showing the model a picture, it extracts shape and connector data from the Excel file's XML and feeds that text to an LLM. In a single proof-of-concept diagram, the text-based method correctly listed all connectors while GPT-4o's image-based analysis added several nonexistent connections.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The comparative claim rests on one diagram, and Section 3.2 discards exact connector-to-shape metadata in favor of coordinate-direction inference, so the method's advantage over VLM is unproven for less convenient diagram layouts.","rationale":"The reader identifies Section 3.2's coordinate/cardinal-direction decoding as the fragile assumption; I agree with that location and share the concern. My read sharpens it: the parser is discarding exact connection references (stCxn/endCxn) that are already present in the source XML, so the method is not fully exploiting the source-file advantage it claims. The single-case comparison in Section 4.3 demonstrates VLM hallucination on bent connectors, but it does not stress the coordinate-direction heuristic. The paper is candid about limitations in Appendix A.1 and open-sources its code, so conditional acceptance is fair. However, the headline claim about 'more accurate answers' should be explicitly scoped to diagrams with ungrouped shapes whose connectors terminate at shape boundaries until the proposed test is run. This does not change the reader's CONDITIONAL verdict; it specifies a concrete condition that should be part of the acceptance criteria.","tokens_in":11236,"tokens_out":3401,"duration_ms":34885,"concrete_test":"Build a small corpus of about 20 xlsx diagrams in which connector endpoints are placed at controlled offsets from shape boundaries (e.g., 0, 5, 15, 30 pixels), with a subset using grouped shapes or SmartArt. For each file, run the published parser and have the LLM enumerate all connectors; compare against ground truth recovered directly from the <a:stCxn>/<a:endCxn> elements in the same XML. If the coordinate-direction inference produces any wrong connection on a case where image-input GPT-4o is correct, or if supplying the exact connection metadata changes the output, then the central claim must be narrowed to diagrams whose connectors abut shape boundaries with ungrouped shapes.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the XML-driven approach yields 'more accurate answers' for diagram-structure questions—is supported only by the single self-selected diagram in Fig. 2. Within that example the pipeline succeeds, but Section 3.2 makes a load-bearing design choice: it explicitly declines to use saved connector-to-shape connection information and instead decodes each connector endpoint into one of four cardinal directions from coordinate data. In DrawingML/OOXML, connectors can carry stCxn/endCxn elements that name the exact connected shape IDs; the paper discards this exact topology and reconstructs it geometrically. That reconstruction is exact only when connectors terminate at or near shape boundaries and shapes are ungrouped rectangles/text boxes. Real diagrams often violate this: floating endpoints, grouped shapes, SmartArt, overlapping connectors, and text boxes whose bounding boxes overlap multiple shapes create ambiguous nearest-shape decisions. Appendix A.1 explicitly concedes that grouped or composite shapes, SmartArt, and specialized WordArt may not be fully supported, and that evaluation is case-based rather than quantitative. Because the claimed superiority over VLM is specifically about 'detailed comprehension of diagram structures,' the connection-inference heuristic is the weakest link: if it degrades on less convenient layouts, the method loses its stated advantage.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an XML-driven diagram understanding approach that bypasses vision-language models (VLMs) by extracting shape, connector, and text information from editable Office source files (in this proof-of-concept, .xlsx) and feeding a structured JSON representation to an LLM. The approach is evaluated on a single system-design diagram created in Microsoft Excel, with qualitative comparisons against GPT-4o image input. The authors report that the XML-driven method correctly enumerated components and connectors without hallucination, whereas the VLM misidentified a bent connector's target, missed a connector to a URL text box, and invented connectors to unrelated components. The paper concludes that the text-driven framework yields more accurate answers for questions requiring detailed comprehension of diagram structures, and it open-sources the parsing library and experimental code.","tokens_in":11415,"tokens_out":2500,"duration_ms":23370,"significance":"If the central claim is correct, the approach offers a practical alternative for diagram understanding in business documents, where source-file metadata is available and where VLM visual recognition errors are a known bottleneck. The paper's strengths include a reproducible open-source pipeline, a clearly described extraction and transformation process, and a direct comparison against a strong VLM baseline on the same diagram. The idea of using native XML topology instead of image rendering is well motivated by prior work on flowchart-to-Mermaid bottlenecks. However, the evidence is limited to one self-selected diagram, no quantitative metrics are reported, and the parser deliberately discards the exact connector-to-shape connection metadata that the OOXML format provides. The stated advantage over VLMs is therefore demonstrated only for a convenient layout, and the generality of the claim is not established.","major_comments":[{"comment":"The core comparative claim that the proposed method 'yielded more accurate answers' is supported by a single hand-picked diagram and qualitative output in Figs. 4 and 5. There are no accuracy or F1 metrics, no repeated trials, no error bars, and no diversity of diagrams or question types. Appendix A.1 explicitly concedes 'insufficient quantitative evaluation' and that the study is 'case-based examples.' As written, the abstract and conclusion overstate the evidence: a proof-of-concept can report an existence proof, but it cannot claim general superiority over VLM-based methods without a systematic evaluation.","section":"§4.3 and Abstract"},{"comment":"The parser chooses not to use the explicit connector-to-shape connection information that OOXML/DrawingML can store (e.g., stCxn and endCxn elements referencing shape IDs), and instead reconstructs connections by decoding each connector endpoint into one of four cardinal directions from coordinate data. This reconstruction is exact only when connectors terminate near shape boundaries and shapes are ungrouped rectangles or text boxes. Grouped shapes, SmartArt, WordArt, floating endpoints, and overlapping bounding boxes make nearest-shape inference ambiguous. Appendix A.1 itself admits that 'Grouped or composite shapes, SmartArt, and specialized WordArt objects may not be fully supported.' Since the paper's stated advantage is specifically about 'detailed comprehension of diagram structures,' this design choice is load-bearing: on less convenient layouts, the connection-inference heuristic can fail, and the claimed advantage over VLMs may collapse. The authors should either use the available connection metadata, or evaluate the heuristic on a set of diagrams that exercises these failure modes.","section":"§3.2"},{"comment":"The conclusion states that the approach 'proved successful in enabling LLMs to understand the diagram without being affected by the detection errors inherent in visual recognition.' This is too strong given that only one diagram was tested and the comparison is qualitative. The manuscript's own limitation statement in A.1 contradicts this degree of certainty. I recommend softening the claim to a proof-of-concept result and explicitly stating the scope of the evidence.","section":"§5"}],"minor_comments":[{"comment":"The text contains a placeholder reference 'Fig. XX' where the JSON format is described; this should be replaced with the actual figure number.","section":"§3.2"},{"comment":"There are typos in the text: 'halucinate' and 'halcination' should be 'hallucinate' and 'hallucination'.","section":"§2.1"},{"comment":"The section heading 'Comparision with VLM' should be 'Comparison with VLM'.","section":"§4.3"},{"comment":"The figure text is very small and the shape/connector IDs are overlaid on a dense diagram, making the outputs difficult to read. Consider enlarging the annotated diagram or presenting the outputs in a tabular form.","section":"Figures 4 and 5"},{"comment":"Several references contain formatting errors, such as 'Information F orensics' with a stray space and 'Lingustics' for 'Linguistics'; these should be corrected.","section":"References"},{"comment":"The prompts for the entity-understanding and relation-understanding stages are shown in Figure 6, but it is unclear whether the system message is the same for both stages and whether the human messages in the figure include both task instructions. Clarify the exact prompt composition used in each of the two stages.","section":"Appendix A.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is an honest proof-of-concept with open-source code, which is a positive feature. However, the abstract and conclusion go beyond the evidence by claiming 'more accurate answers' based on one diagram and no quantitative evaluation. The design choice in §3.2 to ignore explicit connection metadata is the weakest point; for a journal submission, the authors should either use the available stCxn/endCxn information or demonstrate that their geometric decoding is robust on a broader set of diagrams. The paper might be better framed as a short systems or application paper rather than a full research article, unless the evaluation is substantially expanded."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a modest, honest proof-of-concept that extracting xlsx DrawingML shape and connector data into JSON lets an LLM answer relation questions about one system-design diagram where GPT-4o's image input hallucinated. That is a real result, but the comparative claim in the abstract is broader than the evidence, and the pipeline's topology-reconstruction choice may not survive contact with messier real diagrams.\n\nWhat's new: the specific pipeline — parse the xlsx zip/XML, normalize coordinates and colors, emit JSON with shapes and connectors, feed to GPT-4o text — is, as far as I can tell, not in the prior work they cite. The closest analogues are Mermaid/HTML/docx representations, but xlsx connector topology is a different beast, and they show a concrete failure mode of GPT-4o (curved connector near a corner) where the text version succeeds. The code is open-sourced, which makes the feasibility claim checkable. The related-work citations are appropriate and not padded.\n\nSoft spots, in order. First, the evidence is one diagram, one VLM, one prompt set. The abstract says 'experimental comparisons demonstrated more accurate answers', but there is no accuracy metric, no repeated trials, no second diagram. The limitations section concedes this, so the paper is honest, but the abstract oversells. Second, Section 3.2 explicitly declines to use the saved stCxn/endCxn connection info and reconstructs topology from connector endpoint directions and coordinates. The stress-test is right that this is the fragile part: for floating endpoints, grouped shapes, SmartArt, or overlapping shapes, the nearest-shape inference can be wrong, and Appendix A.1 admits those object types aren't fully supported. It's a real limitation, but it is disclosed, and for the one diagram it works. I wouldn't call it load-bearing to the point of invalidating the paper; it does limit generality. Third, minor: the placeholder 'Fig. XX' in Section 3.2 suggests sloppy finalization.\n\nWho is this for? Someone building business-document QA pipelines over Office files, especially Excel-centric workflows. It won't redirect research, but it is a useful data point that source-file metadata can beat VLM pixels on connector-relation questions. It deserves peer review as a proof-of-concept; a serious referee should push for a small benchmark (a dozen diagrams, a few formats, some with grouped shapes) and a quantitative comparison before publication.\n\nRecommendation: send it out, but expect major revision. The authors should either narrow the abstract or add that benchmark.","headline":"A modest, honest proof-of-concept: XML extraction from xlsx beats GPT-4o on one diagram, but the abstract overclaims and the connector-inference heuristic is the fragile part.","tokens_in":11978,"tokens_out":2717,"would_cite":false,"duration_ms":25093,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that diagram understanding can bypass visual recognition entirely by parsing the diagram's source XML into structured text for an LLM.","keywords":["diagram understanding","vision-language models","large language models","XML parsing","xlsx source files","connector relation extraction","Office Open XML","proof of concept"],"falsifier":"Run the same text-driven prompt on xlsx diagrams that use grouped shapes or SmartArt and compare connector accuracy with the image-input baseline; if accuracy drops to or below the VLM's, the coordinate-based direction decoding is the reason.","tokens_in":11007,"feed_emoji":"📊","tokens_out":7093,"duration_ms":61030,"temperature":0.7,"pith_summary":"This paper proposes that diagrams stored in editable Office source files can be understood by large language models without any image processing. The authors parse the XML inside an .xlsx file, convert shapes and connectors into structured JSON, and feed that text to an LLM. In a proof-of-concept on an Excel-drawn system architecture diagram, the text-driven method recovered every connector correctly, while an image-input vision-language model misidentified a curved connector, invented two nonexistent connections, and missed a real one. The claim is that reliable diagram question answering does not require visual recognition when the diagram's source file is available, and that the same principle should carry over to .pptx and .docx.","feed_headline":"XML beats pixels for diagram question answering","feed_subtitle":"A proof-of-concept recovers every connector from an Excel diagram's source file while the image-based VLM hallucinates.","key_machinery":"The load-bearing mechanism is a parser for Office Open XML that turns a diagram's drawing elements into a compact JSON schema of shapes and connectors. For each connector, the parser records start and end coordinates plus a derived cardinal direction for each endpoint, deliberately ignoring the file's explicit connector-to-shape attachment records because diagram creators may leave connectors and shapes logically disconnected. For each shape, it records type, fill and border color, edge coordinates, and contained text, after converting rotation, flip, scale, and theme colors so the numbers match the rendered appearance. This JSON, accompanied by attribute descriptions in the prompt, is what lets an LLM reconstruct which shapes a connector links without ever seeing pixels.","core_discovery":"On the paper's own terms, the central discovery is that the bottleneck in diagram understanding is visual recognition, and that bottleneck can be bypassed by replacing the rendered image with the diagram's source-level representation. Using a system design diagram drawn in Excel, the authors show that a vision-language model given the rendered image lists the components correctly but hallucinates when asked which components are connected: it misreads a curved connector as pointing to a nearby but unrelated component, omits the connector to the URL text box, and invents connections to Key Vault and Microsoft Entra that do not exist. The same model, given the JSON parsed from the XML, identifies all seven connectors with their correct endpoints and annotations. The paper therefore claims that XML-driven text input yields more accurate answers for questions requiring detailed structural comprehension of diagrams.","pith_inferences":["If the approach generalizes, diagram understanding could be treated as a data-extraction problem rather than a perception problem, which would make accuracy easier to audit because each claimed connection can be traced to a JSON record.","A testable extension would be to add explicit connector-to-shape attachment metadata when the source file stores it, and compare against the coordinate-decoding heuristic on grouped shapes and SmartArt.","The same XML-to-JSON pipeline could feed retrieval-augmented generation over a corpus of design documents, letting a search index store diagram elements alongside tables and paragraphs.","Hybrid designs could delegate icons and free-form artwork to a vision-language model while trusting XML for connectors and shapes, limiting visual hallucination to elements that genuinely need image understanding."],"forward_implications":["An LLM equipped with parsed source files can answer relationship questions about a diagram without a vision-language model, so diagram QA no longer has to pass through image recognition.","The same XML-parsing strategy transfers to other Office formats whose source is XML, such as .pptx and .docx, so the approach is not specific to Excel.","The known failure modes of vision-language models on curved connectors, overlapping elements, and fine-grained line relations can be sidestepped by supplying coordinates and endpoint directions as text.","Because the method needs only shape and connector attributes, it can work with commodity text-only LLMs rather than requiring multimodal models.","Document loaders that extract only tables and paragraphs miss diagram information; parsing drawing elements as first-class text unlocks that information."],"supporting_citations":[{"why":"Documents that large vision-language models still struggle with geometric perception of shapes, motivating the paper's shift away from image input.","marker":"Kamoi et al. (2024)"},{"why":"Shows the bottleneck in flowchart QA is textualizing geometric structure and that intermediate text representations help, the main precedent for XML-driven input.","marker":"Ye et al. (2024)"},{"why":"Reports VLMs misidentify complex geometric primitives such as overlapping lines and shapes, the specific failure class the text-driven method avoids.","marker":"Rahmanzadehgervi et al. (2024)"},{"why":"Provides evidence that VLMs struggle to grasp line-based relationships and hallucinate when visual input conflicts with learned knowledge, the failure reproduced in the comparison.","marker":"Giledereli et al. (2024)"},{"why":"Flowchart understanding benchmark showing image-to-text conversion is the hard step, supporting the choice to skip image rendering entirely.","marker":"Pan et al. (2024)"},{"why":"In GUI understanding, supplying layout and positional text alongside screenshots improves recognition, precedent for augmenting models with source-file metadata.","marker":"You et al. (2024)"},{"why":"Extends the same layout-plus-text idea to universal UI understanding, further support that source metadata helps multimodal models.","marker":"Li et al. (2024)"},{"why":"The GPT-4o model report; GPT-4o is the model used for both the VLM image-input baseline and the XML-driven LLM evaluation.","marker":"OpenAI et al. (2024b)"}],"fun_headline_variants":["XML from diagram source beats pixels for QA","Text-driven diagram reading outperforms VLM in test","Skip vision: XML diagram data answers correctly","Proof-of-concept: XML trumps image for diagram queries","Diagram XML source yields accurate QA, pixel model fails"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes it can guess which shapes a connector touches from coordinates and four direction labels, so diagrams with grouped shapes, SmartArt, or connectors floating near edges may lose the relationships it is supposed to recover.","fun_headline_variants_meta":{"raw":{"variants":["XML from diagram source beats pixels for QA","Text-driven diagram reading outperforms VLM in test","Skip vision: XML diagram data answers correctly","Proof-of-concept: XML trumps image for diagram queries","Diagram XML source yields accurate QA, pixel model fails"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000341,"raw_usage":{"total_tokens":1890,"prompt_tokens":967,"completion_tokens":923,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":850}},"tokens_in":583,"tokens_out":923,"duration_ms":9340,"temperature":1.0,"reasoning_tokens":850,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T04:06:17.463607+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same text-driven prompt on xlsx diagrams that use grouped shapes or SmartArt and compare connector accuracy with the image-input baseline; if accuracy drops to or below the VLM's, the coordinate-based direction decoding is the reason.","supporting_citations":[],"review_version":1}