{"id":"f434e881-f6c6-4efd-9699-bdb0886311db","arxiv_id":"2507.22938","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Flowchart images are converted to graphs by a fine-tuned vision model, then embedded as text chunks, letting a text-based retriever answer flowchart-related questions in telecom documents.","lead":"This paper describes a system that turns flowchart images in telecom manuals into text descriptions of their boxes and arrows, so a text-only search engine can find the right diagram when someone asks a question. A smart generalist might care because it is a practical way to make document question-answering work on diagrams without running a large image model at query time.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Prompt in Sec. 3.2.2 appears to put the ground-truth flowchart JSON in the model input, so the reported GED drop (10.21 -> 2.74) may reflect answer-copying rather than learned graph parsing; the central claim is not reliably evidenced.","rationale":"The reader's stated weakest assumption is that node shapes, edge styles, and spatial layout are discarded in the graph representation, reducing semantic fidelity. That is a legitimate secondary concern. But the first-order threat is the prompt in Sec. 3.2.2, which appears to include the ground-truth JSON of the flowchart being converted. If that reading is correct, the GED numbers in Table 3 are not evidence of VLM graph-understanding at all; they are evidence of copying a supplied answer. Because the retrieval experiments in Sec. 4.3 consume the same VLM-generated JSONs and the QA benchmark is proprietary with no public subset, there is no independent confirmation that the graphs are produced without the answer in the prompt. The concern is concrete and testable: rerun the Flowlearn test with a clean production prompt. If the clean-prompt GED returns to about 10.21, the central claim is unsupported. If it stays near 2.74, the worry is resolved. I mark the verdict UNVERDICTED rather than REJECT because the issue may be a reporting ambiguity fixable by a corrected evaluation, but as written the evidence cannot be assessed. My agreement with the reader is partial because the reader's headline weakest assumption was shape information, while I see the prompt condition as more load-bearing; the reader's rationale did, however, flag the prompt issue among the unresolved problems.","tokens_in":9451,"tokens_out":6720,"duration_ms":67841,"concrete_test":"Reproduce the Flowlearn test-set GED evaluation using the exact prompt that would be used in production, with the ground-truth image_json field removed from the input (or filled with a fixed unrelated example that is never the test image). If GED rises to roughly the base-model value of 10.21, the Table 3 gain is explained by answer-copying; if GED stays near 2.74, the concern is resolved. Also report the literal prompt string sent to the model for test-time inference, including exactly what is inserted into image_json.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central claim is that a fine-tuned VLM converts flowchart images into graph JSONs accurately enough that a text-only retriever can support flowchart QA without a VLM at inference. The direct evidence for the conversion step is Table 3: GED drops from 10.21 (base Qwen2-VL) to 2.74 (fine-tuned) on Flowlearn, and to 3.14 on proprietary PI Docs flowcharts. However, the prompt quoted in Sec. 3.2.2 reads: \"I have uploaded an image of a flowchart and here is its ground truth JSON representation, image_json ={}. Now generate JSON for the next image...\" If image_json is populated with the ground-truth graph of the image being converted, the model is handed the answer before it generates anything. In that case, the GED improvement measures the model's ability to copy a supplied target, not its ability to parse flowchart images. The same VLM-generated JSONs are used in the retrieval experiments of Sec. 4.3, so the leakage would propagate to the top-1 accuracy of about 57% and invalidate the end-to-end demonstration. The wording is ambiguous enough that it could instead describe a one-shot example with ground truth from a separate image, but the manuscript does not say which interpretation is correct, and the phrase \"Now generate JSON for the next image\" strongly suggests the target is in context. This is the most load-bearing unresolved issue: if it lands, the main empirical support for the pipeline disappears; if it does not, the concern is resolved by a simple clarification.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an end-to-end pipeline for incorporating flowchart images into text-based RAG question answering in the telecom domain. Images extracted from documents are first classified with a fine-tuned DiT model; flowchart images are then converted into graph-structured JSON representations using a fine-tuned Qwen2-VL model, with the Fine-tuned model trained on the public Flowlearn synthetic flowchart dataset augmented by the authors. The graph JSONs are embedded as text using either bge-large or the domain-adapted TeleRoBERTa, under three chunking strategies, and retrieval is evaluated on 502 SME-curated QA pairs built from 105 proprietary telecom flowcharts. The main reported results are a reduction in average graph edit distance on the Flowlearn test set from 10.21 (base Qwen2-VL) to 2.74 (fine-tuned), a GED of 3.14 on proprietary flowcharts, and a top-1 retrieval accuracy of about 57% with TeleRoBERTa when each graph node is embedded as one chunk. The paper claims that the approach enables image retrieval for flowchart QA without invoking a VLM at inference time.","tokens_in":9739,"tokens_out":6577,"duration_ms":66146,"significance":"If the reported results hold, the paper offers a practical and cost-effective way to bring flowchart images into text-only RAG systems, which is a genuinely useful contribution for industrial document QA. The strongest evidence is the graph-edit-distance comparison on Flowlearn, which uses an external public test set and shows a large, clear improvement from fine-tuning. The paper also makes a sensible design choice by representing flowcharts as attributed directed graphs and by comparing several chunking strategies. The proprietary QA dataset, built with SME involvement, is a useful resource even though it is not publicly released. However, the end-to-end claim is currently supported only partially: the retrieval benchmark may use ground-truth graphs rather than the VLM-generated graphs that would appear in a deployed pipeline, the retrieval correctness criteria differ across chunking methods, and the LoRA hyperparameters are selected on the test set. These issues are fixable with additional clarification and experiments, so the central idea remains defensible.","major_comments":[{"comment":"The prompt quoted in Section 3.2.2 is ambiguous in a way that affects the validity of the central GED result: it states 'here is its ground truth JSON representation, image_json ={}' and then says 'Now generate JSON for the next image'. If the placeholder is ever populated with the ground-truth JSON of the image being converted, the model would be given the answer before generating anything, and the improvement from 10.21 to 2.74 in Table 3 would measure copying rather than parsing. If instead the placeholder is always empty and the text is simply a poorly worded one-shot instruction, the concern is resolved. Please state explicitly whether any ground-truth JSON is ever present in the prompt at training, validation, or test time, and provide the exact prompt template used for inference on the Flowlearn test set and on the PI Docs flowcharts.","section":"§3.2.2"},{"comment":"It is unclear whether the retrieval experiments in Section 4.3 embed the ground-truth graph JSONs associated with the 105 PI Docs flowcharts or the graph JSONs produced by the fine-tuned VLM. The dataset description mentions ground-truth graph structures, while the pipeline description in Section 2.3 refers to graphs 'obtained from the previous step,' i.e., from the VLM. If only ground-truth graphs are used, the reported top-1 accuracy of 57.17% does not reflect the actual end-to-end system, because errors in the VLM output (average GED 3.14 on PI Docs) are not propagated. Please clarify which graph source is used, and report retrieval results on VLM-generated graphs to substantiate the claim that a text-only retriever can support flowchart QA without a VLM at inference.","section":"§3.1.3, §4.3"},{"comment":"The fine-tuning section states that 'the fine-tuning was performed for few choices of parameters R and alpha, and the best fine-tuned model was used in the pipeline,' but no validation split is reported and Table 3 reports test-set GED for every hyperparameter setting. Selecting the LoRA rank and alpha based on test-set performance introduces optimism bias in the reported best GED of 2.74. Please perform hyperparameter selection on the Flowlearn validation split (or a held-out subset) and report the corresponding test-set GED, or otherwise justify that the reported value is not a test-set artifact.","section":"§3.2.2, Table 3"},{"comment":"The retrieval correctness criteria are not comparable across the three chunking approaches. For 'each node as one chunk,' retrieval is considered correct if any node from the top-k retrieved graphs appears in the ground truth; for 'all nodes as one chunk,' all ground-truth nodes must appear; and for 'entire graph JSON as one chunk,' the retrieved graph must correspond to the ground truth. These are very different standards of correctness, so the top-k accuracies in Table 4 cannot be directly compared to conclude that one chunking strategy is better. Please use a single, well-defined criterion, such as whether the ground-truth flowchart's chunk appears in the top-k, or report node-level precision/recall and graph-level accuracy separately.","section":"§2.4, Table 4"}],"minor_comments":[{"comment":"The graph representation deliberately discards node shapes, but the QA benchmark includes a large category of 'decision-related' questions. Since decision semantics in flowcharts are sometimes conveyed by the shape of the decision node (e.g., a diamond), the paper should discuss whether this information loss could affect retrieval or downstream answering, even if the QA pairs were created from the graph representations.","section":"§2.2"},{"comment":"The QA category counts in Table 1 sum to 1325, far exceeding the 502 total QA pairs, which means questions can belong to multiple categories. Please clarify how a question can be both 'node related' and 'edge related,' and state whether the per-category retrieval results in Table 5 are computed on overlapping subsets.","section":"§3.1.3, Table 1"},{"comment":"In the lower half of Table 4 (interspersed with text), the values for bge-large are given without percent signs (e.g., 41.05, 55.53, 59.76), while the upper half uses percentages. Please format consistently.","section":"Table 4"},{"comment":"The GED values in Table 3 are reported as averages without standard deviations or confidence intervals, and the number of test flowcharts is not stated for either the Flowlearn test set or the PI Docs set. Adding these would help assess the stability of the reported improvements.","section":"§4.2, Table 3"},{"comment":"The synthetic augmentation of the Flowlearn training set is described only qualitatively. Please provide details on the number of generated images, the distribution of added node shapes and edge types, and how the augmented data were split to avoid overlap with the test set.","section":"§3.1.2"},{"comment":"Reference [7] is cited for the 'microsoft/dit-base' model, but the cited paper (Lewis et al., 2006, 'Building a test collection for complex document information processing') is not the Document Image Transformer. The correct DiT citation appears to be missing; please fix this reference error.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is an industry submission and relies on a proprietary dataset that cannot be released; this limits reproducibility but is not by itself a reason for rejection. The most important point to verify in revision is whether the retrieval benchmark uses ground-truth or VLM-generated graphs, because this determines whether the end-to-end claim is actually supported. The prompt-leakage concern raised during review appears less severe than it might seem, since the printed prompt contains an empty placeholder, but the authors must clarify the exact inference-time prompt to rule out answer-copying. Given that the core GED result on the public Flowlearn set is strong and the retrieval issues are addressable, major revision is appropriate rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things about this one. First, it fills a real gap in the practical RAG literature: instead of running a VLM at query time, you pre-convert flowchart images to graph JSONs and retrieve them with a text embedder. That's a cost-saving move that others have only gestured at. Second, the headline GED result (10.21 -> 2.74) has a suspicious prompt behind it, and until that's clarified, the main quantitative claim is not reliable.\n\nWhat is genuinely new: Flowlearn stops at graph conversion, FlowVQA assumes the flowchart is already given. This paper adds the retrieval step over VLM-generated graph JSONs and evaluates it on a telecom QA set. The chunking comparison is also a nice touch. Credit where due: the GED improvement on the public Flowlearn test set is large, and the model carries over to their proprietary flowcharts with a low GED (3.14), which suggests some real generalization rather than pure overfitting. The retrieval numbers, around 57% top-1, are moderate but plausible for a first attempt.\n\nNow the soft spots, in order of severity.\n\n1. The prompt in Section 3.2.2 appears to put the ground truth JSON in the model input: 'here is its ground truth JSON representation, image_json ={}. Now generate JSON for the next image'. If that {} gets filled with the current image's target JSON, then the model is copying the answer, and the GED drop is meaningless. If it's meant as a one-shot example from a different image, the paper needs to say so explicitly. Right now, it's ambiguous, and the ambiguity is load-bearing because the retrieval results in Section 4.3 use these same JSONs.\n\n2. LoRA rank and alpha appear to be chosen on the test set (Section 3.2.2: 'best fine-tuned model was used' after reporting results for several settings). No validation split is reported. This inflates the reported GED, though the PI Docs result helps.\n\n3. The retrieval correctness criteria change across chunking approaches (Section 2.4). For 'each node as one chunk', any matching node counts; for 'all nodes as one chunk', all ground-truth nodes must appear; for 'whole JSON', a graph match. You can't compare chunking strategies when the scoring rules differ. Use one rule: does the source graph of the retrieved chunk match the ground-truth graph?\n\n4. The QA benchmark is proprietary and self-annotated, with no error bars. That's a limitation, not a fatal flaw, but it means we can't independently check the retrieval claims.\n\nOverall: this is a useful industrial paper for people building RAG over technical documents with flowcharts. The pipeline is sensible and the direction is worth pursuing. But the prompt ambiguity and the evaluation inconsistencies mean the central claim is not yet convincingly evidenced. I'd send it to review with a request for a major revision: clarify the inference prompt, add a validation split, fix the matching rule, and ideally release an anonymized subset.","headline":"Useful RAG pipeline for flowchart QA, but the prompt ambiguity in Sec. 3.2.2 puts the headline GED result in question.","tokens_in":10337,"tokens_out":4125,"would_cite":false,"duration_ms":39111,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fine-tuned vision-language model turns flowchart images into graph JSONs, and the paper shows that a text-only retriever can then answer flowchart questions without calling the vision model at query time.","keywords":["multi-modal question answering","flowchart understanding","graph representations","retrieval augmented generation","visual language models","telecom technical documents","graph edit distance","text embedding retrieval"],"falsifier":"Build two flowcharts with identical node and edge labels that differ only where the semantics live—decision diamonds versus plain rectangles, or dashed versus solid links—and ask a question whose answer depends on that difference; if the text-embedding retriever ranks them interchangeably, the claim that these graph JSONs carry enough flowchart semantics fails.","tokens_in":9245,"feed_emoji":"📊","tokens_out":11780,"duration_ms":110642,"temperature":0.7,"pith_summary":"This paper proposes that a flowchart image can be treated as a text document by first converting it to a directed graph: each block becomes a node holding the block's text, each link becomes an edge holding any link text, and the whole graph is serialized as JSON. It then argues that embedding that JSON as ordinary text chunks is enough for a text-only retrieval system to find the correct flowchart in response to a question, without calling a vision model at query time. The conversion is done by a fine-tuned vision-language model, which lowers graph edit distance from 10.21 to 2.74 on a public synthetic-flowchart test set and to 3.14 on proprietary telecom flowcharts. On 502 curated QA pairs from 105 telecom flowcharts, the best text retriever places the right flowchart in its top-1 result about 57 percent of the time. The payoff, if the claim holds, is that multimodal flowchart QA can run on the same text embedding pipeline that already serves document RAG.","feed_headline":"Flowchart QA runs on text retrieval once a vision model graphs it","feed_subtitle":"Converting flowcharts to graph text lets deployed QA systems skip vision models at query time, saving cost.","key_machinery":"The load-bearing object is the flowchart graph JSON: a directed graph in which each block becomes a node carrying the block's text, each link becomes a directed edge carrying any text on the link, and block shape is deliberately omitted. Because the graph is serialized as JSON, it can be chunked and embedded by the same text-embedding machinery used for paragraphs. A LoRA-fine-tuned open-source vision-language model is the converter, trained on synthetic flowcharts with ground-truth JSON; graph edit distance scores conversion fidelity by comparing generated nodes and edges to ground truth. On the retrieval side, three chunking strategies—each node as one chunk, all nodes as one chunk, and the entire JSON as one chunk—are embedded with general-purpose and telecom-domain text embedding models, and top-k accuracy measures whether the correct graph JSON is retrieved. The key move is serialization itself: a graph written as JSON enters the vector store without any multimodal index.","core_discovery":"On its own terms, the central claim is that a flowchart's question-answering content survives being reduced to a directed graph of node texts and edge texts, and that this reduced representation, embedded as text, is enough for retrieval. The supporting pipeline has three stages: a fine-tuned document-image classifier selects flowchart images from mixed technical documents; a fine-tuned vision-language model converts each flowchart into a graph JSON whose nodes carry block text and whose edges carry link text; and three chunking schemes (per node, all nodes, whole JSON) put those JSONs into a standard vector store. The reported evidence is quantitative: fine-tuning cuts graph edit distance from 10.21 to 2.74 on a public synthetic-flowchart test set, the same model reaches 3.14 on unseen proprietary telecom flowcharts, and retrieval on 502 curated QA pairs reaches 57.17 percent top-1 accuracy with a telecom-domain text embedding model. The consequence the authors emphasize is operational: because the graph is stored as text, the vision model is needed at ingestion time only, not at query time.","pith_inferences":["The same graph-JSON-as-text trick should extend to other structured diagram types, such as sequence diagrams, provided their ordering semantics can be serialized into nodes and edges; the paper names this direction as future work.","Because the QA benchmark scores retrieval against graph JSONs rather than original images, the reported top-1 figure is a retrieval number, not an end-to-end answer-accuracy number; feeding the retrieved JSON to a generator is the natural next measurement.","A direct testable extension is to fine-tune the converter with extra weight on decision-heavy and edge-heavy synthetic flowcharts; if decision- and edge-related retrieval accuracy still trails node-related accuracy, the bottleneck lies in the text representation itself rather than in the vision-to-graph conversion."],"forward_implications":["A deployed telecom RAG system can serve flowchart questions with the same text retriever it already runs, with the vision model invoked only when documents are ingested.","Whole-JSON chunking gives higher top-3 and top-5 retrieval coverage, while per-node chunking gives the highest top-1 accuracy, so the chunking choice can be tuned to the cost of a wrong first result.","Fine-tuning on synthetic flowcharts transfers to proprietary telecom flowcharts, so the conversion stage does not require large amounts of domain-labeled flowchart images.","Node-related questions retrieve more reliably than decision-related or edge-related questions, indicating where the graph representation currently carries the most information."],"supporting_citations":[{"why":"supplies the exact graph edit distance algorithm used to score the VLM's graph output against ground truth.","marker":"[2]"},{"why":"provides the telecom-domain-adapted text embedding model that achieves the highest top-1 retrieval in the experiments.","marker":"[4]"},{"why":"provides the document-image transformer model fine-tuned into the classifier that separates flowchart images from other telecom document images.","marker":"[7]"},{"why":"supplies the synthetic flowchart images and ground-truth graph JSON used to fine-tune and test the vision-language model.","marker":"[10]"},{"why":"prior work converting flowcharts to graphs for QA that assumes the correct flowchart is already available; this paper adds the missing retrieval stage.","marker":"[14]"},{"why":"prior observations on chunking text and tables that motivate the three graph-JSON chunking strategies tested for retrieval.","marker":"[16]"},{"why":"the open-source vision-language model that is fine-tuned for graph generation and whose un-fine-tuned output provides the baseline edit distance.","marker":"[17]"},{"why":"the general-purpose text embedding model used as the non-domain retrieval baseline.","marker":"[18]"}],"fun_headline_variants":["Graph flowcharts to text, skip vision models at query time","Flowchart QA as text retrieval: vision only at ingestion","Turn flowcharts into graphs, then use plain text retrieval","Cut inference cost: embed flowchart graphs, not images","Graph-based flowchart QA: 57% top-1 without runtime vision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise, stated in Section 2.2, is that a flowchart's meaning for question answering survives being reduced to a directed graph of node texts and edge texts, with block shapes and edge line styles deliberately left out.","fun_headline_variants_meta":{"raw":{"variants":["Graph flowcharts to text, skip vision models at query time","Flowchart QA as text retrieval: vision only at ingestion","Turn flowcharts into graphs, then use plain text retrieval","Cut inference cost: embed flowchart graphs, not images","Graph-based flowchart QA: 57% top-1 without runtime vision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000191,"raw_usage":{"total_tokens":1354,"prompt_tokens":964,"completion_tokens":390,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":306}},"tokens_in":580,"tokens_out":390,"duration_ms":4430,"temperature":1.0,"reasoning_tokens":306,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:02:34.642905+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build two flowcharts with identical node and edge labels that differ only where the semantics live—decision diamonds versus plain rectangles, or dashed versus solid links—and ask a question whose answer depends on that difference; if the text-embedding retriever ranks them interchangeably, the claim that these graph JSONs carry enough flowchart semantics fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the telecom-domain-adapted text embedding model that achieves the highest top-1 retrieval in the experiments."},{"cited_title":"Lewis, Gady Agam, Shlomo Engelson Argamon, Ophir Frieder, David A","cited_arxiv_id":null,"evidence_quote":"provides the document-image transformer model fine-tuned into the classifier that separates flowchart images from other telecom document images."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the synthetic flowchart images and ground-truth graph JSON used to fine-tune and test the vision-language model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"prior observations on chunking text and tables that motivate the three graph-JSON chunking strategies tested for retrieval."}],"review_version":2}