Pith. sign in

REVIEW 4 major objections 6 minor 20 references

A Graph-based Approach for Multi-Modal Question Answering from Flowcharts in Telecom Documents

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A fine-tuned vision-language model turns flowchart images into graph JSONs, and the paper shows that a text-only retriever can then answer flowchart questions without calling the vision model at query time.

desk verdict Useful RAG pipeline for flowchart QA, but the prompt ambiguity in Sec. 3.2.2 puts the headline GED result in question. read the letter →

arxiv 2507.22938 v1 pith:S3LDVVL2 submitted 2025-07-25 cs.CL cs.AI

Sumit Soman , H. G. Ranjani , Sujoy Roychowdhury , Venkata Dharma Surya Narayana Sastry , Akshat Jain , Pranav Gangrade , Ayaaz Khan This is my paper · ORCID
classification cs.CLcs.AI
keywords multi-modalquestionansweringflowchartunderstandinggraphrepresentationsretrievalaugmentedgenerationvisuallanguagemodelstelecomtechnicaldocumentseditdistancetextembedding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes that a flowchart image can be treated as a text document by first converting it to a directed graph: each block becomes a node holding the block's text, each link becomes an edge holding any link text, and the whole graph is serialized as JSON. It then argues that embedding that JSON as ordinary text chunks is enough for a text-only retrieval system to find the correct flowchart in response to a question, without calling a vision model at query time. The conversion is done by a fine-tuned vision-language model, which lowers graph edit distance from 10.21 to 2.74 on a public synthetic-flowchart test set and to 3.14 on proprietary telecom flowcharts. On 502 curated QA pairs from 105 telecom flowcharts, the best text retriever places the right flowchart in its top-1 result about 57 percent of the time. The payoff, if the claim holds, is that multimodal flowchart QA can run on the same text embedding pipeline that already serves document RAG.

What carries the argument

The load-bearing object is the flowchart graph JSON: a directed graph in which each block becomes a node carrying the block's text, each link becomes a directed edge carrying any text on the link, and block shape is deliberately omitted. Because the graph is serialized as JSON, it can be chunked and embedded by the same text-embedding machinery used for paragraphs. A LoRA-fine-tuned open-source vision-language model is the converter, trained on synthetic flowcharts with ground-truth JSON; graph edit distance scores conversion fidelity by comparing generated nodes and edges to ground truth. On the retrieval side, three chunking strategies—each node as one chunk, all nodes as one chunk, and the entire JSON as one chunk—are embedded with general-purpose and telecom-domain text embedding models, and top-k accuracy measures whether the correct graph JSON is retrieved. The key move is serialization itself: a graph written as JSON enters the vector store without any multimodal index.

What would settle it

Build two flowcharts with identical node and edge labels that differ only where the semantics live—decision diamonds versus plain rectangles, or dashed versus solid links—and ask a question whose answer depends on that difference; if the text-embedding retriever ranks them interchangeably, the claim that these graph JSONs carry enough flowchart semantics fails.

Watch

Extended reading notes

Core claim

On its own terms, the central claim is that a flowchart's question-answering content survives being reduced to a directed graph of node texts and edge texts, and that this reduced representation, embedded as text, is enough for retrieval. The supporting pipeline has three stages: a fine-tuned document-image classifier selects flowchart images from mixed technical documents; a fine-tuned vision-language model converts each flowchart into a graph JSON whose nodes carry block text and whose edges carry link text; and three chunking schemes (per node, all nodes, whole JSON) put those JSONs into a standard vector store. The reported evidence is quantitative: fine-tuning cuts graph edit distance from 10.21 to 2.74 on a public synthetic-flowchart test set, the same model reaches 3.14 on unseen proprietary telecom flowcharts, and retrieval on 502 curated QA pairs reaches 57.17 percent top-1 accuracy with a telecom-domain text embedding model. The consequence the authors emphasize is operational: because the graph is stored as text, the vision model is needed at ingestion time only, not at query time.

Load-bearing premise

The load-bearing premise, stated in Section 2.2, is that a flowchart's meaning for question answering survives being reduced to a directed graph of node texts and edge texts, with block shapes and edge line styles deliberately left out.

Editorial extensions

If this is right

  • A deployed telecom RAG system can serve flowchart questions with the same text retriever it already runs, with the vision model invoked only when documents are ingested.
  • Whole-JSON chunking gives higher top-3 and top-5 retrieval coverage, while per-node chunking gives the highest top-1 accuracy, so the chunking choice can be tuned to the cost of a wrong first result.
  • Fine-tuning on synthetic flowcharts transfers to proprietary telecom flowcharts, so the conversion stage does not require large amounts of domain-labeled flowchart images.
  • Node-related questions retrieve more reliably than decision-related or edge-related questions, indicating where the graph representation currently carries the most information.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same graph-JSON-as-text trick should extend to other structured diagram types, such as sequence diagrams, provided their ordering semantics can be serialized into nodes and edges; the paper names this direction as future work.
  • Because the QA benchmark scores retrieval against graph JSONs rather than original images, the reported top-1 figure is a retrieval number, not an end-to-end answer-accuracy number; feeding the retrieved JSON to a generator is the natural next measurement.
  • A direct testable extension is to fine-tune the converter with extra weight on decision-heavy and edge-heavy synthetic flowcharts; if decision- and edge-related retrieval accuracy still trails node-related accuracy, the bottleneck lies in the text representation itself rather than in the vision-to-graph conversion.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes an end-to-end pipeline for incorporating flowchart images into text-based RAG question answering in the telecom domain. Images extracted from documents are first classified with a fine-tuned DiT model; flowchart images are then converted into graph-structured JSON representations using a fine-tuned Qwen2-VL model, with the Fine-tuned model trained on the public Flowlearn synthetic flowchart dataset augmented by the authors. The graph JSONs are embedded as text using either bge-large or the domain-adapted TeleRoBERTa, under three chunking strategies, and retrieval is evaluated on 502 SME-curated QA pairs built from 105 proprietary telecom flowcharts. The main reported results are a reduction in average graph edit distance on the Flowlearn test set from 10.21 (base Qwen2-VL) to 2.74 (fine-tuned), a GED of 3.14 on proprietary flowcharts, and a top-1 retrieval accuracy of about 57% with TeleRoBERTa when each graph node is embedded as one chunk. The paper claims that the approach enables image retrieval for flowchart QA without invoking a VLM at inference time.

Significance. If the reported results hold, the paper offers a practical and cost-effective way to bring flowchart images into text-only RAG systems, which is a genuinely useful contribution for industrial document QA. The strongest evidence is the graph-edit-distance comparison on Flowlearn, which uses an external public test set and shows a large, clear improvement from fine-tuning. The paper also makes a sensible design choice by representing flowcharts as attributed directed graphs and by comparing several chunking strategies. The proprietary QA dataset, built with SME involvement, is a useful resource even though it is not publicly released. However, the end-to-end claim is currently supported only partially: the retrieval benchmark may use ground-truth graphs rather than the VLM-generated graphs that would appear in a deployed pipeline, the retrieval correctness criteria differ across chunking methods, and the LoRA hyperparameters are selected on the test set. These issues are fixable with additional clarification and experiments, so the central idea remains defensible.

major comments (4)
  1. [§3.2.2] The prompt quoted in Section 3.2.2 is ambiguous in a way that affects the validity of the central GED result: it states 'here is its ground truth JSON representation, image_json ={}' and then says 'Now generate JSON for the next image'. If the placeholder is ever populated with the ground-truth JSON of the image being converted, the model would be given the answer before generating anything, and the improvement from 10.21 to 2.74 in Table 3 would measure copying rather than parsing. If instead the placeholder is always empty and the text is simply a poorly worded one-shot instruction, the concern is resolved. Please state explicitly whether any ground-truth JSON is ever present in the prompt at training, validation, or test time, and provide the exact prompt template used for inference on the Flowlearn test set and on the PI Docs flowcharts.
  2. [§3.1.3, §4.3] It is unclear whether the retrieval experiments in Section 4.3 embed the ground-truth graph JSONs associated with the 105 PI Docs flowcharts or the graph JSONs produced by the fine-tuned VLM. The dataset description mentions ground-truth graph structures, while the pipeline description in Section 2.3 refers to graphs 'obtained from the previous step,' i.e., from the VLM. If only ground-truth graphs are used, the reported top-1 accuracy of 57.17% does not reflect the actual end-to-end system, because errors in the VLM output (average GED 3.14 on PI Docs) are not propagated. Please clarify which graph source is used, and report retrieval results on VLM-generated graphs to substantiate the claim that a text-only retriever can support flowchart QA without a VLM at inference.
  3. [§3.2.2, Table 3] The fine-tuning section states that 'the fine-tuning was performed for few choices of parameters R and alpha, and the best fine-tuned model was used in the pipeline,' but no validation split is reported and Table 3 reports test-set GED for every hyperparameter setting. Selecting the LoRA rank and alpha based on test-set performance introduces optimism bias in the reported best GED of 2.74. Please perform hyperparameter selection on the Flowlearn validation split (or a held-out subset) and report the corresponding test-set GED, or otherwise justify that the reported value is not a test-set artifact.
  4. [§2.4, Table 4] The retrieval correctness criteria are not comparable across the three chunking approaches. For 'each node as one chunk,' retrieval is considered correct if any node from the top-k retrieved graphs appears in the ground truth; for 'all nodes as one chunk,' all ground-truth nodes must appear; and for 'entire graph JSON as one chunk,' the retrieved graph must correspond to the ground truth. These are very different standards of correctness, so the top-k accuracies in Table 4 cannot be directly compared to conclude that one chunking strategy is better. Please use a single, well-defined criterion, such as whether the ground-truth flowchart's chunk appears in the top-k, or report node-level precision/recall and graph-level accuracy separately.
minor comments (6)
  1. [§2.2] The graph representation deliberately discards node shapes, but the QA benchmark includes a large category of 'decision-related' questions. Since decision semantics in flowcharts are sometimes conveyed by the shape of the decision node (e.g., a diamond), the paper should discuss whether this information loss could affect retrieval or downstream answering, even if the QA pairs were created from the graph representations.
  2. [§3.1.3, Table 1] The QA category counts in Table 1 sum to 1325, far exceeding the 502 total QA pairs, which means questions can belong to multiple categories. Please clarify how a question can be both 'node related' and 'edge related,' and state whether the per-category retrieval results in Table 5 are computed on overlapping subsets.
  3. [Table 4] In the lower half of Table 4 (interspersed with text), the values for bge-large are given without percent signs (e.g., 41.05, 55.53, 59.76), while the upper half uses percentages. Please format consistently.
  4. [§4.2, Table 3] The GED values in Table 3 are reported as averages without standard deviations or confidence intervals, and the number of test flowcharts is not stated for either the Flowlearn test set or the PI Docs set. Adding these would help assess the stability of the reported improvements.
  5. [§3.1.2] The synthetic augmentation of the Flowlearn training set is described only qualitatively. Please provide details on the number of generated images, the distribution of added node shapes and edge types, and how the augmented data were split to avoid overlap with the test set.
  6. [References] Reference [7] is cited for the 'microsoft/dit-base' model, but the cited paper (Lewis et al., 2006, 'Building a test collection for complex document information processing') is not the Document Image Transformer. The correct DiT citation appears to be missing; please fix this reference error.

Circularity Check

0 steps flagged · score 0.0 of 10

No demonstrated circularity: GED improvement is measured on the external Flowlearn test set, and retrieval uses manually curated ground-truth graphs; the Sec. 3.2.2 prompt does not establish a leakage-by-construction.

full rationale

The paper's core derivation chain is: classify images, fine-tune Qwen2-VL on synthetic Flowlearn data to convert flowchart images to graph JSONs, embed those JSONs with text retrievers, and evaluate retrieval on 502 QA pairs from 105 PI-Doc flowcharts. The decisive intermediate result—the GED drop from 10.21 to 2.74—is computed on the external Flowlearn test set, which is disjoint from the fine-tuning split and is therefore an independent benchmark rather than a fitted parameter renamed as a prediction. The retrieval experiments use manually associated ground-truth graphs and SME-curated questions; the QA pairs and ground-truth graphs are inputs to the evaluation, not outputs of the system, so there is no self-definitional circularity. The paper's self-citations ([11,12,15,16]) are background for RAG in the telecom domain and do not carry the derivation. The only potentially circular artifact is the prompt in Sec. 3.2.2, which mentions 'ground truth JSON representation, image_json ={}' and then asks to 'generate JSON for the next image'; if the placeholder were filled with the target image's ground truth, the GED numbers would be meaningless. However, the wording is consistent with a one-shot example from a separate image, and the paper does not state that the placeholder is the target's ground truth. Without evidence of that specific reduction, the claimed circularity is not established, so no circular step is recorded.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The pipeline rests on the assumption that a flowchart's semantics survive a graph JSON representation that ignores node shapes, that synthetic Mermaid training transfers to proprietary telecom drawings, and that retrieval of the correct graph is a valid proxy for QA success. The only hand-tuned numeric input is the LoRA rank and alpha pair, selected on the test set.

free parameters (1)
  • LoRA rank R and alpha = R=512, alpha=512
    Section 3.2.2 reports a small search over R and alpha; Table 3 shows the GED on the Flowlearn test set for each choice, indicating the best hyperparameters were picked using test set results.
assumptions (4)
  • domain assumption A directed graph with node text and edge text, ignoring node shapes, preserves the information needed for flowchart QA.
    Section 2.2 states node shapes are not captured; the entire retrieval and QA evaluation is done on graph JSONs rather than images.
  • domain assumption Fine-tuning on synthetic Mermaid flowcharts transfers to proprietary telecom flowcharts.
    Section 3.1.2 and 3.2.2 train on Flowlearn plus augmented synthetic images and evaluate on PI Docs without domain-specific fine-tuning.
  • domain assumption The manual ground-truth graph JSONs associated with the 105 PI Docs flowcharts are correct and complete.
    Section 3.1.3 describes SME input for QA pairs but no inter-annotator agreement or validation protocol for the graphs.
  • domain assumption Retrieval of the correct graph JSON is a valid measure of QA performance.
    Section 2.4 evaluates only retrieval; answer generation with the JSON as context is deferred to future work in Section 5.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Graph-based Approach for Multi-Modal Question Answering from Flowcharts in Telecom Documents." pith.science (2026). https://pith.science/paper/S3LDVVL2

@misc{pith2026250722938,
  author       = {Pith},
  title        = {Pith review of: A Graph-based Approach for Multi-Modal Question Answering from Flowcharts in Telecom Documents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S3LDVVL2}},
  note         = {Machine review of arXiv:2507.22938}
}
read the original abstract

Question-Answering (QA) from technical documents often involves questions whose answers are present in figures, such as flowcharts or flow diagrams. Text-based Retrieval Augmented Generation (RAG) systems may fail to answer such questions. We leverage graph representations of flowcharts obtained from Visual large Language Models (VLMs) and incorporate them in a text-based RAG system to show that this approach can enable image retrieval for QA in the telecom domain. We present the end-to-end approach from processing technical documents, classifying image types, building graph representations, and incorporating them with the text embedding pipeline for efficient retrieval. We benchmark the same on a QA dataset created based on proprietary telecom product information documents. Results show that the graph representations obtained using a fine-tuned VLM model have lower edit distance with respect to the ground truth, which illustrate the robustness of these representations for flowchart images. Further, the approach for QA using these representations gives good retrieval performance using text-based embedding models, including a telecom-domain adapted one. Our approach also alleviates the need for a VLM in inference, which is an important cost benefit for deployed QA systems.

Figures

Figures reproduced from arXiv: 2507.22938 by the authors.

Figure 1
Figure 1. A sample description of synthetic flowchart from Flowlearn using GPT4 Vision model. The incorrect part of description [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The end-to-end approach for multi-modal QA, our approach for images is indicated in black-shaded boxes. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. A sample flowchart and its graph representation in JSON format. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Chunking approaches for flowchart JSONs. [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 4
Figure 4. Figure 4: Fine-tuning the VLM with a publicly available [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 6
Figure 6. Figure 6: Performance of the classifier on the test set for various image categories from PI docs. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Confusion matrix on the test set for categorization [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Representative images for various categories from the proprietary telecom dataset. (Note: content has been obfuscated [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: A sample of ambiguous flowchart image in 3GPP document. One can consider this to also be a block diagram. [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 11 canonical work pages

  1. [1]

    3GPP. 2022. 3GPP Release 18. Technical Report. Accessed: 2024-05-19

    work page 2022

  2. [2]

    Zeina Abu-Aisheh, Romain Raveaux, Jean-Yves Ramel, and Patrick Martineau

  3. [3]

    Haoyu Han, Yu Wang, Harry Shomer, Kai Guo, Jiayuan Ding, Yongjia Lei, Ma- hantesh Halappanavar, Ryan A Rossi, Subhabrata Mukherjee, Xianfeng Tang, et al

  4. [4]

    Henrik Holm. 2021. Bidirectional encoder representations from transformers (bert) for question answering in the telecom domain.: Adapting a bert-like lan- guage model to the telecom domain using the electra pre-training approach

    work page 2021

  5. [5]

    Athanasios Karapantelakis, Mukesh Thakur, Alexandros Nikou, Farnaz Moradi, Christian Olrog, Fitsum Gaim, Henrik Holm, Doumitrou Daniil Nimara, and Vin- cent Huang. 2024. Using large language models to understand telecom standards. In 2024 IEEE International Conference on Machine Learning for Communication and Networking (ICMLCN). IEEE, 440–446

    work page 2024

  6. [6]

    Aryan Keskar, Srinivasa Perisetla, and Ross Greer. 2025. Evaluating multimodal vision-language model prompting strategies for visual question answering in road scene understanding. In Proceedings of the Winter Conference on Applications of Computer Vision. 1027–1036

    work page 2025

  7. [7]

    Lewis, Gady Agam, Shlomo Engelson Argamon, Ophir Frieder, David A

    David D. Lewis, Gady Agam, Shlomo Engelson Argamon, Ophir Frieder, David A. Grossman, and Jefferson Heard. 2006. Building a test collection for complex document information processing. Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval(2006)

    work page 2006

  8. [8]

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. 2024. Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13299–13308

    2024

Show all 20 references
  1. [9]

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. LLaVA-NeXT: Improved reasoning, OCR, and world knowl- edge. https://llava-vl.github.io/blog/2024-01-30-llava-next/

    2024

  2. [10]

    Huitong Pan, Qi Zhang, Cornelia Caragea, Eduard Dragut, and Longin Jan Late- cki. 2024. Flowlearn: Evaluating large vision-language models on flowchart understanding. In ECAI 2024. IOS Press, 73–80

    2024

  3. [11]

    Sujoy Roychowdhury, Nishkarsh Jain, and Sumit Soman. 2024. Unlocking tele- com domain knowledge using llms. In 2024 16th International Conference on COMmunication Systems & NETworkS (COMSNETS) . IEEE, 267–269

    2024

  4. [12]

    Sujoy Roychowdhury, Sumit Soman, HG Ranjani, Neeraj Gunda, Vansh Chhabra, and Sai Krishna Bala. [n. d.]. Evaluation of RAG Metrics for Question Answering in the Telecom Domain. In ICML 2024 Workshop on Foundation Models in the Wild

    2024

  5. [13]

    Saraiva, M

    T. Saraiva, M. Sousa, P. Vieira, and A. Rodrigues. 2025. Telco-DPR: A Hybrid Dataset for Evaluating Retrieval Models of 3GPP Technical Specifications. In 2025 IEEE Wireless Communications and Networking Conference (WCNC) . 01–06. doi:10.1109/WCNC61545.2025.10978393

    2025

  6. [14]

    Shubhankar Singh, Purvi Chaurasia, Yerram Varun, Pranshu Pandya, Vatsal Gupta, Vivek Gupta, and Dan Roth. 2024. Flowvqa: Mapping multimodal logic in visual question answering with flowcharts. arXiv preprint arXiv:2406.19237 (2024)

    2024 arXiv

  7. [15]

    Sumit Soman and Ranjani HG. 2023. Observations on LLMs for telecom domain: capabilities and limitations. In Proceedings of the Third International Conference on AI-ML Systems. 1–5

    2023

  8. [16]

    Sumit Soman and Sujoy Roychowdhury. [n. d.]. Observations on Building RAG Systems for Technical Documents. In The Second Tiny Papers Track at ICLR 2024

    2024

  9. [17]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191 (2024)

    2024 arXiv

  10. [18]

    Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. C-Pack: Packaged Resources To Advance General Chinese Embedding. arXiv:2309.07597 [cs.CL] A Image Categories in Telecom Dataset Representative examples of various types of images present in the telecom (PI) d...

    2023 arXiv

  11. [2015]

    In 4th International Conference on Pattern Recognition Applications and KDD ’25 Workshop on Structured Knowledge for LLMs, August 04, 2025, Toronto, CA Soman et al

    An exact graph edit distance algorithm for solving pattern recognition problems. In 4th International Conference on Pattern Recognition Applications and KDD ’25 Workshop on Structured Knowledge for LLMs, August 04, 2025, Toronto, CA Soman et al. Chunking Approach Top-1 Top-3 T...

    2025

  12. [2024]

    arXiv preprint arXiv:2501.00309 (2024)

    Retrieval-augmented generation with graphs (graphrag). arXiv preprint arXiv:2501.00309 (2024)

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.