Pith. sign in

REVIEW 3 major objections 5 minor 26 references

MuDoC: An Interactive Multimodal Document-grounded Conversational AI System

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper's central claim: a conversational agent can answer questions about a long PDF with interleaved text and the document's own figures, each clickable back to its source page.

desk verdict MuDoC is a clearly described integration of standard retrieval pieces plus a clickable PDF navigation UI; the evidence is qualitative and the 'document-grounded' wording overreaches, but the system is a credible, honest prototype worth revising. read the letter →

arxiv 2502.09843 v1 pith:BXBKU3TY submitted 2025-02-14 cs.AI cs.HCcs.MM

Karan Taneja , Ashok K. Goel This is my paper · ORCID
classification cs.AIcs.HCcs.MM
keywords multimodalconversationalAIdocument-groundeddialogueretrieval-augmentedgenerationinterleavedtextandimagesGPT-4otoolcallsDPRretrievalCLIPembeddingsPDFlayoutanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MuDoC is a conversational system that answers questions about a long PDF by composing responses that mix written explanations with figures taken directly from the document. The paper claims this multimodal, document-grounded behavior can be achieved without fine-tuning: PDF pages are preprocessed into text chunks and image snippets with embeddings, retrieval finds content relevant to the query and chat history, and a GPT-4o-based generator assembles an interleaved answer whose image tags are replaced by the actual retrieved figures. The accompanying interface makes the answer verifiable: clicking any figure or paragraph scrolls the PDF to the source page and highlights it, so the user can check the AI's claim against the document. Based on qualitative testing with a 357-page textbook, the paper reports that this promotes trust and works well overall, while acknowledging imperfect figure placement and occasional hallucinated image descriptions.

What carries the argument

The load-bearing mechanism is the tool-call retrieval loop: GPT-4o receives the user query and system instructions, emits structured text- and image-retrieval queries, the embedding index returns top-ranked text chunks and figures, and the model composes a blog-style answer containing HTML image tags that the renderer replaces with the actual retrieved images. Around this loop, a preprocessing pipeline supplies the index—Mask R-CNN layout detection on PDF pages, OCR with GPT-3.5 cleaning and summarization, DPR embeddings for text and captions, and CLIP embeddings for images—and a post-processing step maps answer paragraphs back to source text snippets by cosine similarity, enabling click-to-source navigation.

What would settle it

Take the 357-page test textbook, run a fixed set of questions through MuDoC, and check every generated figure reference against the retrieval output: any image tag whose filename does not match a retrieved image, or any textual description of a figure that contradicts the figure's source caption, falsifies the trustworthiness claim. A sharper experiment holds the question fixed and varies the number of images included in one GPT-4o query, counting hallucinated image descriptions; the paper's own observation predicts more hallucinations in the condition with several images.

Watch

Extended reading notes

Core claim

The paper's central claim is that endowing a conversational agent with document-grounded image retrieval and clickable verification changes what a user can do with a long document. MuDoC treats figures as first-class evidence: layout analysis finds them, CLIP and DPR embeddings index them together with cleaned text, and GPT-4o's tool calls let the model ask for whichever texts and images it needs before writing an answer. The authors argue that reusing the document's own figures, rather than generating new images, preserves the technical content that diagrams carry, and that making every figure and paragraph navigable back to the source page addresses the trust problem that plagues text-only document chatbots.

Load-bearing premise

The system's trustworthiness claim rests on GPT-4o reliably following the prompt—issuing the right retrieval calls, writing image tags that exactly match the retrieved filenames, and describing images without hallucinating—and the paper itself reports one observed hallucinated image description, noting such errors happen more often when multiple images are passed in a single query.

Editorial extensions

If this is right

  • A user can ask a technical question about a textbook topic and receive an answer that includes the textbook's own relevant diagrams, not just prose or an AI-generated illustration.
  • Because every figure and most paragraphs link back to a specific page, a skeptical reader can verify each piece of the answer against the original document, providing a concrete check against fabrication.
  • Since the system uses no model fine-tuning, a new long document can be made conversational simply by running the preprocessing and indexing pipeline on it.
  • The reported tendency for hallucinations to increase when several images are passed in one query implies that capping the number of images sent to GPT-4o is a natural way to improve reliability.
  • If the qualitative results hold, multimodal document dialogue is a realistic alternative to text-only chatbots for learning and reference use with long documents.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The navigation feature verifies provenance, not correctness: seeing the source figure or paragraph tells the user the content exists in the document, but the user still has to judge whether the generated text faithfully reflects it.
  • A natural extension the paper does not test is to filter retrieved images before generation, such as keeping only the one or two figures with highest retrieval scores, which would directly target the observed multi-image hallucination pattern.
  • Because the pipeline's behavior depends on a proprietary model's tool-calling and image-tagging discipline, the qualitative results may not transfer to another model version or to an open-weight model without prompt redesign.
  • A quantitative evaluation comparing MuDoC to a text-only document chatbot on answer faithfulness and user verification speed would settle whether the extra retrieval and navigation complexity pays off; the paper leaves that comparison open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper describes MuDoC, an interactive multimodal document-grounded conversational AI system built on GPT-4o. The system preprocesses long PDF documents by extracting text and image snippets via layout analysis, OCR, GPT-based cleaning/summarization/captioning, and DPR/CLIP embeddings. At query time, GPT-4o tool calls are used to issue retrieval queries, the retrieved text and images are returned to the model, and the model generates an interleaved text-and-image response; rendered image tags are replaced with actual figures. A React-based user interface supports chat alongside a PDF viewer, including clickable navigation from response paragraphs and images back to the source document, as well as Summarize and ELI10 features. The paper reports preliminary qualitative observations from a small set of users on a 357-page AI textbook, noting strengths such as relevant figure retrieval and appreciated navigation, and limitations such as suboptimal figure placement, occasional hallucinated image descriptions, and imperfect paragraph mapping.

Significance. If substantiated, MuDoC would be a useful contribution to multimodal document-grounded dialogue, particularly in its design choice to retrieve and reuse figures from the source document rather than generating potentially unreliable images. The architecture is described in sufficient detail to be reproduced, it uses publicly available models without fine-tuning, and the user-interface idea of clickable navigation from generated responses to source documents is practically valuable. The paper is also honest in reporting limitations. However, the central advertised properties—'document-grounded responses' and 'promotes trustworthiness'—are currently supported only by qualitative examples and anecdotal feedback, and the paper's own observations include a direct counterexample to robust grounding (a hallucinated image description). No quantitative metrics, error rates, or controlled user-study results are reported. The significance therefore depends on future evaluation that this version does not provide.

major comments (3)
  1. [Abstract; GPT-4o Tool Calls and Prompting; Preliminary Observations] The abstract's claims that MuDoC generates 'document-grounded responses' and 'promotes trustworthiness' are not established by the evidence in the paper. Grounding is implemented only as an instruction in the GPT-4o system message ('answer only using texts and images retrieved using search tools'); there is no post-generation verification that every statement is supported by the retrieved context, that an HTML image tag names an actually retrieved file, or that a text description matches the referenced image. The paper's own Preliminary Observations report that 'a text describing an image was hallucinated' and that this occurs more often when multiple images are provided in one query. This is a direct counterexample to a strong reading of 'document-grounded.' Since no error rates are reported, the frequency and severity of such failures are unknown. The authors should either add a verification mechanism that enforces the grounding property, or substantially soften the trustworthiness/grounding claims and report measured failure rates.
  2. [Preliminary Observations] The only evaluation is qualitative and underspecified. The paper says 'We asked some subjects to use MuDoC' but does not state the number of subjects, their background, the task protocol, the number of queries, or any analysis method. Claims such as 'MuDoC responses contain relevant images' and 'navigation ... was highly appreciated' are based on unquantified observations. Without retrieval accuracy metrics, response-groundedness metrics (e.g., human ratings or automatic faithfulness scores), or navigation-success rates, the paper cannot support its conclusion that MuDoC is 'promising' for trustworthy use. A small quantitative evaluation on a held-out set of queries, even using existing document-grounded dialogue measures, would materially strengthen the central claim.
  3. [User Interface / Navigation Using Images and Texts] The verification claim is weaker than the abstract suggests because not all response content is navigable. The paper states that paragraphs shorter than 100 characters or with cosine similarity below a threshold are not mapped and 'cannot be clicked for navigation.' This means that a user cannot verify a substantial subset of the generated text through the interface, precisely the cases where hallucination risk may be highest (short, paraphrased, or low-similarity statements). The paper should report how often paragraphs fall into the unclickable category, or provide an alternative verification signal (e.g., a 'not found' indicator) so that non-verifiable content is clearly marked rather than appearing to be grounded.
minor comments (5)
  1. [Preliminary Observations] There is a typo in 'paragraphs is some responses'; it should read 'paragraphs in some responses.'
  2. [Document Preprocessing] The phrase 'at least 500 characters overlap' is ambiguous: specify whether overlaps are variable and how the system determines actual overlap at chunk boundaries, since this affects the retrieval index and reproducibility.
  3. [Response Generation] The paper should clarify how GPT-4o is prevented from emitting image tags that do not correspond to filenames in the retrieval output. Currently the system message is the only guard; a simple post-processing check of image tags against retrieved filenames would be a meaningful safeguard, and its presence or absence should be stated explicitly.
  4. [Preliminary Observations] The external link 'tinyurl.com/MuDoCSamples' is brittle and the paper does not say how long the examples will remain available. If the qualitative examples are the primary evidence, they should be included in supplementary material.
  5. [Related Work] The related work section would benefit from a clearer statement of how MuDoC differs from visually-grounded document QA systems such as KOSMOS-2.5; the current discussion focuses on generative interleaved-image models rather than retrieval-based document-grounded systems.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MuDoC is an engineering pipeline with no derivation or fitted parameters whose predictions reduce to inputs.

full rationale

MuDoC is a systems paper; there is no formal derivation chain, no fitted parameters, and no uniqueness theorem invoked to force a design choice. The preprocessing pipeline (layout analysis, OCR, GPT-3.5 text cleaning and summarization, GPT-4 captions and descriptions, DPR and CLIP embeddings) and the retrieval/response pipeline (embedding-based top-5 retrieval followed by GPT-4o tool calls with a system message instructing grounded answers) are described concretely, and the claimed outputs—interleaved text and figures with clickable navigation—are not defined in terms of the evaluation. The authors test on an ebook co-authored by one of the authors, but they report only qualitative observations and make no quantitative claim that would be statistically forced by that choice. Self-citations to prior work by the same group (Jill Watson, Textbook of Tomorrow) appear in related work and are not load-bearing premises. The paper's own Preliminary Observations concede that a text describing an image was hallucinated and that paragraph-to-document navigation can fail for short or low-similarity text; this is a correctness and validation concern, not circularity. Because no step reduces by construction to its own inputs, the appropriate finding is no significant circularity with score 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no fitted model parameters in the usual sense, but several engineering constants are hand-chosen. The central claim rests on standard domain assumptions about layout detection, retrieval quality, and proprietary model behavior rather than on derivations. MuDoC is an engineered system, not an invented theoretical entity.

free parameters (5)
  • Text chunk size = 2000 characters
    Fixed retrieval granularity chosen by hand; no ablation or sensitivity analysis reported.
  • Text chunk overlap = 500 characters
    Chosen to preserve continuity across chunk boundaries; no sensitivity analysis reported.
  • Retrieved-text top-k = 5
    Number of text chunks passed to GPT-4o; hand-chosen.
  • Retrieved-image top-k = 5
    Number of candidate images passed to GPT-4o; hand-chosen.
  • Conversation context window = 64K characters
    Truncation limit for API context; set to control cost.
assumptions (4)
  • domain assumption PubLayNet-trained Mask R-CNN accurately detects figures, text, tables, and lists in the target PDFs.
    Layout extraction in Document Preprocessing assumes the detector generalizes to the ebook pages.
  • domain assumption DPR and CLIP embeddings retrieve content that is relevant to the query and useful to GPT-4o.
    Response quality depends on retrieval being good enough for the LLM to select from.
  • domain assumption GPT-4o will follow the system prompt and emit well-formed image tags with the exact filenames returned by retrieval.
    The rendering pipeline substitutes image tags after generation; malformed or hallucinated tags break interleaving.
  • domain assumption OCR plus GPT-3.5 cleaning and GPT-4 captioning preserve the meaning of source figures and text.
    Erroneous captions or descriptions would propagate into retrieval and response grounding.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MuDoC: An Interactive Multimodal Document-grounded Conversational AI System." pith.science (2026). https://pith.science/paper/BXBKU3TY

@misc{pith2026250209843,
  author       = {Pith},
  title        = {Pith review of: MuDoC: An Interactive Multimodal Document-grounded Conversational AI System},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BXBKU3TY}},
  note         = {Machine review of arXiv:2502.09843}
}
read the original abstract

Multimodal AI is an important step towards building effective tools to leverage multiple modalities in human-AI communication. Building a multimodal document-grounded AI system to interact with long documents remains a challenge. Our work aims to fill the research gap of directly leveraging grounded visuals from documents alongside textual content in documents for response generation. We present an interactive conversational AI agent 'MuDoC' based on GPT-4o to generate document-grounded responses with interleaved text and figures. MuDoC's intelligent textbook interface promotes trustworthiness and enables verification of system responses by allowing instant navigation to source text and figures in the documents. We also discuss qualitative observations based on MuDoC responses highlighting its strengths and limitations.

Figures

Figures reproduced from arXiv: 2502.09843 by the authors.

Figure 1
Figure 1. Document Preprocessing: PDF document layouts are detected to extract text and image snippets which are processed using OCR, GPT-3/4 and embedding models to create text and image embeddings for retrieval during response generation. alog with text-only inputs and outputs. Previous work such as (Lv et al. 2024) has also explored visually-grounded chat. MuDoC Architecture and User Interface Given a query and a chat cont… view at source ↗
Figure 2
Figure 2. Response Generation: GPT-4o tool calls feature is used to create text and image retrieval queries. Outputs from embedding-based retrieval are used for response generation and image references are replaced with actual images. sine similarity across raw, cleaned, and summary text is used as the text chunk score, and the top-5 chunks are used as retrieval output. For image retrieval, both DPR and CLIP embeddings are us… view at source ↗
Figure 3
Figure 3. UI Features: Chat-PDF display in (a) shows chat area on left, text box on bottom-left, and PDF on right. Yellow boxes and arrows describe features including summarize, Explain-it-Like-I’m-10 (ELI10), PDF navigation using text and images. figures illustrating AI concepts and algorithms. MuDoC re￾sponses contain long texts and multiple figures which leads to long conversations that cannot be included here. We pro￾vide… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 19 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Betker, J.; Goh, G.; Jing, L.; Brooks, T.; Wang, J.; Li, L.; Ouyang, L.; Zhuang, J.; Lee, J.; Guo, Y.; Manassra, W.; Dhariwal, P.; Chu, C.; Jiao, Y.; and Ramesh, A. 2023. Improving Image Generation with Better Captions

    work page 2023

  4. [4]

    Braunschweiler, N.; Doddipatla, R.; Keizer, S.; and Stoyanchev, S. 2023. Evaluating Large Language Models for Document -grounded Response Generation in Information - Seeking Dialogues . In Association for Computational Linguistics Workshop on Taming Large Language Models : Controllability in the era of Interactive Assistants , 46--55

    work page 2023

  5. [5]

    Chern, E.; Su, J.; Ma, Y.; and Liu, P. 2024. ANOLE : An Open , Autoregressive , Native Large Multimodal Models for Interleaved Image - Text Generation . ArXiv:2407.06135 [cs]

    arXiv 2024

  6. [6]

    Dong, R.; Han, C.; Peng, Y.; Qi, Z.; Ge, Z.; Yang, J.; Zhao, L.; Sun, J.; Zhou, H.; Wei, H.; Kong, X.; Zhang, X.; Ma, K.; and Yi, L. 2024. DreamLLM : Synergistic Multimodal Comprehension and Creation . In International Conference on Learning Representations 2024 Spotlight

    work page 2024

  7. [7]

    Feng, J.; Sun, Q.; Xu, C.; and et al. 2023. MMDialog : A Large -scale Multi -turn Dialogue Dataset Towards Multi -modal Open -domain Conversation . In Association for Computational Linguistics 2023 , 7348--7363

    work page 2023

  8. [8]

    Feng, S.; Wan, H.; Gunasekara, C.; Patel, S.; Joshi, S.; and Lastras, L. 2020. doc2dial: A Goal - Oriented Document - Grounded Dialogue Dataset . In Empirical Methods in Natural Language Processing 2020 , 8118--8128. Online

    work page 2020

Show all 26 references
  1. [9]

    Goel, A.; and Joyner, D. 2018. Knowledge-based Artificial Intelligence : Cognitive Systems (e-book) . Georgia Institute of Technology

    2018

  2. [10]

    He, K.; Gkioxari, G.; Dollár, P.; and Girshick, R. 2017. Mask R - CNN . In International Conference on Computer Vision 2017 , 2980--2988

    2017

  3. [11]

    Kakar, S.; Maiti, P.; Taneja, K.; Nandula, A.; Nguyen, G.; Zhao, A.; Nandan, V.; and Goel, A. 2024. Jill Watson : Scaling and Deploying an AI Conversational Agent in Online Classrooms . In ITS 2024 , 78--90

    2024

  4. [12]

    Karpukhin, V.; Oguz, B.; Min, S.; Lewis, P.; Wu, L.; Edunov, S.; Chen, D.; and Yih, W.-t. 2020. Dense Passage Retrieval for Open - Domain Question Answering . In Empirical Methods in Natural Language Processing 2020 , 6769--6781

    2020

  5. [13]

    Kong, F.; Wang, P.; Feng, S.; Wang, D.; and Zhang, Y. 2024. TIGER : A Unified Generative Model Framework for Multimodal Dialogue Response Generation . In Joint International Conference on Computational Linguistics, Language Resources and Evaluation 2024 , 16135--16141

    2024

  6. [14]

    Lee, J.; Ko, J.; Baek, J.; Jeong, S.; and Hwang, S. J. 2024. Unified Multimodal Interleaved Document Representation for Retrieval . ArXiv:2410.02729 [cs]

    2024

  7. [15]

    Lee, N.; Shin, S.; Choo, J.; Choi, H.-J.; and Myaeng, S.-H. 2021. Constructing Multi - Modal Dialogue Dataset by Replacing Text with Semantically Relevant Images . In ACL International Joint Conference on Natural Language Processing 2021 , 897--906

    2021

  8. [16]

    Lv, T.; Huang, Y.; Chen, J.; Zhao, Y.; Jia, Y.; Cui, L.; Ma, S.; Chang, Y.; Huang, S.; Wang, W.; Dong, L.; Luo, W.; Wu, S.; Wang, G.; Zhang, C.; and Wei, F. 2024. KOSMOS -2.5: A Multimodal Literate Model . ArXiv:2309.11419 [cs]

    2024 arXiv

  9. [17]

    Olson, A.; Maiti, P.; and Goel, A. 2025. The Textbook of Tomorrow : Rethinking Course Material Interfacing in the Era of GPT . ArXiv:2501.03618 [cs]

    2025 arXiv

  10. [18]

    OpenAI . 2022. Introducing ChatGPT . http://openai.com/index/chatgpt. Accessed: 2025-02-13

    2022

  11. [19]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision . In International Conference on Machine Learning ...

    2021

  12. [20]

    M.; and Sankaranarayanan, K

    Saha, A.; Khapra, M. M.; and Sankaranarayanan, K. 2018. Towards building large scale multimodal domain-aware conversation systems. In Association for the Advancement of Artificial Intelligence Symposium on Educational Advances in Artificial Intelligence 2018 , 696--704

    2018

  13. [21]

    Saharia, C.; Chan, W.; Saxena, S.; Li, L.; Whang, J.; Denton, E.; Ghasemipour, S. K. S.; Ayan, B. K.; Mahdavi, S. S.; Lopes, R. G.; Salimans, T.; Ho, J.; Fleet, D. J.; and Norouzi, M. 2022. Photorealistic Text -to- Image Diffusion Models with Deep Language Understanding . In N...

    2022

  14. [22]

    Taneja, K.; Maiti, P.; Kakar, S.; Guruprasad, P.; Rao, S.; and Goel, A. K. 2024. Jill Watson : A Virtual Teaching Assistant Powered by ChatGPT . In International Conference on Artificial Intelligence in Education 2024 , 324--337

    2024

  15. [23]

    Thoppilan, R.; De Freitas, D.; Hall, J.; Shazeer, N.; Kulshreshtha, A.; Cheng, H.-T.; Jin, A.; and others . 2022. LaMDA : Language Models for Dialog Applications . ArXiv:2201.08239 [cs]

    2022 arXiv

  16. [24]

    Zang, X.; Liu, L.; Wang, M.; Song, Y.; Zhang, H.; and Chen, J. 2021. PhotoChat : A Human - Human Dialogue Dataset With Photo Sharing Behavior For Joint Image - Text Modeling . In ACL International Joint Conference on Natural Language Processing 2021 , 6142--6152

    2021

  17. [25]

    Zheng, K.; He, X.; and Wang, X. E. 2024. MiniGPT -5: Interleaved Vision -and- Language Generation via Generative Vokens . ArXiv:2310.02239 [cs]

    2024

  18. [26]

    Zhong, X.; Tang, J.; and Jimeno Yepes, A. 2019. PubLayNet : Largest Dataset Ever for Document Layout Analysis . International Conference on Document Analysis and Recognition 2019, 1015--1022

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.