{"id":"2bf41db5-963c-451b-ac30-392de326fcdb","arxiv_id":"2502.09843","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MuDoC combines document layout analysis, DPR and CLIP retrieval, and GPT-4o to answer PDF questions with interleaved source text and figures, plus clickable provenance.","lead":"The authors built MuDoC, a chatbot that answers questions about a long PDF by pulling in both text and figures from the document and blending them into its replies. It is worth a look as a practical pattern for making AI answers about textbooks more visual and verifiable.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MuDoC's 'document-grounded' guarantee is an instruction to GPT-4o, not an enforced property; the paper's own Preliminary Observations report hallucinated image descriptions, and no automated check catches this before the user sees the response.","rationale":"The reader's weakest_assumption is exactly the concern I identify: GPT-4o must reliably follow MuDoC's prompts, and the authors' own observations show it does not always do so. I agree with the CONDITIONAL verdict because the paper offers a clear architecture and encouraging qualitative demonstrations, but the central trustworthiness claim needs quantitative evaluation and reproducible artifacts. I considered alternative concerns, such as domain mismatch between PubLayNet training data and a textbook layout, or the lack of similarity thresholds in retrieval causing loosely connected figures; both are secondary limitations that do not strike at the core claim as directly as hallucinated content in the final answer. The navigation feature is a genuine strength and partially mitigates the problem by letting users check sources, but it shifts verification burden to the user and does not make responses themselves grounded. The paper honestly discloses the hallucination and the imperfect paragraph mapping, so this is not a hidden flaw; it is nonetheless load-bearing because it directly contradicts the abstract's trustworthiness framing. A concrete grounding-evaluation experiment would settle whether the observed hallucination is a rare edge case or a systematic failure mode, and would determine whether the CONDITIONAL verdict should be upgraded to ACCEPT or downgraded to REJECT.","tokens_in":6902,"tokens_out":4714,"duration_ms":41210,"concrete_test":"Run MuDoC on a fixed set of 50 questions drawn from the Knowledge-based AI ebook (Goel and Joyner 2018). For each response, record every image filename in the rendered HTML and check it is present in the retrieval output; ask two annotators to label each sentence as supported, partially supported, or unsupported by the retrieved text chunks and image captions/descriptions. Compute per-response precision of grounded sentences and exact-match accuracy of image references. Also compare conditions with 1, 3, and 5 retrieved images to test the paper's hypothesis that hallucinations increase with image count. If supported-sentence precision is below, say, 0.9 or if any hallucinated image description occurs in the sampled runs, the claimed document-grounded trustworthiness does not hold as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MuDoC generates document-grounded, trustworthy responses and enables verification. The architecture in 'GPT-4o Tool Calls and Prompting' relies on a system message instructing GPT-4o to answer only from retrieved texts and images and to use exact image filenames in HTML tags. This is a soft prompt-dependence: there is no post-generation verification that every statement is supported by the retrieved context, that an image reference names an actually retrieved file, or that a description matches the image. The paper's Preliminary Observations then report that 'a text describing an image was hallucinated' and that this happens more often when multiple images are passed in one query. Since hallucinated content can appear in the final user-facing response, the abstract's assertion that MuDoC is 'document-grounded' and 'promotes trustworthiness' is not established; at most the UI enables a user to verify some claims after the fact, and the navigation mapping itself can fail for short or low-similarity paragraphs. The observed hallucination is not an incidental bug: it is a direct counterexample to the system's grounding property under the exact architecture described. No metrics, error rates, or user-study results are reported, so the frequency and severity of such failures are unknown, leaving the load-bearing assumption of reliable GPT-4o behavior untested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes MuDoC, an interactive multimodal document-grounded conversational AI system built on GPT-4o. The system preprocesses long PDF documents by extracting text and image snippets via layout analysis, OCR, GPT-based cleaning/summarization/captioning, and DPR/CLIP embeddings. At query time, GPT-4o tool calls are used to issue retrieval queries, the retrieved text and images are returned to the model, and the model generates an interleaved text-and-image response; rendered image tags are replaced with actual figures. A React-based user interface supports chat alongside a PDF viewer, including clickable navigation from response paragraphs and images back to the source document, as well as Summarize and ELI10 features. The paper reports preliminary qualitative observations from a small set of users on a 357-page AI textbook, noting strengths such as relevant figure retrieval and appreciated navigation, and limitations such as suboptimal figure placement, occasional hallucinated image descriptions, and imperfect paragraph mapping.","tokens_in":7139,"tokens_out":3542,"duration_ms":37099,"significance":"If substantiated, MuDoC would be a useful contribution to multimodal document-grounded dialogue, particularly in its design choice to retrieve and reuse figures from the source document rather than generating potentially unreliable images. The architecture is described in sufficient detail to be reproduced, it uses publicly available models without fine-tuning, and the user-interface idea of clickable navigation from generated responses to source documents is practically valuable. The paper is also honest in reporting limitations. However, the central advertised properties—'document-grounded responses' and 'promotes trustworthiness'—are currently supported only by qualitative examples and anecdotal feedback, and the paper's own observations include a direct counterexample to robust grounding (a hallucinated image description). No quantitative metrics, error rates, or controlled user-study results are reported. The significance therefore depends on future evaluation that this version does not provide.","major_comments":[{"comment":"The abstract's claims that MuDoC generates 'document-grounded responses' and 'promotes trustworthiness' are not established by the evidence in the paper. Grounding is implemented only as an instruction in the GPT-4o system message ('answer only using texts and images retrieved using search tools'); there is no post-generation verification that every statement is supported by the retrieved context, that an HTML image tag names an actually retrieved file, or that a text description matches the referenced image. The paper's own Preliminary Observations report that 'a text describing an image was hallucinated' and that this occurs more often when multiple images are provided in one query. This is a direct counterexample to a strong reading of 'document-grounded.' Since no error rates are reported, the frequency and severity of such failures are unknown. The authors should either add a verification mechanism that enforces the grounding property, or substantially soften the trustworthiness/grounding claims and report measured failure rates.","section":"Abstract; GPT-4o Tool Calls and Prompting; Preliminary Observations"},{"comment":"The only evaluation is qualitative and underspecified. The paper says 'We asked some subjects to use MuDoC' but does not state the number of subjects, their background, the task protocol, the number of queries, or any analysis method. Claims such as 'MuDoC responses contain relevant images' and 'navigation ... was highly appreciated' are based on unquantified observations. Without retrieval accuracy metrics, response-groundedness metrics (e.g., human ratings or automatic faithfulness scores), or navigation-success rates, the paper cannot support its conclusion that MuDoC is 'promising' for trustworthy use. A small quantitative evaluation on a held-out set of queries, even using existing document-grounded dialogue measures, would materially strengthen the central claim.","section":"Preliminary Observations"},{"comment":"The verification claim is weaker than the abstract suggests because not all response content is navigable. The paper states that paragraphs shorter than 100 characters or with cosine similarity below a threshold are not mapped and 'cannot be clicked for navigation.' This means that a user cannot verify a substantial subset of the generated text through the interface, precisely the cases where hallucination risk may be highest (short, paraphrased, or low-similarity statements). The paper should report how often paragraphs fall into the unclickable category, or provide an alternative verification signal (e.g., a 'not found' indicator) so that non-verifiable content is clearly marked rather than appearing to be grounded.","section":"User Interface / Navigation Using Images and Texts"}],"minor_comments":[{"comment":"There is a typo in 'paragraphs is some responses'; it should read 'paragraphs in some responses.'","section":"Preliminary Observations"},{"comment":"The phrase 'at least 500 characters overlap' is ambiguous: specify whether overlaps are variable and how the system determines actual overlap at chunk boundaries, since this affects the retrieval index and reproducibility.","section":"Document Preprocessing"},{"comment":"The paper should clarify how GPT-4o is prevented from emitting image tags that do not correspond to filenames in the retrieval output. Currently the system message is the only guard; a simple post-processing check of image tags against retrieved filenames would be a meaningful safeguard, and its presence or absence should be stated explicitly.","section":"Response Generation"},{"comment":"The external link 'tinyurl.com/MuDoCSamples' is brittle and the paper does not say how long the examples will remain available. If the qualitative examples are the primary evidence, they should be included in supplementary material.","section":"Preliminary Observations"},{"comment":"The related work section would benefit from a clearer statement of how MuDoC differs from visually-grounded document QA systems such as KOSMOS-2.5; the current discussion focuses on generative interleaved-image models rather than retrieval-based document-grounded systems.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The paper reads as an early system demonstration rather than a finished empirical study. The architectural description is valuable and likely useful to practitioners, but the mismatch between the abstract's trustworthiness claims and the absence of quantitative evidence is the main obstacle. If the venue has a lower bar for system reports, a minor revision with softened claims might suffice; under standard evaluation expectations, the missing evaluation justifies major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: MuDoC is a sensible integration of existing pieces—layout analysis, DPR/CLIP retrieval, GPT-4o tool calls, and a clickable PDF viewer—and it is the integration plus the navigation UI that is the contribution. It is not a new algorithm and it has no quantitative evaluation. The paper's own observations admit a hallucinated image description, which undercuts the abstract's trustworthiness claim if read as a guarantee. Still, this is an honest system paper with a clear architecture and a useful UI idea, and it deserves referee attention with a request for evaluation and artifacts.\n\nThe genuinely new bit is that responses interleave retrieved figures with generated text and every paragraph and figure is clickable, scrolling the source PDF and highlighting it. That is a practical contribution for long textbooks. The preprocessing pipeline is described in enough detail to reproduce in principle: Mask R-CNN layout analysis, Tesseract OCR, GPT-3.5 cleaning and summarization, GPT-4 captions, DPR and CLIP embeddings, and top-5 retrieval. The use of GPT-4o tool calls is a reasonable design choice, and the referenced samples reportedly show relevant images.\n\nWhere it is soft: the evidence is qualitative and anecdotal. There are no retrieval accuracy numbers, no response quality metrics, and no user study details beyond 'some subjects' and one quoted preference. The stress-test point is not overblown: the grounding constraint lives in a system prompt, and there is no post-generation check that image filenames match retrieved files or that descriptions match images. The paper reports exactly that failure. That does not kill the system, but it means the abstract's wording is too strong. 'Enables verification' is fair if the navigation works; 'document-grounded' and 'trustworthy' overpromise. The paragraph navigation also has a known gap: short or low-similarity paragraphs cannot be clicked. Figure placement and loose image relevance are acknowledged but not measured, so we do not know how often they hurt.\n\nThe citation pattern is not a real problem. The authors cite their own textbook and prior Jill Watson work as context, not as evidence for MuDoC's claims. There is no circular derivation and no fitted parameters.\n\nVerdict: conditionally useful. This paper belongs in a peer-review track for an interactive or educational AI venue, with required additions: evaluation on a public document set, error rates for hallucination and navigation failures, and release of code and prompts. I would rather see it revised than desk-rejected.","headline":"MuDoC is a clearly described integration of standard retrieval pieces plus a clickable PDF navigation UI; the evidence is qualitative and the 'document-grounded' wording overreaches, but the system is a credible, honest prototype worth revising.","tokens_in":7701,"tokens_out":1938,"would_cite":true,"duration_ms":18652,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim: a conversational agent can answer questions about a long PDF with interleaved text and the document's own figures, each clickable back to its source page.","keywords":["multimodal conversational AI","document-grounded dialogue","retrieval-augmented generation","interleaved text and images","GPT-4o tool calls","DPR retrieval","CLIP embeddings","PDF layout analysis"],"falsifier":"Take the 357-page test textbook, run a fixed set of questions through MuDoC, and check every generated figure reference against the retrieval output: any image tag whose filename does not match a retrieved image, or any textual description of a figure that contradicts the figure's source caption, falsifies the trustworthiness claim. A sharper experiment holds the question fixed and varies the number of images included in one GPT-4o query, counting hallucinated image descriptions; the paper's own observation predicts more hallucinations in the condition with several images.","tokens_in":6677,"feed_emoji":"🖼️","tokens_out":7812,"duration_ms":67144,"temperature":0.7,"pith_summary":"MuDoC is a conversational system that answers questions about a long PDF by composing responses that mix written explanations with figures taken directly from the document. The paper claims this multimodal, document-grounded behavior can be achieved without fine-tuning: PDF pages are preprocessed into text chunks and image snippets with embeddings, retrieval finds content relevant to the query and chat history, and a GPT-4o-based generator assembles an interleaved answer whose image tags are replaced by the actual retrieved figures. The accompanying interface makes the answer verifiable: clicking any figure or paragraph scrolls the PDF to the source page and highlights it, so the user can check the AI's claim against the document. Based on qualitative testing with a 357-page textbook, the paper reports that this promotes trust and works well overall, while acknowledging imperfect figure placement and occasional hallucinated image descriptions.","feed_headline":"Ask a PDF a question; it answers with its own figures","feed_subtitle":"MuDoC retrieves text and figures from a long PDF, composes interleaved answers, and links every part back to its source page.","key_machinery":"The load-bearing mechanism is the tool-call retrieval loop: GPT-4o receives the user query and system instructions, emits structured text- and image-retrieval queries, the embedding index returns top-ranked text chunks and figures, and the model composes a blog-style answer containing HTML image tags that the renderer replaces with the actual retrieved images. Around this loop, a preprocessing pipeline supplies the index—Mask R-CNN layout detection on PDF pages, OCR with GPT-3.5 cleaning and summarization, DPR embeddings for text and captions, and CLIP embeddings for images—and a post-processing step maps answer paragraphs back to source text snippets by cosine similarity, enabling click-to-source navigation.","core_discovery":"The paper's central claim is that endowing a conversational agent with document-grounded image retrieval and clickable verification changes what a user can do with a long document. MuDoC treats figures as first-class evidence: layout analysis finds them, CLIP and DPR embeddings index them together with cleaned text, and GPT-4o's tool calls let the model ask for whichever texts and images it needs before writing an answer. The authors argue that reusing the document's own figures, rather than generating new images, preserves the technical content that diagrams carry, and that making every figure and paragraph navigable back to the source page addresses the trust problem that plagues text-only document chatbots.","pith_inferences":["The navigation feature verifies provenance, not correctness: seeing the source figure or paragraph tells the user the content exists in the document, but the user still has to judge whether the generated text faithfully reflects it.","A natural extension the paper does not test is to filter retrieved images before generation, such as keeping only the one or two figures with highest retrieval scores, which would directly target the observed multi-image hallucination pattern.","Because the pipeline's behavior depends on a proprietary model's tool-calling and image-tagging discipline, the qualitative results may not transfer to another model version or to an open-weight model without prompt redesign.","A quantitative evaluation comparing MuDoC to a text-only document chatbot on answer faithfulness and user verification speed would settle whether the extra retrieval and navigation complexity pays off; the paper leaves that comparison open."],"forward_implications":["A user can ask a technical question about a textbook topic and receive an answer that includes the textbook's own relevant diagrams, not just prose or an AI-generated illustration.","Because every figure and most paragraphs link back to a specific page, a skeptical reader can verify each piece of the answer against the original document, providing a concrete check against fabrication.","Since the system uses no model fine-tuning, a new long document can be made conversational simply by running the preprocessing and indexing pipeline on it.","The reported tendency for hallucinations to increase when several images are passed in one query implies that capping the number of images sent to GPT-4o is a natural way to improve reliability.","If the qualitative results hold, multimodal document dialogue is a realistic alternative to text-only chatbots for learning and reference use with long documents."],"supporting_citations":[{"why":"Supplies Dense Passage Retrieval encoders used to embed text chunks, summaries, captions, and queries for text retrieval.","marker":"Karpukhin et al. 2020"},{"why":"Supplies CLIP encoders used to embed images, captions, and descriptions and to score image relevance.","marker":"Radford et al. 2021"},{"why":"Provides the Mask R-CNN detection model used for layout analysis of PDF pages.","marker":"He et al. 2017"},{"why":"Supplies the PubLayNet dataset on which the layout detector is trained to recognize text, figures, lists, and tables.","marker":"Zhong, Tang, and Jimeno Yepes 2019"},{"why":"Supplies the 357-page textbook used as the test document in the paper's qualitative observations.","marker":"Goel and Joyner 2018"}],"fun_headline_variants":["Chat with a PDF, get answers with its own figures","MuDoC: PDF chat cites figures from the source","Your PDF's figures become the answers in MuDoC","Ask a document, get its figures in the reply","MuDoC: Document chat with clickable figure proof"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The system's trustworthiness claim rests on GPT-4o reliably following the prompt—issuing the right retrieval calls, writing image tags that exactly match the retrieved filenames, and describing images without hallucinating—and the paper itself reports one observed hallucinated image description, noting such errors happen more often when multiple images are passed in a single query.","fun_headline_variants_meta":{"raw":{"variants":["Chat with a PDF, get answers with its own figures","MuDoC: PDF chat cites figures from the source","Your PDF's figures become the answers in MuDoC","Ask a document, get its figures in the reply","MuDoC: Document chat with clickable figure proof"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000157,"raw_usage":{"total_tokens":1149,"prompt_tokens":800,"completion_tokens":349,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":416,"completion_tokens_details":{"reasoning_tokens":279}},"tokens_in":416,"tokens_out":349,"duration_ms":3592,"temperature":1.0,"reasoning_tokens":279,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T20:17:12.183063+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 357-page test textbook, run a fixed set of questions through MuDoC, and check every generated figure reference against the retrieval output: any image tag whose filename does not match a retrieved image, or any textual description of a figure that contradicts the figure's source caption, falsifies the trustworthiness claim. A sharper experiment holds the question fixed and varies the number of images included in one GPT-4o query, counting hallucinated image descriptions; the paper's own observation predicts more hallucinations in the condition with several images.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Dense Passage Retrieval encoders used to embed text chunks, summaries, captions, and queries for text retrieval."},{"cited_title":"W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I","cited_arxiv_id":null,"evidence_quote":"Supplies CLIP encoders used to embed images, captions, and descriptions and to score image relevance."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the PubLayNet dataset on which the layout detector is trained to recognize text, figures, lists, and tables."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 357-page textbook used as the test document in the paper's qualitative observations."}],"review_version":1}