{"id":"94b38906-99ca-4568-8274-c2a315475d28","arxiv_id":"2506.21604","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A weighted combination of text, image, caption, and OCR embeddings improves a custom retrieval score by 57.3% over text-only retrieval on a private 19-question enterprise HR document set.","lead":"This paper reports that a weighted mix of text, images, captions, and OCR text in a document retrieval system lifts a custom performance score by 57.3% over text-only retrieval. A generalist might read it to see whether multimodal enterprise search can be evaluated more realistically, but the measurement design makes the headline gain difficult to trust.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 57.3% gain is not a validated effect: the weights were optimized on the same 19-question test set, and the composite metric grants the text-only baseline zero credit on image, caption, and OCR components, so most of the improvement is built into the scoring design.","rationale":"I read the paper in good faith. The framework measures retrieval quality with a composite similarity score, and the paper reports a large gain for a specific weighting. The load-bearing condition for the central claim is that this score is an unbiased measure of cross-modal performance and that the weights are not tuned on the evaluation set. That condition is not met: the baseline cannot receive credit for the three visual components, and the weights are optimized on the same 19 questions used to report the final result (Table 3). Therefore the 57.3% number conflates genuine retrieval improvement with metric design and in-sample optimization. The internal inconsistency between the pseudocode weights and the headline weights strengthens the concern. A held-out evaluation with fixed a priori weights would settle whether the effect is real. These considerations do not change the reader's REJECT verdict.","tokens_in":14804,"tokens_out":5401,"duration_ms":49127,"concrete_test":"Hold out a random subset of the 19 questions (e.g., 9) before any weight selection; fix the component weights a priori (e.g., uniform 0.25 each) and also, separately, re-optimize weights only on the remaining 10 questions. Report the composite score on the held-out 9 questions for the text-only baseline and the full multimodal pipeline. If the 57.3% gain shrinks substantially or reverses under a priori weights or on the held-out split, the headline is an artifact of in-sample weight fitting and asymmetric scoring. As a secondary check, recompute Table 2's final row using the §3.5 pseudocode weights (0.3/0.15/0.35/0.2); if the weighted score differs from 42.74, the paper's stated 'optimal' weighting is internally inconsistent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract; §5) is that the optimized modality weighting (30% text, 15% image, 25% caption, 30% OCR) improves performance by 57.3% over the text-only baseline. For that claim to hold, the composite score must be a fair, comparable measure and the weights must not be fit to the same data used for the headline. Both conditions fail. Section 4.2 defines the score as a weighted sum of text match, image similarity, caption match, and OCR relevance. Appendix D, Table 2 shows the text-only baseline receives dashes (zero credit) for image, caption, and OCR, while multimodal systems receive non-negative scores on all four; a substantial part of the gain is therefore guaranteed by the metric's component structure. More decisively, Sections 4.3.2–4.3.4 describe the weights as 'optimized' (55/45 for text+image, 35/20/45 for text+image+caption, and 30/15/25/30 for the final system) on the same 19-question set reported in Table 3, with no held-out split, no search procedure, and no significance test. The reported 57.3% is thus an in-sample estimate. In addition, the pseudocode in §3.5, line 13 uses weights 0.3/0.15/0.35/0.2, contradicting the 0.25 caption and 0.30 OCR weights used in the abstract and Table 2; the paper does not state which weighting produced Table 1. The claim that this weighting 'improves performance' is therefore not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VisualRAG, a multimodal retrieval-augmented generation pipeline for enterprise document understanding, together with a quantitative evaluation framework that scores systems on a weighted combination of text match, image similarity, caption match, and OCR relevance. The authors report progressive gains from text-only retrieval (0.2387) through text+image (0.2511), text+image+caption (0.3572 with Claude 3.5 Sonnet), and finally text+image+caption+OCR (0.3754), claiming a 57.3% improvement over the text-only baseline under an 'optimized' modality weighting of 30% text, 15% image, 25% caption, and 30% OCR. They also compare foundation models for caption generation and OCR extraction and assert business-oriented trust improvements. The central claim appears in the abstract and is repeated in Section 5.","tokens_in":15121,"tokens_out":3411,"duration_ms":34371,"significance":"If the central claim were supported, the framework would be a useful contribution to enterprise multimodal RAG evaluation: it addresses a real gap in evaluating visually rich documentation, it provides per-question results in Table 3, and its appendices document prompts and example outputs, which is helpful for reproducibility. The comparative benchmarking of caption and OCR models is also practically relevant. However, the headline 57.3% improvement and the claimed optimal weighting are not established by the evidence presented, because the scoring metric structurally disadvantages the baseline and the weights are fit on the same small test set used for the final measurement. The significance of the work is therefore conditional on a fundamentally revised evaluation design.","major_comments":[{"comment":"The composite score is defined as a weighted sum of text match, image similarity, caption match, and OCR relevance, but the text-only baseline row in Table 2 shows dashes (zero credit) for image, caption, and OCR components, while all multimodal systems receive non-negative scores on those components. This makes the baseline structurally unable to earn credit on three of the four scoring components, so a large share of the reported 57.3% improvement is guaranteed by the metric definition rather than by better retrieval or answer quality. To support the central claim, the authors must either use a common scoring rubric in which all systems can receive credit on all components, or justify why a baseline that cannot access certain modalities should be compared on a composite that rewards exactly those modalities.","section":"§4.2, Appendix D Table 2"},{"comment":"The 'optimized' weights (55/45 for text+image, 35/20/45 for text+image+caption, and 30/15/25/30 for the full system) are described as derived from experimental results, but Section 4.3 gives no held-out split, no cross-validation, and no description of the weight search procedure. The final scores in Table 3 are computed on the same 19-question set that was used to select these weights, so the reported 57.3% improvement is an in-sample estimate. Without out-of-sample validation or at least a clearly separated validation/test protocol, the optimality claim in the abstract is not established.","section":"§4.3.2–4.3.4, Table 1, Table 3"},{"comment":"The pseudocode in Section 3.5 combines embeddings with weights 0.3 text, 0.15 image, 0.35 caption, and 0.2 OCR, which contradicts the 30% text, 15% image, 25% caption, and 30% OCR weights stated in the abstract, Section 4.3.4, and Table 2. The manuscript does not state which weighting actually produced the final score of 0.3754 in Table 1. This internal inconsistency must be resolved before any specific modality weighting can be attributed to the reported result.","section":"§3.5, Algorithm line 13"},{"comment":"The metrics described in Section 4.2 measure similarity between the query and retrieved component embeddings (text match, image similarity, caption match, OCR relevance), not the factual correctness or trustworthiness of the final generated response. Claims in Section 4.4 and Section 4.6 such as a 58% reduction in hallucination rates and a 72% improvement in Visual Context Preservation are not accompanied by any described measurement protocol, inter-annotator agreement, or data. The paper therefore does not establish the stated link between retrieval-side component scores and user-centric trust.","section":"§4.2, §4.4, §4.6"},{"comment":"The evaluation uses only 19–20 questions and reports means and standard deviations without significance tests or confidence intervals. Differences such as 0.3754 vs. 0.3731 for LLM vs. Tesseract OCR and 0.3040 vs. 0.2546 for BLIP vs. ViT-GPT2 are presented as conclusive, but with this sample size and the metric construction issues above, formal significance testing or bootstrap intervals are necessary to support comparative conclusions.","section":"Table 3, §4.4"}],"minor_comments":[{"comment":"The cross-references to 'Appendix 4', 'Appendix.5', and 'Appendix 6' are incorrect; these should refer to the appropriate subsections of Appendix D or to the corresponding figures.","section":"§4.3.1, §4.3.2, §4.3.3"},{"comment":"The number of test questions is inconsistent: Section 4.3.4 says '20 diverse questions', while Table 3 lists 19 questions, and Section 4.4 says 'across our test dataset' without a count. The authors should state the exact number of evaluation questions and use it consistently.","section":"§4.3.4, §4.4, Table 3"},{"comment":"The average for the LLM-OCR column is reported as 0.3755 in Table 3 but as 0.3754 in the text and Table 1; this numerical discrepancy should be corrected.","section":"Table 3"},{"comment":"The phrase 'Tessaract' appears as a typo for 'Tesseract' in Appendix D.4; please correct throughout.","section":"Appendix D"},{"comment":"Several reference entries have malformed author lists (for example, 'Kenton Lee, Mandar Joshi Masoud, ...' in the Pix2struct entry) and inconsistent formatting; the reference list should be checked against a consistent style.","section":"References"},{"comment":"The dataset is described as proprietary HR knowledge bases but also includes a live public URL example; the authors should clarify which portion of the corpus is proprietary and whether any privacy or consent considerations apply.","section":"§4.1"}],"recommendation":"reject","confidential_remarks":"The manuscript's central quantitative claim is not supported by the evidence as presented: the scoring metric gives the baseline zero credit on three of four components, and the weights are fit on the same test set used for the headline measurement. These are load-bearing issues that would require a new evaluation design rather than a local revision. I also note that the reported business-impact numbers (e.g., 40% decrease in help desk tickets) appear without any measurement protocol, which further reduces confidence in the paper's framing. The per-question data in Table 3 and the prompt documentation are useful, but they do not rescue the central claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline result here is not a result. The 57.3% gain over text-only retrieval is built into the scoring scheme and the tuning procedure. The composite score in Section 4.2 gives the text-only baseline zero credit on image, caption, and OCR components (Appendix D, Table 2), while the multimodal system gets non-negative scores on all four. So the baseline is structurally prevented from scoring on 75% of the metric. On top of that, the weights are 'optimized' on the same 19-question set used for the final measurement, with no held-out split and no significance test. That makes the headline an in-sample artifact, not a measured effect. There is also a straightforward inconsistency: the pseudocode in Section 3.5 uses caption 0.35 and OCR 0.2, while the abstract and text use 0.25 and 0.30. No statement says which weighting produced Table 1.\n\nThat said, the paper is not empty. The progressive framing—text, then text+image, then +caption, then +OCR—is a sensible structure for thinking about multimodal RAG evaluation. And the qualitative comparison of caption generators (BLIP, ViT-GPT2, LLM) and OCR (Tesseract vs LLM) does give some useful signal about where foundation-model choices matter, even if the numbers are not trustworthy as published.\n\nThe other soft spots are consistent with the main one. The 'trustworthiness' language is not matched by any actual human trust measurement; the composite is similarity between query and retrieved components. The business-impact claims (40% fewer tickets, 58% fewer hallucinations, 50% faster search) appear as assertions with no experimental protocol. The dataset is private and only about 19 questions, so external validation is impossible.\n\nMy take: this deserves a desk reject, not a referee. The central claim is not just weak—it is circular. If the authors came back with held-out validation, a scoring rule that gives all systems credit on the same components, and any of the code or data, it could become a modest applied contribution. As it stands, it is a cautionary example of how easy it is to manufacture improvement with a mismatch between metric and baseline.\n\nRegards.","headline":"The 57.3% gain is a measurement artifact: the baseline is zero-scored on three of four metric components and the weights are fit on the same test set, so the paper's central claim does not stand.","tokens_in":15667,"tokens_out":2815,"would_cite":false,"duration_ms":28546,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multimodal document search that blends text, image, caption, and OCR embeddings reports a 57.3% gain over text-only retrieval on an enterprise help-document benchmark, with weights 30/15/25/30.","keywords":["VisualRAG","multimodal retrieval-augmented generation","cross-modal evaluation","trustworthiness assessment","enterprise document understanding","OCR","image captioning","modality weighting"],"falsifier":"Take the paper's enterprise question set, keep the retrieval systems unchanged, and score only the final returned item (or human-rated answer quality) rather than the per-component weighted composite; if the multimodal pipeline's advantage over text-only is well below 57.3% or vanishes, the headline gain is an artifact of the component-weighted metric. A second check is a held-out split: tune the 30/15/25/30 weights on one subset and measure on another, and see whether the improvement survives.","tokens_in":14570,"feed_emoji":"📄","tokens_out":12166,"duration_ms":102291,"temperature":0.7,"pith_summary":"The paper sets out to show that a retrieval-augmented document search system becomes measurably more trustworthy when visual modalities are added to text one step at a time, and that a particular weighting of those modalities is the best configuration found. On an enterprise HR documentation benchmark, the full pipeline scores 0.3754 against 0.2387 for text-only retrieval, a 57.3% improvement, with weights of 30% surrounding text, 15% image, 25% caption, and 30% OCR text. The paper also claims that LLM-generated captions and OCR extraction outperform classical computer-vision tools, and that the evaluation method connects technical scores to user-facing outcomes such as fewer help-desk tickets and shorter search times. A reader should care because, if true, this gives enterprise teams a concrete recipe and a reusable way to measure whether multimodal document understanding is worth deploying.","feed_headline":"Blending text, images, captions, and OCR lifts search scores 57.3%","feed_subtitle":"Enterprise help-document search gains from a 30/15/25/30 modality mix; the 57.3% gap is partly built into the metric.","key_machinery":"The load-bearing object is a hierarchical multimodal embedding in which every document image is represented four ways using CLIP encoders: its surrounding text, the image itself, an LLM-generated caption, and OCR-extracted embedded text. The four vectors are combined into one stored embedding as $0.30 \\cdot \\text{text} + 0.15 \\cdot \\text{image} + 0.25 \\cdot \\text{caption} + 0.30 \\cdot \\text{OCR}$. At query time the same text encoder encodes the question and the system retrieves the closest stored bundles; the same weighted-composite score is used both to rank results and to measure quality. These weights are the constants the whole 57.3% claim rests on.","core_discovery":"The central claim is that cross-modal retrieval performance in visually rich enterprise documents is best captured by a four-part embedding combination rather than by text alone. The paper reports that combining surrounding-text, image, caption, and OCR embeddings with weights 0.30, 0.15, 0.25, and 0.30 yields an average composite score of 0.3754, a 57.3% gain over the 0.2387 text-only baseline, and that the LLM-powered caption and OCR components drive most of the gain: Claude 3.5 Sonnet captions score 0.3572 versus 0.3040 for BLIP and 0.2546 for ViT-GPT2, while LLM OCR at 0.3754 marginally beats Tesseract at 0.3731. The score itself is a weighted sum of exact text match, CLIP image similarity, semantic caption match, and OCR relevance. The paper interprets these numbers as evidence that optimizing modality weights can be a foundation for trustworthy enterprise RAG.","pith_inferences":["The 57.3% figure is partly a construction of the metric: the text-only baseline receives zero credit on image, caption, and OCR components, so a large share of the gap is guaranteed by the scoring rule rather than by retrieval quality; a fair comparison would score the final answer for both systems on the same yardstick.","Because the modality weights were chosen using the same question set on which the final score was measured, the specific 30/15/25/30 numbers are likely overfit to this HR corpus; the transferable discovery is the evaluation method, not the exact weights.","A natural test would be to hold out a set of questions, tune weights on one subset, and report the score on the held-out subset; if the gain falls sharply, the per-modal contributions are less robust than the headline suggests.","The caption and OCR quality gap between LLMs and classical tools suggests that foundation-model choice, not just modality presence, is a major trust lever—an implication the paper states but does not isolate causally."],"forward_implications":["Enterprise document-intelligence teams can treat 30/15/25/30 (text/image/caption/OCR) as a starting configuration for visually rich corpora, rather than tuning from scratch.","Organizations deciding between cheap generic captioners and expensive multimodal LLMs get a quantified gap: LLM captions add roughly 17% over BLIP and 40% over ViT-GPT2 on this benchmark.","The progressive evaluation design (text, then +image, then +caption, then +OCR) provides per-modality returns, so a team can stop adding modalities when the score gain no longer justifies cost.","If the link between the composite score and user trust holds, the reported business effects—40% fewer help-desk tickets, 50% lower search time, and 4/5 human ratings—are the practical payoff of the weighted pipeline."],"supporting_citations":[{"why":"Supplies the CLIP text and image encoders that produce the jointly aligned embeddings used in every modality combination.","marker":"[Radford et al.(2021)]"},{"why":"Provides the BLIP-2 baseline caption generator that the LLM captions must outperform on the caption match component.","marker":"[Li et al.(2023)]"},{"why":"Defines the OCR text component (TextOCR) and the idea that embedded text in images needs explicit extraction for retrieval.","marker":"[Singh et al.(2021)]"},{"why":"Establishes the layout-aware document understanding that motivates preserving spatial and textual relationships in the pipeline.","marker":"[Xu et al.(2020)]"},{"why":"Supplies the multimodal RAG (MuRAG) approach that the VisualRAG pipeline extends by adding captions and OCR.","marker":"[Chen et al.(2023)]"},{"why":"Frames the survey of multimodal RAG that the proposed hierarchy of document, section, and detail embeddings builds on.","marker":"[Mei et al.(2025)]"}],"fun_headline_variants":["57.3% retrieval lift from a four-modality mix in enterprise docs","Optimal text-image-caption-OCR blend boosts RAG scores 57.3%","Modality weighting key: 30/15/25/30 boosts VisualRAG by 57%","Cross-modal RAG: 57.3% gain over text-only with tuned weights","Enterprise RAG: four embeddings beat text alone by 57.3%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim's load-bearing premise is that the weighted composite score measures the same thing for every system; the text-only baseline gets no credit on image, caption, or OCR components by design, so part of the reported 57.3% gap is baked into the metric rather than earned by better answers.","fun_headline_variants_meta":{"raw":{"variants":["57.3% retrieval lift from a four-modality mix in enterprise docs","Optimal text-image-caption-OCR blend boosts RAG scores 57.3%","Modality weighting key: 30/15/25/30 boosts VisualRAG by 57%","Cross-modal RAG: 57.3% gain over text-only with tuned weights","Enterprise RAG: four embeddings beat text alone by 57.3%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001082,"raw_usage":{"total_tokens":4516,"prompt_tokens":926,"completion_tokens":3590,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":3478}},"tokens_in":542,"tokens_out":3590,"duration_ms":24269,"temperature":1.0,"reasoning_tokens":3478,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:24:19.855102+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the paper's enterprise question set, keep the retrieval systems unchanged, and score only the final returned item (or human-rated answer quality) rather than the per-component weighted composite; if the multimodal pipeline's advantage over text-only is well below 57.3% or vanishes, the headline gain is an artifact of the component-weighted metric. A second check is a held-out split: tune the 30/15/25/30 weights on one subset and measure on another, and see whether the improvement survives.","supporting_citations":[],"review_version":2}