{"id":"3037be73-4174-4b3c-9375-b545917a267e","arxiv_id":"2505.05666","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On a new degraded-document dataset (DocDeg), OCR-based RAG with Llama 3.2 90B outperforms vision-based ColQwen2 retrieval on all quality levels, while the vision pipeline remains more memory-efficient.","lead":"OCR-based retrieval, using a 90-billion-parameter LLM for text extraction, beat a vision-language retrieval model on a new dataset of degraded documents, both in finding the right page and in answering questions about it. The text-first approach was also faster per query, though it used much more memory and took longer to index.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Semantic 'generation' comparison is OCR-based for both pipelines, so Table 3 does not support the claim that OCR-based RAG is better at generation.","rationale":"The paper's central empirical claim has two parts: retrieval and generation. Retrieval is tested fairly on DocDeg (though ColQwen2 is used without DocDeg fine-tuning while the OCR pipelines are zero-shot; this is a reasonable deployment comparison). The generation claim, however, is not tested as a vision-based system: the VLM-retrieved images are OCRed before answer generation (§4.1), so Table 3 scores reflect retrieval quality plus a shared OCR/LLM reader. The conclusion in §5 that OCR-based approaches offer improved generation performance is therefore overbroad. This is not a statistical quibble: if the authors had used a VLM QA model, the semantic ranking could differ, especially on degraded pages where OCR drops text. The model-size disparity flagged by the reader is real but secondary; the more direct flaw is the missing vision-based answer path. A useful revision would either add a VLM QA condition or explicitly rescope the conclusion to retrieval quality. Since the retrieval comparison and DocDeg dataset are still valuable, the CONDITIONAL verdict stands.","tokens_in":13676,"tokens_out":6081,"duration_ms":59792,"concrete_test":"Keep the retrieval stage fixed and replace the answer path for the VLM pipeline: for a stratified random sample of DocDeg queries (e.g., 200 per degradation level), feed the images retrieved by ColQwen2 to a vision-language QA model such as Qwen2-VL-7B or Llama 3.2 11B Vision (no OCR in the answer path), and compare Exact Match/BLEU/ROUGE-1/ROUGE-L against the OCR-based Table 3 rows. If VLM-generated answers match or exceed the OCR-based scores on Levels 2 and 3, the paper's 'generation performance in all evaluated settings' claim fails. Also report the same metrics restricted to queries where ColQwen2 retrieved the correct page, to isolate generation ability from retrieval.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in §5 states that 'without task-specific fine-tuning, OCR-based approaches offer improved retrieval and generation performance in all evaluated settings.' The retrieval half is directly tested, but the generation half is not. §4.1 says: 'For this study’s semantic answer evaluation, we used the OCR text from the images retrieved by ColPali as context for the LLM, in order to enable a fair comparison across pipelines.' Thus, the VLM pipeline's answers in Table 3 are generated from OCR-transcribed text, not from document images, and the answer generator appears to be the same Qwen2-7B used in the OCR pipeline. Under this design, BLEU/ROUGE/EM differences are driven almost entirely by which pages were retrieved, not by any vision-based answer-generation capability. The 'generation' component is therefore a second retrieval measurement with a shared OCR+LLM reader. The conclusion that vision-based RAG has worse generation performance is unsupported; a vision-language QA model could exploit layout and visual robustness and reverse the semantic ranking, especially on Levels 2-3. This is an internal mismatch between the stated claim and the evaluation procedure, not a disagreement with external consensus. The model-size confound (7B VLM vs 90B OCR) is a related but separate limitation; even equalizing model size would not fix the semantic evaluation flaw.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares a vision-based RAG pipeline (ColPali/ColQwen2, 7B) against OCR-based pipelines (Nougat OCR and Llama 3.2 90B OCR followed by Qwen2 7B embeddings) on a new dataset, DocDeg, of 4,196 manually labeled documents at four degradation levels. It reports retrieval metrics (MRR, Recall@5, NDCG@5) and semantic answer metrics (EM, BLEU, ROUGE-1, ROUGE-L) under a no-fine-tuning protocol, plus a supplementary ViDoRe/DocVQA evaluation and computational efficiency measurements. The central claim is that OCR-based RAG generalizes better to unseen degraded documents than VLM-based retrieval, and that this advantage extends to generation quality.","tokens_in":13937,"tokens_out":3616,"duration_ms":38299,"significance":"If the central claim holds, this is a practically important result: it would tell RAG practitioners that OCR-dependent pipelines with strong OCR models are preferable to lightweight VLM-based retrieval for collections containing degraded scans, and it would temper claims made for ColPali-style direct visual embedding. The paper contributes a new public-domain-derived dataset with explicit degradation labels, a dual evaluation protocol, and a slide-removal robustness check. The retrieval comparison is, on its face, a reasonable out-of-the-box comparison of representative model sizes, and the supplementary ViDoRe experiment usefully shows that fine-tuning on the target benchmark changes the ranking. However, the paper's generation claim is not supported by the experiment as designed, because the 'VLM pipeline' answer generation is performed on OCR text rather than on images; and the computational efficiency analysis contains an internal contradiction that must be resolved before the trade-off conclusions can be trusted.","major_comments":[{"comment":"The claim that OCR-based RAG offers 'improved retrieval and generation performance in all evaluated settings' is not supported for the generation component. Section 4.1 states that for the semantic answer evaluation, 'we used the OCR text from the images retrieved by ColPali as context for the LLM, in order to enable a fair comparison across pipelines.' Thus Table 3 compares two OCR+LLM readers applied to different retrieved page sets; the VLM pipeline's generation is never tested. The observed BLEU/ROUGE/EM differences therefore reflect retrieval differences, not vision-based generation ability. The conclusion in Section 5 that OCR-based approaches offer improved 'generation performance' is an overstatement. The authors should either add a condition in which a vision-language QA model answers from retrieved images, or explicitly restrict the claim to retrieval performance and reframe Table 3 as a downstream-retrieval-utility measure.","section":"§4.1, §4.2, Table 3, §5"},{"comment":"The computational efficiency numbers are internally inconsistent. Table 4 reports VLM retrieval latency of 0.04252s/query versus OCR latency of 0.0311s/query, which means the OCR pipeline is faster at query time. However, Section 4.3 states that 'the VLM-based system achieved ... substantially reduced retrieval latency (0.0010s per query vs. 0.0311s per query),' and Section 4.4 repeats 'significantly faster retrieval latency (0.001s vs 0.031s).' The conclusion then says OCR pipelines are 'actually faster at query time compared to VLM-based pipelines like ColPali,' which agrees with Table 4 but contradicts the section 4.3/4.4 text. Since the latency comparison feeds directly into the radar plot and the practical guidance, these values must be reconciled and the correct measurement reported with a clear description of what is timed (end-to-end RAG query, retrieval-only, or embedding-only).","section":"§4.3, Table 4, §4.4, Figure 4"},{"comment":"The slide-removal robustness check in Table 2 contains implausible values that appear internally inconsistent with Table 1. For Level 3, VLM-Based Recall@5 is 0.3112 and MRR is 0.2098—identical to Table 1—but NDCG@5 changes from 0.2350 (Table 1) to 0.2997 (Table 2). For OCR-Based (Llama), Recall@5 is 0.5925 and MRR is 0.4520 (identical to Table 1), but NDCG@5 drops from 0.4872 to 0.2474. NDCG@5 is determined by the same ranking and relevance labels that produce MRR and Recall@5, so such large changes with unchanged MRR/Recall are not coherent. This suggests a copy or aggregation error. The slide-removal claim (that both ColQwen and Llama degrade on level-0 when slides are excluded) depends on this table and should be re-verified.","section":"§4.1, Table 2"},{"comment":"The paper's central generalization claim is conditioned on a model-size disparity that is acknowledged but not analyzed. The vision pipeline uses ColQwen2 (7B) while the OCR pipeline uses Llama 3.2 (90B) for OCR plus Qwen2 7B for embedding; the conclusion that 'OCR-based approaches offer improved retrieval ... in all evaluated settings' should be stated only for the compared configurations. Because the authors argue that a 90B VLM is impractical (Section 1), the claim is a statement about deployable configurations, but it should be explicitly labeled as such in the abstract and conclusion. As written, a reader could infer that OCR-based retrieval is intrinsically better than vision-based retrieval, which is not established by this design.","section":"§1, §5"}],"minor_comments":[{"comment":"The paragraph after the first table contains a broken reference: 'Nougat outperformed the VLM-based RAG pipeline on levels one through three and on the weighted average. Interestingly, Nougat performed worse on the highest quality documents. 1 shows retrieval results with slideshows removed.' The '1' should read 'Table 2' and the sentence structure should be repaired.","section":"§4.1"},{"comment":"The ViDoRe results show exactly identical NDCG@5, MAP@5, MRR@5, and Recall@5 for OCR+Nougat and OCR+Llama (0.3373, 0.3147, 0.3164, 0.4058). The text attributes this to the OCR output not being the bottleneck, but identical four-decimal values across all four metrics merit a comment on whether the OCR outputs were actually identical or whether the same embedding was used by accident.","section":"Table 5"},{"comment":"The NDCG@k formula renders with garbled symbols ('˝' instead of summation and fraction notation). Please replace with a standard typeset formula.","section":"§3.3.1"},{"comment":"The manual review of approximately 40 question-answer pairs is described as qualitative verification, but no details are given on how the review was performed, inter-annotator agreement, or what was confirmed; a sentence on criteria would help.","section":"§4.2"},{"comment":"The radar plot normalization is described as a 'faithful, monotonic transformation,' but since the normalization uses the mean and standard deviation across all runs, the 'relative strengths' visualization depends on the particular mix of pipelines; this should be noted in the caption or text to avoid overinterpreting absolute differences.","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper's retrieval-side comparison is a useful contribution, and the authors are honest about the model-size disparity and the OCR requirement for the VLM pipeline's answer evaluation. The main blockage is the unsupported generation claim and the contradictory latency numbers; both are fixable with rewording and a corrected table/section, respectively. The Table 2 inconsistency should also be checked before publication. If the authors restrict the claim to retrieval performance and reconcile the efficiency numbers, a revision could be acceptable. I would not recommend rejection, as the dataset and experimental protocol have value for the document engineering community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is worth reading for the DocDeg dataset and the retrieval comparison, but the central 'generation' claim is not backed by the experiment as run. The semantic evaluation in Table 3 uses OCR text for both pipelines, so it only tests which pages were retrieved, not whether a vision model can generate better answers from images directly.\n\nWhat's genuinely new: DocDeg is a 4,196-page manually annotated degradation benchmark from real DOE documents, with four quality levels and 12 structural features. That is a useful resource for the document retrieval community. The paper is also the first to compare ColQwen2 against a modern Llama 3.2 90B OCR pipeline, rather than Tesseract. The retrieval protocol is careful: they report MRR, Recall@5, NDCG@5 across levels, and they include a slide-removal robustness check that changes the ranking on level-0 documents, which is honest. The retrieval result itself—Llama OCR pipeline beats ColQwen2 on degraded docs—is credible for the model sizes tested, and the ViDoRe control (ColQwen2 wins when fine-tuned) is a good sanity check.\n\nThe soft spots are real. First and most important: the abstract and conclusion say OCR-based approaches offer improved retrieval and generation performance. The retrieval half is supported; the generation half is not. Section 4.1 states that for semantic evaluation they fed the OCR text from the images retrieved by ColPali into the LLM. So Table 3 is a second retrieval measurement using a shared OCR+Qwen reader. A VLM-based QA model could plausibly do better, especially on degraded scans, and the paper's own future-work section admits this. The claim as written overreaches.\n\nSecond, the latency numbers are internally inconsistent: Table 4 gives VLM retrieval latency as 0.04252s, while Section 4.4 and the radar plot say 0.0010s. That is a factor of forty, and needs fixing before anyone trusts the efficiency trade-off.\n\nThird, the model-size confound is acknowledged but not tested: 7B VLM vs 90B OCR. The practical argument that a 90B VLM is too slow may be right, but the conclusion is conditional on it. The paper should say this more loudly.\n\nMinor: DocDeg is not actually released (no link or DOI); the QA prompts and annotation protocol should be included if the dataset is to be reusable.\n\nBottom line: the retrieval comparison and the dataset deserve referee time. The generation claim should either be removed or re-run with a vision-language QA model. With that fix, this would be a solid contribution to the document retrieval RAG literature.","headline":"A useful new degraded-document benchmark and a credible retrieval comparison, but the paper's generation claim overreaches because the semantic evaluation is OCR-based for both pipelines.","tokens_in":14478,"tokens_out":2540,"would_cite":false,"duration_ms":26105,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Without task-specific fine-tuning, OCR-based retrieval-augmented generation outperforms vision-language retrieval on documents of varying quality.","keywords":["retrieval-augmented generation","vision-language models","OCR","document retrieval","document degradation","late interaction","semantic answer evaluation","DocDeg"],"falsifier":"Run a vision-language retriever at roughly 90B parameter scale (or fine-tune ColQwen2 on DocDeg) and evaluate it on the same degradation-level splits; if its MRR or ROUGE-L matches or beats the Llama-OCR pipeline without task-specific fine-tuning, the paper's central claim fails.","tokens_in":13510,"feed_emoji":"📄","tokens_out":5399,"duration_ms":53330,"temperature":0.7,"pith_summary":"This paper sets out to settle which retrieval paradigm—direct visual embedding of document images or traditional OCR-then-embed—better serves retrieval-augmented generation on real scanned documents of mixed quality. It compares a ColPali-style vision pipeline (ColQwen2, 7B) with OCR pipelines that use Nougat or Llama 3.2 (90B) for text extraction followed by Qwen2 dense embeddings, on a new 4,196-document dataset with four hand-labeled degradation levels and a semantic question-answering benchmark. The central finding is that, without fine-tuning on the target corpus, OCR-based RAG achieves better retrieval and answer-generation scores at every degradation level, while vision-based retrieval only wins on the clean ViDoRe benchmark after being fine-tuned on it. The paper concludes that practitioners should prefer modern-OCR pipelines for heterogeneous scanned collections, accepting slower indexing and larger memory in exchange for accuracy and faster query-time latency.","feed_headline":"OCR-based RAG beats vision retrieval on degraded documents","feed_subtitle":"Without fine-tuning, OCR wins on every quality level tested; vision models only win after in-domain tuning.","key_machinery":"The load-bearing object is the paired pipeline comparison built around three components: the VLM pipeline embeds non-overlapping image patches with ColQwen2 and scores query–patch similarity through ColBERT-style late interaction; the OCR pipeline converts each page to text with Llama 3.2 or Nougat and scores pages by cosine similarity of Qwen2 text embeddings; and the evaluation stack measures retrieval with MRR, Recall@5, and NDCG@5, then measures downstream answer quality with Exact Match, BLEU, and ROUGE, then measures speed and memory. The DocDeg dataset (4,196 pages, four manually graded degradation levels, ten Llama-generated QA pairs per page) is the instrument that makes the comparison possible, and the ViDoRe/DocVQA subset is the clean-domain control that isolates fine-tuning effects.","core_discovery":"The paper's core claim is that the robustness advantage commonly attributed to vision-language retrieval does not survive contact with unseen, degraded documents. On the DocDeg dataset, the OCR pipeline with Llama 3.2 OCR and Qwen2 embeddings outperformed the VLM pipeline (ColQwen2) on MRR, Recall@5, and NDCG@5 at all four degradation levels, and also produced higher Exact Match, BLEU, ROUGE-1, and ROUGE-L scores in end-to-end question answering. The VLM pipeline's only win came on the ViDoRe/DocVQA benchmark, where it had been fine-tuned on that benchmark's own query-page pairs—evidence that its strength is in-domain memorization rather than general visual robustness. The paper also reports that OCR-based retrieval is faster at query time despite slower indexing, while the VLM pipeline uses less memory.","pith_inferences":["If a vision encoder at the same parameter scale as the 90B OCR model became practical, the generalization ranking could plausibly reverse; the paper's comparison bundles architecture choice with model size.","Because the VLM pipeline's answers were generated from OCR text of retrieved images, the 'end-to-end' comparison still contains an OCR step; a vision-language QA model used directly on retrieved images might narrow the semantic gap.","DocDeg could serve as a reusable robustness benchmark for future retrievers, especially for measuring zero-shot transfer across degradation levels and for testing whether fine-tuning closes the gap.","The near-identical Nougat and Llama scores on ViDoRe suggest that on clean documents the binding constraint is query–embedding semantic alignment rather than OCR fidelity, pointing future work toward better encoders rather than better OCR."],"forward_implications":["In production RAG over scanned or heterogeneous document collections, a modern OCR pipeline should be the default choice when no fine-tuning budget exists for a vision retriever.","Vision-language retrieval, used out of the box, cannot be assumed to handle degraded documents better than OCR; its measured advantage is specific to the distribution it was fine-tuned on.","The OCR engine itself matters: Llama 3.2 OCR substantially beat Nougat OCR on the same retrieval tasks, so pipeline comparisons should report which OCR is used.","Query-time latency favors the OCR pipeline, while indexing time and memory favor the VLM pipeline; the choice is a resource trade-off, not a one-sided win.","Retrieval accuracy drops as degradation increases, but semantic answer quality does not decline monotonically, suggesting the generator compensates for some retrieval noise."],"supporting_citations":[{"why":"Defines the ColPali architecture and the ViDoRe benchmark that the paper uses as the vision baseline and clean-domain control.","marker":"[6]"},{"why":"Supplies the ColBERT late-interaction scoring mechanism that ColPali imports and that the paper contrasts with cosine-similarity dense retrieval.","marker":"[9]"},{"why":"Provides the Llama 3.2 and Llama 3.3 models used for OCR extraction and for generating the QA pairs in DocDeg.","marker":"[7]"},{"why":"Establishes that OCR noise degrades RAG performance, the motivation for testing modern OCR and vision retrieval on degraded documents.","marker":"[22]"},{"why":"Shows character-level OCR errors reduce QA accuracy, supporting the need for a semantic answer benchmark beyond retrieval metrics.","marker":"[13]"},{"why":"Identifies Tesseract as the outdated OCR baseline used in earlier ColPali comparisons, which the paper replaces with stronger OCR engines.","marker":"[15]"}],"fun_headline_variants":["OCR beats vision RAG on unseen degraded documents","Vision retrieval wins only after in-domain fine-tuning","OCR-based RAG more robust on degraded unseen docs","OCR beats VLM on unseen docs; vision needs tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that OCR-based RAG generalizes better rests on treating a 90B-parameter OCR model as the representative of OCR pipelines and a 7B vision model as the representative of VLM pipelines; if a comparably large vision encoder existed at acceptable cost, the ranking might reverse.","fun_headline_variants_meta":{"raw":{"variants":["OCR beats vision RAG on unseen degraded documents","Vision retrieval wins only after in-domain fine-tuning","OCR-based RAG more robust on degraded unseen docs","OCR beats VLM on unseen docs; vision needs tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000417,"raw_usage":{"total_tokens":2144,"prompt_tokens":934,"completion_tokens":1210,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":1148}},"tokens_in":550,"tokens_out":1210,"duration_ms":10355,"temperature":1.0,"reasoning_tokens":1148,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:59:07.278806+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a vision-language retriever at roughly 90B parameter scale (or fine-tune ColQwen2 on DocDeg) and evaluate it on the same degradation-level splits; if its MRR or ROUGE-L matches or beats the Llama-OCR pipeline without task-specific fine-tuning, the paper's central claim fails.","supporting_citations":[{"cited_title":"XAI for Skin Cancer Detection with Prototypes and Non-Expert Supervision","cited_arxiv_id":"2402.01410","evidence_quote":"Defines the ColPali architecture and the ViDoRe benchmark that the paper uses as the vision baseline and clean-domain control."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ColBERT late-interaction scoring mechanism that ColPali imports and that the paper contrasts with cosine-similarity dense retrieval."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Identifies Tesseract as the outdated OCR baseline used in earlier ColPali comparisons, which the paper replaces with stronger OCR engines."}],"review_version":1}