Pith. sign in

REVIEW 4 major objections 5 minor 25 references

The paper claims that spoken questions can drive end-to-end retrieval and answering over document images, with OCR, ASR, and TTS entirely removed from the pipeline.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

TextlessRAG retrieves document image pages and generates spoken answers directly from an audio query, without OCR, ASR, or TTS, and introduces a bilingual speech-document RAG benchmark.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection The task is new and the SV-DOC dataset is a real contribution, but the accuracy claim rests on a comparison that omits the obvious ASR baseline and mostly loses to the text-query SOTA. the 4 major comments →

arxiv 2509.07538 v2 pith:LZUYXTNF submitted 2025-09-09 cs.CV

TextlessRAG: End-to-End Visual Document RAG by Speech Without Text

Peijin Xie , Shun Qian , Bingquan Liu , Dexin Wang , Lin Sun , Xiangzheng Zhang This is my paper
classification cs.CV
keywords speech-based visual document RAGtextless pipelineaudio-visual retrievalmultimodal retrieval-augmented generationlayout-aware rerankingbilingual benchmarkspoken question answeringdocument image retrieval
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

TextlessRAG is the first retrieval-augmented generation pipeline that answers spoken questions over document images without ever transcribing the speech, reading page text, or synthesizing the reply from text. The paper claims that a speech-and-vision encoder can rank whole document pages by matching an audio query directly against page images, and that an omni-modal generator can then produce an audio answer from the retrieved pages. Removing OCR, ASR, and TTS does more than shorten the pipeline—the reported latency breakdown attributes the largest costs to those three modules, and the end-to-end system is faster than text-based visual RAG while staying close to its accuracy. To test this, the paper contributes a bilingual benchmark spanning seven datasets in English and Chinese, including a newly built Chinese document RAG dataset. If the claim is right, hands-free question answering over PDFs, slides, and reports becomes practical with just a microphone.

Core claim

The central claim is that speech works as a first-class query modality for visual document retrieval, so the standard cascades—speech-to-text, OCR, text retrieval, and text-to-speech—can be dropped without a proportional loss in accuracy. The retriever, ColQwen-Omni, encodes the audio question and each document-page image into token-level embeddings; a MaxSim score, the mean of maximum token-level similarities, ranks the pages. A layout-aware reranker then splits the top pages into chart, table, text, and natural-image blocks using DocLayout-YOLO, retains blocks whose similarity to the query exceeds a threshold, and re-sorts the pages by those block scores. The generator, Qwen2.5-Omni, takes

What carries the argument

The load-bearing mechanism is the shared speech–image embedding space of ColQwen-Omni combined with ColBERT-style MaxSim late interaction: each spoken query and each page is represented as a set of token vectors, and page-level relevance is the mean of maximum token similarities. Around this core sits a layout-aware reranker that uses DocLayout-YOLO to decompose each top-k page into chart, table, text, and natural-image blocks, keeps blocks whose query similarity exceeds a threshold, and re-orders pages by those block scores. The generator, Qwen2.5-Omni, consumes the spoken query and reranked page images directly and synthesizes an audio answer, so no intermediate text ever appears.

Load-bearing premise

The benchmark's audio queries are TTS renderings of written questions; if real spontaneous speech, accents, or background noise were used, the measured speech-to-image alignment could drop, and the practical claim of hands-free retrieval would not transfer.

What would settle it

Re-run TextlessRAG on the same seven retrieval pools with human-recorded spontaneous questions (varied accents, background noise, disfluencies) instead of TTS audio, and compare nDCG@5 and GPT-4o-judged QA accuracy; a substantial drop would falsify the reliability claim. A complementary test would compare the textless pipeline against a strong ASR + text-visual retriever on the same audio queries, isolating whether the advantage comes from the speech–image alignment or from the benchmark's audio being easy.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Voice becomes a viable input modality for knowledge-base QA over visual documents, enabling hands-free retrieval from PDFs, slides, and reports without typing or manual transcription.
  • Document collections can be indexed once as page-image embeddings; each spoken query consists of one audio encode and two retrieval passes, keeping per-query compute low.
  • Layout reranking consistently improves answer quality across datasets, and on some datasets reranked top-5 inputs outperform gold-page inputs, suggesting block-level selection is a stronger evidence signal than whole-page relevance.
  • Because OCR is absent, structured content such as tables and charts is retrieved through visual features, avoiding errors from text extraction.
  • The released SV-DOC benchmark, with TTS-augmented English datasets and the Chinese CDR dataset, provides a common testbed for future speech-based visual RAG systems.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The benchmark's audio queries are TTS renderings of written text; a direct test with spontaneous, accented, or noisy speech is needed to know whether the retrieval margins survive outside the studio, and would naturally extend the dataset.
  • Because the retriever and generator are independently pretrained omni-models, the same two-stage design could be re-instantiated with future speech–vision encoders, making TextlessRAG a template rather than a fixed stack.
  • The layout-aware reranking thresholds block-level similarity, so the same idea could be pushed deeper: retrieving from within a single long document page, or answering directly from the single chart or table that carries the evidence.
  • A textless pipeline avoids producing a text transcript of the user's voice, shifting the privacy profile of voice-query systems, though the embeddings themselves remain a potential side channel.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. TextlessRAG proposes an end-to-end speech-query visual-document RAG pipeline that removes ASR, TTS, and OCR. It uses ColQwen-Omni to embed audio queries and document page images, MaxSim top-k retrieval, DocLayout-YOLO layout decomposition with a thresholded reranking stage, and Qwen2.5-Omni to generate spoken answers. The authors introduce SV-DOC, a bilingual benchmark obtained by TTS-augmenting ChartQA, DUDE, InfoVQA, SlideVQA, MMLongBench-Doc, Vidoseek, and a new Chinese CDR dataset. Experiments report retrieval nDCG@5, QA accuracy judged by GPT-4o, and latency analysis; the paper claims substantial improvements in both efficiency and accuracy, with layout reranking delivering gains and state-of-the-art results on ChartQA and Vidoseek.

Significance. If the experimental claims hold, the paper opens a useful and underexplored direction: spoken query retrieval over document image collections without cascaded ASR/OCR, and it contributes a first bilingual benchmark and a new Chinese document RAG dataset. The design is simple and the efficiency motivation for removing ASR/TTS/OCR is plausible. The main weakness is that the reported experiments do not yet establish the accuracy superiority asserted in the abstract and conclusion: the comparison set lacks an ASR-based speech-query baseline, several text-query baselines outperform the proposed system on multiple datasets, and key parameter choices are unreported.

major comments (4)
  1. [§4.2, Table 3; abstract] The central accuracy claim is not supported because no ASR-to-text speech-query baseline is included. All main comparisons are against text-query systems, so the effect of removing ASR is confounded with the choice of retriever/generator. In Table 3, the top-5 baseline ViDoRAG (text query, Qwen2.5-VL) outperforms TextLessRAG* on DUDE (87.4 vs 81.3), InfoVQA (82.6 vs 79.4), SlideVQA (84.2 vs 82.6), MMLong (47.3 vs 36.7), and CDR (70.1 vs 47.2); the gold-page rows show the same direction on most datasets (e.g., CDR 70.1 vs 61.3). Since a fair speech-query counterpart could be ASR plus ViDoRAG, the current evidence suggests a possible accuracy trade-off rather than the 'substantial improvements in both efficiency and accuracy' claimed in the abstract.
  2. [§4.1–4.2, Tables 2 and 3] All results are single runs with no variance, confidence intervals, or significance tests. The QA accuracy is GPT-4o averaged judgments, but the number of judgment samples, scoring prompt, and inter-rater agreement are not reported. Retrieval differences as small as 0.3–0.7 points (e.g., Table 2: CDR 87.4 vs 87.7; MMLong 66.5 vs 67.0) are treated as decisive. Because several claims depend on small margins, the paper needs at least a few repeats or an error analysis to establish that the reported rankings are stable.
  3. [§2, Eq. (7)] The layout reranking threshold theta is a free parameter whose value is never reported. The Table 3 rows marked '*' are the primary evidence for the reranking contribution, so without theta and a sensitivity analysis the gains could reflect tuning on the test sets. Report the chosen theta, its selection procedure, and results across a range of thresholds.
  4. [§3, step 5] The SV-DOC audio queries are TTS renderings of written questions using Doubao's TTS API. This means the entire evaluation uses clean synthetic speech. The paper's practical claim that the system 'directly interprets speech' is therefore only demonstrated for one TTS voice style; spontaneous speech, accents, disfluencies, and background noise are untested. A small human-speech test set, or at least an ASR baseline on the same TTS audio, is needed to assess whether the reported retrieval and QA numbers transfer to realistic input.
minor comments (5)
  1. [§4.1, Table 2] 'Rertrive result' should be 'Retrieval result'; 'MMlong' appears as 'MMLong' elsewhere.
  2. [Figure 1] The figure caption contains an untranslated Chinese example; English readers cannot interpret it without translation.
  3. [Figure 4] The text says 'VdocRAG is slightly slower than ViDocRAG'; likely a typo for VDocRAG vs ViDoRAG. Also, no absolute latency numbers or error bars are given, so the 'substantial speedup' claim is hard to verify quantitatively.
  4. [§2, Eq. (8)] The notation Sort-down-Score is not formal; specify the key, ordering direction, and tie-breaking behavior.
  5. [§3] The paper calls CDR 'manually annotated' but step 3 uses commercial VLMs to generate QA candidates. Please clarify the amount and role of human annotation relative to API-generated content.

Circularity Check

0 steps flagged

No significant circularity: off-the-shelf components and direct empirical evaluation.

full rationale

TextlessRAG is an empirical systems paper. The retrieval encoder (ColQwen-Omni), generator (Qwen2.5-Omni), and layout parser (DocLayout-YOLO) are used as off-the-shelf components; no mathematical claim is derived from a fitted parameter. The only free parameter is the reranking threshold θ in Eq. 7, but the paper does not report its value or a tuning procedure, so there is no exhibited reduction of a 'prediction' to a fitted value; this is a reproducibility concern, not circularity. The ViDoRAG baseline (Ref. [10]) shares an author with this paper, but it is used only as a comparison system, not to justify the central premise; the central claims (speech query encoding, textless pipeline) are supported by direct experiments in Tables 2–3 and Figure 4. The audio queries are TTS-generated from text, which raises external-validity concerns about spontaneous speech, but that is not a circularity. Therefore no step reduces to its own input by construction.

Axiom & Free-Parameter Ledger

2 free parameters · 6 axioms · 0 invented entities

The system's behavior rests on the alignment of three off-the-shelf models plus a thresholded reranking step. None of these priors are formally verified, and the threshold is unreported, so readers must accept the model alignment and benchmark construction as given.

free parameters (2)
  • Layout reranking threshold theta
    Introduced in Eq. 7 as a lower bound on block-level MaxSim scores during layout reranking. The optimal value is never reported, so the reported reranking gains may depend on a tuned hyperparameter.
  • Top-k page count = 5
    The generator receives the top-5 retrieved pages (Section 4.2). This is a manually chosen evaluation setting and results depend on it.
axioms (6)
  • domain assumption Pretrained ColQwen-Omni embeds speech and document images in a shared space such that MaxSim retrieval is meaningful
    Invoked in Section 2, Eqs. 1-3. The entire retrieval stage assumes the audio query embedding and image page embeddings are directly comparable without any task-specific training.
  • domain assumption Qwen2.5-Omni can consume a spoken query plus document page images and generate a high-quality answer
    Used in Section 2, Eq. 4. The QA generation stage assumes the Omni model's audio and visual understanding is sufficient for document-based answers.
  • domain assumption DocLayout-YOLO segments pages into chart, table, text, and image blocks accurately enough for reranking
    Used in Section 2, Eqs. 5-6. Reranking quality depends on the layout detector's block boundaries and type labels.
  • domain assumption TTS-generated audio queries are representative of real spoken queries
    Used in Section 3, step 5. All spoken queries in SV-DOC are synthetic TTS renderings, and the paper assumes these reflect natural speech conditions.
  • domain assumption GPT-4o average judgments measure QA correctness
    Used in Section 4. QA accuracy is reported as GPT-4o's averaged judgments with no human agreement or prompt details.
  • domain assumption Existing English dataset text answers remain valid when the question is rendered as speech
    Used in Section 3. The English benchmark extension assumes that paraphrasing a written question into TTS audio does not change the expected answer.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of TextlessRAG: End-to-End Visual Document RAG by Speech Without Text." pith.science (2026). https://pith.science/paper/LZUYXTNF

@misc{pith2026250907538,
  author       = {Pith},
  title        = {Pith review of: TextlessRAG: End-to-End Visual Document RAG by Speech Without Text},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LZUYXTNF}},
  note         = {Machine review of arXiv:2509.07538}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Document images encapsulate a wealth of knowledge, while the portability of spoken queries enables broader and flexible application scenarios. Yet, no prior work has explored knowledge base question answering over visual document images with queries provided directly in speech. We propose TextlessRAG, the first end-to-end framework for speech-based question answering over large-scale document images. Unlike prior methods, TextlessRAG eliminates ASR, TTS and OCR, directly interpreting speech, retrieving relevant visual knowledge, and generating answers in a fully textless pipeline. To further boost performance, we integrate a layout-aware reranking mechanism to refine retrieval. Experiments demonstrate substantial improvements in both efficiency and accuracy. To advance research in this direction, we also release the first bilingual speech--document RAG dataset, featuring Chinese and English voice queries paired with multimodal document content. Both the dataset and our pipeline will be made available at repository:https://github.com/xiepeijinhit-hue/textlessrag

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 21 canonical work pages · 1 internal anchor

  1. [1]

    INTRODUCTION The development of Vision-Language Models (VLMs) has been remarkably rapid[1, 2, 3, 4]. With the introduction of dynamic-resolution visual encoders, recent VLMs have made significant advances in understanding text-rich images[5] These models not only achieve higher accuracy in direct vi- sual document understanding but also eliminate the need...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  2. [2]

    METHOD As shown in Figure 2, TextLessRAG is designed to be both concise and efficient focusing specifically on the retrieval and QA generation components.On Retrieval side, we em- ploy ColQwen-Omni1 as a retrieve encoderEnc(·)to encode both document image pages from document knowedge base I={P 1, P2...Pn}and spoken queryqto embeddings as E={e 1, e2...en}a...

    work page

  3. [3]

    QA” and “Pool

    DA TA ENGINE We developed a data engine to construct the SV-DOC bench- mark. As summarized in Table 1, SV-DOC comprises (i) extensions of existing English benchmarks augmented with TTS, and (ii) CDR, the first Chinese Document RAG dataset created from scratch. CDR is built following a five-step pipeline (illustrated on the right of Figure 3), while the fi...

    work page

  4. [4]

    T”, “I”, and “A

    EXPERIMENT & RESULT We evaluate our approach along three dimensions:Retrieval, QA, andLatency. Retrieval is measured using nDCG@5, while QA accuracy is assessed based on GPT-4o’s averaged judgments. Latency is benchmarked on a single 80GB A100 GPU to ensure consistency. 4.1. Retrival Result Table 2 reportsRetrievalresults, comparing three text-only method...

    work page

  5. [5]

    We also introduce the first bilingual bench- mark for this task and release the first open-source Chinese visual document RAG dataset

    CONCLUSION In this work, we present TextLessRAG, the first RAG pipeline for visual document knowledge bases with speech-based in- put and output. We also introduce the first bilingual bench- mark for this task and release the first open-source Chinese visual document RAG dataset. Experimental results demon- strate that our textless design eliminates the n...

    work page

  6. [6]

    Qwen2.5-vl technical report,

    X. Liu J. Wang W. Ge S. Song K. Dang P. Wang S. Wang J. Tang H. Zhong Y . Zhu M. Yang Z. Li J. Wan P. Wang W. Ding Z. Fu Y . Xu J. Ye X. Zhang T. Xie Z. Cheng H. Zhang Z. Yang H. Xu S. Bai, K. Chen and J. Lin, “Qwen2.5-vl technical report,”arXiv preprint arXiv:2502.13923, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Qwen2-vl: Enhancing vision-language model’s per- ception of the world at any resolution,

    S. Tan S. Wang Z. Fan J. Bai K. Chen X. Liu J. Wang W. Ge Y . Fan K. Dang M. Du X. Ren R. Men D. Liu C. Zhou J. Zhou P. Wang, S. Bai and J. Lin, “Qwen2-vl: Enhancing vision-language model’s per- ception of the world at any resolution,”arXiv preprint arXiv:2409.12191, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models,

    Z. Chen Z. Liu S. Ye L. Gu H. Tian Y . Duan W. Su J. Shao Z. Gao E. Cui X. Wang Y . Cao Y . Liu X. Wei H. Zhang H. Wang W. Xu H. Li J. Wang N. Deng S. Li Y . He T. Jiang J. Luo Y . Wang C. He B. Shi X. Zhang W. Shao J. He Y . Xiong W. Qu P. Sun P. Jiao H. Lv L. Wu K. Zhang H. Deng J. Ge K. Chen L. Wang M. Dou L. Lu X. Zhu T. Lu D. Lin Y . Qiao J. Dai J. Z...

    work page 2025

  9. [9]

    Qwen2.5-omni technical report,

    J. He H. Hu T. He S. Bai K. Chen J. Wang Y . Fan K. Dang B. Zhang X. Wang Y . Chu J. Lin J. Xu, Z. Guo, “Qwen2.5-omni technical report,”arXiv preprint arXiv:2503.20215, 2025

    Pith/arXiv arXiv 2025

  10. [10]

    Multimodal large language models for text-rich image understanding: A comprehensive review,

    Z. Wang Z. Guo C. Duan H. Sun B. Chen J. Ma Q. Jiang K. Zhou P. Fu, T. Guan and J. Luo, “Multimodal large language models for text-rich image understanding: A comprehensive review,” 2025

    work page 2025

  11. [11]

    Mmlongbench-doc: Benchmarking long-context doc- ument understanding with visualizations,

    Y .B. Ma, Y . Zang, L. Chen, M. Chen, Y . Jiao, X.Z. Li, X.Y . Lu, Z.Y . Liu, Y . Ma, X.Y . Dong, . Zhang, L.M. Pan, Y .G Jiang, J.Q Wang, Y .X. Cao, and A.X Sun, “Mmlongbench-doc: Benchmarking long-context doc- ument understanding with visualizations,” 2024

    work page 2024

  12. [12]

    Slidevqa: A dataset for document visual question answering on multiple im- ages,

    K. Nishida R. Tanaka, K. Nishida, “Slidevqa: A dataset for document visual question answering on multiple im- ages,” inAAAI, 2023

    work page 2023

  13. [13]

    Visrag: Vision-based retrieval-augmented gener- ation on multi-modality documents,

    S. Yu, C.Y . Tang, B.K. Xu, J.B. Cui, J.H. Ran, Y .K. Yan, Z.H. Liu, S. Wang, X. Han, Z.Y . Liu, and M.S. Sun, “Visrag: Vision-based retrieval-augmented gener- ation on multi-modality documents,” 2025

    work page 2025

  14. [14]

    Vdocrag: Retrieval-augmented generation over visually-rich documents,

    T. Hasegawa K. Nishida K. Saito J. Suzuki R. Tanaka, T. Iki, “Vdocrag: Retrieval-augmented generation over visually-rich documents,” inCVPR, 2025

    work page 2025

  15. [15]

    Vidorag: Visual document retrieval-augmented generation via dynamic iterative reasoning agents,

    Q.C. Wang, R.X. Ding, Z.H. Chen, W.Q. Wu, S.H. Wang, P.J. Xie, and F. Zhao, “Vidorag: Visual document retrieval-augmented generation via dynamic iterative reasoning agents,”arXiv preprint arXiv:2502.18017, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Towards multilingual spoken visual question answering system using cross-attention,

    A. Roy C. hury, R. Tonmoy, and S. Sanjeev, “Towards multilingual spoken visual question answering system using cross-attention,” inProceedings of the 31st Inter- national Conference on Computational Linguistics, Abu Dhabi, UAE, Jan. 2025, pp. 9165–9175, Association for Computational Linguistics

    work page 2025

  17. [17]

    Spoken question answering for visual queries,

    N. Shabtay, Z. Kons, A. Dekel, H. Aronowitz, R. Hoory, and A. Arbelle, “Spoken question answering for visual queries,” 2025

    work page 2025

  18. [18]

    ChartQA: A benchmark for question answer- ing about charts with visual and logical reasoning,

    Ahmed M., D.X. Long, J.Q. Tan, S. Joty, and H. Ena- mul, “ChartQA: A benchmark for question answer- ing about charts with visual and logical reasoning,” in Findings of the Association for Computational Linguis- tics: ACL 2022, Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, Eds., Dublin, Ireland, May 2022, pp. 2263–2279, Association for Computat...

    work page 2022

  19. [19]

    Infographicvqa,

    M. Mathew, V . Bagal, R. P. Tito, D. Karatzas, E. Val- veny, and C. V Jawahar, “Infographicvqa,” 2021

    work page 2021

  20. [20]

    ICDAR 2023 competition on document understanding of everything (dude),

    Jain R. Kise K. Zanibbi R. Fink, G.A., “ICDAR 2023 competition on document understanding of everything (dude),” inProceedings of the ICDAR 2023, 2023

    work page 2023

  21. [21]

    The probabilistic relevance framework: Bm25 and beyond,

    R. Stephen and Z. Hugo, “The probabilistic relevance framework: Bm25 and beyond,”F ound. Trends Inf. Retr ., vol. 3, no. 4, pp. 333–389, Apr. 2009

    work page 2009

  22. [22]

    Text embeddings by weakly-supervised contrastive pre-training,

    L. Wang, N. Yang, X.L. Huang, B. Jiao, L.J. Yang, D.X. Jiang, R.G. Majumder, and F.R. Wei, “Text embeddings by weakly-supervised contrastive pre-training,” 2024

    work page 2024

  23. [23]

    Nv-embed: Improved tech- niques for training llms as generalist embedding mod- els,

    C. Lee, R. Roy, M.Y . Xu, J. Raiman, M. Shoeybi, B. Catanzaro, and W. Ping, “Nv-embed: Improved tech- niques for training llms as generalist embedding mod- els,” 2025

    work page 2025

  24. [24]

    Learning transferable vi- sual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable vi- sual models from natural language supervision,” 2021

    work page 2021

  25. [25]

    Uni- fying multimodal retrieval via document screenshot em- bedding,

    X.G Ma, S. Lin, M.H. Li, W.H. Chen, and J. Lin, “Uni- fying multimodal retrieval via document screenshot em- bedding,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, Eds., Miami, Florida, USA, Nov. 2024, pp. 6492–6505, Association for Computational Linguistics

    work page 2024

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.