{"id":"664620ab-4294-4816-8359-958400096c27","arxiv_id":"2507.23217","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A training-free RAG system with LLM-generated pseudo-TOCs and hierarchical retrieval reaches 64.7% on MMLongBench-Doc, surpassing prior SOTA by 15 points.","lead":"DocsRay is a training-free document understanding system that uses a language model to build a pseudo table of contents and then does hierarchical retrieval to answer questions about long, mixed-content documents. On the MMLongBench-Doc benchmark it reports 64.7 percent accuracy, close to human expert performance and well above prior systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The only controlled same-backbone ablation (Table 5) shows the pseudo-TOC reduces accuracy (62.8% vs. 63.5%), and no Pro-level flat-retrieval control exists, so the headline accuracy gain is not yet attributable to the proposed pseudo-TOC-guided mechanism.","rationale":"The reader's weakest_assumption (coarse search pruning irrecoverable content) is a real mechanism-level risk and is acknowledged in Section 4.4, but it is not the most load-bearing issue for the headline claim. The headline claim is that a pseudo-TOC-guided system surpasses much larger LVLMs by 15 points. That claim has two load-bearing conditions: (1) the 64.7% number is reproducible, and (2) the system's pseudo-TOC component is actually responsible for the gain. Condition (2) is directly undermined by the only same-backbone ablation available (Table 5), and the paper itself states in Appendix G.1 that the core contribution—superiority of hierarchical semantic retrieval over flat retrieval—was not quantitatively validated. The missing Pro-level flat-retrieval control is therefore the single experiment that would settle whether the proposed mechanism matters for accuracy. I agree with the reader's CONDITIONAL verdict: the paper is a solid system description with honest limitations, but acceptance should hinge on code release, error-bar reporting, and this flat-retrieval control.","tokens_in":24261,"tokens_out":8982,"duration_ms":99431,"concrete_test":"Run DocsRay-Pro on MMLongBench-Doc with the pseudo-TOC stage disabled (flat retrieval over all chunks), keeping the same Gemma-3 27B backbone, dual embeddings, query refinement, top-10 chunk selection, and evaluation protocol; compare accuracy to the reported 64.7%. If flat retrieval attains the same or higher accuracy, the pseudo-TOC-guided mechanism is not the source of the headline accuracy; if it is substantially lower (e.g., below 60%), the coarse-to-fine pruning is essential at Pro scale and the reverse direction of the Base ablation requires explanation. Report per-seed mean and standard deviation for both conditions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution is a pseudo-TOC-guided hierarchical RAG system, and Section 1 explicitly claims \"This design improves both accuracy and efficiency.\" Yet the only controlled same-backbone comparison, Table 5, shows DocsRay-Base accuracy drops from 63.5% (no pseudo-TOC, flat retrieval) to 62.8% (with pseudo-TOC). The paper frames this as a \"marginal 0.7 percentage point decrease\" and an efficiency/accuracy tradeoff, but it is the sole same-backbone evidence available. For the headline DocsRay-Pro 64.7% versus GPT-4.1 49.7%, no equivalent flat-retrieval control is reported, so the 15-point gap cannot be attributed to the pseudo-TOC-guided retrieval; it could be driven by the Gemma-3 27B backbone, the dual embeddings, query refinement, or the multimodal text-centric conversion. Appendix G.1 concedes this: \"This limitation prevented us from quantitatively validating our core technical contribution, which lies in demonstrating the superiority of hierarchical semantic retrieval over flat retrieval methods.\" The efficiency claim (45% latency reduction) is separately supported by Table 5, but the accuracy attribution is not. The result is also reported as a point estimate despite Appendix I.4 stating results are averaged over 5 seeds, so run-to-run variability is unknown.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DocsRay, a training-free document understanding system that combines LLM-generated pseudo-TOCs, text-centric multimodal content conversion, dual embeddings, hierarchical coarse-to-fine retrieval, and iterative query refinement. On MMLongBench-Doc, DocsRay-Pro reports 64.7% accuracy, close to the human expert score of 65.8% and 15.0 points above the strongest prior baseline (GPT-4.1 at 49.7%), with a claimed 45% query latency reduction. The paper also reports scale-based analyses and ablations of the pseudo-TOC and embedding components.","tokens_in":24572,"tokens_out":3297,"duration_ms":36493,"significance":"If the headline results are robust, the paper would demonstrate a practically valuable training-free RAG system that approaches human-level long-document QA while reducing latency. The work is clearly presented, with exact prompts, an algorithm listing, and a commitment to code release, all of which support reproducibility. However, the central attribution claim—that pseudo-TOC-guided hierarchical retrieval improves accuracy—is currently not supported by the controlled evidence, and the authors themselves acknowledge this gap. The significance of the contribution therefore remains conditional on additional same-backbone experiments and statistical reporting.","major_comments":[{"comment":"The only same-backbone controlled comparison shows that removing the pseudo-TOC gives higher accuracy (63.5%) than using it (62.8%). This directly contradicts the Introduction's claim that the design 'improves both accuracy and efficiency.' Because no flat-retrieval control is reported for DocsRay-Pro, the 15-point improvement over GPT-4.1 in Table 2 cannot be attributed to the pseudo-TOC-guided hierarchical retrieval; it may stem from the Gemma-3 27B backbone, the dual embeddings, query refinement, or the text-centric multimodal conversion. Please provide a DocsRay-Pro flat-retrieval variant with the same backbone, same embeddings, and same query refinement to establish attribution.","section":"Section 4.4, Table 5"},{"comment":"The paper explicitly concedes: 'This limitation prevented us from quantitatively validating our core technical contribution, which lies in demonstrating the superiority of hierarchical semantic retrieval over flat retrieval methods.' Since this is exactly the core contribution claimed in Sections 1 and 3.4, the current evidence does not support the central scientific claim. Qualitative expert assessment does not replace a controlled quantitative comparison, especially because the one quantitative same-backbone comparison (Table 5) shows a small accuracy decrease.","section":"Appendix G.1"},{"comment":"Appendix I.4 states that all reported results are averaged over 5 independent runs, yet Tables 2, 5, and 6 report only point estimates without standard deviations or confidence intervals. Additionally, the many free hyperparameters (β=0.3, chunk size 550, overlap 25, top-5 sections, top-10 chunks, two refinement iterations, vector-graphics thresholds) appear to be chosen against the target benchmark without a reported held-out development set. This makes it impossible to assess whether the headline numbers reflect a genuine advantage or overfitting to MMLongBench-Doc; please report per-seed variation and perform any hyperparameter selection on a validation split.","section":"Section 4.4 and Appendix I.4"},{"comment":"The comparison in Table 2 mixes external baselines under potentially different protocols: some are LVLMs evaluated with full-document context, others are OCR+LLM pipelines, and the paper does not state whether these numbers were taken from the official leaderboard or re-run under DocsRay's own evaluation setup. Please clarify the source of each baseline and, where feasible, run the strongest baselines under the same document preprocessing and retrieval protocol to make the SOTA claim apples-to-apples.","section":"Table 2"}],"minor_comments":[{"comment":"Appendix G.1 references 'Section 4.6' for source attribution, but the main text has no Section 4.6; the relevant discussion appears to be in Section 3.4 or Appendix J.","section":"Appendix G.1"},{"comment":"The statement that results are averaged over 5 seeds is not reflected in any table; please add error bars or per-seed results, or remove the statement if only one run was used.","section":"Appendix I.4"},{"comment":"Table 1 uses √ and × symbols but never defines them in the caption; please add a legend (e.g., √ = supported, × = not supported).","section":"Table 1"},{"comment":"There is a terminology mismatch: Section 3.3 describes chunks of 500–600 tokens, while Algorithm 1 uses 'initial chunk size k=5' referring to pages. Please clarify the two levels of chunking and define the notation consistently.","section":"Section 3.3 vs Algorithm 1"},{"comment":"The abstract says 'pseudo Table of Contents (TOC)' but later text uses 'pseudo-TOC' and 'Table of Contents' with inconsistent capitalization; please standardize.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper's own appendix (G.1) concedes the missing quantitative validation of the core retrieval contribution, and the only same-backbone ablation (Table 5) shows a small accuracy decrease rather than the claimed improvement. I recommend that the editors require: (1) a same-backbone flat-retrieval control for DocsRay-Pro, (2) variance reporting over the stated 5 seeds, and (3) clarification of baseline protocol consistency, before considering the headline SOTA claim. These are fixable with additional experiments, so I do not recommend rejection at this stage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"DocsRay is a training-free RAG system that builds a pseudo-TOC with two prompts, indexes chunks with concatenated dual embeddings, and does coarse-to-fine retrieval. The paper is worth a look for the 64.7% on MMLongBench-Doc and for an unusually candid limitations section. But the central attribution claim does not survive its own ablation: the only same-backbone control, Table 5, shows pseudo-TOC slightly reduces accuracy on the Base model (62.8 vs. 63.5), and there is no Pro-level flat-retrieval control. The 15-point gap over GPT-4.1 is therefore not shown to come from the pseudo-TOC guidance; it could be the Gemma-3 27B backbone, the dual embeddings, query refinement, or the text-centric conversion. The authors concede exactly this in Appendix G.1, saying they could not quantitatively validate the hierarchical retrieval contribution. That concession is to their credit, but it means the paper's central claim is under-supported.\n\nWhat is solid: the dual embedding ablation is a controlled result, with concatenation beating single embeddings by 8–9 points. The latency reduction (3.89 to 2.12 seconds) is well-supported by the same table. The qualitative case studies are genuinely informative, including the non-monotonic scaling example where the Pro model undercounts charts. Prompts are given in full, and the limitations are stated in unusual detail.\n\nSoft spots beyond the attribution gap: no variance is reported even though Appendix I.4 says results are averaged over 5 seeds. Hyperparameters look selected against the benchmark, with no held-out development set. Code is promised but not released. The SlideVQA numbers (17.1% EM) are honestly framed as out-of-scope, but they do show the approach's limits on multi-image reasoning.\n\nThis is a legitimate systems contribution with an important but unverified headline. The efficiency story is solid; the accuracy story needs a same-backbone flat-retrieval control for the Pro model, plus error bars. I would send this to peer review, conditional on those additions. Without them, the paper should be framed as an efficiency/accuracy tradeoff rather than a claim that pseudo-TOC improves accuracy. For now, the honest limitations section makes it a useful data point for anyone building practical document QA systems.","headline":"A solid, honest systems paper whose headline accuracy gain is not yet attributable to its core pseudo-TOC mechanism, per its own ablation.","tokens_in":25118,"tokens_out":1701,"would_cite":false,"duration_ms":21735,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DocsRay claims a training-free, pseudo-TOC-guided RAG system reaches 64.7% accuracy on MMLongBench-Doc, 15 points above GPT-4.1 and near human expert score.","keywords":["document understanding","retrieval-augmented generation","pseudo table of contents","hierarchical retrieval","multimodal LLM","zero-shot","MMLongBench-Doc","dual embedding"],"falsifier":"Take any long document with an answer-bearing passage located inside a section whose pseudo-TOC title is generic (for example 'Introduction') and check whether coarse retrieval with top-5 sections recovers that passage; if it is routinely pruned, the pseudo-TOC itself is the error source. A cleaner test is to compare DocsRay's retrieval recall against a flat retriever on a document retrieval benchmark with annotated relevant chunks, which the paper notes is currently missing.","tokens_in":24055,"feed_emoji":"📄","tokens_out":5942,"duration_ms":58368,"temperature":0.7,"pith_summary":"This paper claims that a completely training-free document understanding system can outperform far larger multimodal models on long, multi-page document question answering. The system, DocsRay, first prompts a multimodal LLM to build a pseudo table of contents for an unstructured document, then uses that structure for coarse-to-fine retrieval before answering. On MMLongBench-Doc, DocsRay-Pro reaches 64.7% accuracy, beating the strongest reported prior baseline (GPT-4.1 at 49.7%) and approaching the human expert score of 65.8%, while cutting query latency from 3.89 to 2.12 seconds. If these numbers hold, orchestration of prompting, semantic structuring, and hierarchical retrieval can substitute for massive model scale and task-specific training.","feed_headline":"Training-free RAG hits 64.7% on long-doc QA","feed_subtitle":"Pseudo-TOC-guided retrieval beats GPT-4.1 by 15 points and cuts query latency 45%.","key_machinery":"The pseudo-TOC is a semantic outline generated by prompting an LLM to decide whether consecutive page excerpts change topic and to name each resulting section; it is what lets retrieval be hierarchical rather than flat. Section representations pair a title embedding with the average of chunk embeddings, combined with weight $\\beta$, and the coarse search compares the query to these section vectors before fine search looks inside the top sections. This two-stage design is the mechanism that cuts complexity from $O(N)$ to $O(S + k_1 \\cdot N_s)$ and keeps chunks topically coherent, and the concatenated dual embedding is what makes both lexical and semantic matches visible to the retriever.","core_discovery":"The central claim is that the pseudo-TOC is the load-bearing structure: an LLM detects topic boundaries between page chunks, merges small sections, and titles each section using only two prompts, turning a long heterogeneous document into a navigable hierarchy. Retrieval then works in two stages: coarse search scores section-level embeddings (a weighted blend of title and average content embeddings), and fine search retrieves chunks only inside the top-ranked sections. This reduces retrieval complexity from $O(N)$ to $O(S + k_1 \\cdot N_s)$ and, with dual embeddings formed by concatenating BGE-M3 and Multilingual-E5-Large, yields 64.7% accuracy on MMLongBench-Doc. The paper also demonstrates that the speed-accuracy tradeoff favors hierarchy: removing the pseudo-TOC raises accuracy slightly (62.8% to 63.5%) but increases query time by 45%.","pith_inferences":["The paper explains most errors through coarse retrieval missing sections; a natural extension it leaves implicit is to widen or refine the coarse selection when section titles are generic, or to add a second coarse pass after fine search fails.","Comparing the pseudo-TOC against a human-written TOC on the same documents would isolate how much of the 15-point gain comes from the structuring step itself rather than from hierarchical retrieval in general; the paper does not run this control.","The text-centric conversion of images to captions suggests the accuracy claim should not be generalized to benchmarks requiring quantitative comparison across multiple simultaneous images, where the paper reports only 17.1% exact match on SlideVQA.","The dual-embedding gain (62.8% with concatenation versus roughly 54% for either model alone) suggests that pairing a lexical-leaning and a semantic-leaning encoder is the active ingredient; this likely transfers to other RAG systems regardless of pseudo-TOC."],"forward_implications":["A 27B parameter Gemma-3-based system beats much larger LVLMs on MMLongBench-Doc, so structure-guided retrieval can dominate raw scale on long-document QA.","Hierarchical retrieval cuts Stage 2 query latency by about 45% (3.89 to 2.12 seconds) while losing only 0.7 accuracy points versus flat retrieval, making the tradeoff attractive for interactive use.","Even a 4B variant surpasses several large LVLMs, suggesting the pseudo-TOC compensates for limited model capacity on simple factual queries.","Because the pipeline is training-free, it can be applied to new document types, languages, and domains without collecting task-specific data, as long as the backbone LLM responds to the prompts.","Section-level 'References' in the output allow users to check which document sections actually informed an answer."],"supporting_citations":[{"why":"Defines the MMLongBench-Doc benchmark and its evaluation protocol, the dataset that produces all headline accuracy numbers.","marker":"(Ma et al. 2024)"},{"why":"Supplies the Gemma-3 backbone models (4B/12B/27B) used for the Pro, Base, and Lite variants.","marker":"(Team et al. 2025a)"},{"why":"Provides the BGE-M3 embedding model, one half of the dual embedding whose concatenation is ablated to 62.8% accuracy.","marker":"(Chen et al. 2024)"},{"why":"Provides Multilingual-E5-Large, the complementary embedding model in the dual embedding setup.","marker":"(Wang et al. 2024)"},{"why":"Shows that an LLM can infer semantic breakpoints in long documents; DocsRay extends this idea into hierarchical titled sections.","marker":"(Duarte et al. 2024)"},{"why":"Establishes the two-stage document-then-passage retrieval pattern that DocsRay adapts with its pseudo-TOC sections.","marker":"(Liu et al. 2021)"},{"why":"GPT-4.1 is the strongest prior baseline (49.7%) that DocsRay-Pro surpasses by 15 points.","marker":"(OpenAI 2025)"}],"fun_headline_variants":["Pseudo-TOC guides RAG to 64.7% on long docs","DocsRay: zero-shot TOC-guided RAG hits 64.7%","Training-free TOC-guided retrieval cuts latency 45%","Pseudo-TOC + hierarchical RAG: 64.7% MMLongBench","DocsRay: TOC-guided RAG sets new SOTA on long-doc QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The system assumes the pseudo-TOC partitions the document so that every query-relevant chunk falls inside a section the coarse search selects; if boundary detection produces misaligned sections or generic titles, relevant content is pruned before fine search and cannot be recovered.","fun_headline_variants_meta":{"raw":{"variants":["Pseudo-TOC guides RAG to 64.7% on long docs","DocsRay: zero-shot TOC-guided RAG hits 64.7%","Training-free TOC-guided retrieval cuts latency 45%","Pseudo-TOC + hierarchical RAG: 64.7% MMLongBench","DocsRay: TOC-guided RAG sets new SOTA on long-doc QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001298,"raw_usage":{"total_tokens":5307,"prompt_tokens":966,"completion_tokens":4341,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":4237}},"tokens_in":582,"tokens_out":4341,"duration_ms":34032,"temperature":1.0,"reasoning_tokens":4237,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:56:36.513715+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take any long document with an answer-bearing passage located inside a section whose pseudo-TOC title is generic (for example 'Introduction') and check whether coarse retrieval with top-5 sections recovers that passage; if it is routinely pruned, the pseudo-TOC itself is the error source. A cleaner test is to compare DocsRay's retrieval recall against a flat retriever on a document retrieval benchmark with annotated relevant chunks, which the paper notes is currently missing.","supporting_citations":[],"review_version":1}