Pith. sign in

REVIEW 3 major objections 6 minor 2 references

Three Sides of Retrieval: Factorial Evidence for Document-Side, Query-Side, and Answer-Side Complementarity in RAG

T0 review · 3 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A table-of-contents page index, built without LLM calls, is a significant and additive RAG quality lever.

desk verdict The ToC retrieval method is sensible and cheap, but the evaluation's reference answers were generated with ToC enabled, so the headline effects are likely inflated. read the letter →

arxiv 2607.24781 v1 pith:PDYQU3MI submitted 2026-06-19 cs.IR cs.AI

Ng S. T. Chong This is my paper
classification cs.IRcs.AI
keywords retrieval-augmentedgenerationtableofcontentsdocumentstructurepageretrievallayoutanalysisfactorialexperimentfeaturecomplementarityenterprisedocuments
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that chunk-based retrieval destroys document structure, and that a second retrieval channel that matches queries against layout-inferred headings and loads whole pages restores that structure at zero LLM cost. It reports factorial experiments showing this ToC channel is a statistically significant main effect on answer quality, that it stacks additively with answer-side verification, and that the document-side + answer-side pairing beats the query-side + answer-side pairing. A sympathetic reader would care because this is a cheap, deployment-ready enhancement and the first factorial evidence that document-side, query-side, and answer-side RAG improvements are complementary rather than redundant.

What carries the argument

The central object is the ToC-guided page retrieval algorithm: a three-tier heading extractor (PDF bookmarks, layout analysis inferring heading hierarchy from font sizes/weights/spacing, or first-line fallback) that runs without LLM calls, a parallel heading index embedded with the same model as chunks, cosine-similarity heading matching with a threshold and top-k, and full-page loading with a page budget and deduplication. This channel bypasses the chunk index entirely, which is what lets it recover cousin-section and coarse-heading content that chunk retrieval misses. The parallel-index design, as opposed to score blending, makes each channel's contribution directly measurable and requires

What would settle it

Generate reference answers with ToC disabled and score ToC-on vs ToC-off conditions against those references; if the ToC main effect shrinks to null, the reported effect depends on the evaluation anchor. Alternatively, run ToC+CC vs AS+CC on a corpus of multi-document multi-hop queries where query-side decomposition is designed to shine; a reversal would bound the paper's claim that document-side expansion is the better complement to verification.

Watch

Extended reading notes

Core claim

The central claim is that Table-of-Contents-guided page retrieval — a document-side retrieval channel that extracts headings from visual formatting or PDF bookmarks, embeds them as a parallel index, and loads full pages for matched headings — improves RAG answer quality on enterprise documents. In a 2³ factorial experiment, ToC is a significant main effect (d = +0.41, p = 0.031) with no interactions, indicating additivity with Section Expansion and Completeness Check. Full-page loading improves completeness (+0.20) and accuracy (+0.11) rather than adding noise. In head-to-head comparison, ToC+CC outperforms AS+CC (d = +0.32, p = 0.036), and 20% of citations come from ToC-loaded pages while T

Load-bearing premise

The scoring rubrics are reference answers generated with all features enabled, including ToC; if those references contain content retrievable only through the ToC channel, every ToC-off configuration is judged against a standard that already includes ToC's unique output, which would inflate the main effect and pairwise comparisons.

Editorial extensions

If this is right

  • Deploying ToC+CC is the recommended configuration: it achieves the highest composite quality in both experiments, and ToC alone gives d = +0.41 at near-zero latency cost.
  • ToC's gains concentrate in completeness and usefulness (+0.40 each) with smaller accuracy gains, and full-page loading does not dilute answers as coarser granularity might.
  • The additive structure (no significant interactions) means document-side and answer-side enhancements are independent levers that can be combined without tuning.
  • Quality gains are directionally larger on longer documents (up to +1.50 on a 118-page manual), suggesting the value of structure-aware retrieval grows with document complexity.
  • Default parameters are near-optimal: no parameter in a 480-condition sweep reaches significance, so ToC can be deployed without corpus-specific tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the complementarity holds broadly, heading-aware query decomposition — using the heading index to shape sub-queries — could yield further gains on multi-section questions, which the paper identifies as the hardest query type.
  • The layout-analysis extraction could be extended to a hybrid strategy that falls back to LLM-generated headings only for unformatted texts, preserving ToC's cost advantage on most documents while covering edge cases.
  • The evidence that ToC fires on 93% of queries and contributes 20% of citations suggests that structural retrieval may also reduce downstream verification load, a cost interaction the paper does not directly measure.
  • For long-context systems operating near window limits, ToC's ability to deliver coherent sections in fewer tokens than scattered chunks could be a bigger practical win than the quality scores alone indicate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces ToC-guided page retrieval, a document-side RAG enhancement that extracts headings from PDF bookmarks or visual layout without LLM calls, indexes them separately, and loads full pages for matched headings. It reports four experiments on an 8-document enterprise corpus (1,280 conditions): (1) a significant main effect of ToC on answer quality (d = +0.41, ANOVA p = 0.031, η²p = 0.012), with the largest gains on completeness and usefulness; (2) a head-to-head advantage of ToC+CC over AS+CC (d = +0.32, p = 0.036); (3) a 20% citation share from ToC-loaded pages; (4) a directionally positive but non-significant document-length trend (p = 0.52); and (5) a parameter sensitivity analysis finding no significant effects (all p > 0.38). The paper concludes that document-side, query-side, and answer-side enhancements are complementary and additive.

Significance. If the empirical claims held, the contribution would be useful: a zero-LLM-cost heading extraction method, a parallel-index architecture that avoids score blending, and factorial evidence on the interaction of retrieval-side enhancements. The paper is transparent in several respects: prompts, query sets, configuration matrices, parameters, and statistical procedures are disclosed; effect sizes come with bootstrap confidence intervals; multiple comparisons are FDR-corrected; and three of the eight source documents are public. However, the evaluation methodology has load-bearing problems: reference answers are generated with all features enabled (including ToC) and then used as the scoring rubric, the three-way ANOVA ignores the repeated-measures structure of the data, and several reported p-values are inconsistent with their confidence intervals. These issues currently prevent the paper from establishing its central empirical claims.

major comments (3)
  1. [§5.2, App. E, App. D.4] The scoring is circular in a treatment-dependent way. Reference answers are generated with all features enabled (SE+ToC+AS+CC) and manually verified only for factual correctness. The scoring prompt (App. D.4) then defines completeness as 'what fraction of the reference answer’s content the candidate covers.' Because ToC adds 2.9 pages per query and contributes 20% of citations, the reference may contain correct but non-essential facts that the ToC channel surfaces and that a human-authored minimal answer would omit. ToC-off candidates are penalized for missing those facts, biasing the ToC main effect (Table 5, d = +0.41) and the ToC+CC vs AS+CC contrast (Table 8, d = +0.32). Manual verification checks correctness, not necessity per query; Limitation 5 acknowledges LLM-judge bias but not this reference-generation circularity. The paper provides no test that the reference content is obtain
  2. [§6.1, App. E] The three-way ANOVA treats each of the 384 condition scores as independent, but Experiment 1 is a repeated-measures design: the same 24 queries and 2 models are measured under all 8 configurations. The paired Wilcoxon tests used elsewhere show the paper is aware of pairing, so the ANOVA's F statistics (e.g., ToC main effect F = 4.69, p = 0.031, η²p = 0.012; no-interaction claim p = 0.317) are computed with inflated degrees of freedom and ignore query/model correlation. Reanalysis with a repeated-measures ANOVA or a mixed-effects model with query and model as random effects is required. This is load-bearing for the main-effect and additivity claims.
  3. [§6.1, Table 5; §6.4, Table 8] There are internal inconsistencies between bootstrap confidence intervals and p-values. ToC vs Baseline has d = +0.41 with 95% CI [+0.00, +0.87] and p < 0.001; a two-sided 95% CI whose lower bound is exactly zero is not compatible with p < 0.001. Similarly, ToC+CC vs ToC ([−0.17, +0.65], p = 0.016) and ToC+CC vs AS+CC ([−0.08, +0.77], p = 0.036) show CIs that include zero with quite small p-values. Report the exact Wilcoxon statistics and explain the discrepancy, or correct the p-values. This matters because the headline effect sizes and their claimed significance are central to the paper.
minor comments (6)
  1. [§6.5, Table 9] Experiment 3 is explicitly underpowered (n = 8, p = 0.52). The paper is appropriately hedged in the text, but the abstract's phrase 'gains are directionally larger on longer documents' should be marked as exploratory rather than as a finding.
  2. [§6.6] The sensitivity conclusion that 'defaults are near-optimal' is stronger than the evidence: 8 queries and a single model give low power; p > 0.38 only means no effect was detected. Suggest softening to 'no evidence of a parameter effect in this sample.'
  3. [§4.3, Table 1] The latency row is formatted confusingly ('60s total — 68s total 177s total'); align the columns with the configuration labels so that per-method values are readable.
  4. [References] The Barnett et al. reference contains a typo ('Brber'); check the author spelling.
  5. [§6.2, Table 6] Indicate whether the p-values for query-type subgroups were FDR-corrected; as reported they appear to be raw p-values.
  6. [§6.3] The 20% citation share is descriptive and may reflect the extra pages added by ToC; consider normalizing by token contribution before interpreting it as evidence of channel quality.

Circularity Check

1 steps flagged · score 6.0 of 10

Evaluation rubric is generated with ToC enabled, inflating ToC's measured effect by construction

  1. self definitional [Section 5.2 / Appendix E / Appendix D.4 (scoring prompt)]
    "Reference answers were generated by querying the RAG system with all features enabled (SE + ToC + AS + CC) and manually verified for factual correctness against source documents. // Scoring prompt (D.4): "completeness: What fraction of the reference answer's content does the candidate cover?""

    Every ToC-OFF configuration is scored for completeness as the fraction of a reference that was itself produced with ToC ON. If the ToC channel contributed any content (the paper reports 250 ToC citations, 20.4% of citations), that content is included in the reference and is required for full completeness even when it is not essential to answer the query. The ToC main effect and the ToC+CC vs AS+CC contrast therefore compare configurations against a rubric that already contains the treatment's output; the measured d=+0.41 and d=+0.32 are partly an artifact of the reference-construction rule. Manual verification checks factual correctness, not whether each sentence is a necessary answer facet, so it does not remove the bias. Limitation 5 acknowledges LLM-judge bias but not this reference-gen

full rationale

The only load-bearing circular step I can exhibit is the evaluation-reference construction. The paper's central effect sizes are not derived from an independent gold standard: the rubric is generated with ToC enabled and then used to score ToC-off candidates. This is a self-definitional benchmark: completeness is defined as coverage of a reference that contains ToC-channel content, so the ToC main effect and the ToC+CC contrast are partly forced by the rubric. I do not see a derivation-level circularity in the retrieval algorithm itself: ToC extraction is rule-based, the cosine heading matching is independent of the answer scores, and the parameter sensitivity analysis is not a prediction. Self-citations are not load-bearing. Because the bias affects the central empirical claim but does not fully determine the sign or magnitude (a real ToC effect could survive a clean rubric), the score is 6 rather than 8.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the heading-extraction machinery and the evaluation design. The default ToC parameters are hand-chosen but sensitivity analysis mitigates their importance. The more serious dependencies are the unvalidated heading-relevance assumption and the evaluation circularity.

free parameters (4)
  • toc_page_budget = 5 (default)
    Maximum number of ToC-loaded pages per query; chosen by hand, not fitted; sensitivity analysis shows no significant effect.
  • toc_min_similarity = 0.25/0.3 (default, stated inconsistently)
    Heading match threshold; chosen by hand; sensitivity analysis no effect.
  • toc_match_top_k = 3 (default)
    Number of matched headings; chosen by hand; trend with top_k non-significant.
  • layout_analysis_thresholds = not disclosed
    Font-size/weight/spacing thresholds used by the heading extractor; implicit free parameters that determine extracted headings.
assumptions (4)
  • domain assumption Documents exhibit detectable visual heading structure
    Section 4.1 and 7.2: the method relies on PDF bookmarks or typographic cues; internal documents all have such structure, but the method fails on plain-text or unformatted documents.
  • domain assumption Heading embeddings are informative for query relevance
    Section 4.1: queries are matched to headings via cosine similarity; there is no independent validation that heading text is the right level of relevance.
  • ad hoc to paper LLM-as-judge with system-generated reference answers is a valid measure of quality
    Section 5.2 and Appendix D.4: scoring uses Claude Opus 4.6 against references generated with all features enabled; this is a choice specific to this paper and is not validated against human judgments.
  • ad hoc to paper Repeated conditions in the ANOVA are independent
    Section 6.1: three-way ANOVA treats each configuration-query-model as an independent observation, ignoring the repeated-measures structure of the same 24 queries across configurations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Three Sides of Retrieval: Factorial Evidence for Document-Side, Query-Side, and Answer-Side Complementarity in RAG." pith.science (2026). https://pith.science/paper/PDYQU3MI

@misc{pith2026260724781,
  author       = {Pith},
  title        = {Pith review of: Three Sides of Retrieval: Factorial Evidence for Document-Side, Query-Side, and Answer-Side Complementarity in RAG},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PDYQU3MI}},
  note         = {Machine review of arXiv:2607.24781}
}
read the original abstract

RAG systems rely on chunking, which destroys structural information in documents. Existing heading-based retrieval (Jeong et al., 2025) requires multiple LLM calls per document and returns sub-chunks within matched sections. We introduce ToC-guided page retrieval, which infers headings from visual formatting without LLM calls, embeds them as a parallel index, and loads full page sections. Across 1,280 conditions on 8 enterprise documents (5 to 195 pages), we find: (1) ToC is a significant main effect on answer quality (d = +0.41, p = 0.031), with the largest gains in completeness (+0.40) and usefulness (+0.40); (2) combined with answer-side verification, it outperforms query-side decomposition + verification (d = +0.32, p = 0.036); (3) ToC contributes 20% of citations despite adding only 2.9 pages per query; (4) gains are directionally larger on longer documents (up to +1.50 on 118 pages), though the trend does not reach significance with 8 documents; and (5) a 480-condition sensitivity analysis finds no significant parameter effects (all p > 0.38), confirming defaults are near-optimal. The contribution is both methodological (a new zero-LLM-cost retrieval algorithm) and empirical: factorial evidence that document-side, query-side, and answer-side enhancements are complementary, a three-way interaction not previously studied.

Figures

Figures reproduced from arXiv: 2607.24781 by the authors.

Figure 2
Figure 2. ToC effect size (Cohen's d, ON vs OFF) by query type. Positive values indicate ToC improvement. Multi-section (d=+0.33) and simple (d=+0.31) queries show the largest effects. 6.3 Retrieval-Level Analysis To understand how ToC affects the retrieval pipeline, not just answer quality, Table 6b compares retrieval metrics between ToC-ON and ToC-OFF configurations [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Retrieval contribution breakdown. ToC contributes 20% of citations while firing on 93% of queries. 6.4 Experiment 2: ToC+CC Outperforms AS+CC [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figure 4
Figure 4. Head-to-head composite quality (Exp 2). ToC+CC outperforms AS+CC (d=+0.32, p=0.036) [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Quality vs latency trade-off (Exp 2). ToC achieves high quality at low latency; CC adds quality at substantial latency cost. 6.5 Experiment 3: Document-Length Scaling [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: ToC quality gain by document length (Exp 3). Largest gain on UN Staff Rules (118pp, +1.50). 6.6 Experiment 4: Parameter Sensitivity [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Parameter sensitivity heatmaps (Exp 4). Composite quality by page budget and min similarity for each top_k level. Uniform shading confirms parameter robustness [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Marginal parameter effects (Exp 4). Error bars show standard error. Dashed line shows overall mean (4.35). No parameter reaches significance. 7. Discussion 7.1 The Three-Sided Complementarity Our results reveal a complementarity structure across the three RAG enhanceme…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

2 extracted references · 1 linked inside Pith

  1. [2]

    all types of leave

    Retrieval granularity: full pages vs. within-section chunks. DocsRay matches headings but then retrieves individual chunks within matched sections, preserving section matching at the cost of re-fragmenting the very content that section-level retrieval was meant to keep intact. We load the entire page section, preserving cross-paragraph context: definition...

    2025

  2. [3]

    complexity

    Enterprise corpus. The 8-document corpus consists of institutional documents where layout-based heading extraction succeeds reliably. While the layout analysis approach covers a broad class of documents with visual heading formatting, ToC may be less effective on scanned PDFs without OCR, completely unformatted raw text, or chat logs. These are cases wher...

    arXiv 2025

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.