{"id":"f1051a13-df20-4466-83a9-572cbeecbaa9","arxiv_id":"2601.18116","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A budget-controlled two-path hierarchical retrieval method reports near-full-context accuracy on multi-hop QA with up to 94% token savings, but the reported numbers are internally inconsistent and no artifacts are released.","lead":"This paper introduces FABLE, a retrieval system that builds a summary tree for each document and lets the LLM zoom between coarse summaries and fine chunks at query time, using a fraction of the tokens needed for full-context reading. On multi-hop question-answering benchmarks it reports accuracy close to or better than full-context LLMs using up to 94% fewer tokens, though the results currently ship without code, data, or error bars.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline efficiency claim rests on an uncontrolled cross-model comparison and an internally inconsistent same-generator ablation.","rationale":"I focused on the central empirical claim: structured evidence allocation under a fixed budget matches or beats full-context reading. That claim requires a controlled comparison, and the paper's headline comparison is cross-model. The only same-generator ablation that would rescue it is internally inconsistent: 92.5% appears alongside a 65.7% 'upper bound' for the same generator. This is not an external-consensus disagreement or a style complaint; it is an internal inconsistency in the evidence presented. The reader's weakest_assumption — that LLM-generated internal nodes are lossless proxies for leaf content — is a real algorithmic risk, but it is not the most immediate blocker: even if summaries are faithful, the reported numbers do not yet cleanly support the efficiency claim. I would not move to reject because the method is plausible and the inconsistency is addressable with released artifacts and a re-run; conditional acceptance remains right. I did not treat the missing code as the primary concern, though releasing it would resolve the proposed test.","tokens_in":14919,"tokens_out":7238,"duration_ms":81968,"concrete_test":"Run a single three-way reproduction on DragonBall/DragBalance with the same evaluation script, same judge, and the same Qwen3-32B generator: (a) full concatenated documents, (b) FABLE(docs) at 4K budget, (c) llm-chunks at 8K budget. Check whether (b) reproduces ≈92.5% while (a) ≈65.7% and (c) ≈64.5%. If (b) does not reproduce, the controlled efficiency gain is an artifact; if it does reproduce, then the 'upper bound' framing in Sec. 5.3.1 is wrong, and the central comparison should be rerun with matched generators before the headline claim is accepted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline efficiency claim — FABLE at 31K tokens beats Gemini-2.5-Pro full-context 92.07% vs 91.05% — is not established by a controlled comparison. Table 1's FABLE(docs) row uses DeepSeek-V3.2 for DragonBall (Sec. 4.2), while the full-context baselines are Qwen3-32B and Gemini models, so the gap could reflect generator strength rather than retrieval architecture. The only same-generator control is Sec. 5.3, which claims FABLE(docs) at 4K reaches 92.5% using Qwen3-32B. But Sec. 5.3.1 states the same generator with full document access has a 65.7% 'upper bound' and llm-chunks at 8K reaches 64.5%. Under the same metric and generator, a retrieval architecture cannot be described as approaching a 65.7% full-document upper bound while the same section reports 92.5% with the same generator. This internal inconsistency means the central efficiency demonstration is not currently coherent. If the 92.5% ablation is wrong, the claim that FABLE beats full-context under a fixed generator collapses; if the 65.7% figure is not truly an upper bound, the efficiency interpretation and the 'approaching upper bound' language in Sec. 5.3.1 need correction.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FABLE, a retrieval framework that builds a per-document hierarchical 'semantic forest' offline: documents are LLM-chunked, an LLM generates a tree with titles, summaries, and leaf chunks, and embeddings are indexed. At query time, retrieval runs two paths at document and node levels: LLM-guided traversal over non-leaf nodes and vector/structural propagation (TreeExpansion with fixed equal weights), with a budget controller that decides whether to return whole documents or node-level evidence. Claims: on DragonBall, FABLE(docs) reaches 92.07% completeness and low hallucination, surpassing Gemini-2.5-Pro full-context (91.05%) while using 31K tokens vs 517K (94% reduction); on HotpotQA and 2Wiki it is competitive or state-of-the-art among retrieval baselines; on BrowseComp-plus it improves agent accuracy and recall. Ablations attribute gains to hierarchy and bi-path fusion rather than semantic chunking alone.","tokens_in":15111,"tokens_out":9351,"duration_ms":84297,"significance":"Conditional significance is high. If the results survive scrutiny, FABLE would demonstrate that query-time token budgets, not context length, are the bottleneck for multi-document reasoning, and that LLM-built hierarchies plus bi-path retrieval can beat indiscriminate full-context inference at a fraction of the cost. The design is clearly specified, and the TreeExpansion score uses intentionally un-tuned uniform 1/3 weights, which is a genuine strength: the reported gains are not from overfitting weights to the benchmark. The evaluation spans synthetic QA, multi-hop QA, and an agent setting, giving breadth. However, the paper does not release code or data, and the central efficiency claim is currently supported only by cross-model comparisons and an internally inconsistent ablation. The contribution is therefore promising but not yet established.","major_comments":[{"comment":"The headline claim that FABLE matches full-context LLM performance (92.07% vs Gemini-2.5-Pro's 91.05%) is an uncontrolled cross-model comparison. Per §4.2, DragonBall uses DeepSeek-V3.2 throughout, so FABLE(docs) is generated by DeepSeek-V3.2, while the full-context baselines in Table 1 are Qwen3-32B and Gemini models. The 1.02-point gap could be entirely due to generator strength. The statement in §4.2 that 'relative comparisons are always performed under identical LLM settings' is not honored for this headline comparison. Please report a same-generator full-context DeepSeek-V3.2 row and derive the token-reduction claim from that controlled comparison.","section":"§1 bullet; Table 1; §4.2"},{"comment":"The ablation section states that all ablation experiments use Qwen3-32B for LLM generation. §5.3.1 reports llm-chunks at 8K reaching 64.5% and explicitly calls 65.7% 'the upper bound of qwen3-32B with full document access'; this 65.7% matches Table 1's Qwen3-32B full-context completeness 65.73. Yet §5.3.2 reports FABLE(docs) at 4K reaching 92.5% with the same Qwen3-32B generator. A retrieval method cannot exceed the full-document upper bound on the same dataset and metric. Either 92.5% is wrong, 65.7% is not an upper bound, or §5.3.2 uses a different generator/split than §5.3.1. This is not a presentation nit: the paper's central efficiency claim ('retrieval beats full context') depends on reconciling these numbers.","section":"§5.3.1 vs §5.3.2"},{"comment":"The contribution bullet reports '93.65% completeness' while Table 1 lists FABLE(docs) at 92.07%, with both sharing the same hallucination (5.37%) and irrelevance (2.52%) values. The paper therefore contains two different headline results for the same configuration. The authors should correct the bullet or the table, and all derived claims (e.g., 'surpassing Gemini' and the EM deltas) should be recomputed from the corrected value.","section":"§1 Contributions; Table 1"},{"comment":"The retrieval architecture assumes that LLM-generated internal nodes (toc_path + summary) are faithful, lossless proxies for descendant leaf content: both Eq. (5) embedding retrieval and LLM navigation operate on these summaries. If a summary omits, distorts, or hallucinates evidence present only in a descendant chunk, no downstream step can recover it. The paper provides no analysis of summary fidelity or error propagation. This is especially relevant for the batch-wise construction in Eqs. (3)-(4), which is invoked for documents exceeding the LLM context window; the claim that merging partial trees 'maintains cross-part semantic coherence' is asserted but not validated. Since the efficiency advantage over long-context inference depends on compressing long documents into bounded summaries, please add an analysis or ablation measuring how often gold evidence is absent from internal nodes","section":"§3.1, Eqs. (3)-(5)"},{"comment":"On DragonBall, Completeness, Hallucination, and Irrelevance are evaluated using DeepSeek-V3.2 as the LLM judge. Per §4.2, DeepSeek-V3.2 is also the backbone that constructs the semantic forests and serves as the generator for FABLE's rows. Using the same model both to build the index and to grade the output creates a potential circularity: the judge may favor the structure that the same model produced. Please report agreement with an alternative judge (e.g., a different model or human annotation) or justify why this is not a confound.","section":"§4.3"}],"minor_comments":[{"comment":"The text says scores are computed 'via Eq. 4', but the composite score is defined in Eq. (7). Eq. (4) is the partial-tree merge operation. Please correct the cross-reference.","section":"§3.2, TreeExpansion Details"},{"comment":"Gemini-2.5-Pro's completeness is cited as 91.1% in the ablation text but 91.05% in Table 1. Rounding is fine, but the values should be consistent across the paper.","section":"§5.3.2, §5.3.3"},{"comment":"References [13] and [14] are the same paper (Ho et al., 2020, COLING) and should be merged.","section":"References"},{"comment":"The high-level claims in §5.3.4 (e.g., FABLE(Nodes) average 97.7%, +22.8, +35.5) are not backed by a table or error bars in the figures. Please provide the underlying numbers or a small table for reproducibility.","section":"§5.3.4, Figures 4-5"},{"comment":"The paper states that code, prompts, and constructed forests 'will be released upon acceptance.' For a reproducibility-oriented evaluation, making the semantic forests or at least a hosted index available with the submission would substantially strengthen the reader's ability to verify the claims.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The central inconsistency between §5.3.1 and §5.3.2 is severe enough that the authors should be asked to supply the raw per-budget completeness values and the exact dataset split used in the ablation before re-review. If the 92.5% figure is erroneous, the paper's headline claim should be downgraded. A same-generator full-context comparison is mandatory for the 94% token-reduction claim. The paper may be publishable once these issues are resolved and the controlled comparison is presented."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the one-line version: FABLE is a plausible and well-motivated retrieval architecture, but the paper as written has an internal contradiction in its central ablation, and until that is fixed the efficiency claims cannot be taken at face value.\n\nWhat's actually new: the combination of LLM-built hierarchical forests, query-conditioned bi-path traversal (LLM-guided navigation plus structure-aware propagation), and budget-adaptive routing. Individually those pieces exist in RAPTOR, TreeRAG, HippoRAG2, but the fusion is not in the cited prior work, and the systematic ablations across domains and languages (Figures 4-5) are a real strength. The BrowseComp-plus agent results are also a useful data point: swapping in FABLE improves accuracy and recall over the Qwen3-Embed-8B retriever while keeping the agent LLM fixed.\n\nThe soft spots are not minor. Section 5.3.1 states that Qwen3-32B with full document access has a 65.7% completeness upper bound. Section 5.3.2 then reports FABLE(docs) at 4K tokens reaches 92.5% completeness using the same generator. Those two numbers cannot both be right under the same metric. The manuscript even uses 92.5% to argue FABLE surpasses Gemini-2.5-Pro (91.1%), while the abstract and intro cite 92.07% vs 91.05%. The 93.65% in the contributions bullet disagrees with Table 1's 92.07%. Also, the headline comparison to full-context inference is not controlled: FABLE(docs) on DragonBall uses DeepSeek-V3.2, while the full-context baselines are Qwen3-32B and Gemini models. So part of the gap could simply be generator strength. There are no error bars, no code or data released, and Table 1 has several 'not reported' cells including the TreeRAG row.\n\nOn the positive side, the method is not fundamentally broken; the contradictions look fixable with a careful re-run and artifact release. The uniform 1/3 weights in TreeExpansion are explicitly untuned, which is honest, and the paper is transparent about its hyperparameters.\n\nBottom line: this deserves a serious referee only after the authors reconcile the numbers and release the code and forests. As submitted, the central efficiency claim is unsupported. If the same-generator ablation holds up after correction, the idea is a solid contribution to structured RAG.","headline":"FABLE is a plausible new retrieval architecture, but the paper's central efficiency claim is undermined by an internal contradiction between the 65.7% full-document upper bound and the 92.5% ablation result for the same generator.","tokens_in":15760,"tokens_out":2913,"would_cite":false,"duration_ms":29783,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Multi-document reasoning is budget-constrained, not context-constrained, and a hierarchical bi-path retriever can match full-context LLMs using 94% fewer tokens.","keywords":["retrieval-augmented generation","multi-document reasoning","hierarchical semantic indexing","LLM-guided navigation","budget-constrained evidence allocation","long-context LLM","multi-hop question answering","tree-based retrieval"],"falsifier":"Build a synthetic corpus where each gold answer sits in a single known chunk, then generate summaries that systematically omit that chunk's key fact while summarizing the rest; if the framework's completeness on those queries at its claimed 8K budget collapses toward zero while full-context inference still answers, the central proxy assumption is refuted. Simpler check: compare retrieval via internal-node embeddings against retrieval over leaf chunks and measure how often the gold evidence is found only through the leaf.","tokens_in":14668,"feed_emoji":"🌲","tokens_out":11652,"duration_ms":112096,"temperature":0.7,"pith_summary":"This paper is trying to establish that multi-document reasoning is limited not by how much text a model can read, but by how a fixed query-time evidence budget is allocated across documents and semantic granularities. It argues that full-context inference is non-selective and expensive, while flat chunk retrieval returns locally relevant passages that lack the structure needed for cross-document synthesis. In response, it proposes a framework that pre-builds LLM-generated hierarchical semantic indexes and, at query time, navigates them coarse-to-fine through two complementary paths under an explicit token budget. If the reported results hold, the framework matches or beats a much larger full-context model while using about six percent of the tokens, and the efficiency gain comes from the hierarchy and bi-path design rather than from semantic chunking alone. A sympathetic reader would care because this reframes the retrieval-versus-long-context debate as an allocation problem with measurable cost savings.","feed_headline":"Structured retrieval beats full-context LLM at 94% fewer tokens","feed_subtitle":"A hierarchical index plus bi-path retrieval answers multi-doc queries as well as a 517K-token prompt in 31K tokens.","key_machinery":"Key machinery is the LLM-built semantic forest: per-document trees with typed internal nodes (ToC-style title + summary) and semantic-chunk leaves, embedded as title-path-plus-summary vectors. Query-time retrieval has two paths: depth-limited LLM-guided selection/navigation over high-level abstractions, and TreeExpansion, which ranks nodes by a three-signal score—direct similarity with depth decay, ancestor-inherited relevance, and child-aggregated relevance—then greedily fills the budget. A budget-adaptive router stops at document level when coarse results fit, otherwise descends to node level; fusion dedupes ancestor/descendant pairs and orders chunks positionally, LLM picks first. The mec","core_discovery":"Central claim: multi-document reasoning is budget-constrained, not context-constrained—under a fixed query-time token budget, coarse-to-fine retrieval over LLM-built semantic hierarchies can match or beat full-context inference. The framework builds per-document trees (internal nodes = ToC title + summary; leaves = semantic chunks) and retrieves via two complementary paths: depth-limited LLM-guided selection and structure-aware propagation down tree edges, with budget-adaptive routing between document and node granularity. Reported: 92.07% completeness at 8K generation tokens vs 91.05% for a 517K-token full-context run (94% reduction); best retrieval-based EM/F1 on 2Wiki; +22.14 accuracy on","pith_inferences":["Beyond the paper: the 94% token savings become real cost savings only if the LLM doing selection/navigation is cheaper than the full-context feed; otherwise the economic win is smaller than the token reduction suggests.","Because internal titles/summaries are the only view of document content, a summarizer that systematically omits the exact fact a question needs would defeat the framework—a stress test for the lossless-proxy premise.","The structural scoring weights are uniform and untuned; the paper's own finding that LLM-only beats fusion at 1K tokens suggests query-adaptive weighting of the two paths is a natural improvement it does not explore.","The position-preserving ordering of retrieved chunks (LLM picks before expansion picks, both in document order) implies evidence ordering, not just selection, contributes to the gains—connecting to known positional-bias effects in long-context models."],"forward_implications":["If the reported 94% token reduction holds, long-context inference is not a substitute for structured retrieval on multi-document reasoning; retrieval architecture—not context size—becomes the dominant lever for faithful answers.","Budget-adaptive routing means per-query token cost scales with evidence complexity rather than corpus size: when document-level evidence fits the budget, the system stops before fine-grained traversal.","The bi-path ablations show LLM reasoning and structural propagation have complementary strengths—LLM-only wins at very tight budgets, fusion wins at moderate budgets—so the best path depends on the budget tier.","On BrowseComp-plus, replacing the retriever alone raised agent accuracy by 22 points with the same agent LLM, indicating retriever quality is a first-order constraint on agent performance.","The paper itself states the advantages diminish on highly unstructured corpora or queries that rely solely on keyword matching."],"fun_headline_variants":["BEAR matches full-context with 94% fewer tokens","94% token cut: BEAR beats full-context on multi-doc QA","Budgeted evidence allocation: 94% less input, same accuracy","Coarse-to-fine retrieval outperforms full-context at 31K tokens"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Everything rests on the assumption that an LLM-generated title-plus-summary is a lossless proxy for the content in its descendant chunks: if a summary omits, distorts, or invents a fact the answer depends on, both the LLM navigation path and the summary-embedding path will miss it, and the saving disappears for that query.","fun_headline_variants_meta":{"raw":{"variants":["BEAR matches full-context with 94% fewer tokens","94% token cut: BEAR beats full-context on multi-doc QA","Budgeted evidence allocation: 94% less input, same accuracy","Coarse-to-fine retrieval outperforms full-context at 31K tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000227,"raw_usage":{"total_tokens":1318,"prompt_tokens":764,"completion_tokens":554,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":474}},"tokens_in":508,"tokens_out":554,"duration_ms":6845,"temperature":1.0,"reasoning_tokens":474,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T08:05:10.984631+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a synthetic corpus where each gold answer sits in a single known chunk, then generate summaries that systematically omit that chunk's key fact while summarizing the rest; if the framework's completeness on those queries at its claimed 8K budget collapses toward zero while full-context inference still answers, the central proxy assumption is refuted. Simpler check: compare retrieval via internal-node embeddings against retrieval over leaf chunks and measure how often the gold evidence is found only through the leaf.","supporting_citations":[],"review_version":1}