{"id":"29efa557-fc0c-4d4f-ae1d-51819800a706","arxiv_id":"2505.06569","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"MacRAG retrieves small slices of compressed summaries, scales up candidate chunks, and merges neighboring chunks, achieving higher average F1 than RAPTOR and LongRAG on three LongBench multi-hop QA datasets.","lead":"MacRAG is a new retrieval system that compresses long documents into summaries and small overlapping slices, then recombines the most relevant pieces into a compact query-specific context. The authors report average F1 gains over RAPTOR and LongRAG on three multi-hop question-answering benchmarks with Llama-3.1-8B, Gemini-1.5-pro, and GPT-4o.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported evidence does not establish 'consistently surpasses': several configurations in Tables 3/6/7 are negative for MacRAG, and no uncertainty is quantified; paired significance testing is needed before accepting the central claim.","rationale":"Reader and I agree the paper has real strengths: matched hyperparameters, multiple LLMs and rerankers, seven generation modes, and a shared code link. I do not see a fatal flaw in the architecture. The compression-fidelity concern raised by the reader is legitimate: Section 3.1.2's abstractive summarization is lossy, and Section 4.2 reports only a 7.2% precision gain with no recall/fidelity measurement. However, it is not the most load-bearing issue for the central claim, because MacRAG's h-hop merging and large k1 can partly recover facts even if a summary omits them; the end-to-end claim could survive imperfect compression. The least secure condition is instead the empirical claim itself: the paper asks us to believe 'consistently surpasses,' but its own tables contain multiple negative cells and there is no uncertainty quantification anywhere. A paired significance test directly targets that condition. If the test shows the negatives are noise and the average gains are significant, the claim survives; if not, the claim must be softened. This does not change the overall verdict: CONDITIONAL remains appropriate pending that analysis.","tokens_in":20064,"tokens_out":8565,"duration_ms":91057,"concrete_test":"Run a paired, question-level evaluation of MacRAG versus LongRAG for every configuration in Tables 3, 6, and 7 (same k1, k2, reranker, and generation mode), using identical decoding settings. For each question, record the F1 delta, then compute paired bootstrap 95% confidence intervals and a paired permutation test over questions for each cell. If the majority of intervals include zero, or if the number of significantly negative cells is comparable to chance, the abstract should be revised from 'consistently surpasses' to 'improves on average in these evaluations,' and the current CONDITIONAL verdict remains appropriate pending significance and re-analysis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is an empirical one: MacRAG consistently surpasses LongRAG and RAPTOR across datasets, models, and single-/multi-step generation schemes. The evidence for 'consistently' is the set of F1 tables, and those tables are internally mixed. In Table 3 (GPT-4o, marco-miniLM), 2WikiMultihopQA R&B is 59.00 vs LongRAG 59.97 (-0.97); in Table 6 (Gemini-1.5-pro), HotpotQA R&B is 63.02 vs 63.59 (-0.57) and 2Wiki R&B is 58.38 vs 60.13 (-1.75); Table 7 adds further negative cells (e.g., Musique R&L -1.40 at k2=7/bge, HotpotQA R&B -0.69 at k2=12/bge). No error bars, confidence intervals, or paired tests are reported for any cell. The ablation study deepens the problem: removing Propagation&Merging improves Musique Full_E&F by +2.17 F1 (52.26 vs 50.09), and removing Scaling-up improves several other cells, yet Section 4.2 states that removing these components causes drops. So the strongest reading of the abstract—consistent superiority in every reported regime—is contradicted by the paper's own numbers. The weaker claim of positive average gains may well be true, but the reported evidence does not establish it with enough precision to support 'consistently.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"MacRAG is a hierarchical RAG framework that builds an offline index by compressing each chunk into a summary, slicing the summary into fine-grained units, and then at query time retrieving slices, mapping them to parent chunks, reranking with a cross-encoder, scaling up candidate chunks, selecting top documents, and merging h-hop neighbor chunks into a bounded final context. The paper evaluates MacRAG on LongBench versions of HotpotQA, 2WikiMultihopQA, and Musique using Llama-3.1-8B, Gemini-1.5-pro, and GPT-4o under seven generation modes, comparing against RAPTOR and LongRAG. The headline claim is that MacRAG consistently surpasses baseline RAG pipelines in single- and multi-step generation.","tokens_in":20339,"tokens_out":6283,"duration_ms":57460,"significance":"If substantiated, MacRAG would be a useful modular contribution: it is simple, gives positive average F1 gains over LongRAG in most settings, keeps the final context bounded, and ships with released code and matched reranker/hyperparameter controls that mitigate the usual comparison confounds. The gains on Musique and 2WikiMultihopQA with stronger LLMs are particularly encouraging. However, the paper's central claim of 'consistent' superiority is stronger than the evidence: there are multiple negative cells, no paired significance testing, and an ablation narrative that is contradicted by parts of the ablation table. With corrected claims and uncertainty quantification, this can become a solid empirical study, but in its current form the main conclusion is overstated.","major_comments":[{"comment":"The ablation narrative is not supported by the reported numbers. The text states that 'each component contributes to cumulative performance gains' and that removing Propagation&Merging causes drops of up to about 5 F1, but Table 3 shows improvements from removing this component in several cells, e.g., Musique Full_E&F (52.26 vs 50.09, +2.17), 2Wiki R&B_E&F (72.43 vs 71.40, +1.03), and HotpotQA Fil (68.06 vs 65.44, +2.62). This directly contradicts the claim that removing propagation and merging hurts performance and needs to be reconciled, either by revising the text or by explaining why these cells behave differently.","section":"Section 4.2, Table 3"},{"comment":"The claim that MacRAG 'consistently surpasses' LongRAG and RAPTOR is contradicted by the paper's own results. For example, Table 6 shows MacRAG behind LongRAG on HotpotQA R&B (63.02 vs 63.59) and on 2Wiki R&B (58.38 vs 60.13) with Gemini-1.5-pro, and Table 1 shows MacRAG at 44.87 F1 on 2Wiki with Llama-3.1-8B, below the plain RAG-with-reranking baseline's 46.33. Table 7 adds further negative cells, such as HotpotQA R&B with bge-m3 at k2=7 (-0.08) and Musique R&L with bge-m3 at k2=7 (-1.40). No confidence intervals, error bars, or paired significance tests are reported for any of these comparisons, so the evidence does not support the strong wording in the abstract and Section 4.2. The authors should either provide significance testing, or soften the central claim to positive average gains with explicitly quantified variance.","section":"Abstract, Section 4.2, Tables 6 and 7"},{"comment":"The core retrieval assumption, that compressing each chunk into a summary and then retrieving only from slices of summaries preserves the facts needed for multi-hop reasoning, is never verified. If the summarizer drops a bridging entity or relation, the later parent-chunk and document expansions cannot recover it, because slice-level retrieval only sees summarized content. The preliminary 7.2% precision gain from summaries over raw slices reported in Section 4.2 does not measure recall or evidence fidelity. A direct evaluation of whether gold supporting evidence survives compression (e.g., passage-level or entity-level recall on the retrieved slices) is needed to establish that the compression step is not a hidden source of information loss.","section":"Section 3.1.2, Section 3.2.1, Section 4.2"},{"comment":"The sentence 'regardless of the choice of k2 or reranker, MacRAG maintains substantial advantages in all test settings in Table 5' is not accurate. Table 5 contains negative cells, including HotpotQA R&B with bge-m3 at k2=7 (-0.08) and Musique R&L with bge-m3 at k2=7 (-1.40). This internal inconsistency between the prose and the table must be fixed, and the robustness claim should be limited to the settings where it actually holds.","section":"Section 4.2, Table 5"}],"minor_comments":[{"comment":"The generation-mode naming is inconsistent: Tables 6 and 7 use 'R&B_Ext_Fil' while the text and Table 3 consistently use 'R&B_E&F' for what appears to be the same mode; unify the terminology.","section":"Section 3.3, Tables 6 and 7"},{"comment":"The gain notation in Table 1 is ambiguous because different rows compare against different baselines within the same table; add an explicit legend or baseline column so the reader can tell whether a parenthetical is relative to RAPTOR, LongRAG, or the plain RAG-with-reranking row.","section":"Table 1"},{"comment":"The caption describes the system as 'MacRAG+LongRAG' while all other tables and the text call it 'MacRAG'; clarify whether this is the same method or a distinct pipeline.","section":"Table 7, caption"},{"comment":"The text refers to 'Table 4 in Appendix A.1' but Table 4 appears in the main body, and the appendix reference is confusing; use consistent numbering and placement.","section":"Section 4.2, Appendix A.1"},{"comment":"On the x-axis of Figure 3, the value labeled 'max' is not defined; state what alpha value or selection criterion this corresponds to.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The core idea is reasonable and the controlled comparison setup is a strength, but the abstract and Section 4.2 overclaim consistency, the ablation section contradicts its own table, and the compression-fidelity question is unresolved. These are load-bearing issues for the paper's stated contributions, though they are fixable with a revised narrative, significance testing, and an evidence-fidelity analysis. I would not reject the work on the current evidence, but it needs a substantive revision before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a genuine systems contribution: it combines summary compression, fine-grained slicing, parent-chunk mapping, scaled document selection, and h-hop merging into one pipeline, and that specific composition is not in the prior work. The experiments are extensive—three datasets, three LLMs, two rerankers, two k2 values, seven generation modes—and they did the right thing by matching LongRAG's reported hyperparameters. The latency analysis is a nice addition, and the ablation study, whatever its interpretation, shows they are not hiding the per-configuration numbers.\n\nThe problem is the abstract says MacRAG 'consistently surpasses' baselines, and the paper's own tables don't support that. There are multiple negative cells: R&B is worse than LongRAG on 2Wiki with Llama-3.1-8B (-1.46), on HotpotQA with Gemini (-0.57), on 2Wiki with Gemini (-1.75), and on several bge-m3 and k2=12 settings in Table 7. The ablation in Table 3 is messier than the text admits: removing Propagation&Merging improves Musique Full_E&F by +2.17, and removing Scaling-up improves several cells, yet Section 4.2 says removing these components causes drops. That is a real internal inconsistency, not a nitpick. There are also no error bars, confidence intervals, or paired significance tests anywhere. Given those negative cells, the strong reading of 'consistently' cannot survive.\n\nThat said, the weaker claim—positive average gains over LongRAG and RAPTOR—is plausible and probably true. The averages in Tables 2, 5, and 6 are positive across nearly every row, and the gains on Musique are sizable. The circularity burden is low: they are comparing against external baselines with matched rerankers and hyperparameters, not tuning alpha and h against test results. The one assumption I'd flag is that summarization preserves the bridging facts needed for multi-hop reasoning; they report a 7.2% precision gain from summaries over raw slices but never measure summary fidelity or show that gold evidence survives compression. That is a legitimate weakness, though it can be addressed in revision.\n\nWho is this for? Researchers building hierarchical RAG systems and engineers looking for a modular retrieval recipe. It is not a paradigm shift, but it is a useful, carefully documented system. The code is promised but not verified, so reproducibility is plausible, not guaranteed.\n\nRecommendation: send it to peer review, but with a clear revision request. Soften 'consistently' to 'on average' or report per-configuration results honestly, add paired significance tests or at least standard errors, reconcile the ablation text with the table, and release the code. A serious referee should engage with this; the core pipeline is worth discussing even if the headline needs to be recalibrated.","headline":"A solid, well-engineered RAG pipeline with unusually broad evaluation, but the paper's own tables contradict the 'consistently surpasses' claim, and the missing significance testing makes the central result provisional.","tokens_in":21009,"tokens_out":1531,"would_cite":false,"duration_ms":17260,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MacRAG claims that starting retrieval at fine summary slices and expanding to parent chunks and neighboring text improves multi-hop RAG accuracy across three benchmarks and three LLMs.","keywords":["retrieval-augmented generation","multi-hop question answering","hierarchical retrieval","long-context language models","context compression","multi-scale retrieval","LongBench","adaptive context construction"],"falsifier":"Run MacRAG on HotpotQA, 2WikiMultihopQA, and Musique with the summarizer replaced by an identity function, retrieving raw slices instead of summarized slices while keeping chunking, slicing, reranking, and neighbor merging fixed; if F1 stays roughly the same, compression is not load-bearing. Alternatively, check whether all gold evidence passages referenced by the dataset questions survive the summarization step; if a large fraction of bridging entities or relations disappear from the summaries, the claimed gains cannot be attributed to preserving facts through compression.","tokens_in":19770,"feed_emoji":"🔍","tokens_out":9639,"duration_ms":83809,"temperature":0.7,"pith_summary":"MacRAG is a retrieval pipeline designed to fix a specific failure: standard RAG either retrieves chunks too coarse to be precise or too fine to carry the connecting facts needed for multi-hop questions. The paper claims that a hierarchical index, documents split into chunks, each chunk compressed into a summary, each summary sliced into small overlapping units, lets retrieval start at the finest level and then expand upward to parent chunks, neighboring chunks, and entire documents. This compress, slice, and scale-up order, the paper argues, gives both precision and coverage while keeping the final context bounded. Evaluated on LongBench versions of HotpotQA, 2WikiMultihopQA, and Musique, MacRAG consistently beats RAPTOR and LongRAG in F1 across Llama-3.1-8B, Gemini-1.5-pro, and GPT-4o in both single-step and multi-step generation. If true, this would make multi-hop QA more accurate without simply feeding the model more tokens.","feed_headline":"Slice-first retrieval lifts multi-hop QA accuracy across three LLMs","feed_subtitle":"Compress chunks to summaries, retrieve fine slices, then merge neighbors: consistent F1 gains over RAPTOR and LongRAG.","key_machinery":"The load-bearing object is a four-level hierarchy, document to chunk to summary to slice, built once offline, combined with a bottom-up retrieval procedure that walks back up that hierarchy at query time. Each document is split into overlapping chunks of roughly 200 to 500 tokens; each chunk is abstractively summarized; and each summary is split into overlapping slices of roughly 50 to 200 tokens that are embedded and stored with metadata linking each slice to its parent chunk and document. Retrieval then runs in five steps: slice-level top-$k_1$ search, unique parent-chunk mapping, cross-encoder chunk reranking, scaled top-($k_2 \\times \\alpha$) chunk selection followed by document ranking to pick $k_2$ distinct documents, and finally merging each chosen chunk with its $h$-hop neighbors to form bounded final contexts. The mechanism's work is to let the cheap, precise slice search decide where to look and then let the cheap index-based lookups decide how much context to add, so the model sees a query-specific, length-bounded assembly rather than whole documents or a static chunk list.","core_discovery":"Starting from the finest retrieval granularity and expanding only as needed is enough to lift multi-hop reasoning accuracy above existing hierarchical RAG baselines. Concretely, MacRAG first retrieves the top $k_1$ summary slices by query similarity, maps them to their parent chunks, reranks those chunks with a cross-encoder, widens the candidate set by a factor $\\alpha$ to catch borderline chunks, ranks the source documents, and finally merges each selected chunk with its $h$-hop neighbors to form the final top-$k_2$ merged contexts. On the LongBench versions of HotpotQA, 2WikiMultihopQA, and Musique, this yields average F1 gains over LongRAG of about +2.05 with Llama-3.1-8B, +6.10 with Gemini-1.5-pro, and +5.29 with GPT-4o when paired with LongRAG's extract-and-filter generation, and a +2.37 average F1 gain over RAPTOR with Llama-3.1-8B; the gains are largest on Musique, the dataset the paper identifies as the hardest retrieval setting. The paper also reports a 7.2% precision gain from indexing summaries rather than raw slices, sub-second retrieval and reranking at 0.23 seconds on average, and ablation results in which removing the propagation-and-merging step costs up to about 5% F1 and removing the scaling-up step costs up to about 6%.","pith_inferences":["A testable extension: measure summary fidelity directly by checking whether gold evidence entities and relations from each question survive the chunk-to-summary compression; if they often do not, the reported gains would have to come from the expansion steps rather than from compression preserving facts.","The $h$-hop neighbor merge suggests a cheap alternative to explicit graph construction: linear adjacency in document order may capture many bridging relations that graph-based indexes are built to find, at lower indexing cost.","Because the final context size is controlled by $k_2$ and $\\alpha$, MacRAG looks well suited as a context-assembly module inside iterative or agentic RAG loops, where each step's token budget must stay stable; the paper proposes this direction but does not evaluate it.","The 7.2% precision gain from summaries over raw slices is preliminary; a controlled sweep over summarization models and compression ratios would reveal whether abstractive compression is the load-bearing ingredient or whether slicing alone drives most of the gain."],"forward_implications":["Multi-hop QA systems can raise F1 by starting retrieval at summary slices and expanding to parent chunks and neighbors, rather than by retrieving whole documents or fixed chunks.","The gains persist across an 8B open model and two large proprietary models, so the retrieval architecture's benefit is largely complementary to model scale.","Because the final context is bounded to $k_2$ merged chunks, MacRAG keeps retrieval and reranking near 0.23 seconds per query, and its intermediate generation variant can beat LongRAG's full-document extraction while using less context.","The method drops by roughly 5% F1 when neighbor propagation is removed and roughly 6% when scaling-up is removed, so both expansion steps are load-bearing for multi-hop coverage.","MacRAG's retrieved contexts can feed seven different single- and multi-step generation schemes, so it behaves as a drop-in retrieval module rather than a single end-to-end recipe."],"supporting_citations":[{"why":"It provides the LongRAG baseline, its seven generation modes, and the k1=100, k2=7 hyperparameters that MacRAG adopts for controlled comparison.","marker":"Zhao et al., 2024"},{"why":"It provides the RAPTOR hierarchical-clustering baseline that MacRAG compares against on the three LongBench datasets.","marker":"Sarthi et al., 2024"},{"why":"It provides the LongBench benchmark and the three multi-hop QA datasets used in all evaluations.","marker":"Bai et al., 2023"},{"why":"It supplies the comparative analysis of hierarchical RAG methods that justifies selecting RAPTOR and LongRAG as the baselines.","marker":"Zhang et al., 2025"},{"why":"It identifies Musique as a challenging retrieval condition and motivates the inference-scaling trade-off that MacRAG addresses.","marker":"Yue et al., 2024"},{"why":"It documents the lost-in-the-middle effect that MacRAG's bounded context assembly is designed to mitigate.","marker":"Liu et al., 2024"},{"why":"It defines the Llama-3.1-8B-instruct model used as one of the three evaluation LLMs.","marker":"Dubey et al., 2024a"},{"why":"It defines Gemini-1.5-pro, one of the three evaluation LLMs.","marker":"Team et al., 2024"},{"why":"It defines GPT-4o, one of the three evaluation LLMs.","marker":"OpenAI, 2024"}],"fun_headline_variants":["Fine-first RAG beats LongRAG by up to 6 F1","Adaptive slicing lifts multi-hop QA across three LLMs","Slice, rerank, merge: MacRAG improves RAG on hard QA","Sub-second fine-to-coarse retrieval wins on multi-hop","Go fine-to-coarse: better RAG for long-document reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that compressing each chunk into a summary and then slicing the summary preserves the specific facts, the entities and relations, needed for multi-hop reasoning, because retrieval only ever sees summarized slices; if summarization drops a bridging fact, no later expansion can recover it.","fun_headline_variants_meta":{"raw":{"variants":["Fine-first RAG beats LongRAG by up to 6 F1","Adaptive slicing lifts multi-hop QA across three LLMs","Slice, rerank, merge: MacRAG improves RAG on hard QA","Sub-second fine-to-coarse retrieval wins on multi-hop","Go fine-to-coarse: better RAG for long-document reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000834,"raw_usage":{"total_tokens":3710,"prompt_tokens":1087,"completion_tokens":2623,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":703,"completion_tokens_details":{"reasoning_tokens":2530}},"tokens_in":703,"tokens_out":2623,"duration_ms":18255,"temperature":1.0,"reasoning_tokens":2530,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:38:43.382795+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MacRAG on HotpotQA, 2WikiMultihopQA, and Musique with the summarizer replaced by an identity function, retrieving raw slices instead of summarized slices while keeping chunking, slicing, reranking, and neighbor merging fixed; if F1 stays roughly the same, compression is not load-bearing. Alternatively, check whether all gold evidence passages referenced by the dataset questions survive the summarization step; if a large fraction of bridging entities or relations disappear from the summaries, the claimed gains cannot be attributed to preserving facts through compression.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the LongRAG baseline, its seven generation modes, and the k1=100, k2=7 hyperparameters that MacRAG adopts for controlled comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It defines GPT-4o, one of the three evaluation LLMs."}],"review_version":1}