REVIEW 3 major objections 6 minor 48 references
HF-RAG: Hierarchical Fusion-based RAG with Multiple Sources and Rankers
T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read The paper claims that a two-stage fusion of retrieval results - reciprocal rank fusion within each evidence source, then z-score standardization across sources - consistently improves RAG-based fact verification over the best individual ran
desk verdict HF-RAG is a clean, incremental fusion recipe with a plausible central claim, but the evidence needs variance estimates to be convincing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the two-stage fusion. Stage one applies reciprocal rank fusion (RRF), summing 1/rank over each retriever's top-k list, separately for labeled and unlabeled sources, so each source yields one fused ranked list. Stage two standardizes each fused list's RRF scores by subtracting the source mean and dividing by the source standard deviation (z-score), producing a total order across the two non-overlapping lists. The z-score step is justified by treating inter-source comparison as a stochastic comparison problem and assuming each list's scores are roughly Gaussian.
What would settle it
Take a fact-verification benchmark where per-source retrieval scores are heavily skewed, and compare HF-RAG against the same pipeline with quantile normalization replacing z-scores, plus against the best single source/ranker; if quantile normalization changes the winner or if HF-RAG fails to beat the single best configuration under resampling, the comparability assumption is not holding.
Extended reading notes
Core claim
On its own terms, the central claim is that combining evidence hierarchically - rank fusion inside each source, standardized-score fusion across sources - yields a better and more transferable context for RAG than any single retriever or single source. In the paper's tables, HF-RAG beats the oracle single-source single-ranker bound RAG-OptSel in all six LLM-by-dataset settings, and it is the top method on SciFact, the most domain-shifted setting. The authors interpret this as evidence that labeled examples capture task-specific semantics while unlabeled documents provide broader grounding, and that aggregating multiple rankers improves the relevance of both.
Load-bearing premise
The z-score merge assumes that after subtracting each source's mean and dividing by its standard deviation, the two sources' retrieval scores are comparable enough to be ranked against each other; if the per-source score distributions are not roughly Gaussian or not comparable under standardization, the inter-source ranking becomes an arbitrary re-ranking.
Editorial extensions
If this is right
- Any RAG system with access to several retrievers and two evidence pools can adopt HF-RAG at inference time without retraining or prompt tuning.
- Fusing multiple rankers within a source is consistently better than picking the best single ranker, so retrieval diversity itself is a lever for downstream accuracy.
- Combining labeled and unlabeled context outperforms an oracle that selects the best single source/ranker with test labels, suggesting the fused context contains something no individual configuration provides.
- Out-of-domain gains, especially on scientific claims, imply the method partially decouples task-specific semantics from domain-specific language.
- Retrieval-quality gains from fusion track downstream F1 gains, so the hierarchical merge is not just a ranking trick but a way to supply more useful evidence to the generator.
Reading between the lines
- Editor's inference: the Gaussian-or-comparable assumption behind z-score merging is the main place the method could gain or lose; on skewed retrieval-score distributions a rank-based or quantile normalization might behave differently and is directly testable.
- Editor's inference: because the method is inference-only, it should transfer to other knowledge-intensive tasks such as open-domain QA and claim generation, where labeled and unlabeled contexts are complementary; this is untested in the paper.
- Editor's inference: the two fixed fusion rules (RRF then z-score) could be replaced by learned weights or learned score transforms, but the paper's result suggests even unlearned parameter-free fusion has a strong baseline effect.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes HF-RAG, a hierarchical fusion method for retrieval-augmented fact verification. For each of two sources (labeled examples and unlabeled Wikipedia passages), the outputs of four retrievers (BM25, Contriever, ColBERT, MonoT5) are merged with reciprocal rank fusion (RRF). The two source-level fused lists are then merged by z-score normalization of the RRF scores. The method is evaluated on FEVER, Climate-FEVER, and SciFact with two LLMs (Llama-2-70B and Mistral-7B). The main empirical claim is that HF-RAG consistently outperforms the best individual ranker/source configuration (including the oracle RAG-OptSel bound) and shows better out-of-domain generalization. The paper also reports ablations (LU-RAG-α, single-source RRF variants) and sensitivity to context size.
Significance. If the empirical claims are sustained, HF-RAG would be a simple, parameter-free-inference-time contribution: it combines existing rankers and heterogeneous retrieval sources without additional training, and the authors provide code. The hierarchical RRF-then-z-score design is a reasonable and practical heuristic, and the comparison against an oracle single-source/single-ranker bound is a strong experimental idea. The main value would be in showing that inference-time fusion of labeled and unlabeled evidence can improve LLM fact verification, including out-of-domain transfer. However, the current evidence is not yet load-bearing because the central comparison rests on unreplicated point estimates and the z-score comparability assumption is untested.
major comments (3)
- [Table 1, §4 (RQ-1)] The central claim that HF-RAG 'consistently improves' over the best single-source/single-ranker configuration rests on six macro-F1 point estimates. Two of the six margins against the oracle RAG-OptSel are extremely small: Climate-FEVER/Mistral 0.5019 vs 0.5001 (Δ=0.0018) and FEVER/Mistral 0.5628 vs 0.5584 (Δ=0.0044). No confidence intervals, bootstrap estimates, or significance tests are reported, and no repeated decoding/retrieval runs are described. Since RAG-OptSel is selected using test labels, small wins against it could easily be chance outcomes in a single run. The paper should report variance estimates and paired significance tests (e.g., bootstrap over claims or repeated generation seeds), or qualify the claim accordingly.
- [§2, Eq. (2): Z-score for Inter-Source Combination] The inter-source merge assumes that RRF scores within each source are comparable after z-score standardization, appealing to a Gaussian assumption in dueling bandits. This assumption is load-bearing for the method: if the two score distributions are non-Gaussian or have different shapes, the z-score merge is an arbitrary re-ranking, and other normalizations (min-max, rank-only, quantile transform) could yield different results. The paper provides no distributional diagnostics and no ablation comparing z-score with alternative standardizations. Please add an empirical check of score distributions and at least one alternative normalization/merge strategy.
- [§2, Eq. (1) and §4 (RQ-4)] Two implementation details needed for reproducibility are missing. First, the RRF missing-document rank M in Eq. (1) is never given a value; the paper only says M≫k. Second, while k is varied in Figure 4a, sensitivity to M is not discussed. Since M affects the contribution of documents absent from some ranked lists, and therefore the fused list, its default value should be stated. Please specify M and, if practical, include a sensitivity check.
minor comments (6)
- [Abstract and §4] 'Consistently improves' is too strong given Table 1; the small margins (0.0018 on Climate-FEVER/Mistral) should be described as 'tends to improve' until significance is established.
- [§2, Eq. (2)] The dueling-bandits analogy is somewhat forced because there is no reward or active selection policy; the authors acknowledge this, but the paragraph could be shortened to avoid implying a stronger theoretical grounding than is needed.
- [Figure 3, RQ-3] The correlation between nDCG@10 and F1 is asserted from visual inspection. Please report a correlation coefficient (e.g., Spearman) over the shown points, or state that the relation is qualitative.
- [Figure 1 caption] The red/blue highlights are described but not visible in grayscale printing; consider adding markers or labels.
- [§3, Retriever and Generators] The phrase 'LLaMA 2.0 (70B)' should be 'LLaMA-2-70B'; also specify the exact model variants (e.g., base vs chat) and decoding settings (temperature, number of generated tokens) for reproducibility.
- [References] Some citations appear reused for different baselines (e.g., [27], [29] for L-RAG). Please verify the citation-to-method mapping so readers can reproduce the exact prompting and retrieval configurations.
Circularity Check
No significant circularity: HF-RAG's hierarchical fusion uses no fitted parameters and its gains are measured against held-out benchmarks, so the derivation is self-contained.
full rationale
The paper's derivation chain is self-contained. Equation 1 (RRF) combines ranked lists using only ranks, and Equation 2 (z-score) standardizes scores using the mean and standard deviation of each source-specific list; neither equation is fitted to or defined by the target F1 results. The reported comparisons in Table 1 are against external test splits (FEVER, Climate-FEVER, SciFact) and include RAG-OptSel, an oracle selected using test labels; HF-RAG is not fitted to those labels, so beating this oracle is not a tautology. The Gaussian score-distribution assumption in Section 2 is a stated statistical modeling assumption, not a reduction of the conclusion to an input. The self-citations (e.g., [36], [37]) support background framing about labeled/unlabeled RAG and are not used to justify the hierarchical-fusion claim, to define the result, or to forbid alternatives. The LU-RAG-alpha ablation uses a grid-searched alpha, but alpha is not part of HF-RAG, so it does not constitute a fitted parameter renamed as a prediction. Consequently no circular step is present; concerns about small margins without significance tests are statistical robustness issues, not circularity.
Assumptions & free parameters
free parameters (2)
- Context size k =
10
- RRF missing-document rank M =
large, unspecified (M >> k)
assumptions (4)
- domain assumption Retrieval score distributions within each source are comparable after z-score standardization and are assumed to follow a Gaussian distribution.
- domain assumption Reciprocal rank fusion additively combines reciprocal ranks from multiple rankers into a meaningful relevance score.
- domain assumption Labeled examples and unlabeled documents provide complementary evidence for fact verification.
- domain assumption The retrieval lists from labeled and unlabeled sources are non-overlapping, making RRF across sources inapplicable.
Cite this review
Pith. "Pith review of HF-RAG: Hierarchical Fusion-based RAG with Multiple Sources and Rankers." pith.science (2026). https://pith.science/paper/FFL7NLQN
@misc{pith2026250902837,
author = {Pith},
title = {Pith review of: HF-RAG: Hierarchical Fusion-based RAG with Multiple Sources and Rankers},
year = {2026},
howpublished = {\url{https://pith.science/paper/FFL7NLQN}},
note = {Machine review of arXiv:2509.02837}
}
read the original abstract
Leveraging both labeled (input-output associations) and unlabeled data (wider contextual grounding) may provide complementary benefits in retrieval augmented generation (RAG). However, effectively combining evidence from these heterogeneous sources is challenging as the respective similarity scores are not inter-comparable. Additionally, aggregating beliefs from the outputs of multiple rankers can improve the effectiveness of RAG. Our proposed method first aggregates the top-documents from a number of IR models using a standard rank fusion technique for each source (labeled and unlabeled). Next, we standardize the retrieval score distributions within each source by applying z-score transformation before merging the top-retrieved documents from the two sources. We evaluate our approach on the fact verification task, demonstrating that it consistently improves over the best-performing individual ranker or source and also shows better out-of-domain generalization.
Figures
Reference graph
Works this paper leans on
-
[1]
Meta AI. 2023. LLaMA 2: Open Foundation and Chat Models. https://huggingface. co/meta-llama/Llama-2-7b. Accessed: 2025-06-03
work page 2023
-
[2]
Avi Arampatzis and Stephen Robertson. 2011. Modeling score distributions in information retrieval. Information Retrieval 14 (2011), 26–46
work page 2011
-
[3]
Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. 2023. Retrieval-based language models and applications. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts) . 41–46
work page 2023
-
[4]
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...
2020
-
[5]
Manish Chandra, Debasis Ganguly, and Iadh Ounis. 2025. One size doesn’t fit all: Predicting the number of examples for in-context learning. In European Conference on Information Retrieval . Springer, 67–84
work page 2025
-
[6]
Xinran Chen, Sei-Ching Joanna Sin, Yin-Leng Theng, and Chei Sian Lee. 2015. Why do social media users share misinformation?. In Proceedings of the 15th ACM/IEEE-CS joint conference on digital libraries . 111–114
work page 2015
-
[7]
Gordon V Cormack, Charles LA Clarke, and Stefan Buettcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. 758–759
2009
-
[8]
Ronan Cummins. 2014. Document score distribution models for query perfor- mance inference and prediction. ACM Transactions on Information Systems (TOIS) 32, 1 (2014), 1–28
work page 2014
Show all 48 references
-
[9]
Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. 2020. Climate-fever: A dataset for verification of real-world climate claims. arXiv preprint arXiv:2012.00614 (2020)
2020 arXiv
-
[10]
Mohamed Farah and Daniel Vanderpooten. 2007. An outranking approach for rank aggregation in information retrieval. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. 591–598
2007
-
[11]
Edward Fox and Joseph Shaw. 1994. Combination of multiple searches. NIST special publication SP (1994), 243–243
1994
-
[12]
Sebastian Hofstätter, Jiecao Chen, Karthik Raman, and Hamed Zamani. 2023. Fid- light: Efficient and effective retrieval-augmented text generation. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1437–1447
2023
-
[13]
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)
2021 arXiv
-
[14]
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bo- janowski, Armand Joulin, and Edouard Grave. 2021. Unsupervised dense in- formation retrieval with contrastive learning. arXiv preprint arXiv:2112.09118 (2021)
2021 arXiv
-
[15]
Gautier Izacard and Edouard Grave. 2020. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282 (2020)
2020 arXiv
-
[16]
Gautier Izacard and Edouard Grave. 2021. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Paola Merlo, Jorg Tiede...
2021 doi
-
[17]
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, De- vendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thoma...
2023 arXiv
-
[18]
Kelvin Jiang, Ronak Pradeep, and Jimmy Lin. 2021. Exploring listwise evidence reasoning with t5 for fact verification. In Proceedings of the 59th Annual Meeting of the ACL and the 11th IJCNLP (Volume 2: Short Papers) . 402–410
2021
-
[19]
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. arXiv:2503.09516 [cs.CL] https://arxiv.org/abs/2503.09516
2025 arXiv
-
[20]
Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bin Liu
-
[21]
Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, China)(SIGIR ...
2020
-
[22]
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems 35 (2022), 22199–22213
2022
-
[23]
Srijan Kumar, Robert West, and Jure Leskovec. 2016. Disinformation on the web: Impact, characteristics, and detection of wikipedia hoaxes. In Proceedings of the 25th international conference on World Wide Web . 591–602
2016
-
[24]
Tiziano Labruna, Jon Ander Campos, and Gorka Azkune. 2024. When to Re- trieve: Teaching LLMs to Utilize Information Retrieval Effectively. arXiv preprint arXiv:2404.19705 (2024)
2024 arXiv
-
[25]
Eunchan Lee, Changhyeon Lee, and Sangtae Ahn. 2022. Comparative Study of Multiclass Text Classification in Research Proposals Using Pretrained Language Models. Applied Sciences (2022). https://api.semanticscholar.org/CorpusID: 248471302
2022
-
[26]
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing S...
2020
-
[27]
Xiaoqian Li, Ercong Nie, and Sheng Liang. 2023. From classification to generation: Insights into crosslingual retrieval augmented icl. arXiv preprint arXiv:2311.06595 (2023)
2023 arXiv
-
[28]
Yinhan Liu. 2019. Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692 (2019)
2019 arXiv
-
[29]
Quanyu Long, Wenya Wang, and Sinno Jialin Pan. 2023. Adapt in Contexts: Retrieval-Augmented Domain Adaptation via In-Context Learning.arXiv preprint arXiv:2311.11551 (2023)
2023 arXiv
-
[30]
or: How I learned to stop worrying and love
Andrew Parry, Debasis Ganguly, and Manish Chandra. 2024. In-Context Learning" or: How I learned to stop worrying and love" Applied Information Retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval . 14–25
2024
-
[31]
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al. 2020. KILT: a benchmark for knowledge intensive language tasks. arXiv preprint arXiv:2009.02252 (2020)
2020 arXiv
-
[32]
Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. 2021. The expando-mono-duo design pattern for text ranking with pretrained sequence-to-sequence models. arXiv preprint arXiv:2101.05667 (2021)
2021 arXiv
-
[33]
Zackary Rackauckas. 2024. Rag-fusion: a new take on retrieval-augmented generation. arXiv preprint arXiv:2402.03367 (2024)
2024 arXiv
-
[34]
Zackary Rackauckas, Arthur Câmara, and Jakub Zavrel. 2024. Evaluating rag-fusion with ragelo: an automated elo-based framework. arXiv preprint arXiv:2406.14783 (2024)
2024 arXiv
-
[35]
Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Frame- work: BM25 and Beyond. Found. Trends Inf. Retr. 3, 4 (April 2009), 333–389. doi:10.1561/1500000019
2009 doi
-
[36]
The Absence of Evidence is Not the Evidence of Absence
Payel Santra, Madhusudan Ghosh, Debasis Ganguly, Partha Basuchowdhuri, and Sudip Kumar Naskar. 2024. “The Absence of Evidence is Not the Evidence of Absence”: Fact Verification via Information Retrieval-Based In-Context Learn- ing. In Big Data Analytics and Knowledge Discovery...
2024 doi
-
[37]
Curious Case of Contexts
Payel Santra, Madhusudan Ghosh, Debasis Ganguly, Partha Basuchowdhuri, and Sudip Kumar Naskar. 2025. The “Curious Case of Contexts” in Retrieval- Augmented Generation With a Combination of Labeled and Unlabeled Data. Wiley Interdisciplinary Reviews: Data Mining and Knowledge D...
2025
-
[38]
Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin C! robust fact verification with contrastive evidence.arXiv preprint arXiv:2103.08541 (2021)
2021 arXiv
-
[39]
Tal Schuster, Darsh J Shah, Yun Jie Serene Yeo, Daniel Filizzola, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. arXiv preprint arXiv:1908.05267 (2019)
2019 arXiv
-
[40]
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663 (2021)
2021 arXiv
-
[41]
TheBloke. 2023. Llama-2-70B-Chat-AWQ. https://huggingface.co/TheBloke/ Llama-2-70B-Chat-AWQ. Accessed: 2025-06-01
2023
-
[42]
TheBloke. 2023. Mistral-7B-Instruct-v0.2-AWQ. https://huggingface.co/ TheBloke/Mistral-7B-Instruct-v0.2-AWQ. Accessed: 2025-06-01
2023
-
[43]
James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2018. The Fact Extraction and VERification (FEVER) Shared Task. CIKM ’25, November 10–14, 2025, Seoul, Republic of Korea Santra et al. In Proceedings of the First Workshop on Fact Ext...
2018
-
[44]
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)
2023 arXiv
-
[45]
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or Fiction: Verifying Scientific Claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , Bonnie Webber...
2020 doi
-
[46]
Yisong Yue, Josef Broder, Robert Kleinberg, and Thorsten Joachims. 2012. The K-armed dueling bandits problem. J. Comput. Syst. Sci. 78, 5 (2012), 1538–1556
2012
-
[47]
Delvin Ce Zhang and Dongwon Lee. 2025. CORRECT: Context- and Reference- Augmented Reasoning and Prompting for Fact-Checking. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologi...
2025
-
[2023]
In International Conference on Learning Representations
Continual Pre-training of Language Models. In International Conference on Learning Representations. https://api.semanticscholar.org/CorpusID:258079422
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.