RAGtifier: Evaluating RAG Generation Approaches of State-of-the-Art RAG Systems for the SIGIR LiveRAG Competition

Hailay Teklehaymanot; Oleh Astappiev; Tim Cofala; William Xion

REVIEW 3 major objections 6 minor 1 cited by

RAGtifier: Evaluating RAG Generation Approaches of State-of-the-Art RAG Systems for the SIGIR LiveRAG Competition

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper reports that InstructRAG, paired with Pinecone dense retrieval and BGE reranking, placed third in the SIGIR 2025 LiveRAG Challenge with correctness 1.13 and faithfulness 0.55.

desk verdict A credible, verifiable competition result undermined by an internal evaluation protocol that feeds the golden answer and document to the judge, which can change relative rankings and leaves the component-level claims unsupported. read the letter →

arxiv 2506.14412 v2 pith:SOXIFCWJ submitted 2025-06-17 cs.IR cs.AIcs.LG

Tim Cofala , Oleh Astappiev , William Xion , Hailay Teklehaymanot This is my paper

classification cs.IRcs.AIcs.LG

keywords retrievalaugmentedgenerationInstructRAGLivechallengeDataMorganaLLM-as-a-judgererankingPineconecontextordering

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is a competition report: it evaluates five RAG generation strategies under the LiveRAG 2025 constraints, where Falcon-3-10B generates answers and a judge LLM scores them for correctness and faithfulness. The central outcome is that InstructRAG—which first elicits a rationale identifying relevant material in messy retrieval and then answers from the denoised context—performed best when combined with a Pinecone dense retriever at 200 documents, a BGE reranker keeping the top five, and inverted context order. On the official live questions this configuration received a correctness score of 1.13 and a faithfulness score of 0.55 in the non-human evaluation, enough for third place overall. The report indicates that selective context denoising plus dense retrieval and a single lightweight reranking pass can outperform multi-round iterative retrieval and knowledge-conflict resolution under the competition's model-size and runtime limits.

What carries the argument

The core mechanism is InstructRAG's two-phase prompting: the model first looks at the retrieved passages and produces a rationale that identifies which parts are relevant, then it answers using only the denoised context. In the submitted pipeline this is combined with inverted context ordering, in which the reranked documents are presented in descending relevance so the highest-ranked document sits immediately before the question, and with Pinecone@200 dense retrieval followed by a BGE reranker that keeps the top five documents. The paper also builds its evaluation around a 'LiveRAG prompt' modeled on the challenge's guidelines, which scores correctness on a -1 to 2 scale and faithfulness on a -1 to 1 scale.

What would settle it

Re-run the exact submitted configuration, InstructRAG with inverted order, Pinecone@200, BGE@5, and Falcon-3-10B, on the official LiveRAG questions and compare per-question correctness and faithfulness scores given by the official Claude-3.5-Sonnet judge with scores given by the internal Gemma-3-27B prompt; weak per-question correlation or a large average gap would show the internal proxy did not predict the official judgments.

Watch

Extended reading notes

Core claim

The paper's own claim is that the InstructRAG generation method becomes a strong LiveRAG submission when paired with Pinecone@200 as a dense retriever, BGE@5 as a reranker, and inverted context ordering of the reranked documents, with the most relevant document placed immediately before the question. In internal comparisons across Simple Prompt, TrustRAG, Astute RAG, InstructRAG, and IterDRAG on 500 DataMorgana-generated single- and multi-hop questions, the authors narrowed the field to InstructRAG and IterDRAG; a live-day head-to-head scored by Gemma-3-27B and manual inspection favored InstructRAG, with internal correctness of 1.91 and faithfulness of 0.93. The official non-human evaluation then gave the submitted InstructRAG pipeline correctness of 1.13 and faithfulness of 0.55, a 40.9% drop the authors attribute to a more capable judge model and to their internal prompt having access to golden documents and answers, and this placed the solution third in the SIGIR 2025 LiveRAG Challenge.

Load-bearing premise

The load-bearing premise is that the team's internal Gemma-3-27B evaluation on its own 500 DataMorgana questions predicted the official Claude-3.5-Sonnet evaluation on the live questions closely enough that choices made on that proxy transferred to the contest; the paper's own reported 40.9% score drop indicates the premise held only partially.

Editorial extensions

If this is right

Under the LiveRAG constraints, InstructRAG with inverted context order, Pinecone@200 retrieval, and BGE@5 reranking was the configuration the authors submitted, and it placed third overall in the official challenge.
The paper's internal results place InstructRAG and IterDRAG ahead of Simple Prompt, TrustRAG, and Astute RAG on DataMorgana questions; on the live day, InstructRAG beat IterDRAG on both correctness and faithfulness.
The choice of Pinecone over OpenSearch is justified by higher gold-document retrieval rates and faster response times at the retrieval depths the team tested.
The reported 40.9% drop from internal to official scores implies that switching the judge from Gemma-3-27B to Claude-3.5-Sonnet can shift both correctness and faithfulness substantially for the same generated answers.
With a fixed retrieval depth, increasing rerank@k produced score variations up to 25%, while inverted context ordering contributed roughly 1% on average.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: the selection rests on only one 500-question internal benchmark, so the reported third-place ranking does not establish that Pinecone@200/BGE@5 is the globally optimal retrieval configuration; a broader sweep with more questions would be needed to separate component effects from benchmark noise.
Inference: the 40.9% judge gap implies that teams using smaller local judges should either calibrate against the official judge on a small sample or keep a score-margin buffer before choosing a final system, rather than trusting absolute internal scores.
Inference: because the authors credit part of the gap to golden documents and answers appearing in the internal judge prompt, ablating that gold information would allow a clean estimate of how much of the score shift comes from the stronger judge alone versus from the evaluation protocol difference.
Inference: the near-impractical 100-second latency of Rank-R1 under live constraints suggests that reasoning-heavy rerankers need faster inference before they can replace BGE-style rerankers in time-bounded RAG competitions; a faster distilled variant of Rank-R1 is a natural test of that trade-off.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is a competition report from the SIGIR 2025 LiveRAG Challenge, and the headline result is externally verifiable: third place with a specific configuration (InstructRAG + Pinecone@200 + BGE@5, inverted context order). That much is solid, and the code is public. It also does a useful thing by cross-comparing five RAG generation strategies and two rerankers under the challenge's model constraints (Falcon-3-10B, 10B limit). The component-level story is the weak part, and the stress-test note lands. The internal evaluation that selects components uses a judge prompt (Figure 5) that includes both the golden answer and the golden document. That is an oracle setting: in the actual challenge, the golden document is not available at answer time. The paper's own live-day evaluation omits both, and under that gold-free prompt InstructRAG beats IterDRAG by 28% in correctness and 27% in faithfulness, whereas under the gold-inclusive prompt the two were 'comparable.' So the gold-inclusive prompt does not just add a constant offset; it changes relative rankings. That means the internal component rankings in Table 1, including the choice of Pinecone@200/BGE@5, are not validated as predictors of the official judge's scores. The external third-place result remains credible, but the paper's explanatory claims about why that configuration worked are undercut. Other soft spots: no error bars on the component comparisons, small samples (100 questions for prompt studies), and the internal benchmark is not released. The paper is transparent about the 40.9% score drop between internal and official scores, which is to its credit, but it doesn't resolve the mismatch. Who is this for? Practitioners assembling RAG pipelines under time and compute constraints, and people working on LLM-as-judge reliability. It is a solid engineering write-up, not a research breakthrough. It deserves a serious referee: the flaws are fixable (release the benchmark, re-run the key comparisons under a gold-free prompt, add variance estimates), and the LiveRAG context makes the result useful to the community.

Referee Report

3 major / 6 minor

Summary. The paper describes the authors' submission to the SIGIR 2025 LiveRAG Challenge, in which they place third overall. The system combines InstructRAG with a Pinecone dense retriever (retrieval@200), a BGE-M3 reranker (rerank@5), and an inverted context order, generating answers with Falcon-3-10B. The authors benchmark five RAG generation strategies, two retrievers, two rerankers, several retrieval/rerank configurations, and three evaluation prompts on a self-built benchmark of 500 DataMorgana-generated QA pairs, judged by Gemma-3-27B and Claude-3.5-Haiku. They report internal correctness/faithfulness scores that are substantially higher than the official judge's scores, and they document a live-day comparison between InstructRAG and IterDRAG that reverses the relative ranking suggested by the internal benchmark.

Significance. If the component-level comparisons were reliable, the paper would be a useful engineering case study of RAG pipeline optimization under a constrained, time-limited competition setting. The authors provide public code, a transparent account of the design space, runtime measurements for retrievers and rerankers, and an external anchor: the third-place result is verifiable from the LiveRAG challenge report [7]. The paper also contributes custom DataMorgana user/question categories and a detailed comparison of evaluator prompts. However, the scientific value of the component-level conclusions is limited by the internal evaluation protocol, which uses an oracle prompt containing the golden document and golden answer. The authors' own live-day experiment shows that removing these oracles changes relative system ranking, calling into question the claim that the reported ablation studies support the final system choice.

major comments (3)

[Section 4.2, Figure 5, Table 1, Section 4.5] The internal evaluation that drives all component selection and the final system choice uses the LiveRAG prompt in Figure 5, which explicitly provides the ground-truth answer and the golden passage to the judge LLM. This is an oracle condition: in the actual challenge, the golden document is not part of the input at answer-generation time, and the authors' own live-day evaluation (Section 4.5) omitted both the golden document and the golden answer. The consequence is not merely a constant offset: under the oracle prompt, InstructRAG and IterDRAG are described as 'perform comparably' in Table 1, but under the gold-free live-day prompt InstructRAG is 28% better in correctness and 27% better in faithfulness. Therefore the selection of Pinecone@200, BGE@5, and InstructRAG as the final pipeline is not supported by the reported internal protocol, because the protocol does not predict the ranking that held during the actual challenge evaluation. The external third-place result is credible, but the paper's component-level conclusions (Sections 4.3-4.5) should be re-derived or re-framed using an evaluation that does not include golden information in the judge prompt.
[Table 1 and Section 4.5] The paper reports differences between RAG methods and between hyperparameter settings without error bars, confidence intervals, or significance tests. For example, in Table 1 the single-hop correctness difference between InstructRAG (4.2%) and IterDRAG (3.4%) is 0.8 percentage points on a sample of 250 questions; the paper uses this and similar small differences to justify proceeding with both systems. Given that the internal benchmark is a single draw of 500 DataMorgana questions and that the evaluation LLM is stochastic, these differences are likely within noise. The subsequent hyperparameter search over Pinecone@{100,200,300}, BGE@{5,8,10,12}, and inverted context order is reported as selecting a configuration without any validation set or repeated trials, so the reported '±2%' and 'up to 25%' variation claims are not accompanied by a measure of uncertainty. The authors should at least report bootstrap intervals or per-question variances, and should temper the component-level claims accordingly.
[Section 4.5] The paper reports a 40.9% drop in both correctness and faithfulness between the internal Gemma-3-27B evaluation (1.91 and 0.93) and the official Claude-3.5-Sonnet evaluation (1.13 and 0.55), and attributes this to a 'more capable judge LLM and accessing the golden document and golden answer.' This sentence is ambiguous and potentially self-contradictory: the internal evaluation includes the golden document and golden answer, while the official evaluation should not have access to the golden document at generation time. Regardless of the intended meaning, the size of the drop shows that the internal metric is miscalibrated as a predictor of the official score. Since the paper's main methodological contribution is 'evaluating RAG generation approaches,' the authors should analyze this miscalibration explicitly: e.g., report per-question agreement, quantify judge bias, or provide a mapping between internal and official scores. Without this, the internal evaluation cannot be taken as a reliable proxy for LiveRAG, which is the load-bearing assumption for most of the experimental section.

minor comments (6)

[Section 4.2] Typo: 'Gamma-3-as-a-Judge' should be 'Gemma-3-as-a-Judge'.
[Figure 3 caption] Typo: 'Faclon3-10B' should be 'Falcon-3-10B'; also 'Pinecone@k 1' should be 'Pinecone@k'.
[References [10] and [11]] References [10] and [11] are the same paper (Cuconasu et al., SIGIR 2024); they should be merged into one entry and cited once.
[Header and author block] The running header shows 'Trovato et al.' and the ACM reference format line also says 'Trovato et al.'; these appear to be leftover template placeholders and should be replaced with the actual author names.
[Table 1] The column headers repeat 'Correctness {1,2}' and 'Faithfulness {0,1}' for both Single-Hop and Multi-Hop, which is confusing. Please use a clearer two-level header, e.g., 'Single-Hop' spanning 'Correctness' and 'Faithfulness', and same for 'Multi-Hop'.
[Section 4.2] The description of the 'simple comparison' prompt says it compares the generated answer with the ground-truth answer, but the text later says 'query-only answers are scored almost as correctly as the best-performing answer generation strategies.' It would help to clarify whether the simple prompt also includes the gold answer, since the later LiveRAG prompt does.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the final configuration is validated by an external official evaluation, and the internal proxy issues are validity concerns, not definitional or self-citation circularity.

full rationale

The paper's central claim (InstructRAG + Pinecone@200 + BGE@5) is selected on an internally generated 500-question DataMorgana benchmark and then independently scored by the LiveRAG organizers' Claude-3.5-Sonnet judge, placing third; the official score is an external benchmark not constructed from the paper's inputs. No component is defined in terms of another, no parameter is fit to the official metric and then reported as a prediction, and the paper contains no load-bearing self-citations. The LiveRAG prompt in Figure 5 does include the golden answer and golden document, and this could bias internal component comparisons; however, the authors transparently report the resulting 40.9% drop against the organizer evaluation in Section 4.5. That divergence is a proxy-validity and calibration problem, not a circular derivation: the component ranking is an empirical observation under a flawed judge prompt, not an equation that reduces to its own inputs. The third-place external result remains independent evidence for the configuration. Accordingly, no circular step meets the required evidentiary bar.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its results rest on the representativeness of the internally generated benchmark, the reliability of the proxy judge, and the transferability of gold-document-based retrieval measurements to the official test setting.

free parameters (5)

Retrieval@k (Pinecone) for final system = 200
Selected from Pinecone@{100,200,300} based on internal benchmark; higher k did not consistently improve correctness (±2%).
Rerank@k (BGE) for final system = 5
Selected from BGE@{5,8,10,12}; variation in rerank@k caused 1-25% correctness swings on internal eval.
Inverted context order = true
Applied to all non-IterDRAG approaches; measured ~1% average improvement.
Generation temperature = 0.1
Fixed setting for Falcon-3-10B; no tuning reported.
IterDRAG iterations and BGE@k = 4-5 iterations, BGE@3-5
Tuned for iterative retrieval steps in the approach that was considered but not submitted.

assumptions (4)

domain assumption The 500 DataMorgana-generated QA pairs with custom user/question categories are representative of the official LiveRAG challenge questions.
Used to evaluate and select systems (Sections 3.1, 4.1); the official test set differs and is only available live, so transferability is assumed.
domain assumption Gemma-3-27B with the LiveRAG prompt is a valid proxy for the official Claude-3.5-Sonnet judge.
Section 3.5 selects this judge; Section 4.5 reports a 40.9% gap between internal and official scores, weakening this assumption.
domain assumption Gold documents are available for development-time retrieval@k measurements, and those measurements transfer to the challenge setting.
Sections 4.3 and 4.4 compare retrievers using gold documents from self-generated QA pairs; actual test questions do not provide gold documents.
domain assumption Falcon-3-10B's context window limits retrieval to about 50 documents without reranking, motivating reranker experiments.
Stated in Section 4.3; the context limit explains the experimental design.

how reviews work

0 comments

Cite this review

Pith. "Pith review of RAGtifier: Evaluating RAG Generation Approaches of State-of-the-Art RAG Systems for the SIGIR LiveRAG Competition." pith.science (2026). https://pith.science/paper/SOXIFCWJ

@misc{pith2026250614412,
  author       = {Pith},
  title        = {Pith review of: RAGtifier: Evaluating RAG Generation Approaches of State-of-the-Art RAG Systems for the SIGIR LiveRAG Competition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SOXIFCWJ}},
  note         = {Machine review of arXiv:2506.14412}
}

read the original abstract

Retrieval-Augmented Generation (RAG) enriches Large Language Models (LLMs) by combining their internal, parametric knowledge with external, non-parametric sources, with the goal of improving factual correctness and minimizing hallucinations. The LiveRAG 2025 challenge explores RAG solutions to maximize accuracy on DataMorgana's QA pairs, which are composed of single-hop and multi-hop questions. The challenge provides access to sparse OpenSearch and dense Pinecone indices of the Fineweb 10BT dataset. It restricts model use to LLMs with up to 10B parameters and final answer generation with Falcon-3-10B. A judge-LLM assesses the submitted answers along with human evaluators. By exploring distinct retriever combinations and RAG solutions under the challenge conditions, our final solution emerged using InstructRAG in combination with a Pinecone retriever and a BGE reranker. Our solution achieved a correctness score of 1.13 and a faithfulness score of 0.55 in the non-human evaluation, placing it overall in third place in the SIGIR 2025 LiveRAG Challenge.

Figures

Figures reproduced from arXiv: 2506.14412 by the authors.

**Figure 5.** Evaluation prompt based on LiveRAG evaluation [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗

**Figure 2.** Pinecone excels OpenSearch in retrieving gold doc [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Figures from the paper (3 more)

**Figure 3.** Figure 3: Gold documents [%] in retrieval for Faclon3-10B [PITH_FULL_IMAGE:figures/full_fig_p006_3.png]

**Figure 4.** Figure 4: OpenSearch and Pinecone retrievers, and BGE [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 6.** Figure 6: Subset of question and user categories we used to [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

SIGIR 2025 -- LiveRAG Challenge Report
cs.CL 2025-07 conditional novelty 3.0 of 10

In the SIGIR 2025 LiveRAG Challenge, all 25 active RAG teams beat the no-RAG baseline on LLM-judged correctness, and LLM scores correlated with human scores at r=0.88.

Reference graph

Works this paper leans on

42 extracted references · 19 canonical work pages · cited by 1 Pith paper

[7]

David Carmel, Simone Filice, Guy Horowitz, et al. 2025. SIGIR 2025 – LiveRAG Challenge Report. arXiv:2507.04942 [cs.CL] https://arxiv.org/abs/2507.04942

arXiv 2025
[1]

[n. d.]. Chatbot Arena — openlm.ai. https://openlm.ai/chatbot-arena/. [Accessed 10-04-2025]

work page 2025
[2]

[n. d.]. Gemma open models | Google AI for Developers — ai.google.dev. https: //ai.google.dev/gemma. [Accessed 10-04-2025]

work page 2025
[3]

[n. d.]. LiveRAG Challenge - Challengedetails — liverag.tii.ae. https://liverag.tii. ae/challenge-details.php. [Accessed 10-04-2025]

work page 2025
[4]

Falcon 3. [n. d.]. Falcon 3 — falconllm.tii.ae. https://falconllm.tii.ae/falcon3/index. html. [Accessed 20-05-2025]

work page 2025
[5]

Anthropic. 2024. Claude Haiku 3.5. https://www.anthropic.com/claude/haiku Accessed: May 10, 2025

work page 2024
[6]

Brown, Benjamin Mann, Nick Ryder, et al

Tom B. Brown, Benjamin Mann, Nick Ryder, et al . 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, et al . (Eds.). https://proceedings.neurips...

work page 2020
[8]

Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu

Show all 42 references

[9]

Cheng-Han Chiang and Hung-yi Lee. 2023. Can Large Language Models Be an Al- ternative to Human Evaluations?. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki...

2023 doi
[12]

Dorner, Vivian Y

Florian E. Dorner, Vivian Y. Nastl, and Moritz Hardt. 2024. Limits to scalable evalu- ation at the frontier: LLM as Judge won’t beat twice the data.CoRR abs/2410.13341 (2024). https://doi.org/10.48550/ARXIV.2410.13341 arXiv:2410.13341

2024 doi
[13]

Hashimoto

Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. 2024. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evalua- tors. CoRR abs/2404.04475 (2024). https://doi.org/10.48550/ARXIV.2404.04475 arXiv:2404.04475

2024 doi arXiv
[14]

Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert
[15]

Benjamin Feuer, Micah Goldblum, Teresa Datta, et al . 2024. Style Out- weighs Substance: Failure Modes of LLM Judges in Alignment Benchmark- ing. CoRR abs/2409.15268 (2024). https://doi.org/10.48550/ARXIV.2409.15268 arXiv:2409.15268

2024 doi arXiv
[16]

Simone Filice, Guy Horowitz, David Carmel, et al . 2025. Generating Diverse Q&A Benchmarks for RAG Evaluation with DataMorgana. CoRR abs/2501.12789 (2025). https://doi.org/10.48550/ARXIV.2501.12789 arXiv:2501.12789

2025 doi arXiv
[17]

FineWeb. [n. d.]. HuggingFaceFW/fineweb· Datasets at Hugging Face — hugging- face.co. https://huggingface.co/datasets/HuggingFaceFW/fineweb. [Accessed 20-05-2025]

2025
[18]

Hambarde and Hugo Proença

Kailash A. Hambarde and Hugo Proença. 2023. Information Retrieval: Recent Advances and Beyond. IEEE Access 11 (2023), 76581–76604. https://doi.org/10. 1109/ACCESS.2023.3295776

2023
[20]

Jaehun Jung, Faeze Brahman, and Yejin Choi. 2024. Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement. CoRR abs/2407.18370 (2024). https://doi.org/10.48550/ARXIV.2407.18370 arXiv:2407.18370

2024 doi arXiv
[21]

Patrick Lewis, Ethan Perez, Aleksandra Piktus, et al. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Infor- mation Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-...

2020
[22]

Alex Mallen, Akari Asai, Victor Zhong, et al. 2023. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL...

2023
[23]

Pinecone. [n. d.]. The vector database to build knowledgeable AI | Pinecone — pinecone.io. https://www.pinecone.io/. [Accessed 20-05-2025]

2025
[24]

Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, and Jimmy Lin. 2025. The Great Nugget Recall: Automating Fact Extrac- tion and RAG Evaluation with Large Language Models. arXiv:2504.15068 [cs.IR] https://arxiv.org/abs/2504.15068

2025 arXiv
[25]

Ofir Press, Muru Zhang, Sewon Min, et al. 2023. Measuring and Narrowing the Compositionality Gap in Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023 , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Ass...

2023 doi
[26]

OpenSearch Project. [n. d.]. OpenSearch — opensearch.org. https://opensearch. org/. [Accessed 20-05-2025]

2025
[27]

Robertson and Hugo Zaragoza

Stephen E. Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Found. Trends Inf. Retr. 3, 4 (2009), 333–389. https://doi.org/10.1561/1500000019

2009 doi
[28]

Venktesh V, Mandeep Rathee, and Avishek Anand. 2025. SUNAR: Se- mantic Uncertainty based Neighborhood Aware Retrieval for Complex QA. CoRR abs/2503.17990 (2025). https://doi.org/10.48550/ARXIV.2503.17990 arXiv:2503.17990

2025 doi arXiv
[29]

Fei Wang, Xingchen Wan, Ruoxi Sun, et al . 2024. Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Models. CoRR abs/2410.07176 (2024). https://doi.org/10.48550/ARXIV.2410.07176 arXiv:2410.07176

2024 doi arXiv
[30]

Zhepei Wei, Wei-Lin Chen, and Yu Meng. 2025. InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationales. In The Thir- teenth International Conference on Learning Representations, ICLR 2025, Singa- pore, April 24-28, 2025 . OpenReview.net. https://o...

2025
[31]

Xiao Yang, Kai Sun, Hao Xin, et al . 2024. CRAG - Comprehensive RAG Benchmark. In Advances in Neural Information Processing Systems 38: An- nual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024 , Amir Globers...

2024
[32]

Jiayi Ye, Yanbo Wang, Yue Huang, et al. 2024. Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge. CoRR abs/2410.02736 (2024). https://doi.org/10.48550/ ARXIV.2410.02736 arXiv:2410.02736

2024 doi arXiv
[33]

Zhenrui Yue, Honglei Zhuang, Aijun Bai, et al . 2025. Inference Scaling for Long-Context Retrieval Augmented Generation. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 . OpenReview.net. https://openreview.net/for...

2025
[34]

Hamed Zamani and Michael Bendersky. 2024. Stochastic RAG: End-to-End Retrieval-Augmented Generation through Expected Utility Maximization. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington...

2024
[35]

Zihan Zhang, Meng Fang, and Ling Chen. 2024. RetrievalQA: Assessing Adap- tive Retrieval-Augmented Generation for Short-form Open-Domain Question Answering. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11...

2024 doi
[36]

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, et al . 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neu- ral Information Processing Systems 36: Annual Conference on Neural In- formation Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, De...

2023
[37]

Ming Zhong, Aston Zhang, Xuewei Wang, et al . 2024. Law of the Weakest Link: Cross Capabilities of Large Language Models. CoRR abs/2409.19951 (2024). https://doi.org/10.48550/ARXIV.2409.19951 arXiv:2409.19951

2024 doi arXiv
[38]

Huichi Zhou, Kin-Hei Lee, Zhonghao Zhan, et al. 2025. TrustRAG: Enhancing Robustness and Trustworthiness in RAG. CoRR abs/2501.00879 (2025). https: //doi.org/10.48550/ARXIV.2501.00879 arXiv:2501.00879

2025 doi arXiv
[39]

Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2023. JudgeLM: Fine-tuned Large Language Models are Scalable Judges. CoRR abs/2310.17631 (2023). https: //doi.org/10.48550/ARXIV.2310.17631 arXiv:2310.17631 SIGIR, July 17, 2025, Padua, IT Trovato et al

2023 doi arXiv
[40]

Shengyao Zhuang, Xueguang Ma, Bevan Koopman, et al . 2025. Rank-R1: En- hancing Reasoning in LLM-based Document Rerankers via Reinforcement Learn- ing. CoRR abs/2503.06034 (2025). https://doi.org/10.48550/ARXIV.2503.06034 arXiv:2503.06034

2025 doi arXiv
[41]

Shengyao Zhuang, Honglei Zhuang, Bevan Koopman, and Guido Zuccon. 2024. A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information ...

2024
[44]

Combines elements of: - **coverage**: portion of vital information, in the ground truth answer which is covered by the generated answer

Correctness. Combines elements of: - **coverage**: portion of vital information, in the ground truth answer which is covered by the generated answer. - **relevance**: portion of the generated response which is directly addressing the question, regardless its factual correctnes...
[45]

correctness

Faithfulness. Assesses whether the response is **grounded in the retrieved passages**. Graded on a continuous scale with the following representative points: - **1:** Full support. All answer parts are grounded - **0:** Partial support. Not all answer parts are grounded - **-1...

2025
[2024]

arXiv:2402.03216 [cs.CL]

BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. arXiv:2402.03216 [cs.CL]

arXiv
[2025]

arXiv:2309.15217 [cs.CL] https://arxiv.org/abs/2309.15217

Ragas: Automated Evaluation of Retrieval Augmented Generation. arXiv:2309.15217 [cs.CL] https://arxiv.org/abs/2309.15217

arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [7]

David Carmel, Simone Filice, Guy Horowitz, et al. 2025. SIGIR 2025 – LiveRAG Challenge Report. arXiv:2507.04942 [cs.CL] https://arxiv.org/abs/2507.04942

arXiv 2025

[2] [1]

[n. d.]. Chatbot Arena — openlm.ai. https://openlm.ai/chatbot-arena/. [Accessed 10-04-2025]

work page 2025

[3] [2]

[n. d.]. Gemma open models | Google AI for Developers — ai.google.dev. https: //ai.google.dev/gemma. [Accessed 10-04-2025]

work page 2025

[4] [3]

[n. d.]. LiveRAG Challenge - Challengedetails — liverag.tii.ae. https://liverag.tii. ae/challenge-details.php. [Accessed 10-04-2025]

work page 2025

[5] [4]

Falcon 3. [n. d.]. Falcon 3 — falconllm.tii.ae. https://falconllm.tii.ae/falcon3/index. html. [Accessed 20-05-2025]

work page 2025

[6] [5]

Anthropic. 2024. Claude Haiku 3.5. https://www.anthropic.com/claude/haiku Accessed: May 10, 2025

work page 2024

[7] [6]

Brown, Benjamin Mann, Nick Ryder, et al

Tom B. Brown, Benjamin Mann, Nick Ryder, et al . 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, et al . (Eds.). https://proceedings.neurips...

work page 2020

[8] [8]

Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu