Pith. sign in

REVIEW 2 major objections 7 minor 2 cited by

SIGIR 2025 -- LiveRAG Challenge Report

T0 review · 2 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM-based judges can evaluate RAG answers as reliably as human annotators, the LiveRAG Challenge report argues.

desk verdict A solid, transparently written challenge report whose central validation claim (r=0.8826) is weaker than it looks because it is computed only on the teams the LLM judge already ranked on top. read the letter →

arxiv 2507.04942 v2 pith:EQZHMMNJ submitted 2025-07-07 cs.CL cs.IR

David Carmel , Simone Filice , Guy Horowitz , Yoelle Maarek , Oren Somekh , Ran Tavory , Mehdi Ghissassi , Edo Liberty
show 1 more author
Roy Miara
This is my paper
classification cs.CLcs.IR
keywords RAGevaluationLLM-as-a-judgeLiveChallengepseudo-liveCorrectnessmetricFaithfulnessmanualFineweb-10BT
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports on the SIGIR 2025 LiveRAG Challenge, a competition in which 25 teams built retrieval-augmented generation (RAG) question-answering systems on a fixed web corpus with a fixed open LLM. Its central aim is to show that an automated LLM-based judge can evaluate RAG answers as reliably as human annotators. The key evidence is a Pearson correlation of 0.8826 between the LLM-computed Correctness score and manual Borda scores over 105 shared questions, with the four top teams identical under both evaluations. The paper also documents that every participating team beat the no-RAG baseline, supporting the practical value of RAG for question answering. A sympathetic reader would take the paper's claim to be that this two-stage evaluation design makes large-scale RAG comparisons both feasible and trustworthy.

What carries the argument

The central machinery is a claim-level evaluation pipeline. The LLM judge extracts atomic claims from reference and generated answers, classifies each claim as Direct, Useful, or Useless, and runs natural-language inference to test whether the generated answer entails or contradicts reference claims. Coverage weighs Direct claims at $\alpha=0.7$ and Useful claims at $0.3$; Relatedness is the share of Direct claims in the generated answer; Correctness is their harmonic mean, scaled to $[-1,2]$. Faithfulness checks whether answer claims are entailed by retrieved passages. Manual evaluation uses three Likert-scaled ratings aggregated by Borda counts. The 105 shared questions between the two sessions provide the common scale on which both evaluations can be correlated.

What would settle it

Manually evaluate all 25 participating teams (or a random sample not selected by the automated score) on the same 105 shared questions and recompute the Pearson correlation between the LLM-based Correctness and the manual Borda score; if the correlation drops substantially below 0.88, the range-restriction concern is confirmed.

Watch

Extended reading notes

Core claim

The paper claims that LLM-as-a-judge evaluation of RAG-generated answers is robust, as validated against manual assessment. Using Claude-3.5-sonnet to compute Correctness (harmonic mean of coverage and relatedness) and Faithfulness, the organizers evaluated 500 answers per team. For the top 13 teams selected by the automated score, human annotators scored coverage, relatedness, and quality on a Likert scale, aggregated via Borda counts. The automated Correctness score and the manual Borda score correlated at r=0.8826 over the 105 shared questions, and the four leading teams under the LLM judge were also the four leading teams under manual evaluation. The paper therefore asserts that its LLM-based methodology is a reliable proxy for human judgment in RAG evaluation.

Load-bearing premise

The whole validation rests on treating the manual annotators' Borda scores as ground truth, while those annotators were only shown answers from the teams that the automated judge had already ranked highest.

Editorial extensions

If this is right

  • Future RAG competitions or benchmarks can rely on LLM-based scoring to rank many teams quickly, reserving manual review for finalists.
  • The 105-question shared-seed design offers a simple calibration mechanism for comparing systems evaluated in separate sessions.
  • The reported correlation suggests that claim-level NLI-based correctness scores capture much of what human annotators mean by answer quality.
  • The result that all 25 active teams beat no-RAG Falcon3 indicates that, on this fixed corpus and generator, retrieval augmentation consistently improves answer correctness.
  • Organizers may reuse the same faithfulness formulation and the first-300-words evaluation budget as a low-cost standard for future live challenges.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The r=0.8826 is computed only on the top 13 teams, which were themselves chosen by the LLM score being validated; restricting the range this way typically inflates correlation, so the true agreement on the full set of 25 teams is probably lower.
  • No confidence interval or inter-annotator agreement is reported, so the precision of 0.88 is unknown; a reader should treat it as an order-of-magnitude estimate rather than a tight number.
  • Because manual review was limited to teams that scored well, the methodology may be most reliable at the top of the leaderboard, which is exactly where winner decisions happen.
  • A natural extension would be to test the same judge-based pipeline on another corpus or with a different generator LLM to see whether the 0.88-level agreement transfers or is specific to this setup.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. The paper reports on the SIGIR 2025 LiveRAG Challenge, a pseudo-live competition in which 25 teams built RAG systems on a fixed corpus (Fineweb-10BT) with a fixed generator (Falcon3-10B-Instruct) and answered 500 unseen questions within two hours. Evaluation was two-stage: an LLM-as-a-judge (Claude-3.5-sonnet) computed Correctness and Faithfulness scores, and then the top-13 teams according to the LLM-based Correctness score were manually evaluated on 105 shared questions using Coverage, Relatedness, Quality, and a Borda aggregate. The paper claims that all active teams outperformed a no-RAG Falcon3 baseline on Correctness, and that manual scores correlated highly with LLM-based Correctness (Pearson r=0.8826), which is presented as supporting the robustness of the LLM-based evaluation methodology.

Significance. If the validation claim held, the paper would provide valuable public evidence on LLM-as-a-judge reliability in a realistic competitive RAG setting, with a fixed corpus and generator and a two-stage evaluation design. The all-teams-beat-baseline claim is directly supported by Table 3, and the evaluation protocol is described in enough detail to be replicated. The correlation between two independent judgment processes is an honest empirical observation, and the paper credits the manual annotators and makes team reports available. However, the central robustness claim is weakened by a selection-on-the-predictor design: manual evaluation was restricted to the 13 teams with the highest LLM-based Correctness scores, so the reported correlation cannot support conclusions about the full team population.

major comments (2)
  1. [Section 6 and Table 5] The claim that manual evaluation results were 'highly consistent' with LLM-based evaluation, 'supporting the robustness of our LLM-based evaluation methodology,' is not supported by the reported correlation. The Pearson r=0.8826 in Table 5 is computed only over the 13 teams selected for manual evaluation because they had the highest LLM-based Correctness scores (Section 4.2). This is a selection-on-the-predictor design: the sample is restricted to the upper tail of the distribution the metric is meant to rank. The correlation therefore provides no evidence about agreement for middle- or bottom-ranked teams, where ranking errors would also matter. The authors should either manually evaluate a random sample of non-top teams on the same 105 shared questions and recompute the correlation over the full set of teams, or substantially temper the claim to consistency among the top-ranked teams only.
  2. [Section 4.2 and Table 5] The report does not provide any uncertainty or reliability measure for the manual evaluation that serves as the ground truth. With n=13, the 95% confidence interval around r=0.8826 is wide, and the manual scores aggregate judgments from 'more than a dozen' annotators without any reported inter-annotator agreement (e.g., Krippendorff's alpha or Cohen's kappa). Without IAA, it is difficult to distinguish genuine agreement between the two methods from noise in the manual labels, especially given the range restriction discussed above. Please report agreement statistics and confidence intervals, and discuss how the restricted sample affects the interpretation.
minor comments (7)
  1. [Section 4.1] The Correctness score is defined as the harmonic mean of Coverage and Relatedness, but the explicit formula is not given; please include it for reproducibility.
  2. [Section 4.1, Eq. (1)] Please clarify how NLI values of -1 (contradiction) are treated in the Coverage sum; the current notation is ambiguous about whether contradictions reduce the score or are clipped.
  3. [Section 4.2] The description of Borda count aggregation is underspecified; please state how the three Likert metrics are converted to Borda counts per question and how ties are handled.
  4. [Section 3.2] There are two typos: '500 synthetic questions questions' should be '500 synthetic questions', and 'the final prompt submitted to to Falcon3' should be 'the final prompt submitted to Falcon3'.
  5. [Section 5] The phrase 'Interestingly, there is a high correlation' is informal for a journal report; consider 'The Pearson correlation is high' or a similar neutral formulation.
  6. [Table 3] Please state how the Falcon3 no-RAG baseline answers were generated (e.g., whether the model received only the question with no retrieval) and whether the baseline was evaluated on the same 500 questions; this is relevant to the 'all teams outperformed' claim.
  7. [Section 4.1] The hand-set parameters (α=0.7, the 300-word answer cap, and the 10-passage cap) are described but not justified; a sensitivity analysis or at least a discussion of their impact would strengthen the metric's credibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the validation correlation is empirical, though limited by selection-on-the-predictor as a validity concern.

full rationale

The paper makes no derivation claims that reduce to their own inputs. The central robustness claim in Section 6 rests on the Pearson correlation r=0.8826 in Table 5, between LLM-based Correctness and manual Borda scores over 105 shared questions. These are independently produced judgments: the LLM score is not computed from the manual scores nor fitted to them, and the alpha=0.7 weight in Eq. 1 is hand-set rather than optimized. The manual metrics share conceptual definitions with the LLM components (coverage and relatedness), but that is convergent validity, not circularity. Self-citations such as DataMorgana and Pinecone describe tools and infrastructure; they are not invoked to justify the evaluation claim. The main limitation is external validity: manual evaluation was restricted to the 13 teams selected by the LLM-based Correctness score (Sections 4.2 and 5), so the correlation is estimated on a range-restricted sample and may not support robustness across all teams. That is a statistical and design concern, not a circularity concern.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

Core evaluation claims rest on the validity of both the LLM judge and human annotators, on session comparability via 105 shared questions, and on evaluation parameters such as α=0.7, the 300-word response cap, and the 10-passage cap. No invented entities are introduced.

free parameters (3)
  • Alpha (α) in Coverage formula = 0.7
    Eq. 1 in Section 4.1 weights Direct claims at 0.7 and Useful claims at 0.3; set by organizers and not justified by data. It changes Correctness scores and therefore the ranking and reported correlation.
  • Response length cap for LLM evaluation = 300 words
    Section 4.1 limits evaluated answers to the first 300 words, a budget-driven choice that can affect scores for longer answers.
  • Passage count cap for Faithfulness = 10 passages
    Section 4.1 only considers the first 10 submitted passages when computing Faithfulness, which changes faithfulness scores and is not derived from data.
assumptions (4)
  • domain assumption Claude-3.5-sonnet can reliably extract atomic claims, label them Direct/Useful/Useless, and perform NLI entailment as required by Eqs. 1-3.
    The entire LLM-based evaluation pipeline in Section 4.1 depends on this ability; no standalone validation of the judge's claim extraction or NLI accuracy is reported.
  • domain assumption Manual annotations by qualified annotators are accurate and consistent ground truth.
    Section 4.2 uses manual Borda scores as the validation signal, but no inter-annotator agreement or adjudication details are reported.
  • domain assumption The 105 shared questions are representative and make the two sessions' benchmarks comparable.
    Section 3.2 uses 105 shared questions for calibration, but no analysis shows that the two session benchmarks are equally difficult or that 105 questions suffice.
  • domain assumption All LiveRAG evaluation questions are answerable from Fineweb-10BT alone.
    Section 2.1 states this guarantee, but the evidence is not shown; it underpins the fairness of the fixed-corpus benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SIGIR 2025 -- LiveRAG Challenge Report." pith.science (2026). https://pith.science/paper/EQZHMMNJ

@misc{pith2026250704942,
  author       = {Pith},
  title        = {Pith review of: SIGIR 2025 -- LiveRAG Challenge Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EQZHMMNJ}},
  note         = {Machine review of arXiv:2507.04942}
}
read the original abstract

The LiveRAG Challenge at SIGIR 2025, held between March and May 2025, provided a competitive platform for advancing Retrieval-Augmented Generation (RAG) technologies. Participants from academia and industry were invited to develop a RAG-based question-answering system using a fixed corpus (Fineweb-10BT) and a common open-source LLM (Falcon3-10B-Instruct). The goal was to facilitate challenging comparisons of retrieval and prompting strategies. During the Live Challenge Day, 70 teams from 27 different countries provided answers and supportive information to 500 unseen questions within a strict two-hour time window. Evaluation was conducted in two stages: first an automated LLM-as-a-judge approach was used to compute correctness and faithfulness score, then a manual review of top ranked submissions was conducted. The finalists were announced on June 12, 2025, with prizes awarded during the LiveRAG Workshop at SIGIR 2025 in Padua, Italy.

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. RMIT-ADM+S at the SIGIR 2025 LiveRAG Challenge

    cs.IR 2025-06 conditional novelty 4.0 of 10

    The RMIT-ADM+S G-RAG system, combining HyDE-style hypothetical answers with hybrid retrieval and pointwise LLM re-ranking, was declared the winner of the SIGIR 2025 LiveRAG Challenge.

  2. RAGtifier: Evaluating RAG Generation Approaches of State-of-the-Art RAG Systems for the SIGIR LiveRAG Competition

    cs.IR 2025-06 conditional novelty 4.0 of 10

    A RAG pipeline using InstructRAG, Pinecone, and BGE placed third in the 2025 LiveRAG Challenge, though internal evaluation only weakly predicted official scores.

Reference graph

Works this paper leans on

32 extracted references · 20 canonical work pages · cited by 2 Pith papers

  1. [1]

    Retrieval-augmented generation for knowledge- intensive NLP tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt ¨aschel, et al. Retrieval-augmented generation for knowledge- intensive NLP tasks. Advances in Neural Information Processing Systems , 33:9459–9474, 2020. 7 SIGIR 2025 – LiveRAG Challenge Report A PREPRINT

    work page 2020

  2. [2]

    Atlas: Few-shot learning with retrieval augmented lan- guage models

    Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented lan- guage models. Journal of Machine Learning Research, 24(251):1–43, 2023

    work page 2023

  3. [3]

    Retrieval-Augmented Generation for Large Language Models: A Survey.https://arxiv.org/ abs/2312.10997, 2024

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-Augmented Generation for Large Language Models: A Survey.https://arxiv.org/ abs/2312.10997, 2024

    arXiv 2024

  4. [4]

    The great nugget recall: Automating fact extraction and RAG evaluation with large language models

    Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, and Jimmy Lin. The great nugget recall: Automating fact extraction and RAG evaluation with large language models. arXiv preprint arXiv:2504.15068, 2025

    arXiv 2025

  5. [5]

    Ragas: Automated evaluation of retrieval augmented generation

    Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. Ragas: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations , pages 150–158, 2024

    work page 2024

  6. [6]

    Generat- ing diverse QA benchmarks for RAG evaluation with DataMorgana

    Simone Filice, Guy Horowitz, David Carmel, Zohar Karnin, Liane Lewin-Eytan, and Yoelle Maarek. Generat- ing diverse QA benchmarks for RAG evaluation with DataMorgana. https://arxiv.org/abs/2501.12789, 2025

    arXiv 2025

  7. [7]

    CRAG-comprehensive RAG benchmark

    Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Gui, Ziran Jiang, Ziyu Jiang, et al. CRAG-comprehensive RAG benchmark. Advances in Neural Information Processing Systems, 37:10470–10490, 2024

    work page 2024

  8. [8]

    Support evaluation for the trec 2024 RAG track: Comparing human versus llm judges

    Nandan Thakur, Ronak Pradeep, Shivani Upadhyay, Daniel Campos, Nick Craswell, and Jimmy Lin. Support evaluation for the trec 2024 RAG track: Comparing human versus llm judges. arXiv preprint arXiv:2504.15205, 2025

    arXiv 2024

Show all 32 references
  1. [9]

    Rahmani, Varsha Ramineni, Nick Craswell, Bhaskar Mitra, and Emine Yilmaz

    Hossein A. Rahmani, Varsha Ramineni, Nick Craswell, Bhaskar Mitra, and Emine Yilmaz. Towards understand- ing bias in synthetic data for evaluation, 2025

    2025

  2. [10]

    Don’t use llms to make relevance judgments

    Ian Soboroff. Don’t use llms to make relevance judgments. Information retrieval research journal , 1(1):10– 54195, 2025

    2025

  3. [11]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydl ´ıˇcek, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Lean- dro V on Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Process...

    2024

  4. [12]

    Text embeddings by weakly-supervised contrastive pre-training, 2024

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training, 2024

    2024

  5. [13]

    Accurate and efficient metadata filtering in pinecone’s serverless vector database

    Amir Ingber and Edo Liberty. Accurate and efficient metadata filtering in pinecone’s serverless vector database. In Proceedings of the 1st Workshop on V ector Databases (V ecDB@ICML2025), Vancouver, Canada, July 2025

    2025

  6. [14]

    The fast johnson–lindenstrauss transform and approximate nearest neighbors

    Nir Ailon and Bernard Chazelle. The fast johnson–lindenstrauss transform and approximate nearest neighbors. SIAM Journal on Computing, 39(1):302–322, 2009

    2009

  7. [15]

    Product quantization for nearest neighbor search

    Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1):117–128, 2011

    2011

  8. [16]

    Y . A. C.Malkov and D. A. Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(4):824–836, 2020

    2020

  9. [17]

    Generating Q&A benchmarks for RAG evaluation in enterprise settings

    Simone Filice, Guy Horowitz, David Carmel, Zohar Karnin, Liane Lewin-Eytan, and Yoelle Maarek. Generating Q&A benchmarks for RAG evaluation in enterprise settings. In Proceedings of the 63st Annual Meeting of the Association for Computational Linguistics (Industry Track) , 2025

    2025

  10. [18]

    RMIT-ADM+S at the SIGIR 2025 LiveRAG Challenge

    Kun Ran, Shuoqi Sun, Khoi Nguyen Dinh Anh, Damiano Spina, and Oleg Zendel. RMIT-ADM+S at the SIGIR 2025 LiveRAG Challenge. https://arxiv.org/abs/2506.14516, 2025

    2025 arXiv

  11. [19]

    RAGtifier: Evaluating RAG Generation Approaches of State-of-the-Art RAG Systems for the SIGIR LiveRAG Competition

    Tim Cofala, Oleh Astappiev, William Xion, and Hailay Teklehaymanot. RAGtifier: Evaluating RAG Generation Approaches of State-of-the-Art RAG Systems for the SIGIR LiveRAG Competition. https://arxiv.org/ abs/2506.14412, 2025

    2025 arXiv

  12. [20]

    CIIR@LiveRAG 2025: Optimizing Multi-Agent Re- trieval Augmented Generation through Self-Training

    Alireza Salemi, Mukta Maddipatla, and Hamed Zamani. CIIR@LiveRAG 2025: Optimizing Multi-Agent Re- trieval Augmented Generation through Self-Training. https://arxiv.org/abs/2506.10844, 2025

    2025 arXiv

  13. [21]

    LTRR: Learning To Rank Retrievers for LLMs

    To Eun Kim and Fernando Diaz. LTRR: Learning To Rank Retrievers for LLMs. https://arxiv.org/abs/ 2506.13743, 2025. 8 SIGIR 2025 – LiveRAG Challenge Report A PREPRINT

    2025 arXiv

  14. [22]

    DoTA-RAG: Dynamic of Thought Aggregation RAG

    Saksorn Ruangtanusak, Natthapath Rungseesiripak, Peerawat Rojratchadakorn, Monthol Charattrakool, and Nat- apong Nitarach. DoTA-RAG: Dynamic of Thought Aggregation RAG. https://arxiv.org/abs/2506. 12571, 2025

    2025

  15. [23]

    PreQRAG – Classify and Rewrite for Enhanced RAG

    Damian Martinez, Catalina Riano, and Hui Fang. PreQRAG – Classify and Rewrite for Enhanced RAG. https: //arxiv.org/abs/2506.17493, 2025

    2025 arXiv

  16. [24]

    RAGentA: Multi-Agent Retrieval-Augmented Generation for Attributed Question Answering

    Ines Besrour, Jingbo He, Tobias Schreieder, and Michael F ¨arber. RAGentA: Multi-Agent Retrieval-Augmented Generation for Attributed Question Answering. https://arxiv.org/abs/2506.16988, 2025

    2025 arXiv

  17. [25]

    TopClustRAG at SIGIR 2025 LiveRAG Challenge

    Juli Bakagianni, John Pavlopoulos, and Aristidis Likas. TopClustRAG at SIGIR 2025 LiveRAG Challenge. https://arxiv.org/abs/2506.15246, 2025

    2025 arXiv

  18. [26]

    Knowledge-aware diverse reranking for cross-source question answering

    Tong Zhou. Knowledge-aware diverse reranking for cross-source question answering. https://arxiv.org/ abs/2506.20476, 2025

    2025 arXiv

  19. [27]

    Leveraging LLM-Assisted Query Understanding for Live Retrieval-Augmented Generation

    Guanting Dong, Xiaoxi Li, Yuyao Zhang, and Mengjie Deng. Leveraging LLM-Assisted Query Understanding for Live Retrieval-Augmented Generation. https://arxiv.org/abs/2506.21384, 2025

    2025 arXiv

  20. [28]

    UiS- IAI@LiveRAG: Retrieval-Augmented Information Nugget-Based Generation of Responses

    Weronika Łajewska, Ivica Kostric, Gabriel Iturra-Bocaz, Mariam Arustashvili, and Krisztian Balog. UiS- IAI@LiveRAG: Retrieval-Augmented Information Nugget-Based Generation of Responses. https://arxiv. org/abs/2506.22210, 2025

    2025 arXiv

  21. [29]

    HLTCOE at LiveRAG: GPT- Researcher using ColBERT retrieval

    Kevin Duh, Eugene Yang, Orion Weller, Andrew Yates, and Dawn Lawrie. HLTCOE at LiveRAG: GPT- Researcher using ColBERT retrieval. https://arxiv.org/abs/2506.22356, 2025

    2025 arXiv

  22. [30]

    A survey on LLM-as-a-judge

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on LLM-as-a-judge. arXiv preprint arXiv:2411.15594, 2024

    2024 arXiv

  23. [31]

    Initial nugget evaluation results for the trec 2024 rag track with the autonuggetizer framework, 2024

    Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, and Jimmy Lin. Initial nugget evaluation results for the trec 2024 rag track with the autonuggetizer framework, 2024

    2024

  24. [32]

    Rank aggregation methods for the web

    Cynthia Dwork, Ravi Kumar, Moni Naor, and Dandapani Sivakumar. Rank aggregation methods for the web. In Proceedings of the 10th international conference on World Wide Web , pages 613–622, 2001. 9

    2001

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.