HLTCOE at LiveRAG: GPT-Researcher using ColBERT retrieval

Andrew Yates; Dawn Lawrie; Eugene Yang; Kevin Duh; Orion Weller

REVIEW 2 major objections 7 minor 1 cited by

HLTCOE at LiveRAG: GPT-Researcher using ColBERT retrieval

T0 review · 2 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper reports that a RAG system using ColBERT/PLAID-X retrieval, LLM-generated follow-up queries, and cosine-similarity snippet filtering finished fifth among participating teams in correctness on the LiveRAG benchmark, with a score…

desk verdict A credible fifth-place shared-task result undermined by a model-name inconsistency that makes the described pipeline unverifiable. read the letter →

arxiv 2506.22356 v1 pith:DULSWRVD submitted 2025-06-27 cs.IR

Kevin Duh , Eugene Yang , Orion Weller , Andrew Yates , Dawn Lawrie This is my paper

classification cs.IR

keywords retrieval-augmentedgenerationdenseretrievalColBERTPLAID-XmultilingualquerysnippetfilteringLiveRAG

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports a retrieval-augmented generation (RAG) system built for the LiveRAG benchmark and claims that its specific configuration is competitive on that task. The pipeline runs an initial ColBERT/PLAID-X search over a local index of the FineWeb-10BT collection, asks an LLM to propose two additional search queries from the top retrieved passages, retrieves three passages per query, and then filters the passages into snippets by cosine similarity against the original question. Accepted snippets are concatenated and passed to a 10-billion-parameter model that writes a 200-word answer. The system finished fifth of the participating teams in correctness with a score of 1.070111, while its faithfulness score was 0.340711, ranking 14th. The paper's claim is that this combination of token-level dense retrieval, LLM-guided query expansion, and lightweight embedding filtering is a viable live RAG configuration.

What carries the argument

The load-bearing mechanism is the ColBERT bi-encoder, which encodes each passage as a collection of dense token vectors rather than a single vector, enabling token-level similarity between query and passage. PLAID-X compresses this index so it can run over a 10-billion-token collection at roughly 400 ms per query on a V100 GPU. Around retrieval, the system wraps three components: a query-generation prompt to Qwen2.5-7B-Instruct that proposes two additional searches from the initial context; a filtering stage that chunks passages into 1000-character snippets with 100-character overlap and keeps only snippets whose m2-bert cosine similarity to the original question is at least 0.35; and a response prompt to Falcon3-10B-Instruct that generates up to 200 words from the concatenated accepted snippets. Snippets are fed in passage-rank order per query, not in relevance order.

What would settle it

Rerun the same pipeline on the LiveRAG questions with the filtering threshold varied over a grid (for example, 0.0, 0.2, 0.35, 0.5, 0.7) while keeping retrieval and generation fixed; if the correctness or faithfulness score moves materially with the threshold, the reported ranking depends on an unexamined choice. A second check would be to compare against a version with no snippet filtering at all.

Watch

Extended reading notes

Core claim

The central claim is that a RAG pipeline organized around per-token dense retrieval with a compressed multilingual index can hold its own on a live question-answering challenge. Concretely, the paper asserts that its architecture—ColBERT/PLAID-X over FineWeb-10BT, LLM-generated follow-up queries, m2-bert-80M-8k-retrieval snippet filtering with a 0.35 cosine threshold, and Falcon3-10B answer generation—produces correct answers often enough to place fifth with a correctness score of 1.070111. It further reports that faithfulness lagged well behind correctness (0.340711, 14th), and that an anecdotal comparison between a 10B and a 70B response model showed no obvious quality difference, suggesting that retrieval and context construction carry the performance.

Load-bearing premise

The system assumes that a cosine similarity cutoff of 0.35 between question and snippet embeddings reliably separates useful context from noise, but the paper provides no calibration or ablation for this threshold.

Editorial extensions

If this is right

If the fifth-place result is representative, then a RAG system built on ColBERT-style dense retrieval plus LLM-generated query expansion is a strong practical baseline for live question answering over web-scale collections.
The described snippet filter is cheap enough to run inline, and the reported end-to-end time of just over one hour for 500 questions shows that the configuration meets live-challenge latency constraints.
The gap between correctness and faithfulness indicates that the bottleneck for this pipeline is not finding relevant passages but keeping the generated answer grounded in them.
The anecdotal 10B-versus-70B response-model comparison implies that, within this architecture, retrieval and context construction deserve optimization effort before scaling the generator.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The 0.35 filtering threshold is a likely leverage point: varying it could change both the number of snippets in the prompt and the final answer, and the paper gives no evidence that 0.35 is better than nearby values.
Reordering accepted snippets by cosine similarity rather than by passage rank is a natural cheap experiment that might improve faithfulness without any retrieval change.
Because the retrieval model was trained for multilingual retrieval, the same pipeline may transfer to non-English LiveRAG questions, though the paper only reports English behavior.
A controlled comparison that ablates the LLM query-generation step would isolate how much of the correctness score comes from query expansion versus the underlying dense retriever.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague—

The paper is a short shared-task write-up of the HLTCOE LiveRAG system. It reports a fifth-place correctness score, which is an externally measured result and the main value of the paper. The system is a clean combination of existing components: ColBERT/PLAID-X retrieval, Qwen-based query generation, m2-bert filtering, and Falcon3-based answer generation. The engineering details—reducing generated queries from four to two, the latency fix for serving passages—are useful for anyone building a live RAG system. The example output and prompt listings give a concrete sense of the pipeline.

The main soft spot is an internal inconsistency in the model identity. Section 2.1 names 'Qwen2.5-7B-Instruct-Turbo' as the query generator, but footnote 1 links to the 72B Instruct model. The abstract and introduction say 7B. These are different checkpoints with different cost and output behavior. If the system that actually ran used the 72B model, the fifth-place result does not validate the 7B configuration. The paper releases no code or index, so the discrepancy can't be resolved from outside. This is a genuine reproducibility flaw, not a quibble.

The other soft spots are more minor. The 0.35 cosine filtering threshold is reported without calibration or ablation. The Falcon-vs-Llama comparison in Table 2 is anecdotal, which the authors acknowledge. The faithfulness score of 0.34 is reported but not discussed, and the paper doesn't say why the system is strong on correctness but weak on faithfulness.

The central claim—fifth place with this overall architecture—holds up as an external benchmark result. But the paper's internal description needs to be corrected before the result can be attributed to a specific pipeline. For a shared-task system description, that's a fixable issue.

This is worth a serious referee if the venue is a workshop or shared-task proceedings. I'd recommend acceptance conditional on the authors clarifying which Qwen model was used, releasing whatever code they can, and ideally adding a small ablation on the threshold. For a general IR audience, it's a modest data point; for someone building a practical RAG system, it's a useful recipe.

Referee Report

2 major / 7 minor

Summary. The paper is a competition/system description for HLTCOE's LiveRAG entry. It describes an end-to-end RAG pipeline: a GPT-Researcher framework with Qwen2.5-7B query generation, ColBERT/PLAID-X multilingual retrieval over FineWeb-10BT, snippet filtering by m2-bert-80M-8k-retrieval cosine similarity with a fixed threshold of 0.35, and Falcon3-10B response generation. The system is reported to have placed 5th among 25 teams in correctness (score 1.070111) and 14th in faithfulness (score 0.340711). The paper includes one detailed example, histograms of prompt lengths, and an anecdotal comparison of Falcon3-10B with Llama3.1-70B.

Significance. If the configuration is as described, the report provides a reproducible recipe for a competitive LiveRAG system, with the official leaderboard score serving as external validation. The main strengths are the clear enumeration of the pipeline stages, the prompts, and several parameter values, as well as the honest reporting of both correctness and faithfulness scores, including a lackluster faithfulness result. The paper's usefulness as a reference point, however, depends on resolving the model identity inconsistency and on providing some support for the filtering threshold; without those, the described system is not unambiguously defined and the contribution of individual components to the fifth-place score remains unclear.

major comments (2)

[Section 2.1 and footnote 1] The query generator is identified inconsistently: the abstract and Section 1 say Qwen2.5-7B-Instruct, Section 2.1 says Qwen2.5-7B-Instruct-Turbo, and footnote 1 links to Qwen/Qwen2.5-72B-Instruct. These are different checkpoints, and the exact configuration that produced the reported fifth-place score is therefore unspecified. Please correct the model name and link, and state the precise checkpoint and quantization actually used in the official submission.
[Section 2.1] The snippet filter threshold of 0.35 is presented as a fixed decision rule with no calibration, ablation, or sensitivity analysis. Because this threshold directly determines the context given to the answer generator, the contribution of the filtering step to the reported result is not established. Please provide at least a small development-set analysis showing how the threshold affects the number and quality of retained snippets, or explicitly frame the threshold as an unoptimized design choice.

minor comments (7)

[Section 4 and Table 2] The Falcon3-10B versus Llama3.1-70B comparison is anecdotal: it is based on three ad-hoc questions with no scoring, no repeated samples, and no controlled context length. The resulting suggestion that retrieval and context construction matter more than generator size is not supported by these data; please soften the claim or provide a quantitative comparison.
[Section 3] The development dataset "DataMorgana" is mentioned without a reference or a URL; please provide a citation or a pointer so that readers can understand what data were used for spot checks.
[Section 4] The response generator is called "Falcon3-10B-Instruct" in the abstract and Section 2.2, but "Falcon-10B Instruct" in Section 4; please use a consistent model name throughout.
[Appendix A] The phrase "Table 2 compares Falcom to Llama generation" contains a typo: "Falcom" should be "Falcon".
[Section 5] The sentence "This may of interesting implcation for RAG tasks" contains two typos; it should read "This may be of interesting implication for RAG tasks" or similar.
[Figure 4] The histograms in Figure 4 lack axis labels and units for the final prompt length; please add these to make the figure interpretable.
[Section 2.2] The expression "⌈length of(Passage)/1000⌉" is awkward; consider reformatting the snippet-count formula and clarifying whether character or token length is meant.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the LiveRAG fifth-place score is an external benchmark result, and no fitted quantity is renamed as a prediction.

full rationale

This paper is a shared-task system description, not a derivation. Its central quantitative claim is the fifth-place correctness score of 1.070111 reported in Section 4 from LiveRAG's automatic evaluation; that result is external to the paper and is not defined in terms of any parameter fitted here. The pipeline components—Qwen2.5 query generation, ColBERT/PLAID-X retrieval, m2-bert cosine filtering, and Falcon3 response generation—are described with fixed choices, and no equation is derived. The 0.35 cosine threshold in Section 2.1 is an uncalibrated engineering choice, not a parameter fitted to the reported score, so it cannot make the score circular. The model-identity inconsistency in Section 2.1 (text says 'Qwen2.5-7B-Instruct-Turbo' while footnote 1 links 'Qwen/Qwen2.5-72B-Instruct') is a reproducibility and internal-consistency gap, not a circular reduction. Several references are to prior work by overlapping authors ([4], [9], [10]), but they are used as component provenance and system-adaptation history, not as proof of the LiveRAG placement; the placement is justified by the external evaluation. No self-definitional, fitted-input-as-prediction, uniqueness-import, or ansatz-smuggling pattern is present. Therefore the derivation chain is self-contained with respect to circularity.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim depends on benchmark-level assumptions, including corpus coverage and model usefulness, and on several hand-set hyperparameters such as the 0.35 threshold, 450-token passages, and 1000-character snippets. None of these are derived; they are engineering choices made during development.

free parameters (6)

snippet similarity threshold = 0.35
A snippet is kept if its cosine similarity to the question is at least 0.35 (Section 2.1). No calibration or sensitivity analysis is reported.
top passages per query = 3
The initial query and each generated query return the top three passages, yielding up to nine passages (Sections 2.1 and 2.2).
max generated queries = 2
The query generator prompt sets max_iterations to 2, reduced from 4 during development for speed (Sections 2.1 and 3).
passage length = 450 tokens
Documents are divided into non-overlapping passages of 450 tokens for indexing (Section 2.1).
snippet chunk size = 1000 characters with 100 overlap
Passages are chunked into snippets of 1000 characters with 100 character overlap for filtering (Section 2.1).
answer word limit = 200 words
The response generation prompt restricts the final answer to at most 200 words (Figure 3).

assumptions (4)

domain assumption FineWeb-10BT contains relevant, current passages for the LiveRAG questions.
The system relies entirely on this corpus for retrieval; no external knowledge source is used (Section 2.1).
domain assumption The multilingual ColBERT/PLAID-X model retrieves useful passages for LiveRAG questions.
Retrieval quality is assumed from the model's Translate-Distill training rather than validated on LiveRAG questions in this paper (Section 2.1).
domain assumption m2-bert-80M-8k-retrieval cosine similarity measures snippet relevance well enough for the 0.35 threshold to be useful.
The filter threshold is asserted without calibration or ablation (Section 2.1).
domain assumption Language models generate useful follow-up queries and final answers from the supplied context.
Prompt quality and model behavior are assumed; only anecdotal examples are shown (Figures 2 and 3, Table 1).

how reviews work

0 comments

Cite this review

Pith. "Pith review of HLTCOE at LiveRAG: GPT-Researcher using ColBERT retrieval." pith.science (2026). https://pith.science/paper/DULSWRVD

@misc{pith2026250622356,
  author       = {Pith},
  title        = {Pith review of: HLTCOE at LiveRAG: GPT-Researcher using ColBERT retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DULSWRVD}},
  note         = {Machine review of arXiv:2506.22356}
}

read the original abstract

The HLTCOE LiveRAG submission utilized the GPT-researcher framework for researching the context of the question, filtering the returned results, and generating the final answer. The retrieval system was a ColBERT bi-encoder architecture, which represents a passage with many dense tokens. Retrieval used a local, compressed index of the FineWeb10-BT collection created with PLAID-X, using a model fine-tuned for multilingual retrieval. Query generation from context was done with Qwen2.5-7B-Instruct, while filtering was accomplished with m2-bert-80M-8k-retrieval. Up to nine passages were used as context to generate an answer using Falcon3-10B. This system placed 5th in the LiveRAG automatic evaluation for correctness with a score of 1.07.

Figures

Figures reproduced from arXiv: 2506.22356 by the authors.

**Figure 2.** Query Generation Prompt Information: "{context}" — Using the above information, answer the following query or task: "{question}" in one or two sentences. Use at most {total_words} words [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Response Generation Prompt • date - the date the software is run • context - a concatenation of the top three ranked passages using the question as the search query • dynamic_examples - based on max_iterations and in this case is "query 1", "query 2" This prompt was issued to the Qwen2.5-7B-Instruct-Turbo model. The top three passages were kept from each of the two queries generated as well as the original query. Th… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Histogram of number of unique documents (left), number of snippets (middle), and corresponding final prompt [PITH_FULL_IMAGE:figures/full_fig_p003_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

SIGIR 2025 -- LiveRAG Challenge Report
cs.CL 2025-07 conditional novelty 3.0 of 10

In the SIGIR 2025 LiveRAG Challenge, all 25 active RAG teams beat the no-RAG baseline on LLM-judged correctness, and LLM scores correlated with human scores at r=0.88.

Reference graph

Works this paper leans on

12 extracted references · 7 canonical work pages · cited by 1 Pith paper

[1]

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guil- laume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116 (2019)

arXiv 2019
[2]

Tobias Domhan, Michael Denkowski, David Vilar, Xing Niu, Felix Hieber, and Kenneth Heafield. 2020. The Sockeye 2 Neural Machine Translation Toolkit at AMTA 2020. In Proceedings of the 14th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track) . Association for Machine Translation in the Americas, Virtual, 110–115

work page 2020
[3]

Dan Fu, Simran Arora, Jessica Grogan, Isys Johnson, Evan Sabri Eyuboglu, Armin Thomas, Benjamin Spector, Michael Poli, Atri Rudra, and Christopher Ré. 2023. Monarch mixer: A simple sub-quadratic gemm-based architecture. Advances in Neural Information Processing Systems 36 (2023), 77546–77603

work page 2023
[4]

Oard, Luca Soldanini, and Eugene Yang

Dawn Lawrie, Sean MacAvaney, James Mayfield, Paul McNamee, Douglas W. Oard, Luca Soldanini, and Eugene Yang. 2025. Overview of the TREC 2024 NeuCLIR Track. In The Thirty-Third Text REtrieval Conference (TREC 2024) Pro- ceedings

work page 2025
[5]

Suraj Nair, Eugene Yang, Dawn Lawrie, Kevin Duh, Paul McNamee, Kenton Murray, James Mayfield, and Douglas W. Oard. 2022. Transfer Learning Ap- proaches for Building Cross-Language Dense Retrieval Models. In Proceed- ings of the 44th European Conference on Information Retrieval (ECIR) . https: //arxiv.org/abs/2201.08471

work page Pith review arXiv 2022
[6]

Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Ran- gan Majumder, and Li Deng. 2016. MS MARCO: A Human Generated MA- chine Reading COmprehension Dataset. arXiv preprint arXiv:1611.09268 (2016). Kevin Duh, et al. arXiv:1611.09268 http://arxiv.org/abs/1611.09268

arXiv 2016
[7]

Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li,...

arXiv 2025
[8]

TII Team. 2024. The Falcon 3 family of Open Models

work page 2024

Show all 12 references

[9]

Oard, and Scott Miller

Eugene Yang, Dawn Lawrie, James Mayfield, Douglas W. Oard, and Scott Miller
[10]

Eugene Yang, Dawn Lawrie, Orion Weller, and James Mayfield. 2025. HLTCOE at TREC 2024 NeuCLIR Track. In The Thirty-Third Text REtrieval Conference (TREC

2025
[12]

unaffected themselves, the host birds can spread the infection to susceptible species, especially domesticated chickens, turkeys and geese

Proceedings. A EXAMPLES Table 1 contains example output. Table 2 compares Falcom to Llama generation. HLTCOE at LiveRAG: GPT-Researcher using ColBERT retrieval Table 1: Example system output. Generated queries are the result from the LLM call shown in Figure 2. Filtered passag...

2025
[2024]

In Proceedings of the 46th European Conference on Information Retrieval (ECIR)

Translate-Distill: Learning Cross-Language Dense Retrieval by Translation and Distillation. In Proceedings of the 46th European Conference on Information Retrieval (ECIR). https://arxiv.org/abs/2401.04810

arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guil- laume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116 (2019)

arXiv 2019

[2] [2]

Tobias Domhan, Michael Denkowski, David Vilar, Xing Niu, Felix Hieber, and Kenneth Heafield. 2020. The Sockeye 2 Neural Machine Translation Toolkit at AMTA 2020. In Proceedings of the 14th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track) . Association for Machine Translation in the Americas, Virtual, 110–115

work page 2020

[3] [3]

Dan Fu, Simran Arora, Jessica Grogan, Isys Johnson, Evan Sabri Eyuboglu, Armin Thomas, Benjamin Spector, Michael Poli, Atri Rudra, and Christopher Ré. 2023. Monarch mixer: A simple sub-quadratic gemm-based architecture. Advances in Neural Information Processing Systems 36 (2023), 77546–77603

work page 2023

[4] [4]

Oard, Luca Soldanini, and Eugene Yang

Dawn Lawrie, Sean MacAvaney, James Mayfield, Paul McNamee, Douglas W. Oard, Luca Soldanini, and Eugene Yang. 2025. Overview of the TREC 2024 NeuCLIR Track. In The Thirty-Third Text REtrieval Conference (TREC 2024) Pro- ceedings

work page 2025

[5] [5]

Suraj Nair, Eugene Yang, Dawn Lawrie, Kevin Duh, Paul McNamee, Kenton Murray, James Mayfield, and Douglas W. Oard. 2022. Transfer Learning Ap- proaches for Building Cross-Language Dense Retrieval Models. In Proceed- ings of the 44th European Conference on Information Retrieval (ECIR) . https: //arxiv.org/abs/2201.08471

work page Pith review arXiv 2022

[6] [6]

Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Ran- gan Majumder, and Li Deng. 2016. MS MARCO: A Human Generated MA- chine Reading COmprehension Dataset. arXiv preprint arXiv:1611.09268 (2016). Kevin Duh, et al. arXiv:1611.09268 http://arxiv.org/abs/1611.09268

arXiv 2016

[7] [7]

Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li,...

arXiv 2025

[8] [8]

TII Team. 2024. The Falcon 3 family of Open Models

work page 2024