REVIEW 4 major objections 4 minor 34 references
GOLFer: Smaller LM-Generated Documents Hallucination Filter & Combiner for Query Expansion in Information Retrieval
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read GOLFer filters hallucinated sentences from small-LM output and reweights the rest, making an 8-billion-parameter model match query-expansion results that previously needed much larger LLMs.
desk verdict A credible engineering contribution whose abstract overstates consistency; the combiner carries the gains while the hallucination filter is unvalidated and its ablation is mixed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the hallucination score $H(s_j^i)=F(s_j^i)\cdot C(s_j^i)$. $F$ is the average, over tokens, of token entropy times the average attention that later tokens in the last Transformer layer pay to that token; $C$ is the mean NLI contradiction probability, computed with DeBERTa-v3-large, of the sentence against every other generated passage. Sentences with $H$ above 0.8 are dropped. The combiner then builds the expanded query: for sparse retrieval it repeats the original query 20 times and adds the five filtered passages, and for dense retrieval it sums document embeddings weighted by average generation probability while fixing the original query weight at $\beta=0.6$. These two components are the mechanism that lets a smaller LM's output be used as evidence without poisoning the query vector.
What would settle it
Have annotators label each sentence in the LLaMA-3-8B-Instruct hypothetical passages as factual or hallucinated, then compute the $H=F\cdot C$ score for those sentences; if the score's separation, say its AUC, is near chance, the filter is not doing the claimed work and the combiner alone explains the results.
Extended reading notes
Core claim
The paper's central claim is that the bottleneck for small-LM query expansion is not the generator but what is done with its output. Removing sentences whose hallucination score $H(s)=F(s)\cdot C(s)$ exceeds 0.8, then combining the filtered passages with the original query through a weight vector, turns LLaMA-3-8B-Instruct into an expansion engine that beats other small-LM methods everywhere it is tested and remains competitive with methods built on GPT-3.5 and GPT-4o. The paper reports gains for BM25, ANCE, Colbert_v2, Aggretriever v1/v2, and Contriever, with the combiner module doing the heavy lifting and the filter adding further gains.
Load-bearing premise
The method rests on assuming that a high product of token uncertainty, attention from later tokens, and contradiction with other generated passages really marks a hallucinated sentence, and the paper gives no human validation that this product tracks factuality.
Editorial extensions
If this is right
- With BM25, GOLFer lifts nDCG@10 beyond query2doc on MS MARCO and most BEIR low-resource datasets.
- With Contriever, GOLFer beats HyDE and query2doc on every low-resource dataset in the paper on nDCG@10 and Recall@100.
- On SciFact and FiQA, GOLFer using LLaMA-3-8B-Instruct exceeds HyDE using GPT-4o in nDCG@10.
- Ablations show both modules are needed, but the combiner alone accounts for most of the gain, with the filter adding a further increase in nDCG@10 and Recall@1k.
- The fixed hyperparameters (threshold 0.8, beta 0.6) are not tuned per dataset, so the method transfers without extra search.
Reading between the lines
- The confidence-weighted combination recipe is a general pattern: treat any LLM-generated text as noisy evidence and down-weight low-confidence spans; the same idea could transfer to retrieval-augmented generation and multi-document QA.
- Because the filter needs last-layer attention scores, it only works on open-weight models; a lightweight predictor trained to imitate H would extend the benefit to black-box APIs.
- The paper tests one generator, LLaMA-3-8B-Instruct; if the filter score tracks factuality, the gains should survive other model families and sizes, which would be a direct test of whether the mechanism, not the model, matters.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes GOLFer, a query expansion method that uses a small open-source LM (LLaMA-3-8B-Instruct) to generate multiple hypothetical documents, filters sentences judged hallucinated by a score that combines token entropy, self-attention, and NLI contradiction (Eqs. 1–8), and then combines the filtered documents with the original query using dataset-independent weights for sparse and dense retrieval (Eqs. 9–11). The method is evaluated on MS MARCO dev, TREC DL19/20, and ten BEIR datasets with several sparse and dense retrievers, comparing against HyDE and query2doc under the same generated documents. The paper claims consistent improvement over other methods using small LMs and competitive performance against large-LLM methods.
Significance. If the reported results hold, GOLFer would be a practically useful contribution: it could lower the cost and access barriers of LLM-based query expansion by using an 8B open model, and the evaluation spans a broad set of datasets and retrievers. The public code release is a concrete strength that supports reproducibility. However, the significance is currently tempered by three issues: the hallucination-filter score, which is the paper's main claimed novelty, is never validated against any ground truth; the ablation study shows that the filter alone can degrade retrieval quality on one of the two test sets; and the 'consistently outperforms' claim is contradicted by several entries in the paper's own tables. These gaps need to be addressed before the central claims can be accepted.
major comments (4)
- [§3.1, Eqs. (1)–(8)] The hallucination-filter score H = F·C is the load-bearing component of the paper's claimed contribution, yet the manuscript provides no evidence that this score actually separates hallucinated sentences from factual ones. There is no comparison with human annotations, no evaluation on a known hallucination benchmark, no calibration analysis, and no sensitivity study for the threshold τ = 0.8. The claim that 0.8 is 'empirically found' (end of §3.1) is not supported by any held-out protocol. Without such validation, the improvements in Tables 1–2 could come from the combiner or from deletion acting as regularization rather than from accurate hallucination detection. In addition, Eq. (6) renormalizes only the entailment and contradiction logits, so neutral sentence-document pairs receive a degenerate contradiction score near 0.5 and add noise to the consistency estimate; this should be corrected or justified.
- [§5, Table 3] The ablation results undercut the claim that the hallucination filter is beneficial. On TREC DL20, 'w/ filter only' decreases Aggretriever_v2's MAP from 47.1 to 45.7 and nDCG@10 from 69.7 to 64.6, and the full model's improvement over 'w/ combiner only' is small (nDCG@10 +0.9 on DL19 and +0.3 on DL20). This pattern is consistent with the combiner (or with deletion as a regularizer) being responsible for the gains, not with accurate detection of hallucinations. The paper should report sentence-level filter accuracy on a labelled hallucination dataset and provide error bars or significance tests for the retrieval metrics to establish that the filter helps beyond chance.
- [§4, Tables 1–2] The abstract and §4 claim that GOLFer 'consistently outperforms' other methods using smaller LMs, but the paper's own tables contain counterexamples. For example, Table 2 shows BM25 nDCG@10 on TREC-COVID of 69.9 for GOLFer versus 72.1 for query2doc, and Table 1 shows ANCE DL19 MAP of 44.1 for GOLFer versus 45.3 for HyDE and 45.0 for query2doc, as well as ANCE DL20 R@1K of 81.2 versus 81.6 for HyDE. Several reported gains are very small (e.g., +0.1 MRR@10 in Table 1), and no variance or significance information is reported for any result. The claims should be qualified to reflect the observed pattern, and the experiments should include multiple runs or paired significance tests so the reader can judge whether differences are meaningful.
- [§3.1–§3.2] The method's key hyperparameters—filter threshold τ = 0.8, combination weight β = 0.6, query repetition count 20, and number of generated documents n = 5—are all fixed values that appear to have been selected by inspecting the evaluated benchmarks. The text says these are not tuned per dataset, but it does not report a tuning protocol, a sensitivity analysis, or a held-out validation set. Given that these values directly control the expanded query, the paper needs to show that performance is stable over reasonable ranges of these parameters, or describe an explicit tuning procedure that does not use the test sets.
minor comments (4)
- [Abstract and §1 contributions] The abstract says 'ten low-resource datasets' while the contribution bullet in §1 says 'nine low-resource datasets form BEIR'; the actual evaluation in §4.1 lists ten BEIR datasets, so the wording should be made consistent.
- [§3.1, Eq. (6)] The notation in Eq. (6) is unclear: C(s_i^j, d_{k≠i}) is used before the set of documents not containing s_i^j is formally described, and the phrase 'across all documents that do not contain s_i^j' would be easier to parse if it were stated before the equation.
- [Table 4] The comparison against GPT-4o uses only four datasets and no significance tests or error bars; the text should acknowledge that this is a small, indicative comparison rather than a broad claim of competitiveness.
- [§5, Table 3] The ablation table reports single numbers for each configuration; adding standard deviations across at least three runs would make the small differences (e.g., 48.4 vs 48.3 MAP on DL19) interpretable.
Circularity Check
GOLFer's consistent-gains claim is partly a fitted outcome: the filter threshold (0.8), combination weight (0.6), and query repetition (20) are set empirically on the same benchmarks used for Tables 1-3, and the hallucination score H=F*C is never validated against external factuality labels; yet the pipeline is not definitionally circular and no self-citation chain is load-bearing.
-
fitted input called prediction
[Section 3.1 (Eq. 8 and threshold 0.8); Section 3.2 (Eq. 11 and beta 0.6); Abstract]
"If the filter score H (si j) is beyond than a certain number, we will delete it since it could be highly hallucinatory. We have empirically found that 0.8 is a generally good value and do not tune it on a dataset basis. ... And we find that β = 0.6 is an effective values for combing 5 hypothesis documents , which we do not tune on a dataset-specific basis."
The two central hyperparameters are 'empirically found' without any held-out protocol, and the same MS MARCO / TREC DL19/20 / BEIR benchmarks are then used in Tables 1-3 to support the abstract's claim that GOLFer 'consistently outperforms'. Because the threshold and beta are selected using the evaluation data, the reported gains are fitted outcomes rather than independent predictions of the method; the claim is partly forced by construction. The filter score H itself is not validated against human hallucination labels, so the only evidence offered for the filter's mechanism is the tuned retrieval performance.
full rationale
No self-citation chain or imported uniqueness theorem is load-bearing: the paper's citations are to external prior work (HyDE, Query2Doc, SelfCheckGPT), and the authors do not cite their own prior results. The core pipeline (entropy x attention factuality score, NLI contradiction, weighted combination) is specified by the paper's own equations rather than derived from the target retrieval scores. The main circularity concern is pattern 2: the free parameters (threshold 0.8, beta 0.6, repetition 20) are described as 'empirically found' on the same benchmarks that are later reported as evidence of consistent gains, and no held-out validation or external hallucination benchmark is used to establish that H=F*C separates factual from hallucinated sentences. Table 3 further shows that filter-only underperforms the backbone on TREC DL20 MAP and nDCG@10, weakening the named mechanism; this is a correctness and generalizability risk rather than a definitional circularity. Overall the paper is not self-referential in a way that makes its derivation equivalent to its inputs, but the headline claim of consistent superiority is partly fitted, so a moderate score is appropriate.
Assumptions & free parameters
free parameters (5)
- Filter threshold tau =
0.8
- Dense combination weight beta =
0.6
- Sparse query repetition count =
20
- Number of generated documents n =
5
- Generation temperature / top-p / max tokens =
0.6 / 0.9 / 128
assumptions (4)
- domain assumption Token entropy negatively correlates with factuality
- domain assumption Average attention from later tokens measures a token's influence on context
- domain assumption NLI contradiction across multiple samples measures inconsistency
- domain assumption Weighted linear combination of query and document embeddings is an effective expansion
Cite this review
Pith. "Pith review of GOLFer: Smaller LM-Generated Documents Hallucination Filter & Combiner for Query Expansion in Information Retrieval." pith.science (2026). https://pith.science/paper/2PD7B5XO
@misc{pith2026250604762,
author = {Pith},
title = {Pith review of: GOLFer: Smaller LM-Generated Documents Hallucination Filter & Combiner for Query Expansion in Information Retrieval},
year = {2026},
howpublished = {\url{https://pith.science/paper/2PD7B5XO}},
note = {Machine review of arXiv:2506.04762}
}
read the original abstract
Large language models (LLMs)-based query expansion for information retrieval augments queries with generated hypothetical documents with LLMs. However, its performance relies heavily on the scale of the language models (LMs), necessitating larger, more advanced LLMs. This approach is costly, computationally intensive, and often has limited accessibility. To address these limitations, we introduce GOLFer - Smaller LMs-Generated Documents Hallucination Filter & Combiner - a novel method leveraging smaller open-source LMs for query expansion. GOLFer comprises two modules: a hallucination filter and a documents combiner. The former detects and removes non-factual and inconsistent sentences in generated documents, a common issue with smaller LMs, while the latter combines the filtered content with the query using a weight vector to balance their influence. We evaluate GOLFer alongside dominant LLM-based query expansion methods on three web search and ten low-resource datasets. Experimental results demonstrate that GOLFer consistently outperforms other methods using smaller LMs, and maintains competitive performance against methods using large-size LLMs, demonstrating its effectiveness.
Figures
Reference graph
Works this paper leans on
-
[1]
Hiteshwar Kumar Azad and Akshay Deepak. 2019. Query expansion techniques for information retrieval: a survey. Information Processing & Management, 56(5):1698–1735. 8
work page 2019
-
[2]
Amos Azaria and Tom Mitchell. 2023. The internal state of an llm knows when it’s lying. arXiv preprint arXiv:2304.13734
arXiv 2023
-
[3]
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, An- drew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268
arXiv 2016
-
[4]
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021. Overview of the trec 2020 deep learning track. Preprint, arXiv:2102.07662
arXiv 2021
-
[5]
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M V oorhees. 2020. Overview of the trec 2019 deep learning track. arXiv preprint arXiv:2003.07820
arXiv 2020
-
[6]
Fernando Diaz, Bhaskar Mitra, and Nick Craswell. 2016. Query expansion with locally-trained word embeddings. arXiv preprint arXiv:1605.07891
arXiv 2016
-
[7]
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783
arXiv 2024
-
[8]
Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166
arXiv 2023
Show all 34 references
-
[9]
Luyu Gao and Jamie Callan. 2021. Unsupervised corpus aware language model pre-training for dense passage retrieval. arXiv preprint arXiv:2108.05540
2021 arXiv
-
[10]
Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2022. Precise zero-shot dense retrieval without relevance labels. arXiv preprint arXiv:2212.10496
2022 arXiv
-
[11]
Zhiguo Gong, Chan Wa Cheang, and U Leong Hou. 2005. Web query expansion by wordnet. In Database and Expert Systems Applications: 16th International Conference, DEXA 2005, Copenhagen, Denmark, August 22-26,
2005
-
[12]
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543
2021 arXiv
-
[13]
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Towards unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118, 2(3)
2021 arXiv
-
[14]
Rolf Jagerman, Honglei Zhuang, Zhen Qin, Xuanhui Wang, and Michael Bendersky. 2023. Query expansion by prompting large language models. arXiv preprint arXiv:2305.03653
2023 arXiv
-
[15]
Vladimir Karpukhin, Barlas O˘guz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen- tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906
2020 arXiv
-
[16]
Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th International ACM SIGIR Conference ...
2021
-
[17]
Sheng-Chieh Lin, Minghan Li, and Jimmy Lin. 2023. Aggretriever: A simple approach to aggregate textual representations for robust dense passage retrieval. Transactions of the Association for Computational Linguistics, 11:436–452
2023
-
[18]
Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. arXiv preprint arXiv:2303.08896
2023 arXiv
-
[19]
Bhaskar Mitra and Nick Craswell. 2017. Neural models for information retrieval.arXiv preprint arXiv:1705.01509
2017 arXiv
-
[20]
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 3...
2022
-
[21]
Dipasree Pal, Mandar Mitra, and Kalyankumar Datta. 2014. Improving query expansion using wordnet. Journal of the Association for Information Science and Technology, 65(12):2469–2478
2014
-
[22]
Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt. 2021. The curious case of hallucinations in neural machine translation. arXiv preprint arXiv:2104.06683
2021 arXiv
-
[23]
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389. 9
2009
-
[24]
Dwaipayan Roy, Debjyoti Paul, Mandar Mitra, and Utpal Garain. 2016. Using word embeddings for automatic query expansion. arXiv preprint arXiv:1606.07608
2016 arXiv
-
[25]
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2021. Colbertv2: Effective and efficient retrieval via lightweight late interaction. CoRR, abs/2112.01488
2021 arXiv
-
[26]
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. Beir: A het- erogenous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663
2021 arXiv
-
[27]
Heyuan Wang, Ziyi Wu, and Junyu Chen. 2019. Multi-turn response selection in retrieval-based chatbots with iterated attentive convolution matching network. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pages 1081–1090
2019
-
[28]
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Simlm: Pre-training with representation bottleneck for dense passage retrieval. arXiv preprint arXiv:2207.02578
2022 arXiv
-
[29]
Liang Wang, Nan Yang, and Furu Wei. 2023. Query2doc: Query expansion with large language models. arXiv preprint arXiv:2303.07678
2023 arXiv
-
[30]
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2019. Neural text generation with unlikelihood training. arXiv preprint arXiv:1908.04319
2019 arXiv
-
[31]
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00808
2020 arXiv
-
[32]
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34:27263–27277
2021
-
[33]
Please write a passage to answer the question. [question_text]
Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji-Rong Wen. 2024. Dense text retrieval based on pretrained language models: A survey. ACM Transactions on Information Systems, 42(4):1–60. A Appendix A.1 Instructions TREC DL19 Instruction message = "Please write a passage to answer ...
2024
-
[2005]
Springer
Proceedings 16, pages 166–175. Springer
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.