arxiv: 2604.11092 · v1 · submitted 2026-04-13 · 💻 cs.IR

Recognition: unknown

ARHN: Answer-Centric Relabeling of Hard Negatives with Open-Source LLMs for Dense Retrieval

Hyewon Choi , Jooyoung Choi , Hansol Jang , Hyun Kim , Chulmin Yun , ChangWook Jun , Stanley Jungkyu Choi

Authors on Pith no claims yet

Pith reviewed 2026-05-10 16:06 UTC · model grok-4.3

classification 💻 cs.IR

keywords dense retrievalhard negative miningLLM-based relabelinganswer-centric signalstraining data cleaningBEIR benchmark

0 comments

The pith

Answer-centric LLM relabeling of hard negatives produces cleaner training data for neural retrievers.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes ARHN, a two-stage method that uses open-source large language models to refine the hard negative passages commonly used when training dense retrieval models. Hard-negative mining frequently includes passages that actually contain answers or are relevant, which creates noisy training signals and hurts model quality. In the first stage the LLM generates a short answer snippet directly from each passage or declares that none exists. In the second stage the same model ranks all candidate passages by how directly they answer the query; passages ranked above the original positive are turned into additional positives, while passages that contain an answer snippet but rank below the positive are dropped from the negative set. Experiments on the BEIR benchmark show that performing both relabeling and filtering together outperforms either step alone and yields more effective retrieval models.

Core claim

ARHN is a two-stage framework that leverages open-source LLMs to refine hard negative samples using answer-centric relevance signals. For each query-passage pair the LLM first generates a passage-grounded answer snippet or indicates that the passage does not support an answer. The LLM then performs listwise ranking over the candidate set to order passages by direct answerability to the query. Passages ranked above the original positive are relabeled as additional positives; among passages ranked below the positive, any that contain an answer snippet are excluded from the negative set. The combined relabeling-plus-filtering strategy consistently improves retrieval effectiveness over either re

What carries the argument

The ARHN two-stage process of LLM-generated answer snippet detection followed by listwise ranking to decide which hard negatives should be relabeled as positives or removed from the negative set.

If this is right

Jointly relabeling false negatives and filtering ambiguous negatives produces cleaner supervision than applying either operation by itself.
The resulting training triplets improve retrieval effectiveness across multiple BEIR datasets.
The pipeline remains practical at large scale because it relies only on open-source models rather than proprietary APIs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same answer-centric signals could be used to audit or repair training sets collected for other retrieval or question-answering tasks.
If the method generalizes, it reduces the need for expensive human verification of hard-negative labels in industrial retrieval pipelines.
Future experiments could test whether the refined negatives also help when the downstream model is a reranker or a generative reader rather than a first-stage retriever.

Load-bearing premise

The open-source LLMs can correctly decide whether a passage supports an answer to the query and can produce accurate listwise rankings without introducing new systematic label errors.

What would settle it

Train a standard dense retriever on triplets refined by ARHN versus the original hard-negative triplets and measure whether nDCG or Recall on the BEIR test sets is lower for the refined version.

Figures

Figures reproduced from arXiv: 2604.11092 by ChangWook Jun, Chulmin Yun, Hansol Jang, Hyewon Choi, Hyun Kim, Jooyoung Choi, Stanley Jungkyu Choi.

**Figure 2.** Figure 2: Overview of the two-stage ARHN pipeline. Given a query [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Figure 3: Stage 1–2 prompts of ARHN. In Stage 1, the LLM ex [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

**Figure 4.** Figure 4: Effect of LLM scale used in ARHN labeling (Stage 1– [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

read the original abstract

Neural retrievers are often trained on large-scale triplet data comprising a query, a positive passage, and a set of hard negatives. In practice, hard-negative mining can introduce false negatives and other ambiguous negatives, including passages that are relevant or contain partial answers to the query. Such label noise yields inconsistent supervision and can degrade retrieval effectiveness. We propose ARHN (Answer-centric Relabeling of Hard Negatives), a two-stage framework that leverages open-source LLMs to refine hard negative samples using answer-centric relevance signals. In the first stage, for each query-passage pair, ARHN prompts the LLM to generate a passage-grounded answer snippet or to indicate that the passage does not support an answer. In the second stage, ARHN applies an LLM-based listwise ranking over the candidate set to order passages by direct answerability to the query. Passages ranked above the original positive are relabeled to additional positives. Among passages ranked below the positive, ARHN excludes any that contain an answer snippet from the negative set to avoid ambiguous supervision. We evaluated ARHN on the BEIR benchmark under three configurations: relabeling only, filtering only, and their combination. Across datasets, the combined strategy consistently improves over either step in isolation, indicating that jointly relabeling false negatives and filtering ambiguous negatives yields cleaner supervision for training neural retrieval models. By relying strictly on open-source models, ARHN establishes a cost-effective and scalable refinement pipeline suitable for large-scale training.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

ARHN is a practical two-stage LLM cleanup for noisy hard negatives in dense retrieval training, but lacks validation that the labels are actually better.

read the letter

The main point is a simple pipeline that uses open-source LLMs to turn some hard negatives into positives and drop ambiguous ones by checking whether passages contain grounded answers to the query. Stage one generates an answer snippet or flags no support; stage two does listwise ranking and promotes anything ranked above the original positive while filtering answer-containing passages below it. They run the three variants on BEIR and report the combination beats either step alone. That specific answer-centric relabeling plus filtering combo is new enough to be worth noting, and sticking to open-source models keeps it usable at scale. The method is easy to understand and applies directly to existing triplet training setups without new model designs. It earns credit for testing the pieces separately and together on a standard benchmark. The soft spot is the missing evidence that the LLM steps actually produce cleaner supervision. The abstract gives no numbers, no error rates on the snippet generation or rankings, and no human checks or inter-annotator agreement. Without that, the reported gains could just reflect shifts in positive/negative counts or LLM quirks rather than reduced label noise. The stress-test concern holds: we need to see whether the LLM decisions are reliable before attributing improvements to better data. This is for retrieval researchers who already mine hard negatives and want a data-cleaning trick to try. It is not a big theoretical step, but the empirical idea is concrete. I would send it to peer review so the authors can add the validation numbers and any ablation on prompt stability.

Referee Report

2 major / 2 minor

Summary. The paper proposes ARHN, a two-stage framework that uses open-source LLMs to refine hard-negative samples for dense retrieval training. In stage 1, the LLM generates a passage-grounded answer snippet or indicates that the passage does not support an answer; in stage 2, an LLM-based listwise ranking orders candidates by answerability, relabeling passages ranked above the original positive as additional positives and excluding passages below the positive that contain answer snippets from the negative set. Experiments on the BEIR benchmark compare relabeling-only, filtering-only, and combined configurations, with the claim that the combined strategy consistently improves over the isolated steps by producing cleaner supervision signals.

Significance. If the LLM decisions prove reliable, ARHN provides a scalable, cost-effective pipeline for reducing label noise in triplet data using only open-source models, which could improve training of neural retrievers on large corpora without proprietary API costs. The empirical focus on BEIR and the explicit comparison of the two stages offer a reproducible test of the joint benefit, though broader adoption would require evidence that gains generalize beyond the evaluated settings and are not artifacts of label-count changes.

major comments (2)

[§4] §4 (Evaluation): The central claim attributes BEIR gains to cleaner supervision from the two-stage LLM process, yet the manuscript supplies no human validation, inter-annotator agreement, or error analysis of the LLM-generated answer snippets and listwise rankings. Without such checks it remains possible that improvements arise from incidental shifts in positive/negative ratios or domain-specific LLM biases rather than genuine noise reduction.
[§4] §4: The abstract and evaluation summary report only that the combined strategy 'consistently improves' without quantitative deltas, per-dataset scores, error bars, or statistical significance tests. These details are load-bearing for assessing whether the observed gains exceed what would be expected from regularization or sampling variance alone.

minor comments (2)

[Abstract] Abstract: The claim of 'consistent gains' would be strengthened by naming the specific BEIR datasets and reporting at least the average nDCG@10 improvement of the combined configuration.
[§3.2] §3.2: The listwise ranking prompt is described at a high level; including the exact prompt template and temperature settings would improve reproducibility of the ranking stage.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on the evaluation section. We address each major comment below and describe the revisions planned for the next version of the manuscript.

read point-by-point responses

Referee: [§4] §4 (Evaluation): The central claim attributes BEIR gains to cleaner supervision from the two-stage LLM process, yet the manuscript supplies no human validation, inter-annotator agreement, or error analysis of the LLM-generated answer snippets and listwise rankings. Without such checks it remains possible that improvements arise from incidental shifts in positive/negative ratios or domain-specific LLM biases rather than genuine noise reduction.

Authors: We acknowledge that the manuscript currently lacks human validation, inter-annotator agreement, or a dedicated error analysis of the LLM outputs. The empirical results on BEIR provide indirect support through consistent gains under the combined configuration, but we agree this does not fully rule out alternative explanations such as label-count shifts. In the revised manuscript we will add a new error-analysis subsection that manually inspects a stratified random sample of LLM-generated answer snippets and listwise rankings (approximately 200 instances across datasets), reports agreement with human judgments, and discusses any observed biases. This addition will strengthen the link between the observed improvements and noise reduction. revision: yes
Referee: [§4] §4: The abstract and evaluation summary report only that the combined strategy 'consistently improves' without quantitative deltas, per-dataset scores, error bars, or statistical significance tests. These details are load-bearing for assessing whether the observed gains exceed what would be expected from regularization or sampling variance alone.

Authors: The full evaluation section already contains per-dataset tables with nDCG@10 and Recall@100 scores for the three configurations. We will revise the abstract to report the average absolute and relative gains across BEIR, and we will augment the tables with standard-error bars and paired significance tests (e.g., Wilcoxon signed-rank) against the baseline. These changes will make the quantitative evidence explicit and allow readers to judge whether the gains exceed sampling variance. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical pipeline with external LLM judgments

full rationale

The paper describes an empirical two-stage framework (answer-snippet generation followed by listwise ranking) that refines hard-negative labels using open-source LLMs, then reports experimental gains on BEIR under relabeling-only, filtering-only, and combined configurations. No equations, fitted parameters, or first-principles derivations appear; the central claim rests on observed performance differences rather than any reduction of outputs to inputs by construction. No self-citations are invoked to justify uniqueness or load-bearing premises, and the method does not rename known results or smuggle ansatzes. The derivation chain is therefore self-contained as a practical pipeline whose validity is tested externally.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that LLM answerability judgments are sufficiently reliable proxies for human relevance labels. No free parameters or invented entities are introduced in the abstract.

axioms (1)

domain assumption LLM-generated answer snippets and listwise rankings provide accurate signals of passage answerability to the query.
Invoked in both stages of the framework description.

pith-pipeline@v0.9.0 · 5594 in / 1247 out tokens · 29830 ms · 2026-05-10T16:06:23.264556+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

36 extracted references · 14 canonical work pages · 3 internal anchors

[1]

Luiz Bonifacio, Hugo Abonizio, Marzieh Fadaee, and Rodrigo Nogueira. 2022. Inpars: Data augmentation for information retrieval using large language models. arXiv preprint arXiv:2202.05144(2022)

work page arXiv 2022
[2]

Yinqiong Cai, Jiafeng Guo, Yixing Fan, Qingyao Ai, Ruqing Zhang, and Xueqi Cheng. 2022. Hard negatives or false negatives: Correcting pooling bias in Choi et al. training neural ranking models. InProceedings of the 31st ACM International Conference on Information & Knowledge Management. 118–127

2022
[3]

Devendra Singh Chaplot. 2023. Albert q. jiang, alexandre sablayrolles, arthur men- sch, chris bamford, devendra singh chaplot, diego de las casas, florian bressand, gianna lengyel, guillaume lample, lucile saulnier, lélio renard lavaud, marie-anne lachaux, pierre stock, teven le scao, thibaut lavril, thomas wang, timothée lacroix, william el sayed.arXiv p...

work page internal anchor Pith review Pith/arXiv arXiv 2023
[4]

Jooyoung Choi, Hyun Kim, Hansol Jang, Changwook Jun, Kyunghoon Bae, Hyewon Choi, Stanley Jungkyu Choi, Honglak Lee, and Chulmin Yun. 2025. LG-ANNA-Embedding technical report.arXiv preprint arXiv:2506.07438(2025)

work page arXiv 2025
[5]

Nachshon Cohen, Hedda Cohen-Indelman, Yaron Fairstein, and Guy Kushilevitz
[6]

InEuropean Conference on Information Retrieval

Indi: Informative and diverse sampling for dense retrieval. InEuropean Conference on Information Retrieval. Springer, 243–258
[7]

Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. The power of noise: Redefining retrieval for rag systems. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 719–729

2024
[8]

Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B Hall, and Ming-Wei Chang. 2022. Promptagator: Few-shot dense retrieval from 8 examples.arXiv preprint arXiv:2209.11755(2022)

work page arXiv 2022
[9]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 4171–4186

2019
[10]

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models.ICLR1, 2 (2022), 3

2022
[11]

Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bo- janowski, Armand Joulin, and Edouard Grave. 2021. Unsupervised dense in- formation retrieval with contrastive learning.arXiv preprint arXiv:2112.09118 (2021)

work page internal anchor Pith review arXiv 2021
[12]

Fan Jiang, Tom Drummond, and Trevor Cohn. 2023. Noisy self-training with synthetic queries for dense retrieval.arXiv preprint arXiv:2311.15563(2023)

work page arXiv 2023
[13]

Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. 2020. Hard negative mixing for contrastive learning.Advances in neural information processing systems33 (2020), 21798–21809

2020
[14]

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick SH Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering.. InEMNLP (1). 6769–6781

2020
[15]

Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, et al. 2024. Gecko: Versatile text embeddings distilled from large language models.arXiv preprint arXiv:2403.20327 (2024)

work page arXiv 2024
[16]

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks.Advances in neural information processing systems33 (2020), 9459–9474

2020
[17]

Chaofan Li, MingHao Qin, Shitao Xiao, Jianlyu Chen, Kun Luo, Yingxia Shao, Defu Lian, and Zheng Liu. 2024. Making text embedders few-shot learners.arXiv preprint arXiv:2409.15700(2024)

work page arXiv 2024
[18]

Gabriel de Souza P Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, and Even Oldridge. 2024. NV-Retriever: Improving text embedding models with effective hard-negative mining.arXiv preprint arXiv:2407.15831 (2024)

work page arXiv 2024
[19]

Ansong Ni, Matt Gardner, and Pradeep Dasigi. 2021. Mitigating false-negative contexts in multi-document question answering with retrieval marginalization. arXiv preprint arXiv:2103.12235(2021)

work page arXiv 2021
[20]

Jingwei Ni, Tobias Schimanski, Meihong Lin, Mrinmaya Sachan, Elliott Ash, and Markus Leippold. 2025. DIRAS: Efficient LLM annotation of document relevance for retrieval augmented generation. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: L...

2025
[21]

Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxi- ang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language ...

2021
[22]

Thilina Chaturanga Rajapakse, Andrew Yates, and Maarten de Rijke. 2024. Nega- tive Sampling Techniques for Dense Passage Retrieval in a Multilingual Setting. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 575–584

2024
[23]

Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models.arXiv preprint arXiv:2104.08663(2021)

work page internal anchor Pith review arXiv 2021
[24]

Nandan Thakur, Crystina Zhang, and Xueguang Ma Jimmy Lin. 2025. Hard Neg- atives, Hard Lessons: Revisiting Training Data Quality for Robust Information Retrieval with LLMs. InFindings of the Association for Computational Linguistics: EMNLP 2025. 9064–9083

2025
[25]

Fei Wang, Xingchen Wan, Ruoxi Sun, Jiefeng Chen, and Sercan O Arik. 2025. Astute rag: Overcoming imperfect retrieval augmentation and knowledge con- flicts for large language models. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 30553–30571

2025
[26]

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Improving text embeddings with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 11897–11916

2024
[27]

Shiqi Wang, Yeqin Zhang, and Cam-Tu Nguyen. 2024. Mitigating the impact of false negative in dense retrieval with contrastive confidence regularization. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 19171–19179

2024
[28]

Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020. Approximate nearest neighbor nega- tive contrastive learning for dense text retrieval.arXiv preprint arXiv:2007.00808 (2020)

work page arXiv 2020
[29]

Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. 2024. Corrective retrieval augmented generation. (2024)

2024
[30]

Zhen Yang, Zhou Shao, Yuxiao Dong, and Jie Tang. 2024. Trisampler: A bet- ter negative sampling principle for dense retrieval. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 9269–9277

2024
[31]

Qiuhai Zeng, Zimeng Qiu, Dae Yon Hwang, Xin He, and William M Campbell
[32]

Unsupervised text representation learning via instruction-tuning for zero- shot dense retrieval.arXiv preprint arXiv:2409.16497(2024)

work page arXiv 2024
[33]

Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma
[34]

InProceed- ings of the 44th international ACM SIGIR conference on research and development in information retrieval

Optimizing dense retrieval model training with hard negatives. InProceed- ings of the 44th international ACM SIGIR conference on research and development in information retrieval. 1503–1512
[35]

Shengming Zhao, Yuheng Huang, Jiayang Song, Zhijie Wang, Chengcheng Wan, and Lei Ma. 2024. Towards understanding retrieval accuracy and prompt quality in rag systems.arXiv preprint arXiv:2411.19463(2024)

work page arXiv 2024
[36]

Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji-Rong Wen. 2024. Dense text retrieval based on pretrained language models: A survey.ACM Transactions on Information Systems42, 4 (2024), 1–60

2024