Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

BiXSE: Improving Dense Retrieval via Probabilistic Graded Relevance Distillation

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a pointwise binary cross-entropy loss over LLM-generated graded relevance probabilities, with a logit bias to offset in-batch negative imbalance, trains dense retrievers that consistently outperform InfoNCE and match

desk verdict BiXSE is a solid, practical idea with broad experiments, but the abstract overstates the pairwise comparison and the manuscript has table and appendix errors that need fixing before the claims are taken at face value. read the letter →

arxiv 2508.06781 v1 pith:IIF4YSVE submitted 2025-08-09 cs.IR cs.AIcs.LG

Christos Tsirigotis , Vaibhav Adlakha , Joao Monteiro , Aaron Courville , Perouz Taslakian This is my paper
classification cs.IR cs.AIcs.LG
keywords dense retrievalgraded relevancebinary cross-entropyknowledge distillationLLM-generated labelscontrastive learningin-batch negativessentence embeddings
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BiXSE trains dense retrieval models with binary cross-entropy over graded relevance probabilities instead of softmax contrastive learning. The paper argues that a single LLM-generated graded label per query, converted to a probability target, carries enough supervision to match pairwise and listwise distillation methods that need several labeled documents per query. Experiments across retrieval and sentence-embedding benchmarks show that BiXSE consistently outperforms InfoNCE on both binary and graded data, matches or exceeds pairwise baselines on LLM-supervised data, degrades more gracefully under label noise, and lets models learn from a wider range of relevance scores without aggressive filtering. The practical stake is that as LLMs make graded relevance labels cheap and abundant, this pointwise loss is a scalable, annotation-efficient way to distill that supervision into fast encoders.

What carries the argument

The mechanism is the BiXSE loss, $$L_{\mathrm{BiXSE}} = -\frac{1}{B}\sum_{i=1}^{B}\sum_{j=1}^{B} \left[ z_{i,j}\log\$\sigma$(s(q_i,d_j)) + (1-z_{i,j})\log\$\sigma$(-s(q_i,d_j)) \right],$$ with $s(q,d)=\alpha q^{\top}d + \beta$, $z_{i,i}$ equal to the graded relevance probability and $z_{i,j}=0$ for $i\neq j$. The continuous target $z$ is what lets one graded label act as richer supervision than a binary label; the learnable logit bias $\beta$ is what lets the loss use all in-batch pairs as negatives without pushing all scores down. Because $\beta$ is optimized at a higher learning rate than the encoder, it absorbs the marginal label distribution, and the encoder must learn query-document content t

What would settle it

Train two BiXSE encoders on the same LightBlue query-document pairs, one with the raw teacher probabilities $z$ as targets and one with $z$ passed through a calibration step that maps teacher probabilities onto human graded relevance distributions from TREC-DL qrels. If the calibrated-target model does not improve nDCG@10 on held-out TREC-DL 2019–2023 queries, then the claim that $z$ is a faithful relevance probability is not what carries BiXSE's gains; the gains would instead come from the loss shape or the logit bias.

Watch

Extended reading notes

Core claim

BiXSE replaces the softmax contrastive objective for dense retrieval with a binary cross-entropy loss over probabilistic graded targets. For each query-document pair, the teacher LLM's token probabilities over discrete ordinal relevance options are averaged into a continuous target $z = \sum_{s\in S} s \, p_{\mathrm{LLM}}(s|q,d)/N$ in $[0,1]$. The student scores a pair as $s(q,d)=\alpha q^{\top} d + \beta$; the loss then penalizes every pair in the batch against $z_{i,i}$ for the labeled document and $0$ for in-batch negatives. The central claim is that this simple pointwise objective extracts more from a single graded label than InfoNCE does from a binary one, and that the learnable logit b

Load-bearing premise

The load-bearing premise is that LLM-generated graded scores, converted to probabilities by averaging the teacher's softmax over discrete options and rescaling to [0,1], are faithful relevance probabilities for every query-document pair; if these targets are miscalibrated or systematically biased, the BCE loss trains the encoder toward a distorted notion of relevance.

Editorial extensions

If this is right

  • If BiXSE is right, dense retrieval training no longer needs multiple labeled comparisons per query: one graded LLM label per query plus in-batch negatives matches pairwise baselines.
  • Because the pointwise BCE loss scales quadratically in batch size while pairwise losses scale cubically, larger batches and larger datasets become feasible within the same memory budget.
  • The noise-robustness result implies that mined hard-negative datasets with false negatives are less harmful under BCE than under InfoNCE, so less aggressive negative mining should be needed.
  • BiXSE narrows the gap to zero-shot LLM rankers: a 3B encoder trained this way lands within about three nDCG@10 points of a 32B ranker, suggesting distillation can substitute for expensive LLM inference at serving time.
  • Training can use a wider spectrum of graded scores without aggressive low-relevance filtering; peak performance comes at a moderate cutoff rather than the strictest one.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if graded LLM labels become the default supervision source, the practical center of gravity in dense retrieval training may shift from contrastive formulations to pointwise BCE, making hard-negative mining a second-order concern.
  • Editorial inference: the logit-bias correction is a general device for any in-batch negative training setup; the same bias-tuned-faster trick could be applied to other BCE-style embedding losses, such as cross-modal alignment, where label imbalance is present.
  • Editorial inference: the paper's noise-robustness gradient argument implies that BCE's advantage should grow with batch size, since one mislabeled pair affects a smaller fraction of BCE terms than of softmax terms; this is directly testable on datasets with known false-negative rates.
  • Editorial extension: because the method's targets are teacher probabilities, a natural next step is calibrating LLM relevance probabilities against human graded judgments; BiXSE's gains should increase if the teacher's probabilities are well calibrated and degrade if they are not.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes BiXSE, a pointwise training objective for dense retrieval that replaces softmax-based InfoNCE with binary cross-entropy (BCE) over graded relevance labels. LLM-produced ordinal judgments are converted to continuous relevance probabilities and used as BCE targets; a scalar logit bias is added to the scoring function to account for the imbalance introduced by in-batch negatives. The authors fine-tune ModernBERT, Llama-3.2, and Qwen 2.5 encoders on English and multilingual mixtures of binary and graded data and evaluate on BEIR, MMTEB, and TREC-DL. They report consistent gains over InfoNCE, favorable comparisons with MarginMSE and Soft InfoNCE, noise-robustness, and competitive performance with pairwise and listwise baselines such as PairwiseBCE and LambdaLoss.

Significance. If supported, the paper would make a useful practical contribution: a simple, scalable, token-efficient way to distill graded LLM relevance judgments into dense encoders, with the notable property that it works with one labeled document per query. The empirical effort is substantial: multiple base models, multilingual and English benchmarks, controlled noise experiments, and a filtering study. The authors also release code. However, the central comparison against pairwise baselines is substantially overstated by the paper's own appendix data, and the main results table contains an arithmetically inconsistent cell. The core pointwise BCE idea is defensible and likely valuable, but the current presentation overclaims its standing relative to LambdaLoss.

major comments (4)
  1. [Table 1; §5 'Main Results'] The Qwen2.5-0.5B-Instruct row reports InfoNCE BEIR=38.25 and BiXSE=73.91 with '+6.0%'. These values are mutually inconsistent: a +6.0% relative gain over 38.25 is about 40.5, not 73.91. Appendix Table 9 lists 73.91 as the BEIR(short) value for BiXSE, so Table 1 appears to have copied the wrong column into the BEIR column. Because Table 1 is the principal evidence for the claim that BiXSE 'consistently outperforms' InfoNCE, this error must be corrected and every aggregate and percentage in the table rechecked against the per-benchmark appendix results.
  2. [Abstract; §5 'Comparison to other training objectives'; Appendix Table 9; Tables 11 and 13] The abstract's claim that BiXSE 'consistently ... matches or exceeds strong pairwise ranking baselines when trained on LLM-supervised data' is not supported by the paper's own tables. In Table 9, LambdaLoss nDCG-v2 beats BiXSE on MTEB Multilingual v1 (55.62 vs. 55.46), TREC 2022 top-100 (37.25 vs. 36.85), and TREC 2023 top-100 (38.23 vs. 38.04); PairwiseBCE also beats BiXSE on TREC 2022. In Tables 11 and 13, LambdaLoss nDCG-v2 wins four of the five hard-negative/batch configurations on both BEIR and MTEB English retrieval, in some cases by more than 3 points. The later acknowledgement that LambdaLoss 'performs close or better' only in the appendix is not enough. The wording should be qualified, e.g., BiXSE is competitive with pairwise losses in some settings and is outperformed by LambdaLoss v2 when ample labeled negatives are available.
  3. [Appendix Tables 11 and 13; 'BGE-M3 Results'] The comparison against pairwise baselines on BGE-M3 varies the number of hard negatives and the batch size jointly and then reports each method's best configuration. This does not isolate the loss function: BiXSE's best BEIR result uses 0 hard negatives with batch 256, while PairwiseBCE and LambdaLoss nDCG-v2 use 1–3 hard negatives with smaller batches. The reported differences may therefore reflect the supervision budget or the batch-size/negative trade-off rather than the intrinsic merit of the objectives. I recommend reporting matched-resource comparisons or explicitly presenting the Pareto frontier over labeled-negative count and batch size.
  4. [Section 4; 'Lightblue Reranker Distillation Dataset'] The paper asserts that the transformed LLM scores z := sum_s s * p_LLM(s|q,d) / N are valid relevance probabilities, but it provides no calibration analysis. This premise is load-bearing for interpreting BiXSE as probabilistic graded-relevance distillation: if the teacher's probabilities are systematically miscalibrated or biased across languages/domains, the BCE targets are distorted. This is not circular because the downstream evaluation is held out, but it is an unvalidated assumption. The paper would be strengthened by a calibration plot or by showing that the conclusions are robust to alternative monotone transformations of the teacher scores.
minor comments (5)
  1. [Section 4, paragraph after Eq. (2)] 'with 0 being most relevant and 1 absolutely relevant' should read '0 being least relevant and 1 absolutely relevant'.
  2. [Section 4, label transformation] For the LightBlue teacher, scores are on 1–5, so the formula z := sum_s s p_LLM(s) / N yields values in [0.2, 1], not [0, 1]. Please clarify the exact affine rescaling used.
  3. [Throughout] The method name is typeset inconsistently as both 'BiXSE' and 'BIXSE' (e.g., abstract and figure captions use different spellings). Please standardize.
  4. [§5, 'Comparison to other training objectives'] The statement 'BIXSE consistently matches or outperforms alternatives like Pairwise BCE and LambdaLoss' is repeated in the main text, but the appendix data contradict a strict reading. Please align the main-text summary with the actual per-benchmark tables.
  5. [Appendix, 'Extended Related Work'] The Deep InfoMax relation is explained well, but the transition from representation learning for images to text retrieval could cite the SigLIP adaptation more explicitly in the main text rather than only in the appendix.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; BiXSE's central claims are empirical and evaluated on held-out benchmarks.

full rationale

BiXSE's claimed improvements are established by direct measurement on held-out benchmarks (BEIR, MMTEB, TREC-DL) after training on LightBlue, E5, and BGE-M3 data. The graded targets z are defined in Section 4 from an external LLM teacher's softmax probabilities (z := sum_s s p_LLM(s|q,d)), not from the student encoder or from evaluation labels; the paper does not fit z, the logit scale alpha, or the logit bias beta to the reported benchmark scores. Alpha and beta are training hyperparameters/optimized variables, with beta explicitly described as modeling the marginal label distribution caused by in-batch negatives—not as a benchmark-tuned parameter. The comparison against InfoNCE, PairwiseBCE, LambdaLoss, and MarginMSE is a measured experimental result with hyperparameter search reported for all methods. Self-citations (LLM2Vec, MMTEB) refer to external published methods or benchmarks and are not used as unverified uniqueness arguments. The only mild selection effect is checkpoint/hyperparameter selection on a BEIR-derived validation subset (Appendix 'Model Selection'), which is standard practice and does not reduce the full-benchmark numbers to construction. The paper's own Appendix Table 9 contradicts the abstract's unqualified 'consistently matches or exceeds'—LambdaLoss nDCG-v2 wins on MTEB Multilingual v1, TREC 2022, and TREC 2023—and the appendix itself acknowledges that LambdaLoss v2 'performs close or better ... on several benchmarks.' That is a correctness/overclaim concern, not circularity: the results are measured, not derived from the method's inputs.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The ledger shows the main free parameters are the logit scale and bias, both of which are tuned rather than derived. The domain assumptions concern the reliability of LLM teacher scores and the validity of in-batch negatives. No new physical or conceptual entities are introduced.

free parameters (2)
  • logit scale alpha = 20
    Inverse temperature in scoring function s(q,d) = alpha * q^T d + beta; tuned by hand and reported as 20.
  • logit bias beta = learned; value not reported
    Added to correct in-batch negative label imbalance; trained with a significantly higher learning rate than the encoder.
assumptions (3)
  • domain assumption LLM-generated graded relevance scores, converted to probabilities and linearly rescaled to [0,1], are valid training targets.
    Core premise of BiXSE; stated in Section 4 where z = sum_s s p_LLM(s|q,d) / N is used as the target.
  • domain assumption In-batch negatives are true negatives, i.e., z_{i,j}=0 for i != j.
    Standard assumption in contrastive training, stated in Section 4; can introduce false negatives.
  • ad hoc to paper A fast-updating logit bias absorbs the marginal label distribution without learning query-document relevance.
    Empirical design choice, supported only by the ablation in Figure 5, not by theoretical derivation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BiXSE: Improving Dense Retrieval via Probabilistic Graded Relevance Distillation." pith.science (2026). https://pith.science/paper/IIF4YSVE

@misc{pith2026250806781,
  author       = {Pith},
  title        = {Pith review of: BiXSE: Improving Dense Retrieval via Probabilistic Graded Relevance Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IIF4YSVE}},
  note         = {Machine review of arXiv:2508.06781}
}
read the original abstract

Neural sentence embedding models for dense retrieval typically rely on binary relevance labels, treating query-document pairs as either relevant or irrelevant. However, real-world relevance often exists on a continuum, and recent advances in large language models (LLMs) have made it feasible to scale the generation of fine-grained graded relevance labels. In this work, we propose BiXSE, a simple and effective pointwise training method that optimizes binary cross-entropy (BCE) over LLM-generated graded relevance scores. BiXSE interprets these scores as probabilistic targets, enabling granular supervision from a single labeled query-document pair per query. Unlike pairwise or listwise losses that require multiple annotated comparisons per query, BiXSE achieves strong performance with reduced annotation and compute costs by leveraging in-batch negatives. Extensive experiments across sentence embedding (MMTEB) and retrieval benchmarks (BEIR, TREC-DL) show that BiXSE consistently outperforms softmax-based contrastive learning (InfoNCE), and matches or exceeds strong pairwise ranking baselines when trained on LLM-supervised data. BiXSE offers a robust, scalable alternative for training dense retrieval models as graded relevance supervision becomes increasingly accessible.

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MemReranker: Reasoning-Aware Reranking for Agent Memory Retrieval

    cs.CL 2026-05 unverdicted novelty 5.0 of 10

    MemReranker applies multi-stage distillation to Qwen3-Reranker to produce reasoning-aware rerankers that outperform baselines on memory tasks with temporal and causal constraints.

  2. MemReranker: Reasoning-Aware Reranking for Agent Memory Retrieval

    cs.CL 2026-05 unverdicted novelty 4.0 of 10

    MemReranker applies multi-teacher pairwise distillation, BCE pointwise training, and InfoNCE contrastive learning on mixed general and memory-specific dialogue data to produce efficient rerankers that improve calibrat...

Reference graph

Works this paper leans on

47 extracted references · 37 canonical work pages · cited by 1 Pith paper

  1. [1]

    Ms marco: A human generated machine reading comprehension dataset

    Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268,

    arXiv

  2. [2]

    Results shown for the best-performing configuration per loss type among BGE-trained models

    TREC-COVID 53.4 50.2 57.1 BioASQ 45.2 42.9 46.8 NFCorpus 38.1 36.5 40.2 NQ 49.8 48.7 52.6 HotpotQA 62.4 60.1 64.0 FiQA-2018 34.9 35.5 38.4 ArguAna 30.6 28.9 34.1 Touch ´e-2020 28.3 26.8 31.7 Quora 89.3 91.2 90.8 DBPedia 42.6 41.3 45.1 SCIDOCS 18.4 17.7 20.5 FEVER 72.9 70.1 75.4 Climate-FEVER 24.7 23.9 27.3 SciFact 59.3 57.2 60.9 CQADupStack 32.1 31.6 34.0...

    work page 2018

  3. [3]

    Your task is to judge how well the passage answers the query. 0 - Irrelevant, 1 - Perfectly relevant, exact answer. Answer with 0, or 1

    0 - Irrelevant, 1 - Relevant topic, but does not contain the answer, 2 - Highly relevant, partial or unclear answer, and 3 - Perfectly relevant, exact answer. Answer with 0, 1, 2 or 3.”. For the binary relevance evaluation (0-1), we apply the following instruction: “ Your task is to judge how well the passage answers the query. 0 - Irrelevant, 1 - Perfect...

    work page arXiv 2025

  4. [5]

    Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith Hall, and Ming-Wei Chang

    URL https://www.microsoft.com/ en-us/research/publication/overview-of-the-trec-2023-deep-learning-track/ . Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith Hall, and Ming-Wei Chang. Promptagator: Few-shot dense retrieval from 8 examples. In The Eleventh International Conference on Learning Representations,

    work page 2023

  5. [6]

    Retrieve the most relevant passages to the given query

    For some of the datasets contained, we define a task instruction that reflects the domain of the query contained in a row of the training dataset. If a specific task instruction has not been defined for a dataset, we fallback to a default one: “Retrieve the most relevant passages to the given query”. In Section 6, we provide the list of instructions we us...

    work page 2025

  6. [8]

    Learning deep representations by mutual information estimation and maximization

    11 Published as a conference paper at COLM 2025 R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations,

    work page 2025

  7. [10]

    Dense passage retrieval for open-domain question an- swering

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question an- swering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),

    work page 2020

  8. [11]

    Nv-embed: Improved techniques for training llms as generalist embedding models

    Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.17428, 2024a. Jinhyuk Lee, Zhuyun Dai, Xiaoqi Ren, Blair Chen, Daniel Cer, Jeremy R. Cole, Kai Hui, Michael Boratko, Rajvi Kapadia, Wen Ding, Yi L...

    arXiv

Show all 47 references
  1. [13]

    Lightblue reranker distillation dataset

    12 Published as a conference paper at COLM 2025 Lightblue. Lightblue reranker distillation dataset. https://huggingface.co/datasets/ lightblue/reranker continuous filt max7 train,

    2025

  2. [14]

    Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, and Sanjeev Arora

    Accessed: 2025-03-27. Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, and Sanjeev Arora. On the SDEs and scaling rules for adaptive gradient algorithms. In Advances in Neural Information Processing Systems,

    2025

  3. [15]

    Generative representational instruction tuning

    Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. arXiv preprint arXiv:2402.09906,

    arXiv

  4. [16]

    Mitigating false-negative contexts in multi- document question answering with retrieval marginalization

    Ansong Ni, Matt Gardner, and Pradeep Dasigi. Mitigating false-negative contexts in multi- document question answering with retrieval marginalization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,

    2021

  5. [17]

    Large dual encoders are generalizable retrievers

    Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang. Large dual encoders are generalizable retrievers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,

    2022

  6. [19]

    RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering

    Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In Proceedings of the 2021 Conference of the North American C...

    2021

  7. [20]

    SQuAD: 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,

    2016

  8. [21]

    Sentence-BERT: Sentence embeddings using Siamese BERT-networks

    Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),

    2019

  9. [22]

    RocketQAv2: A joint training method for dense passage retrieval and passage re-ranking

    Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. RocketQAv2: A joint training method for dense passage retrieval and passage re-ranking. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,

    2021

  10. [23]

    The probabilistic relevance framework: Bm25 and beyond

    13 Published as a conference paper at COLM 2025 Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr.,

    2025

  11. [24]

    doi: 10.1007/978-981-15-5554-1

    ISBN 978-981-15-5554-1. doi: 10.1007/978-981-15-5554-1

    doi

  12. [25]

    ColBERTv2: Effective and efficient retrieval via lightweight late interaction

    Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. ColBERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Lingui...

    2022

  13. [26]

    Repetition improves language model embeddings

    Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan. Repetition improves language model embeddings. arXiv preprint arXiv:2402.15449,

    arXiv

  14. [27]

    Smith, Luke Zettlemoyer, and Tao Yu

    Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. One embedder, any task: Instruction- finetuned text embeddings. In Findings of the Association for Computational Linguistics: ACL 2023,

    2023

  15. [28]

    Is ChatGPT good at search? investigating large language models as re-ranking agents

    Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is ChatGPT good at search? investigating large language models as re-ranking agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing,

    2023

  16. [29]

    FEVER: a large-scale dataset for fact extraction and VERification

    James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language...

    2018

  17. [30]

    Text embeddings by weakly-supervised contrastive pre-training

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533,

    arXiv

  18. [31]

    Improving text embeddings with large language models

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368,

    arXiv

  19. [32]

    14 Published as a conference paper at COLM 2025 Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Confere...

    2025

  20. [33]

    Contrastive learning of sentence embeddings from scratch

    Junlei Zhang, Zhenzhong Lan, and Junxian He. Contrastive learning of sentence embeddings from scratch. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023a. Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. Mr. TyDi: A multi-lingual...

    2023

  21. [34]

    MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages

    Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso- Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages. Transactions of the Association for Computational Linguisti...

    2024

  22. [35]

    Classical retrieval methods like TF-IDF and BM25 (Robertson & Zaragoza,

    15 Published as a conference paper at COLM 2025 Appendix Extended Related Work Neural Dense Retrieval. Classical retrieval methods like TF-IDF and BM25 (Robertson & Zaragoza,

    2025

  23. [37]

    covers multiple embedding-based tasks like retrieval, classification, and clustering across various languages. To encourage generalization across tasks, it is now common to prepend each input with a natural language task description, guiding the encoder to produce task-specifi...

    2023

  24. [38]

    1” through “5

    for general-purpose text embedding tasks. 16 Published as a conference paper at COLM 2025 Lightblue Reranker Distillation Dataset Lightblue (2025) constructed this dataset through a four-step process aimed at creating a di- verse, high-quality resource for evaluating query-tex...

    2025

  25. [44]

    These experiments were carried out on two diverse datasets, LightBlue (multilingual, in- batch only) and BGE-M3 (Chen et al.,

    with two NDCG weighting variants, and MarginMSE (Hofst¨atter et al., 2021). These experiments were carried out on two diverse datasets, LightBlue (multilingual, in- batch only) and BGE-M3 (Chen et al.,

    2021

  26. [45]

    For fair comparison, we run hyperparameter search for all methods to tune for learning rates and logit scales, as well as hyperparameters specific to each training loss

    training datasets (English only subset, with hard negatives), using the QWEN 2.5-0.5B-I NSTRUCT model architecture. For fair comparison, we run hyperparameter search for all methods to tune for learning rates and logit scales, as well as hyperparameters specific to each traini...

    2021

  27. [100]

    once more, we can see that only the negative pairs that are perceived as positive by the model are going to be used and they will 19 Published as a conference paper at COLM 2025 Benchmark BIXSE (OURS) P AIRWISEBCE L AMBDA- NDCGV1 L AMBDA- NDCGV2 BEIR (short) 73.91 72.17 72.43 ...

    2025

  28. [256]

    BGE-M3 Results: English data with mined and scored hard-negatives To investigate training efficiency under fixed compute and memory budgets, we evaluate model perfor- mance across configurations that varied the number of hard negatives and batch size. As we see in Tables 11 an...

    2025

  29. [2005]

    and PairDistill (Huang & Chen, 2024)), LambdaLoss (Wang et al.,

    2024

  30. [2009]

    Neural dense retrieval methods address this by embedding texts into dense semantic vector spaces using pre-trained language models (Karpukhin et al., 2020; Xiong et al., 2021)

    represent texts as sparse, high-dimensional vectors based on lexical overlap, effectively capturing keyword matches but lacking deeper semantic understanding. Neural dense retrieval methods address this by embedding texts into dense semantic vector spaces using pre-trained lan...

    2020

  31. [2015]

    We apply no weight decay

    with β1 = 0.9, β2 = 0.98. We apply no weight decay. For the learning rate scheduler, we adopt a linear warmup over 5% of the total training steps, followed by a linear decay till end of training. We train all models for a constant of 4 epochs. The logit scale α is tuned and we...

    2025

  32. [2017]

    Gabriel de Souza P

    URL https://kaggle.com/competitions/quora-question-pairs. Gabriel de Souza P . Moreira, Radek Osmulski, Mengyao Xu, Ronay Ak, Benedikt Schifferer, and Even Oldridge. Nv-retriever: Improving text embedding models with effective hard-negative mining. arXiv preprint arXiv:2407.15831,

    arXiv

  33. [2018]

    Llm2vec: Large language models are secretly powerful text encoders

    10 Published as a conference paper at COLM 2025 Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders. arXiv preprint arXiv:2404.05961,

    2025 arXiv

  34. [2019]

    SimCSE: Simple contrastive learning of sentence embeddings

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,

    2021

  35. [2021]

    PairDistill: Pairwise relevance distillation for dense retrieval

    Chao-Wei Huang and Yun-Nung Chen. PairDistill: Pairwise relevance distillation for dense retrieval. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,

    2024

  36. [2022]

    Representation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748,

    arXiv

  37. [2023]

    Rahmani, Daniel Campos, Jimmy Lin, Ellen M

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Hossein A. Rahmani, Daniel Campos, Jimmy Lin, Ellen M. Voorhees, and Ian Soboroff. Overview of the trec 2023 deep learning track. In Text REtrieval Conference (TREC). NIST, TREC,

    2023

  38. [2024]

    Improving con- trastive learning of sentence embeddings from AI feedback

    Qinyuan Cheng, Xiaogui Yang, Tianxiang Sun, Linyang Li, and Xipeng Qiu. Improving con- trastive learning of sentence embeddings from AI feedback. In Findings of the Association for Computational Linguistics: ACL 2023,

    2023

  39. [2025]

    Towards general text embeddings with multi-stage contrastive learning

    Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281,

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.