Pith. sign in

REVIEW 5 major objections 6 minor 58 references

Disentangling Locality and Entropy in Ranking Distillation

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Under ranking distillation, sampling choice and teacher entropy are orthogonal, and only minimal negative locality plus mid-range teacher entropy matter.

desk verdict A useful negative-mining ablation wrapped in a generalization bound that does not prove what it claims; the empirics merit a serious look, the theory needs repair or removal. read the letter →

arxiv 2505.21058 v1 pith:3AM5LGCA submitted 2025-05-27 cs.IR

Andrew Parry , Debasis Ganguly , Sean MacAvaney This is my paper
classification cs.IR
keywords rankingdistillationhardnegativeminingteacherentropygeneralizationboundneuralBregmandivergenceknowledgead-hocretrieval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Modern neural ranking pipelines spend large compute on multi-stage hard-negative mining and teacher ensembles, but this paper argues and demonstrates that under distillation these expensive sampling choices are largely redundant. The paper frames sampling and labeling as orthogonal controls and derives a generalisation bound showing that the excess risk of a distilled ranker is governed by the query-space diameter, the teacher's pairwise ranking entropy, and a density-ratio term, not by how hard the sampled negatives are. Across two in-domain benchmarks and the BEIR out-of-domain set, the authors find that once a minimal locality (BM25-level negatives) is enforced, ensemble and cross-encoder sampling add no consistent gains over simple sampling under any of three distillation losses. The active ingredient is instead the entropy of the teacher's targets: ranking examples from the middle of the entropy distribution improves in-domain effectiveness, while extreme-entropy subsets hurt. If right, this redirects training effort from building expensive negative mining cascades to controlling geometry and target entropy.

What carries the argument

The load-bearing object is the ranking-distillation generalisation bound of Theorem 2.1, which decomposes excess risk into a locality term $\zeta L \Delta_Q\, \eta(H(g))$ and a statistical term. $\Delta_Q$ is the essential diameter of the query-conditioned document space under the sampling measure, $H(g)$ is the teacher's pairwise ranking entropy, and $\eta$ is the conversion from entropy to misordering probability via Pinsker's inequality; the biased-sampling version introduces the density ratio $\kappa_Q$ between the miner's measure and the true measure. The bound does the work of separating what sampling can change (the geometry and density ratio) from what only the teacher can change (entropy), which the paper then tests by ablating four sampling sources and by stratifying training pairs into quartiles of teacher entropy.

What would settle it

Take a near-perfect teacher whose pairwise entropy is just below $\log 2$ (so $\eta(H(g))$ is tiny) and a query set with a large measured essential diameter $\Delta_Q$; if a distilled student's excess risk exceeds $L\Delta_Q\,\eta(H(g))$, the theorem's entropy term fails.

Watch

Extended reading notes

Core claim

The central result is a generalisation bound, Theorem 2.1: for an $L$-Lipschitz hypothesis class with VC dimension $d$, the excess risk of a student $\hat{f}$ trained with a Bregman distillation loss against teacher $g$ satisfies $R(\hat{f}) - R(f^\star) \leq \zeta L \Delta_Q\, \eta(H(g)) + C\sqrt{d\log(1/\delta)/n}$, where $\Delta_Q$ is the essential diameter of the query-conditioned document space, $H(g)$ is the teacher's pairwise ranking entropy, and $\eta(H(g))$ is the teacher's entropy-derived misordering probability. The first term says that locality matters only through the geometry of the query manifold, and entropy enters only through the teacher; biased sampling enters only through a density ratio $\kappa_Q$ multiplying the statistical term (Corollary 2.1). The empirical ablation shows that, under distillation, going from BM25 negatives to teacher-scored or ensemble negatives yields no consistent statistically differentiable improvement in or out of domain, while stratifying training pairs by teacher entropy gives consistent in-domain gains for the inner quartiles. The paper's conclusion is that complex, multi-stage hard-negative pipelines yield minimal gains over simpler sampling strategies under distillation.

Load-bearing premise

The theorem assumes that the teacher's excess risk over the best possible student, $R(g)-R(f^\star)$, is bounded above by $L\Delta_Q\,\eta(H(g))$; the appendix only establishes a lower bound on teacher misordering from entropy, so this inequality is the load-bearing premise.

Editorial extensions

If this is right

  • Under RankNet, MarginMSE, and KL distillation, BM25-level negatives are statistically equivalent to teacher-scored and ensemble negatives on TREC Deep Learning 2019/2020, so the extra stages of hard-negative mining buy no consistent effectiveness.
  • Stratifying training pairs by teacher ranking entropy yields consistent in-domain gains for the middle quartiles, while low- and high-entropy outlier quartiles degrade effectiveness.
  • Because the bound separates the locality term from the entropy term, sampling changes only the geometry and density-ratio contributions; once locality is sufficient, more aggressive hard-negative mining cannot further reduce the teacher-entropy bias.
  • The power-law score tails induced by ensemble sampling correlate with lower out-of-domain effectiveness, indicating that apparent in-domain gains may reflect overfitting rather than transferable ranking skill.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A practical corollary the authors hint at but do not develop: replacing multi-stage model cascades with BM25 candidate generation plus a single teacher labeler would cut compute and reduce reproducibility problems while preserving effectiveness.
  • The entropy-quartile result suggests a cheap selection heuristic: compute pairwise target entropy from one teacher and sample the central mass, instead of iteratively re-mining negatives with ever-stronger teachers.
  • The observed connection between score-tail shape and out-of-domain drop could be tested as a diagnostic: train students with varying tail slopes and check whether BEIR effectiveness falls as the tail steepens.
  • The same orthogonality, if it holds, would apply to any pairwise preference distillation beyond ranking, such as reward-model distillation in preference optimization, where hard-pair mining is a common practice.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper studies the interaction between negative sampling locality and teacher ranking entropy in ranking distillation. It claims a PAC-style generalization bound (Theorem 2.1) in which the excess risk of a distilled student is bounded by a locality term ζ L Δ_Q η(H(g)) plus a statistical term C sqrt(d log(1/δ)/n), and uses this to argue that expensive multi-stage hard-negative mining pipelines are largely unnecessary under distillation. The empirical part abates four sampling sources (Random, BM25, Cross-Encoder, Ensemble) and three distillation losses (RankNet, MarginMSE, KL) on TREC DL19/DL20 and BEIR, using TOST equivalence tests and entropy-quartile analyses to identify the role of teacher entropy. The core theoretical claim, however, rests on an unsupported inequality in the proof of Theorem 2.1, and the statistical equivalence procedure is circular, so the paper's main conclusions do not follow from the presented evidence.

Significance. If the theoretical bound and the empirical conclusions held, the paper would be significant: it could justify simplifying expensive multi-stage neural ranking pipelines and redirecting research toward entropy-aware training. The ablation is broad (two architectures, four sampling domains, three losses, in- and out-of-domain benchmarks) and the authors release a repository, which are strengths. However, the central theoretical result is not established: the only step that introduces teacher entropy (Appendix C, Step 1) asserts an inequality whose direction conflicts with the paper's own Pinsker-based lemma and which is false for low-entropy deterministic teachers. The TOST equivalence procedure uses an observed-difference-dependent margin, invalidating the statistical support for the 'minimal gains' claim. The empirical observations may still be of interest, but as presented the main contributions do not withstand scrutiny.

major comments (5)
  1. [Appendix C, Step 1 / Theorem 2.1] The inequality R(g) - R(f*) ≤ L Δ_Q η(H(g)) is asserted but never derived, and it is not a consequence of the stated assumptions. Lemma B.1 and Corollary B.1 provide a lower bound on the teacher's misordering probability in terms of η(H(g)); they do not provide an upper bound on the teacher's excess risk over the Bayes optimal scorer. In fact, for a deterministic teacher that always reverses the true pairwise order, H(g) = 0, R(g) - R(f*) = 1, and η(H(g)) = 1/2 - sqrt(ln 2 / 2) ≈ -0.0887, so the asserted inequality is false and the right-hand side is negative. The proof's informal assumption that 'H(g) is sufficiently high to approximate f*' is not a hypothesis of Theorem 2.1 and is conceptually backwards: high entropy means closer to a uniform ranking, not closer to f*. Since this step is the only place the teacher entropy term enters the bound, Theorem 2.1 is unproved and can be false as stated.
  2. [Appendix C, Step 6 / Theorem 2.1] The proof introduces an unstated non-degeneracy assumption. To merge the locality term with the entropy term, Step 6 assumes η(H(g)) ≥ ε > 0 and then writes ζLΔ_Q ≤ ζLΔ_Q ε^{-1} η(H(g)). This assumption appears nowhere in Theorem 2.1's hypotheses, and it makes the effective constant ζ depend on the teacher g, contradicting the theorem's statement that ζ depends only on the divergence potential φ. Moreover, for a teacher with low entropy, η(H(g)) is negative (e.g., η(0) ≈ -0.0887), so the right-hand side of Eq. (6) can be negative, whereas R(f_hat) - R(f*) ≥ 0 by the definition of f* as the risk minimizer in H. Thus the theorem as stated cannot hold.
  3. [Appendix D, Corollary D.1] The density-ratio scaling in Eq. (8) is not justified by the proof. Step 2 of the proof bounds |w_Q Z_h| ≤ κ_Q L Δ_Q, but from this boundedness alone, the Hoeffding-based argument used in the unbiased case yields a uniform deviation with a factor κ_Q, not sqrt(κ_Q). The stated rate C sqrt(κ_Q d log(1/δ)/n) requires an additional second-moment condition such as E[w_Q^2] ≤ κ_Q, which is neither stated nor proved. Therefore the biased-sampling bound as written is not derived from the given assumptions.
  4. [Appendix E.3 / Tables 1, 5, 6] The TOST equivalence procedure defines the margin as θ = |μ_2 - μ_1| · ε, i.e., a fraction of the observed effect. This is circular: a small observed difference produces a small margin, making equivalence trivially likely, while a large observed difference produces a large margin, making non-equivalence impossible to detect. A valid TOST requires a margin specified independently of the data. Consequently, the superscripted equivalence claims in Tables 1, 5, and 6 do not substantiate the abstract's conclusion that complex, multi-stage hard-negative pipelines yield minimal gains over simpler sampling strategies under distillation. Additionally, Section 3.1 states a 5% bound whereas the Table 1 caption states a 1% bound; this discrepancy needs clarification.
  5. [Table 1, mMSE rows] Even setting aside the TOST issue, the paper's own numbers for cross-encoders under MarginMSE show large differences between sampling domains: BM25 gives nDCG@10 of 0.601 on TREC DL'19 and 0.607 on DL'20, while Ensemble gives 0.717 and 0.736, respectively. These are substantial gaps, not 'minimal gains.' The claim that complex pipelines are largely identical to naive approaches under distillation is not supported by these rows and should be qualified by architecture and loss, or the empirical conclusion should be reconsidered.
minor comments (6)
  1. [Eq. (9)] The LCE loss in Eq. (9) has an apparent typo: the denominator's first term is written 'exp(X_i/τ)' while the numerator uses 'exp(f(X_i)/τ)'; the denominator should also use f(X_i)/τ.
  2. [Definition 2.4] The function η(H(g)) = 1/2 - sqrt((log 2 - H(g))/2) is negative for H(g) < log 2 - 1/2 (e.g., η(0) ≈ -0.0887), so it is not a valid 'misordering probability' on its stated domain. The paper should restrict the domain to H(g) ≥ log 2 - 1/2 or define η as max(0, 1/2 - sqrt((log 2 - H(g))/2)).
  3. [Section 2.4] Corollary 2.1 refers to 'Theorem 1' but the theorem is numbered Theorem 2.1; additionally, Section 3.2 references 'Corollary 3.1.1,' which is never defined.
  4. [Table 2] The estimated density ratios bκ_Q have extremely large standard deviations (e.g., Random: 14.202 ± 556.251, BM25: 12.747 ± 461.588), suggesting the estimator is highly unstable. The paper should discuss whether the reported means are meaningful and how this instability affects the connection to Corollary D.1.
  5. [Appendix C, Step 4 / Lemma A.2] The proof conflates the VC dimension of the hypothesis class H with that of the induced loss class {Z_h}. A short argument showing that the VC dimension of the loss class is O(d) would be needed for the uniform deviation bound to follow.
  6. [Abstract and Section 1] The abstract and introduction use causal language ('clear empirical evidence of the causal factors in model effectiveness') for what is an ablation study without controlled causal inference; given the theoretical issues above, the causal claims should be softened.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular reduction found; the main theorem's entropy term rests on an unsupported proof step, which is a correctness gap rather than a construction-level circularity.

full rationale

This paper is not circular in the fitting or self-citation sense. Theorem 2.1's two bias terms are defined independently: ΔQ is the essential diameter of the query manifold and H(g) is the teacher's pairwise entropy; neither is fitted to the student's empirical risk, and the empirical conclusions are checked against external TREC-DL and BEIR judgments. The self-citations (Parry et al. 2025; Wang et al. 2022) appear only in background motivation and are not load-bearing. The one serious problem found is in Appendix C, Step 1, where the proof asserts 'R(g) − R(f ⋆) ≤ L∆Qη(H(g))' after saying H(g) 'is assumed to be sufficiently high to approximate f ⋆.' This inequality is not derived and goes in the opposite direction of the paper's own Lemma B.1/Corollary B.1, which give a lower bound on teacher misordering via Pinsker. That is a proof gap or correctness risk, not an equivalence-by-construction or a fitted-parameter-as-prediction, so per the circularity rubric it does not raise the circularity score beyond the minor self-citation allowance.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The theory introduces no new entities, but it does introduce an unproven inequality about the teacher and leaves key constants unspecified. The empirical estimates of kappa_Q and Delta_Q are approximations made with the authors' own choices of embedding model and teacher, not fitted parameters in the theoretical derivation.

free parameters (2)
  • zeta (divergence calibration constant) = unspecified
    Introduced in Proof Step 2 of Theorem 2.1 to calibrate the Bregman divergence to the 0-1 risk; never defined or computed, and it absorbs the main slack between the loss and the risk.
  • TOST equivalence margin epsilon = 5% or 1% (text and table caption disagree)
    Chosen by hand; the procedure defines theta from the observed difference, making the margin data-dependent and the equivalence test nonstandard.
assumptions (6)
  • domain assumption The hypothesis class H has VC dimension d and every h in H is L-Lipschitz.
    Invoked in Theorem 2.1; standard in PAC analysis but not verified for transformer rankers.
  • ad hoc to paper Teacher's excess risk over the Bayes optimal scorer is bounded by L Delta_Q eta(H(g)).
    Assumed in Proof Step 1 (Appendix C) to introduce the entropy term; no derivation is provided and the direction conflicts with Lemma B.1.
  • domain assumption Bregman potential phi is strictly convex, C1, with |phi'| <= 1, and its divergence is Lipschitz in the score differences.
    Needed for the calibration step (Step 2) and the population gap (Step 3) of the proof.
  • ad hoc to paper Teacher is non-degenerate: eta(H(g)) >= epsilon > 0.
    Invoked in Step 6 of Proof C.2 to merge the locality and entropy terms into the stated bound; not stated in the theorem itself.
  • standard math Pinsker's inequality applied to Bernoulli teacher preferences to derive eta(H).
    Used in Definition 2.4 and Lemma B.1; the algebra is correct but the result is a lower bound on misordering, not an upper bound on teacher risk.
  • domain assumption Supremum density ratio kappa_Q is finite and applies to pairs.
    Corollary D.1 requires kappa_Q < infinity; under product measures nu_Q^{x2} the pair-level ratio should be kappa_Q squared, so the stated form is questionable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Disentangling Locality and Entropy in Ranking Distillation." pith.science (2026). https://pith.science/paper/3AM5LGCA

@misc{pith2026250521058,
  author       = {Pith},
  title        = {Pith review of: Disentangling Locality and Entropy in Ranking Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3AM5LGCA}},
  note         = {Machine review of arXiv:2505.21058}
}
read the original abstract

The training process of ranking models involves two key data selection decisions: a sampling strategy, and a labeling strategy. Modern ranking systems, especially those for performing semantic search, typically use a ``hard negative'' sampling strategy to identify challenging items using heuristics and a distillation labeling strategy to transfer ranking "knowledge" from a more capable model. In practice, these approaches have grown increasingly expensive and complex, for instance, popular pretrained rankers from SentenceTransformers involve 12 models in an ensemble with data provenance hampering reproducibility. Despite their complexity, modern sampling and labeling strategies have not been fully ablated, leaving the underlying source of effectiveness gains unclear. Thus, to better understand why models improve and potentially reduce the expense of training effective models, we conduct a broad ablation of sampling and distillation processes in neural ranking. We frame and theoretically derive the orthogonal nature of model geometry affected by example selection and the effect of teacher ranking entropy on ranking model optimization, establishing conditions in which data augmentation can effectively improve bias in a ranking model. Empirically, our investigation on established benchmarks and common architectures shows that sampling processes that were once highly effective in contrastive objectives may be spurious or harmful under distillation. We further investigate how data augmentation, in terms of inputs and targets, can affect effectiveness and the intrinsic behavior of models in ranking. Through this work, we aim to encourage more computationally efficient approaches that reduce focus on contrastive pairs and instead directly understand training dynamics under rankings, which better represent real-world settings.

Figures

Figures reproduced from arXiv: 2505.21058 by the authors.

Figure 1
Figure 1. The existing SentenceTransform￾ers [Reimers and Gurevych, 2019] pipeline for hard negatives. Even naive sampling pipelines can often have three stages of prior sampling [Song et al., 2023, Wang et al., 2024]. This practise re￾duces our ability to attribute effectiveness gain to a particular source and more broadly replicate this process due to cross-mining of examples from dif￾ferent stages in the pipeline. Furtherm… view at source ↗
Figure 2
Figure 2. Score versus rank ratio comparing the teacher (LCE) and two RankNet-trained students [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Training loss in the form of the KL divergence (left) and gradient norm of the student [PITH_FULL_IMAGE:figures/full_fig_p022_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: log-log plots of average score at each document rank on MS MARCO passage (TREC DL [PITH_FULL_IMAGE:figures/full_fig_p023_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

58 extracted references · 22 canonical work pages

  1. [1]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762

    arXiv 2023

  2. [2]

    BERT: pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, N...

    2019

  3. [3]

    Pretrained Transformers for Text Ranking: BERT and Beyond

    Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. Pretrained Transformers for Text Ranking: BERT and Beyond . Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers, 2021. ISBN 978-3-031-01053-8. doi:10.2200/S01123ED1V01Y202108HLT053. URL https://doi.org/10.2200/S01123ED1V01Y202108HLT053

    doi:10.2200/s01123ed1v01y202108hlt053 2021

  4. [4]

    MS MARCO: A human generated machine reading comprehension dataset

    Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A human generated machine reading comprehension dataset. In Tarek Richard Besold, Antoine Bordes, Artur S. d'Avila Garcez, and Greg Wayne, editors, Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 ...

    work page 2016

  5. [5]

    Passage re-ranking with BERT

    Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with BERT . CoRR, abs/1901.04085, 2019. URL http://arxiv.org/abs/1901.04085

    arXiv 1901

  6. [6]

    Dense passage retrieval for open-domain question answering

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen - tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November ...

    doi:10.18653/v1/2020.emnlp-main.550 2020

  7. [7]

    Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering

    Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani - T \" u r, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakrab...

    2021

  8. [8]

    Hinton, Oriol Vinyals, and Jeffrey Dean

    Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. CoRR, abs/1503.02531, 2015. URL http://arxiv.org/abs/1503.02531

    arXiv 2015

Show all 58 references
  1. [9]

    Distilling dense representations for ranking using tightly-coupled teachers

    Sheng - Chieh Lin, Jheng - Hong Yang, and Jimmy Lin. Distilling dense representations for ranking using tightly-coupled teachers. CoRR, abs/2010.11386, 2020. URL https://arxiv.org/abs/2010.11386

    2010 arXiv

  2. [10]

    a tter, Sophia Althammer, Michael Schr \

    Sebastian Hofst \" a tter, Sophia Althammer, Michael Schr \" o der, Mete Sertkan, and Allan Hanbury. Improving efficient neural ranking models with cross-architecture knowledge distillation. CoRR, abs/2010.02666, 2020. URL https://arxiv.org/abs/2010.02666

    2010 arXiv

  3. [11]

    Generalization bounds via distillation

    Daniel Hsu, Ziwei Ji, Matus Telgarsky, and Lan Wang. Generalization bounds via distillation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. URL https://openreview.net/forum?id=EGdFhBzmAwB

    2021

  4. [12]

    Retromae: Pre-training retrieval-oriented language models via masked auto-encoder

    Shitao Xiao, Zheng Liu, Yingxia Shao, and Zhao Cao. Retromae: Pre-training retrieval-oriented language models via masked auto-encoder. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Proce...

    2022 doi

  5. [13]

    Rocketqav2: A joint training method for dense passage retrieval and passage re-ranking

    Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji - Rong Wen. Rocketqav2: A joint training method for dense passage retrieval and passage re-ranking. In Marie - Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen - tau Yih, ed...

    2021

  6. [14]

    Doermann, and Junsong Yuan

    Liangchen Song, Xuan Gong, Helong Zhou, Jiajie Chen, Qian Zhang, David S. Doermann, and Junsong Yuan. Exploring the knowledge transferred by response-based teacher-student distillation. In Abdulmotaleb El - Saddik, Tao Mei, Rita Cucchiara, Marco Bertini, Diana Patricia Tobon V...

    2023

  7. [15]

    Fectek: Enhancing term weight in lexicon-based retrieval with feature context and term-level knowledge

    Zunran Wang, Zhonghua Li, Wei Shen, Qi Ye, and Liqiang Nie. Fectek: Enhancing term weight in lexicon-based retrieval with feature context and term-level knowledge. CoRR, abs/2404.12152, 2024. doi:10.48550/ARXIV.2404.12152. URL https://doi.org/10.48550/arXiv.2404.12152

    2024 doi arXiv

  8. [16]

    Rethink training of BERT rerankers in multi-stage retrieval pipeline

    Luyu Gao, Zhuyun Dai, and Jamie Callan. Rethink training of BERT rerankers in multi-stage retrieval pipeline. In Djoerd Hiemstra, Marie - Francine Moens, Josiane Mothe, Raffaele Perego, Martin Potthast, and Fabrizio Sebastiani, editors, Advances in Information Retrieval - 43rd...

    2021 doi

  9. [17]

    Noise-contrastive estimation: A new estimation principle for unnormalized statistical models

    Michael Gutmann and Aapo Hyv \" a rinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Yee Whye Teh and D. Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Stati...

    2010

  10. [18]

    Ciwan Ceylan and Michael U. Gutmann. Conditional noise-contrastive estimation of unnormalised models. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsm \" a ssan, Stockholm, Sweden, July ...

    2018

  11. [19]

    Model compression

    Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Knowledge Discovery and Data Mining, 2006. URL https://api.semanticscholar.org/CorpusID:11253972

    2006

  12. [20]

    Do deep nets really need to be deep? In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D

    Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Syst...

    2014

  13. [21]

    Squeezing water from a stone: A bag of tricks for further improving cross-encoder effectiveness for reranking

    Ronak Pradeep, Yuqi Liu, Xinyu Zhang, Yilin Li, Andrew Yates, and Jimmy Lin. Squeezing water from a stone: A bag of tricks for further improving cross-encoder effectiveness for reranking. In Matthias Hagen, Suzan Verberne, Craig Macdonald, Christin Seifert, Krisztian Balog, Kj...

    2022

  14. [22]

    Rankzephyr: Effective and robust zero-shot listwise reranking is a breeze! CoRR, abs/2312.02724, 2023

    Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. Rankzephyr: Effective and robust zero-shot listwise reranking is a breeze! CoRR, abs/2312.02724, 2023. doi:10.48550/ARXIV.2312.02724. URL https://doi.org/10.48550/arXiv.2312.02724

    2023 doi arXiv

  15. [23]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International ...

    2019 doi

  16. [25]

    An inspection of the reproducibility and replicability of tct-colbert

    Xiao Wang, Sean MacAvaney, Craig Macdonald, and Iadh Ounis. An inspection of the reproducibility and replicability of tct-colbert. In Enrique Amig \' o , Pablo Castells, Julio Gonzalo, Ben Carterette, J. Shane Culpepper, and Gabriella Kazai, editors, SIGIR '22: The 45th Intern...

    2022

  17. [26]

    Bennett, Junaid Ahmed, and Arnold Overwijk

    Lee Xiong, Chenyan Xiong, Ye Li, Kwok - Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In 9th International Conference on Learning Representations, ICLR 2021, Virtu...

    2021

  18. [27]

    Optimizing dense retrieval model training with hard negatives

    Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. Optimizing dense retrieval model training with hard negatives. In Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai, editors, SIGIR '21: The 44th International ...

    2021

  19. [28]

    Christiano, Jan Leike, Tom B

    Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Ad...

    2017

  20. [29]

    Manning, Stefano Ermon, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, e...

    2023

  21. [30]

    Annotating data for fine-tuning a neural ranker? current active learning strategies are not better than random selection

    Sophia Althammer, Guido Zuccon, Sebastian Hofst \" a tter, Suzan Verberne, and Allan Hanbury. Annotating data for fine-tuning a neural ranker? current active learning strategies are not better than random selection. In Qingyao Ai, Yiqin Liu, Alistair Moffat, Xuanjing Huang, Te...

    2023

  22. [31]

    Bruce Croft

    Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Jaap Kamps, and W. Bruce Croft. Neural ranking models with weak supervision. In Noriko Kando, Tetsuya Sakai, Hideo Joho, Hang Li, Arjen P. de Vries, and Ryen W. White, editors, Proceedings of the 40th International ACM SIGIR Co...

    2017

  23. [32]

    Rank-distillm: Closing the effectiveness gap between cross-encoders and llms for passage re-ranking

    Ferdinand Schlatt, Maik Fr \" o be, Harrisen Scells, Shengyao Zhuang, Bevan Koopman, Guido Zuccon, Benno Stein, Martin Potthast, and Matthias Hagen. Rank-distillm: Closing the effectiveness gap between cross-encoders and llms for passage re-ranking. In Claudia Hauff, Craig Mac...

    2025

  24. [33]

    Instruction distillation makes large language models efficient zero-shot rankers

    Weiwei Sun, Zheng Chen, Xinyu Ma, Lingyong Yan, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Instruction distillation makes large language models efficient zero-shot rankers. CoRR, abs/2311.01555, 2023 a . doi:10.48550/ARXIV.2311.01555. URL https://d...

    2023 doi arXiv

  25. [34]

    Is chatgpt good at search? investigating large language models as re-ranking agent

    Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren. Is chatgpt good at search? investigating large language models as re-ranking agent. CoRR, abs/2304.09542, 2023 b . doi:10.48550/ARXIV.2304.09542. URL https://doi.org/10.48550/arXiv.2304.09542

    2023 doi arXiv

  26. [35]

    Knowledge distillation in wide neural networks: Risk bound, data efficiency and imperfect teacher

    Guangda Ji and Zhanxing Zhu. Knowledge distillation in wide neural networks: Risk bound, data efficiency and imperfect teacher. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria - Florina Balcan, and Hsuan - Tien Lin, editors, Advances in Neural Information Process...

    2020

  27. [36]

    Zico Kolter

    Vaishnavh Nagarajan and J. Zico Kolter. Uniform convergence may be unable to explain generalization in deep learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alch \' e - Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information P...

    2019

  28. [37]

    A generalization theory of cross-modality distillation with contrastive learning

    Hangyu Lin, Chen Liu, Chengming Xu, Zhengqi Gao, Yanwei Fu, and Yuan Yao. A generalization theory of cross-modality distillation with contrastive learning. CoRR, abs/2405.03355, 2024. doi:10.48550/ARXIV.2405.03355. URL https://doi.org/10.48550/arXiv.2405.03355

    2024 doi arXiv

  29. [38]

    Robertson, Steve Walker, Micheline Hancock - Beaulieu, Mike Gatford, and A

    Stephen E. Robertson, Steve Walker, Micheline Hancock - Beaulieu, Mike Gatford, and A. Payne. Okapi at TREC-4 . In Donna K. Harman, editor, Proceedings of The Fourth Text REtrieval Conference, TREC 1995, Gaithersburg, Maryland, USA, November 1-3, 1995 , volume 500-236 of NIST ...

    1995

  30. [39]

    Document ranking with a pretrained sequence-to-sequence model

    Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. Document ranking with a pretrained sequence-to-sequence model. In Trevor Cohn, Yulan He, and Yang Liu, editors, Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 20...

    2020 doi

  31. [40]

    Voorhees

    Ellen M. Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness. In W. Bruce Croft, Alistair Moffat, C. J. van Rijsbergen, Ross Wilkinson, and Justin Zobel, editors, SIGIR '98: Proceedings of the 21st Annual International ACM SIGIR Conferenc...

    1998

  32. [41]

    Variations in relevance judgments and the shelf life of test collections

    Andrew Parry, Maik Fr \" o be, Harrisen Scells, Ferdinand Schlatt, Guglielmo Faggioli, Saber Zerhoudi, Sean MacAvaney, and Eugene Yang. Variations in relevance judgments and the shelf life of test collections. CoRR, abs/2502.20937, 2025. doi:10.48550/ARXIV.2502.20937. URL http...

    2025 doi arXiv

  33. [42]

    Chris J.C. Burges. From ranknet to lambdarank to lambdamart: An overview. Technical Report MSR-TR-2010-82, June 2010. URL https://www.microsoft.com/en-us/research/publication/from-ranknet-to-lambdarank-to-lambdamart-an-overview/

    2010

  34. [43]

    Solomon Kullback and R. A. Leibler. On information and sufficiency. Annals of Mathematical Statistics, 22: 0 79--86, 1951. URL https://api.semanticscholar.org/CorpusID:120349231

    1951

  35. [44]

    Amichai Painsky and Gregory W. Wornell. Bregman divergence bounds and universality properties of the logarithmic loss. IEEE Trans. Inf. Theory , 66 0 (3): 0 1658--1673, 2020. doi:10.1109/TIT.2019.2958705. URL https://doi.org/10.1109/TIT.2019.2958705

    2020

  36. [45]

    Voorhees

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M. Voorhees. Overview of the TREC 2019 deep learning track. CoRR, abs/2003.07820, 2020 a . URL https://arxiv.org/abs/2003.07820

    2019 arXiv

  37. [46]

    Overview of the TREC 2020 deep learning track

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. Overview of the TREC 2020 deep learning track. In Ellen M. Voorhees and Angela Ellis, editors, Proceedings of the Twenty-Ninth Text REtrieval Conference, TREC 2020, Virtual Event [Gaithersburg, Maryland, USA], Nove...

    2020

  38. [47]

    BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models

    Nandan Thakur, Nils Reimers, Andreas R \" u ckl \' e , Abhishek Srivastava, and Iryna Gurevych. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Joaquin Vanschoren and Sai - Kit Yeung, editors, Proceedings of the Neural Information P...

    2021

  39. [48]

    a rvelin and Jaana Kek \

    Kalervo J \" a rvelin and Jaana Kek \" a l \" a inen. Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst. , 20 0 (4): 0 422--446, 2002. doi:10.1145/582415.582418. URL http://doi.acm.org/10.1145/582415.582418

    2002

  40. [49]

    Le, and Christopher D

    Kevin Clark, Minh - Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: pre-training text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview....

    2020

  41. [50]

    Kingma and Max Welling

    Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014. URL http://arxiv.org/a...

    2014 arXiv

  42. [51]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  43. [52]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 arXiv

  44. [53]

    Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas K \" o pf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Stei...

    2019

  45. [54]

    Declarative experimentation ininformation retrieval using pyterrier

    Craig Macdonald and Nicola Tonellotto. Declarative experimentation ininformation retrieval using pyterrier. In Proceedings of ICTIR 2020, 2020

    2020

  46. [55]

    L.M. Bregman. The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR Computational Mathematics and Mathematical Physics, 7 0 (3): 0 200--217, 1967. ISSN 0041-5553. doi:https://doi.org/10.1016...

    1967

  47. [56]

    On the optimality of conditional expectation as a bregman predictor

    Arindam Banerjee, Xin Guo, and Hui Wang. On the optimality of conditional expectation as a bregman predictor. IEEE Trans. Inf. Theory , 51 0 (7): 0 2664--2669, 2005. doi:10.1109/TIT.2005.850145. URL https://doi.org/10.1109/TIT.2005.850145

    2005

  48. [57]

    N. Weaver. Lipschitz Algebras. G - Reference,Information and Interdisciplinary Subjects Series. World Scientific, 1999. ISBN 9789810238735. URL https://books.google.co.uk/books?id=45rnwyVjg\_QC

    1999

  49. [58]

    Probability inequalities for sums of bounded random variables

    Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58 0 (301): 0 13--30, 1963. doi:10.1080/01621459.1963.10500830. URL https://www.tandfonline.com/doi/abs/10.1080/01621459.1963.10500830

    1963

  50. [59]

    On the density of families of sets

    N Sauer. On the density of families of sets. Journal of Combinatorial Theory, Series A, 13 0 (1): 0 145--147, 1972. ISSN 0097-3165. doi:https://doi.org/10.1016/0097-3165(72)90019-2. URL https://www.sciencedirect.com/science/article/pii/0097316572900192

    1972

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.