Pith. sign in

REVIEW 20 cited by

Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2010.02666 v2 pith:4XC7OMPF submitted 2020-10-06 cs.IR

Sebastian Hofstätter , Sophia Althammer , Michael Schröder , Mete Sertkan , Allan Hanbury This is my paper · ORCID
classification cs.IR
keywords rankingarchitecturesdistillationknowledgemodelsbertdifferentefficient
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Retrieval and ranking models are the backbone of many applications such as web search, open domain QA, or text-based recommender systems. The latency of neural ranking models at query time is largely dependent on the architecture and deliberate choices by their designers to trade-off effectiveness for higher efficiency. This focus on low query latency of a rising number of efficient ranking architectures make them feasible for production deployment. In machine learning an increasingly common approach to close the effectiveness gap of more efficient models is to apply knowledge distillation from a large teacher model to a smaller student model. We find that different ranking architectures tend to produce output scores in different magnitudes. Based on this finding, we propose a cross-architecture training procedure with a margin focused loss (Margin-MSE), that adapts knowledge distillation to the varying score output distributions of different BERT and non-BERT passage ranking architectures. We apply the teachable information as additional fine-grained labels to existing training triples of the MSMARCO-Passage collection. We evaluate our procedure of distilling knowledge from state-of-the-art concatenated BERT models to four different efficient architectures (TK, ColBERT, PreTT, and a BERT CLS dot product model). We show that across our evaluated architectures our Margin-MSE knowledge distillation significantly improves re-ranking effectiveness without compromising their efficiency. Additionally, we show our general distillation method to improve nearest neighbor based index retrieval with the BERT dot product model, offering competitive results with specialized and much more costly training methods. To benefit the community, we publish the teacher-score training files in a ready-to-use package.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 20 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Tevatron-Elastic: A Unified Abstraction for Training Elastic Retrievers and Rerankers

    cs.CL 2026-08 conditional novelty 7.0 of 10

    Tevatron-Elastic unifies depth, token, and width compression for retrievers and rerankers into one abstraction that reproduces prior elastic methods as special cases and adds a new multi-ratio token compression method (MLTC).

  2. Hypencoder: Hypernetworks for Information Retrieval

    cs.IR 2025-02 conditional novelty 7.0 of 10

    Hypencoder replaces the query vector with a query-generated neural network that scores document vectors, and reports consistent gains over dense retrieval baselines across standard, out-of-domain, and hard retrieval b...

  3. KoVRE: Training an Efficient Embedding Model for Korean Visual Document Retrieval

    cs.AI 2026-08 conditional novelty 6.0 of 10

    A 2B single-vector retriever, KoVRE, trained with hard-negative mining and reranker distillation, achieves top aggregate Korean visual document retrieval scores.

  4. Tevatron Meets Megatron: Expert-Parallel LLM Reranker Training on an Academic Budget

    cs.IR 2026-08 conditional novelty 6.0 of 10

    Tevatron 3.0 adds Megatron expert parallelism to the open Tevatron reranker toolkit, trains a 30B MoE reranker on a small cluster, and shows it matches dense-8B reranking quality at higher serving throughput.

  5. Relevance-Based Embeddings: Lightweight Candidate Retrieval via Heavy-Ranker Calls

    cs.IR 2026-07 conditional novelty 6.0 of 10

    Relevance vectors to support items/queries, mapped by light networks, uniformly approximate any continuous ranker and improve lightweight candidate retrieval over dual encoders.

  6. Disentangling Locality and Entropy in Ranking Distillation

    cs.IR 2025-05 reject novelty 6.0 of 10

    Under ranking distillation, complex hard-negative sampling pipelines yield little or no benefit over BM25 sampling, while intermediate teacher score entropy improves in-domain effectiveness and the paper's generalizat...

  7. Knowledge Distillation for Enhancing Walmart E-commerce Search Relevance Using Large Language Models

    cs.IR 2025-05 conditional novelty 6.0 of 10

    Distilling a 7B LLM teacher into a BERT-base student with Margin-MSE loss on 170M teacher-labeled pairs yields a small student that matches or slightly beats the teacher on NDCG and improves Walmart's tail-query searc...

  8. Provence: efficient and robust context pruning for retrieval-augmented generation

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Provence prunes and reranks retrieved contexts in one pass, compressing 50-80% of the context while keeping question-answering accuracy close to the full-context baseline.

  9. Beyond Questions: Leveraging ColBERT for Keyphrase Search

    cs.IR 2024-12 conditional novelty 6.0 of 10

    Keyphrase-trained ColBERT variants improve ranking on keyphrase queries and match standard ColBERT on question queries.

  10. Towards a Relevance Posterior in Neural Information Access

    cs.IR 2026-07 conditional novelty 5.0 of 10

    Fusing a cached query-independent document-quality prior with BM25 or re-rankers improves TREC DL nDCG, with largest gains for RankZephyr, under a prior–likelihood view of retrieval.

  11. Rationale-Guided Knowledge Distillation for Cross-Lingual Stance Detection

    cs.CL 2026-07 conditional novelty 5.0 of 10

    Distilling LLM-generated reasoning rationales into mBERT via dual-path contrastive distillation improves cross-lingual stance detection by 1–3% accuracy on three benchmarks.

  12. RAMP: Robust Ad Recommendation Under Limited Personalized-Feature Availability via Masking and Alignment Pathways

    cs.IR 2026-07 conditional novelty 5.0 of 10

    RAMP splits ad prediction into personalized and non-personalized towers, then uses a distillation-style loss to transfer knowledge so non-personalized predictions stay accurate when user features are absent.

  13. Score-Only Distillation for Compact Dense Retrieval

    cs.IR 2026-07 conditional novelty 5.0 of 10

    Score-only distillation with a row-centered all-pairs PairMSE objective lets 0.6B bi-encoders recover up to 50% of the base-to-teacher retrieval gap under matched protocols.

  14. Conventional Contrastive Learning Often Falls Short: Improving Dense Retrieval with Cross-Encoder Listwise Distillation and Synthetic Data

    cs.IR 2025-05 conditional novelty 5.0 of 10

    Contrastive fine-tuning often degrades strong dense retrievers, while combining cross-encoder listwise distillation with diverse synthetic queries consistently improves them.

  15. Can Cross Encoders Produce Useful Sentence Embeddings?

    cs.CL 2025-02 conditional novelty 5.0 of 10

    Early layers of cross encoders embed single sentences well enough to initialize a shallow dual encoder that matches a 12-layer baseline while running 5.15x faster.

  16. Remining Hard Negatives for Generative Pseudo Labeled Domain Adaptation

    cs.IR 2025-01 conditional novelty 5.0 of 10

    Periodically refreshing hard negatives with the in-training retriever improves unsupervised domain adaptation of dense retrievers over the static GPL baseline.

  17. QABISAR: Query-Article Bipartite Interactions for Statutory Article Retrieval

    cs.IR 2024-12 conditional novelty 5.0 of 10

    QABISAR uses a query-article bipartite graph augmented with legal hierarchy and knowledge distillation to improve statutory article retrieval on BSARD.

  18. Comparative Analysis of Lion and AdamW Optimizers for Cross-Encoder Reranking with MiniLM, GTE, and ModernBERT

    cs.IR 2025-06 reject novelty 4.0 of 10

    Lion and AdamW are compared for reranker fine-tuning, but the reported Lion gains are confounded by a 10x learning-rate difference and an inverted GPU-utilization metric.

  19. Exp4Fuse: A Rank Fusion Framework for Enhanced Sparse Retrieval using Large Language Model-based Query Expansion

    cs.IR 2025-06 conditional novelty 4.0 of 10

    Exp4Fuse improves sparse retrieval by fusing the ranked lists from the original query and an LLM-expanded query using a modified reciprocal rank fusion.

  20. Semantic Certainty Assessment in Vector Retrieval Systems: A Novel Framework for Embedding Quality Evaluation

    cs.IR 2025-07 reject novelty 3.0 of 10

    A query-level score combining quantization stability and neighborhood density predicts retrieval performance and is claimed to improve Recall@10 by only 2 to 3 percent per dataset.

Pith tools