Pith. sign in

REVIEW 5 cited by

Distilling Dense Representations for Ranking using Tightly-Coupled Teachers

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2010.11386 v1 pith:PTTHWPOR submitted 2020-10-22 cs.IR cs.CL

Sheng-Chieh Lin , Jheng-Hong Yang , Jimmy Lin This is my paper · ORCID
classification cs.IRcs.CL
keywords representationsapproachcolbertdensedistillationmodeleffectivenessknowledge
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present an approach to ranking with dense representations that applies knowledge distillation to improve the recently proposed late-interaction ColBERT model. Specifically, we distill the knowledge from ColBERT's expressive MaxSim operator for computing relevance scores into a simple dot product, thus enabling single-step ANN search. Our key insight is that during distillation, tight coupling between the teacher model and the student model enables more flexible distillation strategies and yields better learned representations. We empirically show that our approach improves query latency and greatly reduces the onerous storage requirements of ColBERT, while only making modest sacrifices in terms of effectiveness. By combining our dense representations with sparse representations derived from document expansion, we are able to approach the effectiveness of a standard cross-encoder reranker using BERT that is orders of magnitude slower.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Disentangling Locality and Entropy in Ranking Distillation

    cs.IR 2025-05 reject novelty 6.0 of 10

    Under ranking distillation, complex hard-negative sampling pipelines yield little or no benefit over BM25 sampling, while intermediate teacher score entropy improves in-domain effectiveness and the paper's generalizat...

  2. Towards a Relevance Posterior in Neural Information Access

    cs.IR 2026-07 conditional novelty 5.0 of 10

    Fusing a cached query-independent document-quality prior with BM25 or re-rankers improves TREC DL nDCG, with largest gains for RankZephyr, under a prior–likelihood view of retrieval.

  3. Score-Only Distillation for Compact Dense Retrieval

    cs.IR 2026-07 conditional novelty 5.0 of 10

    Score-only distillation with a row-centered all-pairs PairMSE objective lets 0.6B bi-encoders recover up to 50% of the base-to-teacher retrieval gap under matched protocols.

  4. Conventional Contrastive Learning Often Falls Short: Improving Dense Retrieval with Cross-Encoder Listwise Distillation and Synthetic Data

    cs.IR 2025-05 conditional novelty 5.0 of 10

    Contrastive fine-tuning often degrades strong dense retrievers, while combining cross-encoder listwise distillation with diverse synthetic queries consistently improves them.

  5. Comparative Analysis of Lion and AdamW Optimizers for Cross-Encoder Reranking with MiniLM, GTE, and ModernBERT

    cs.IR 2025-06 reject novelty 4.0 of 10

    Lion and AdamW are compared for reranker fine-tuning, but the reported Lion gains are confounded by a 10x learning-rate difference and an inverted GPU-utilization metric.

Pith tools