REVIEW 8 cited by
SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
In this paper, we propose SimLM (Similarity matching with Language Model pre-training), a simple yet effective pre-training method for dense passage retrieval. It employs a simple bottleneck architecture that learns to compress the passage information into a dense vector through self-supervised pre-training. We use a replaced language modeling objective, which is inspired by ELECTRA, to improve the sample efficiency and reduce the mismatch of the input distribution between pre-training and fine-tuning. SimLM only requires access to unlabeled corpus, and is more broadly applicable when there are no labeled data or queries. We conduct experiments on several large-scale passage retrieval datasets, and show substantial improvements over strong baselines under various settings. Remarkably, SimLM even outperforms multi-vector approaches such as ColBERTv2 which incurs significantly more storage cost. Our code and model check points are available at https://github.com/microsoft/unilm/tree/master/simlm .
Forward citations
Cited by 8 Pith papers
-
RaDeR: Reasoning-aware Dense Retrieval Models
A math-trained dense retriever and reranker, built from MCTS reasoning trajectories and self-reflection, outperforms strong baselines on reasoning-intensive retrieval benchmarks and beats BM25 on chain-of-thought queries.
-
O1 Embedder: Let Retrievers Think Before Action
A jointly trained retriever that first generates query thoughts and then encodes them improves accuracy on 12 retrieval benchmarks.
-
Matryoshka Re-Ranker: A Flexible Re-Ranking Architecture With Configurable Depth and Width
A single LLM re-ranker can be configured at runtime to different depths and widths, with training tricks that keep compressed variants close to full-scale accuracy.
-
AIR-Bench: Automated Heterogeneous Information Retrieval Benchmark
AIR-Bench uses LLMs to generate retrieval test data over 69 datasets, 9 domains and 13 languages, and reports a 0.82 rank correlation between model rankings on its generated MS MARCO set and human-labeled MS MARCO.
-
State Space Models are Strong Text Rerankers
Mamba-1 and Mamba-2 rerankers match comparably sized transformers on ranking accuracy but are less efficient in training and inference, with Mamba-2 improving on both fronts.
-
GOLFer: Smaller LM-Generated Documents Hallucination Filter & Combiner for Query Expansion in Information Retrieval
GOLFer filters hallucinated sentences from small-LM-generated hypothetical documents and reweights the rest into the query, improving retrieval at lower cost than large LLM expansion.
-
Exp4Fuse: A Rank Fusion Framework for Enhanced Sparse Retrieval using Large Language Model-based Query Expansion
Exp4Fuse improves sparse retrieval by fusing the ranked lists from the original query and an LLM-expanded query using a modified reciprocal rank fusion.
-
LLMs are Also Effective Embedding Models: An In-depth Overview
A structured survey of using decoder-only LLMs as text embedding models, covering prompting, fine-tuning, data construction, benchmarks, and open problems.
Discussion (0). Continue with ORCID to comment.