REVIEW 14 cited by
Bridging the Gap Between Indexing and Retrieval for Differentiable Search Index with Query Generation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
The Differentiable Search Index (DSI) is an emerging paradigm for information retrieval. Unlike traditional retrieval architectures where index and retrieval are two different and separate components, DSI uses a single transformer model to perform both indexing and retrieval. In this paper, we identify and tackle an important issue of current DSI models: the data distribution mismatch that occurs between the DSI indexing and retrieval processes. Specifically, we argue that, at indexing, current DSI methods learn to build connections between the text of long documents and the identifier of the documents, but then retrieval of document identifiers is based on queries that are commonly much shorter than the indexed documents. This problem is further exacerbated when using DSI for cross-lingual retrieval, where document text and query text are in different languages. To address this fundamental problem of current DSI models, we propose a simple yet effective indexing framework for DSI, called DSI-QG. When indexing, DSI-QG represents documents with a number of potentially relevant queries generated by a query generation model and re-ranked and filtered by a cross-encoder ranker. The presence of these queries at indexing allows the DSI models to connect a document identifier to a set of queries, hence mitigating data distribution mismatches present between the indexing and the retrieval phases. Empirical results on popular mono-lingual and cross-lingual passage retrieval datasets show that DSI-QG significantly outperforms the original DSI model.
Forward citations
Cited by 14 Pith papers
-
KuaiSearch: An E-Commerce Search Dataset with Authentic Queries and Product Texts for Recall, Ranking, and Relevance
KuaiSearch is a new large-scale, plaintext e-commerce search dataset from Kuaishou spanning recall, ranking, and relevance tasks.
-
RetroLLM: Empowering Large Language Models to Retrieve Fine-grained Evidence within Generation
RetroLLM unifies retrieval and generation by having an LLM generate corpus-constrained clues and then fine-grained evidence before answering, outperforming RAG baselines on open-domain QA.
-
Token-Level Credit Assignment Optimization for Generative Document Retrieval
TCA, a token-level credit assignment reinforcement learning framework, improves R@1 and MRR@10 over sequence-level RL baselines in generative document retrieval on MS MARCO and Natural Questions.
-
Generative Chinese Statute Retrieval
A generative retriever with multi-granularity structured statute IDs and multi-task training outperforms strong sparse, dense, and legal baselines on the STARD Chinese statute benchmark.
-
Revisiting Text Ranking in Deep Research
In deep-research agents, passage-level BM25 retrieval plus re-ranking beats larger neural rankers, and converting agent queries into natural questions (Q2Q) improves neural ranking.
-
MixLoRA-DSI: Dynamically Expandable Mixture-of-LoRA Experts for Rehearsal-Free Generative Retrieval over Dynamic Corpora
A rehearsal-free generative retriever that expands mixture-of-LoRA experts only when router energy scores flag out-of-distribution tokens, achieving sublinear parameter growth over dynamic corpora.
-
Bridging the Gap: From Ad-hoc to Proactive Search in Conversations
Conv2Query fine-tunes an LLM to convert conversational context into ad-hoc queries, enabling off-the-shelf retrievers to work effectively on proactive search in conversations.
-
Birdie: Natural Language-Driven Table Discovery Using Differentiable Search Index
Birdie applies a differentiable search index to table discovery: a single model is trained to map natural language queries to table identifiers, outperforming dense retrieval baselines on three benchmarks.
-
Replication and Exploration of Generative Retrieval over Dynamic Corpora
Generative retrieval with text-based docids (n-grams, titles, URLs) generalizes to newly added documents far better than numeric docids, and a constrained multi-docid numeric design recovers much of the gap on NQ.
-
DOGR: Leveraging Document-Oriented Contrastive Learning in Generative Retrieval
DOGR combines identifier generation with document-level contrastive learning and a fused relevance score, improving generative retrieval on NQ320k and MS MARCO.
-
Stay Hungry, Stay Foolish: On the Extended Reading Articles Generation with LLMs
An LLM pipeline generates TED-Ed extended reading articles and course recommendations, but its recommendation hit rate drops when the generation stages are added, undercutting the claimed accuracy.
-
Adaptive Two-Phase Finetuning LLMs for Japanese Legal Text Retrieval
A two-phase fine-tuning pipeline (global contrastive pretraining, then domain-specific hard-negative training) improves LLaMA-based dense retrieval on a Gemini-generated Japanese legal dataset and on a subset of MS MARCO.
-
Optimizing Multi-Stage Language Models for Effective Text Retrieval
A language-model-only, two-phase retrieval pipeline with hard-negative training and a grid-searched ensemble is reported to outperform sparse, dense, and generative baselines on a Japanese legal retrieval test set and...
-
Foundations of GenIR
A survey chapter proposing that generative AI reshapes information access through two paradigms, information generation and information synthesis.
Discussion (0). Continue with ORCID to comment.