REVIEW 5 cited by
Scaling Retrieval-Based Language Models with a Trillion-Token Datastore
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Scaling laws with respect to the amount of training data and the number of parameters allow us to predict the cost-benefit trade-offs of pretraining language models (LMs) in different configurations. In this paper, we consider another dimension of scaling: the amount of data available at inference time. Specifically, we find that increasing the size of the datastore used by a retrieval-based LM monotonically improves language modeling and several downstream tasks without obvious saturation, such that a smaller model augmented with a large datastore outperforms a larger LM-only model on knowledge-intensive tasks. By plotting compute-optimal scaling curves with varied datastore, model, and pretraining data sizes, we show that using larger datastores can significantly improve model performance for the same training compute budget. We carry out our study by constructing a 1.4 trillion-token datastore named MassiveDS, which is the largest and the most diverse open-sourced datastore for retrieval-based LMs to date, and designing an efficient pipeline for studying datastore scaling in a computationally accessible manner. Finally, we analyze the effect of improving the retriever, datastore quality filtering, and other design choices on our observed scaling trends. Overall, our results show that datastore size should be considered as an integral part of LM efficiency and performance trade-offs. To facilitate future research, we open-source our datastore and code at https://github.com/RulinShao/retrieval-scaling.
Forward citations
Cited by 5 Pith papers
-
Salient Knowledge Pathways: Sparse Cross-Modal Routing for Efficient Knowledge-Intensive Multimodal Question Answering
A sparse cross-modal routing system matches or exceeds dense retrieval-augmented baselines on five KI-MMQA benchmarks at 3.4–6.8× lower FLOPs.
-
RbFT: Robust Fine-tuning for Retrieval-Augmented Generation against Retrieval Defects
Fine-tuning an LLM with defect detection and utility extraction tasks makes it more robust to noisy, irrelevant, and counterfactual documents in retrieval-augmented generation.
-
Healthier LLMs: Retrieval-Augmented Generation for Public Health Question Answering
Hybrid RAG over UK public health guidance sharply raises MCQA accuracy and free-form faithfulness, letting smaller open models match larger closed models without retrieval.
-
Uncovering Scaling Laws for Large Language Models via Inverse Problems
Proposes using inverse problems to discover LLM scaling laws, but provides no empirical evidence or new results.
-
VeriDebug: A Unified LLM for Verilog Debugging via Contrastive Embedding and Guided Correction
A jointly trained embedding and generation model for Verilog debugging that retrieves candidate buggy lines and types before generating fixes, reported at 64.7% Acc@1 on a new synthetic dataset.
Discussion (0). Continue with ORCID to comment.