RETRO matches GPT-3 and Jurassic-1 performance on the Pile benchmark using 25 times fewer parameters by conditioning on retrieved chunks from a 2-trillion-token database.
InProceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval
2 Pith papers cite this work. Polarity classification is still indexing.
2
Pith papers citing it
verdicts
UNVERDICTED 2representative citing papers
Combining contrastive loss with KLD distillation and adding sparsity regularization improves effectiveness and reduces FLOPS by 2x in conversational search with minimal recall loss.
citing papers explorer
-
Improving language models by retrieving from trillions of tokens
RETRO matches GPT-3 and Jurassic-1 performance on the Pile benchmark using 25 times fewer parameters by conditioning on retrieved chunks from a 2-trillion-token database.
-
Improving the Efficiency and Effectiveness of LLM Knowledge Distillation for Conversational Search
Combining contrastive loss with KLD distillation and adding sparsity regularization improves effectiveness and reduces FLOPS by 2x in conversational search with minimal recall loss.