Minimizing FLOPs to Learn Efficient Sparse Representations
read the original abstract
Deep representation learning has become one of the most widely adopted approaches for visual search, recommendation, and identification. Retrieval of such representations from a large database is however computationally challenging. Approximate methods based on learning compact representations, have been widely explored for this problem, such as locality sensitive hashing, product quantization, and PCA. In this work, in contrast to learning compact representations, we propose to learn high dimensional and sparse representations that have similar representational capacity as dense embeddings while being more efficient due to sparse matrix multiplication operations which can be much faster than dense multiplication. Following the key insight that the number of operations decreases quadratically with the sparsity of embeddings provided the non-zero entries are distributed uniformly across dimensions, we propose a novel approach to learn such distributed sparse embeddings via the use of a carefully constructed regularization function that directly minimizes a continuous relaxation of the number of floating-point operations (FLOPs) incurred during retrieval. Our experiments show that our approach is competitive to the other baselines and yields a similar or better speed-vs-accuracy tradeoff on practical datasets.
This paper has not been read by Pith yet.
Forward citations
Cited by 6 Pith papers
-
Rescaling MLM-Head for Neural Sparse Retrieval
Rescaling the MLM-head projection by a constant factor at initialization resolves scale mismatch in learned sparse retrieval and enables stable use of large-norm backbones such as ModernBERT.
-
Why Advanced Encoders Lag on Sparse Retrieval? The Answer and an Approach to Bridging Vocabulary Gaps
Transferring modern encoders to normalized (lowercased) vocabularies via geometric embedding initialization and activation calibration closes the performance gap in learned sparse retrieval, achieving 52.4 nDCG on BEIR.
-
Efficient Retrieval Scaling with Hierarchical Indexing for Large Scale Recommendation
A jointly learned hierarchical index with cross-attention and residual quantization scales exact retrieval in foundational recommendation models, deployed at Meta with additional performance from test-time training on...
-
Improving the Efficiency and Effectiveness of LLM Knowledge Distillation for Conversational Search
Combining contrastive loss with KLD distillation and adding sparsity regularization improves effectiveness and reduces FLOPS by 2x in conversational search with minimal recall loss.
-
Surface-Form Neural Sparse Retrieval: Robust Fuzzy Matching for Industrial Music Search
A neural sparse retrieval system with granular subword tokenization (max 3 chars) achieves 91.4% recall@10 on a 6M music document corpus versus 57.7% for trigrams, with improved HCI exploration efficiency and zero add...
-
The Role of Vocabularies in Learning Sparse Representations for Ranking
Larger 100K vocabularies in SPLADE models, especially those initialized with ESPLADE pretraining, improve retrieval effectiveness after pruning compared to 32K baselines while keeping similar efficiency.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.