REVIEW 9 cited by
Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring
read the original abstract
The use of deep pre-trained bidirectional transformers has led to remarkable progress in a number of applications (Devlin et al., 2018). For tasks that make pairwise comparisons between sequences, matching a given input with a corresponding label, two approaches are common: Cross-encoders performing full self-attention over the pair and Bi-encoders encoding the pair separately. The former often performs better, but is too slow for practical use. In this work, we develop a new transformer architecture, the Poly-encoder, that learns global rather than token level self-attention features. We perform a detailed comparison of all three approaches, including what pre-training and fine-tuning strategies work best. We show our models achieve state-of-the-art results on three existing tasks; that Poly-encoders are faster than Cross-encoders and more accurate than Bi-encoders; and that the best results are obtained by pre-training on large datasets similar to the downstream tasks.
Forward citations
Cited by 9 Pith papers
-
Learning to Unscramble Feynman Loop Integrals with SAILIR
A self-supervised transformer learns to unscramble Feynman integrals for online IBP reduction, delivering bounded memory use on complex two-loop topologies while matching Kira's speed on the hardest cases tested.
-
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Sentence-BERT adapts BERT with siamese and triplet networks to produce sentence embeddings for efficient cosine-similarity comparisons, cutting computation time from hours to seconds on similarity search while matchin...
-
GrocLM: Grocery Category Recommendation in E-Commerce with Large Language Models
A LoRA-tuned LLM with trie-constrained decoding improves grocery category recommendation and yields a 7.5% cart-add lift in production.
-
HiSAC: Hierarchical Sparse Activation Compression for Ultra-long Sequence Modeling in Recommenders
Hierarchical semantic tokenization with sparse interest-agent voting and soft-routing attention compresses ultra-long user histories for recommenders, reporting a 1.65% online CTR lift at Taobao.
-
Atlas: Few-shot Learning with Retrieval Augmented Language Models
Atlas reaches over 42% accuracy on Natural Questions with only 64 examples, outperforming a 540B-parameter model by 3% with 50x fewer parameters.
-
Unsupervised Dense Information Retrieval with Contrastive Learning
Contrastive learning trains unsupervised dense retrievers that beat BM25 on most BEIR datasets and support cross-lingual retrieval across scripts.
-
Multimodal Video-to-Music Recommendation via Semantic Retrieval and Temporal Reranking
VTMR is a two-stage video-to-music recommender: joint audio-visual-text retrieval of candidates, then temporal-sequence reranking, lifting R@10 to 18.3 and matching commercial preference.
-
Next-Gen Sponsored Search: Crafting the Perfect Query with Inventory-Aware RAG (InvAwr-RAG) Based GenAI
Inventory-aware RAG plus LoRA-tuned Llama-2 rewrites zero-fill sponsored-search queries to 68% fill rate and higher NDCG@8 than GPT-4 on 10k historical Walmart queries.
-
Cross-Subject EEG Emotion Recognition Based on Temporal Asynchronous Alignment Contrastive Learning
Proposes TA2CL framework that uses temporal asynchronous alignment in contrastive learning to improve cross-subject EEG emotion classification, reporting 64.5% accuracy on 9-class FACED, 79.5% binary on FACED, 86.4% o...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.