Uptraining multi-head transformer checkpoints to grouped-query attention models achieves near multi-head quality at multi-query inference speeds using 5% additional compute.
FiDO: Fusion-in-Decoder Optimized for Stronger Performance and Faster Inference
3 Pith papers cite this work. Polarity classification is still indexing.
3
Pith papers citing it
abstract
Fusion-in-Decoder (FiD) is a powerful retrieval-augmented language model that sets the state-of-the-art on many knowledge-intensive NLP tasks. However, the architecture used for FiD was chosen by making minimal modifications to a standard T5 model, which our analysis shows to be highly suboptimal for a retrieval-augmented model. In particular, FiD allocates the bulk of FLOPs to the encoder, while the majority of inference time results from memory bandwidth constraints in the decoder. We propose two simple changes to the FiD architecture to alleviate memory bandwidth constraints, and speed up inference by 7x. This allows us to use a much larger decoder at modest cost. We denote FiD with the above modifications as FiDO, and show that it strongly improves performance over existing FiD models for a wide range of inference budgets. For example, FiDO-Large-XXL performs faster inference than FiD-Base and achieves better performance than FiD-Large.
citation-role summary
background 1
citation-polarity summary
roles
background 1polarities
background 1representative citing papers
SAGE uses offline imitation learning from a latency-quality oracle to choose per-query retrieval budgets, reporting 95% SLO compliance and 51% lower retrieval cost on Natural Questions with 2 points of EM loss.
Yi models are 6B and 34B open foundation models pretrained on 3.1T curated tokens that achieve strong benchmark results through data quality and targeted extensions like long context and vision alignment.
citing papers explorer
-
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Uptraining multi-head transformer checkpoints to grouped-query attention models achieves near multi-head quality at multi-query inference speeds using 5% additional compute.
-
SAGE: SLO-Aware Adaptive Retrieval for Production RAG Systems
SAGE uses offline imitation learning from a latency-quality oracle to choose per-query retrieval budgets, reporting 95% SLO compliance and 51% lower retrieval cost on Natural Questions with 2 points of EM loss.
-
Yi: Open Foundation Models by 01.AI
Yi models are 6B and 34B open foundation models pretrained on 3.1T curated tokens that achieve strong benchmark results through data quality and targeted extensions like long context and vision alignment.