{"paper":{"title":"GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Uptraining multi-head attention checkpoints to grouped-query attention recovers near-original quality with only 5% additional compute and achieves multi-query inference speeds.","cross_cats":["cs.LG"],"primary_cat":"cs.CL","authors_text":"Federico Lebr\\'on, James Lee-Thorp, Joshua Ainslie, Michiel de Jong, Sumit Sanghai, Yury Zemlyanskiy","submitted_at":"2023-05-22T17:16:38Z","abstract_excerpt":"Multi-query attention (MQA), which only uses a single key-value head, drastically speeds up decoder inference. However, MQA can lead to quality degradation, and moreover it may not be desirable to train a separate model just for faster inference. We (1) propose a recipe for uptraining existing multi-head language model checkpoints into models with MQA using 5% of original pre-training compute, and (2) introduce grouped-query attention (GQA), a generalization of multi-query attention which uses an intermediate (more than one, less than number of query heads) number of key-value heads. We show t"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"We show that uptrained GQA achieves quality close to multi-head attention with comparable speed to MQA.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The uptraining recipe with 5% compute is sufficient to recover near-original quality without task-specific degradation or architecture-dependent failures.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"Uptraining multi-head transformer checkpoints to grouped-query attention models achieves near multi-head quality at multi-query inference speeds using 5% additional compute.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Uptraining multi-head attention checkpoints to grouped-query attention recovers near-original quality with only 5% additional compute and achieves multi-query inference speeds.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"083e9c9899dea7d57941e9ab9c3d62927a7ba7c1505b4c5d2ac0c8c264197600"},"source":{"id":"2305.13245","kind":"arxiv","version":3},"verdict":{"id":"0a685e45-ceed-4ec1-98b4-cb5802fe461b","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-11T06:48:00.303359Z","strongest_claim":"We show that uptrained GQA achieves quality close to multi-head attention with comparable speed to MQA.","one_line_summary":"Uptraining multi-head transformer checkpoints to grouped-query attention models achieves near multi-head quality at multi-query inference speeds using 5% additional compute.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The uptraining recipe with 5% compute is sufficient to recover near-original quality without task-specific degradation or architecture-dependent failures.","pith_extraction_headline":"Uptraining multi-head attention checkpoints to grouped-query attention recovers near-original quality with only 5% additional compute and achieves multi-query inference speeds."},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2305.13245/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":40,"sample":[{"doi":"","year":null,"title":"James Bradbury and Roy Frostig and Peter Hawkins and Matthew James Johnson and Chris Leary and Dougal Maclaurin and George Necula and Adam Paszke and Jake Vander","work_id":"f722cabc-fda6-4889-82c5-313113edc5fb","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":null,"title":"Jonathan Heek and Anselm Levskaya and Avital Oliver and Marvin Ritter and Bertrand Rondepierre and Andreas Steiner and Marc van","work_id":"099845dd-6d9a-4e0c-a9f2-0f6843660d5f","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":null,"title":"Alexey Romanov and Chaitanya Shivade","work_id":"b4ce3d71-9be2-4401-8618-528a87b188bd","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2015,"title":"Kingma and Jimmy Ba , editor =","work_id":"03bfe487-67cd-4832-be0e-f98027fded15","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2018,"title":"Adafactor: Adaptive Learning Rates with Sublinear Memory Cost , booktitle =","work_id":"2835d2d4-5323-4fa8-a5df-250f93b86aee","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":40,"snapshot_sha256":"c71236cb8c84b6325f2f3c7c9ee93b91ca4873d7a633cd472e5edd15e77c805d","internal_anchors":11},"formal_canon":{"evidence_count":2,"snapshot_sha256":"934aee5a5c19d1e9498cbc8f71a6f6069daa79f5e1b5ace090291877a0775c6f"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}