{"id":"335655b6-6a70-4dd9-89e4-bb1ec8e7cd55","arxiv_id":"2505.21487","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Grouped-Tied Attention and Grouped Latent Attention reduce KV-cache memory and speed up LLM decoding by up to 2x while matching the quality of GQA and MLA at up to 1.47B parameters.","lead":"This paper introduces two new attention designs, GTA and GLA, that shrink the memory footprint of key-value caches during AI text generation and speed up token-by-token decoding on GPUs. The authors show GLA can serve responses up to twice as fast as the existing DeepSeek MLA approach while keeping model quality about the same.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Serving/kernel benchmarks use GLA-8 with 8x256 latent (4x MLA's 512) and random weights; quality parity is shown only for GLA-2 at <=1.47B, so the 'practical replacement' claim rests on an unvalidated configuration.","rationale":"The paper has genuine strengths: a clean arithmetic-intensity analysis, an open-source kernel implementation, careful paged-KV optimization, quality checks at four scales up to 1.47B, and live-server SGLang benchmarks. I am not questioning the authors' honesty or the validity of the kernel engineering. The concern is about how the evidence is connected.\n\nThe headline claim combines a quality claim and a speed claim into 'practical replacement.' The quality claim is supported for GLA-2 only; the speed claim is supported for GLA-8 only. These are different architectures: GLA-2 has two 256-d latent heads (total 512, same as MLA), while GLA-8 has eight 256-d latent heads (total 2048, 4x MLA). The serving benchmark's GLA-8 also uses randomly initialized weights; the authors state this is intentional because they benchmark performance, not accuracy. But the conclusion then treats the serving result as evidence that GLA can replace MLA, which requires quality at the benchmarked configuration. No such evidence is presented.\n\nThe reader's weakest assumption was scale transfer from 1.47B to 100B+. My concern is more immediate: even within the paper's own scales, the benchmarked architecture differs from the validated one. If GLA-8's extra latent capacity is what allows it to maintain quality while sharding, then the 2x speedup is partly a capacity-vs-duplication trade, not a pure architectural win. The per-device KV reduction in Table 27 comes from GLA-8 storing 256 per device versus MLA's 512 duplicated per device, but GLA-8 stores 2048 total, four times MLA's 512. Thus the comparison is not apples-to-apples on total memory or information capacity.\n\nThis does not refute the paper; it identifies a missing link. The kernel engineering is real, and GLA-2 quality parity at 1.47B is encouraging. But the central 'practical replacement' statement is conditionally supported at best. I would keep the reader's CONDITIONAL verdict and add an explicit condition: validate the exact GLA-8 (or a controlled total-latent-matched GLA) configuration used in serving, or restrict the speed claim to the validated GLA-2 setting.","tokens_in":37414,"tokens_out":8168,"duration_ms":76781,"concrete_test":"Train and evaluate the exact GLA-8 configuration used in the serving benchmark - eight latent heads of dimension 256, total latent 2048, h_q=128 - at 1.47B on FineWeb-Edu, and compare FineWeb-Edu perplexity and average downstream accuracy against MLA in Tables 5 and 17. If GLA-8 fails to match MLA's 10.256 PPL/59.1% average, the Table 27/Figure 4 serving comparison has no valid quality basis for the 'practical replacement' claim; if it matches, rerun the serving test with GLA-2 (total latent 512) to verify the speedup is not due to the 4x larger total latent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that GLA is a practical replacement for MLA is assembled from two non-overlapping experiment sets. Quality parity (Section 5.1, Tables 3-5) is demonstrated for GLA-2: two latent heads of dimension 2d_h, total latent 4d_h = 512 at the 1.47B scale, matching MLA's single 512-d latent, with h_q=16. The serving and kernel benchmarks (Section 5.2-5.3, Tables 27-32, Figure 4) instead use GLA-8: eight latent heads of dimension 256, total 2048 bytes/token of latent, i.e., four times MLA's 512, sharded one head per GPU, and the model is simulated by restructuring DeepSeek-Coder-V2 with randomly initialized weights (Appendix B.6). The reported up-to-2x throughput/latency advantage is therefore produced by a never-trained architecture that also carries 4x the total latent; the comparison does not isolate GLA's grouping/sharding mechanism. The kernel speed claim is likewise measured at query heads=128 (Figure 4), while all quality-validated models use h_q=12-16. Consequently, concluding 'GLA is a practical replacement for MLA' (Section 7) extrapolates across both scale and architecture in one step: no evidence shows the benchmarked GLA-8 preserves MLA-comparable quality at any size.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes two attention mechanisms for efficient LLM decoding: Grouped-Tied Attention (GTA), which ties key and value into a single state shared by a group of query heads with partial RoPE, and Grouped Latent Attention (GLA), which splits MLA's latent into h_c grouped latent heads of dimension 2d_h. The paper derives arithmetic-intensity estimates for these architectures, describes CUDA kernel optimizations (software pipelining, warp specialization, distributed offset calculation for paged KV), and reports: (i) quality comparisons on FineWeb-Edu at 183M–1.47B parameters showing GTA-4 roughly matching or improving on GQA-4 and GLA-2 roughly matching MLA; (ii) kernel benchmarks versus FlashMLA with up to 2x speedup at query length 2 and h_q=128; (iii) online serving benchmarks on a restructured DeepSeek-Coder-V2 model with randomly initialized GLA weights showing up to ~2x throughput/latency gains over MLA in some configurations. The central claim is that GTA is an efficient replacement for GQA and GLA is a practical replacement for MLA.","tokens_in":37743,"tokens_out":9095,"duration_ms":90177,"significance":"If the claims are substantiated, GLA would be a practically important extension of MLA: by grouping latent heads it avoids MLA's per-device duplication under tensor parallelism while preserving the low-KV-cache benefit, and GTA offers a simple tied-KV alternative to GQA. The paper's strengths include a clean arithmetic-intensity framework, parameter-matched quality experiments at several scales, and open-source kernel engineering with measurable optimizations (e.g., page-size-1 paged KV matching page-size-64 speed, Figure 6). The main risk is that the headline 'practical replacement' claim rests on splicing together quality results for GLA-2 (<=1.47B) and system results for an untrained GLA-8 configuration on a different model, with no configuration where both quality and speed are demonstrated.","major_comments":[{"comment":"Quality and system evaluations use disjoint configurations. The quality evidence for GLA is limited to GLA-2 (h_c=2, d_c=2d_h) on models up to 1.471B (Tables 2–5), while the kernel and serving benchmarks use GLA-8 (h_c=8, d_c=256; total latent 2048 bytes/token versus MLA's 512) on a restructured DeepSeek-Coder-V2 model with randomly initialized weights (Appendix B.6, Figure 4). No experiment shows that GLA-8 retains MLA-comparable quality at any scale. The conclusion in Section 7 that 'GLA is a practical replacement for MLA' therefore extrapolates across both configuration and scale. Please either train and evaluate GLA-8 at least at the 1.47B scale, or explicitly restrict the quality-parity claim to GLA-2 and present the system results as workload-level illustrations of an untrained configuration.","section":"§5.1, §5.2–5.3, §7"},{"comment":"Kernel benchmarks use a query-head count not covered by the quality experiments. Section 5.3 and Figure 4 report decoding speed for query heads h_q=128 and query lengths 1 and 2, while all quality-validated models use h_q=12–16 (Tables 6–10). Because the arithmetic intensity grows with h_q (Table 1), the claimed 1.2–2x kernel speedup is only established for h_q=128. The paper should report kernel latencies at the h_q values of the trained models, or provide a sweep over h_q showing the speedup persists at lower h_q.","section":"§5.3, Figure 4"},{"comment":"Online-serving results are for randomly initialized GLA weights. Appendix B.6 states 'To simulate GLA, we restructure the MLA latent dimension to GLA with randomly initializing weights since we benchmark performance, not accuracy, in this phase.' While this is legitimate for a system-level bandwidth study, the title and abstract claim that 'GLA reduces end-to-end latency and increases token throughput by up to 2x' without flagging that the benchmarked model has never been trained. Please state this limitation in the main text alongside the speed claims and avoid using the serving numbers as direct evidence for the 'practical replacement' conclusion.","section":"Appendix B.6"},{"comment":"The GLA-8 versus MLA serving comparison confounds grouping with total latent size. GLA-8 stores 8x256 = 2048 bytes/token of latent across the system, four times MLA's 512 bytes/token; per-device savings come from spreading this larger cache over TP ranks. Consequently Tables 27–32 do not isolate the effect of GLA's grouped-latent sharding from the effect of a larger aggregate cache. A comparison with a GLA variant of the same total latent budget (e.g., h_c=8, d_c=64) would separate these factors; please add such an ablation or soften the interpretation.","section":"§5.2, Tables 27–32"},{"comment":"Quality results lack error bars or multi-seed runs. The parity claims in Section 5.1 rely on small differences (e.g., 1.47B: FineWeb-Edu PPL 10.218 vs 10.256 and downstream 60.0% vs 59.1% for GLA-2 vs MLA, Table 5). Single training runs cannot distinguish these from noise, so statements such as 'GLA consistently matches or exceeds MLA accuracy on all model scales' (Section 7) are stronger than the data support. At minimum, acknowledge this in the limitations and use confidence-aware language.","section":"§5.1, Tables 2–5"}],"minor_comments":[{"comment":"The column headers and arithmetic-intensity formulas are garbled in the formatted text (e.g., the GLA-2 and GLA columns are not clearly separated from the General Variant row); please reformat for readability.","section":"Table 1"},{"comment":"Section 5.2 says GLA-8 achieves 'roughly 2.5x the MLA throughput', but Table 35 reports values implying 2.7x (100.68 vs 37.20 tokens/s); please align the numbers.","section":"§5.2, Figure 5, Table 35"},{"comment":"The caption says both models run with 'eight-way tensor parallelism and four-way data parallel attention', but the configuration is TP=4, DP=2; the caption should match the actual parallelism scheme.","section":"Table 31"},{"comment":"The sentence 'we set the h_c = 2 for GLA' appears in the general methodology, but later experiments use GLA-8; clarify that this sentence refers to the quality experiments only.","section":"§3.3.2"},{"comment":"Several parallelism descriptors are inconsistent: Table 29's caption says 'eight-way tensor parallelism and four-way data parallel attention' while the configuration is TP=2, DP=4; please audit all parallelism descriptors in the appendix.","section":"Appendix B.6.2, Tables 29 and 31"}],"recommendation":"major_revision","confidential_remarks":"The strongest claims in the abstract and conclusion exceed what the experiments support. I would not reject the paper: the core architectural ideas are sound and the engineering contributions are real. However, the authors should be asked to either close the quality-speed gap for GLA-8 or substantially temper the 'practical replacement' language. The manuscript also seems well aligned with a systems-oriented audience; if the journal expects algorithmic novelty, the incremental novelty over GQA/MLA should be argued more sharply."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a good systems paper. The two attention variants (GTA, GLA) are real, and the small-scale quality comparison is honestly conducted. But the headline serving/throughput numbers are produced with a configuration (GLA-8) that was never trained, so the 'practical replacement' claim in the conclusion is not backed by the experiments as written.\n\nWhat's new: GTA ties K and V with partial RoPE, halving KV cache relative to GQA at same group count; GLA splits the latent into per-head groups so tensor-parallel sharding doesn't duplicate the cache. Both are sensible and, as far as I can tell, not in the cited prior work. The paper also contributes a decoding kernel that overlaps memory and compute via warp specialization and a cooperative paged-offset calculation; the kernel speedups against FlashMLA (1.2-2x) are plausible and the code is released.\n\nThe quality experiments are the strongest part. They match parameter counts by widening MLPs, compare against GQA, GTA-4, MLA, MHA, MQA at four scales, and the differences are small but consistently in favor of GTA/GLA at up to 1.47B. No error bars, but the effect sizes are 0.1-0.3 perplexity and 1-2 downstream points, so single runs are a minor concern, not a fatal one.\n\nThe soft spot is the serving benchmark. Tables 27-32 simulate GLA by restructuring DeepSeek-Coder-V2 with randomly initialized GLA weights. They explicitly say they are benchmarking performance, not accuracy, which is honest, but it means the throughput numbers come from an architecture that has never been trained and that carries four times MLA's latent (GLA-8 with 8x256 vs MLA's 512). The quality parity is shown only for GLA-2. So the conclusion that GLA is a practical replacement for MLA extrapolates across both scale and architecture in one step. The kernel speed comparison (Figure 4) also uses 128 query heads, while all quality-validated models use 12-16; that's a legitimate kernel-level benchmark, but it is not a model-level claim.\n\nWho this is for: inference engineers and attention-architecture researchers. It deserves a serious referee; the core architecture ideas are worth debating and the kernel work is reproducible. I would ask for a revision that either trains GLA-8 at a moderate scale or softens the replacement claim.","headline":"Solid systems paper with clean small-scale quality experiments; the serving-benchmark numbers come from a different, untrained configuration, so the 'practical replacement' claim outruns the evidence.","tokens_in":38281,"tokens_out":1965,"would_cite":true,"duration_ms":20095,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two attention designs cut KV cache memory in half with no quality loss and decode up to 2× faster.","keywords":["attention mechanisms","KV cache compression","inference efficiency","tensor parallelism","arithmetic intensity","grouped latent attention","grouped-tied attention","speculative decoding"],"falsifier":"Train a several-hundred-billion-parameter model with GLA and compare it against a matched MLA model on standard downstream benchmarks; a meaningful quality gap would falsify the parity claim, as would serving benchmarks where the GLA kernel is not faster than FlashMLA under the reported query lengths.","tokens_in":37190,"feed_emoji":"⚡","tokens_out":6644,"duration_ms":62767,"temperature":0.7,"pith_summary":"Decoding a large language model is bottlenecked by loading the cached key-value states of previous tokens from memory, not by computation. This paper argues attention should be redesigned so each byte loaded from the KV cache does more arithmetic, and proposes two variants that do so without hurting model quality. Grouped-Tied Attention (GTA) reuses a single tied state as both key and value within each group, cutting the KV cache roughly in half relative to Grouped-Query Attention (GQA) at the same quality. Grouped Latent Attention (GLA) splits the compressed latent KV representation into separate heads that shard across GPUs, matching Multi-head Latent Attention (MLA) quality while fetching a smaller per-device cache and running up to 2× faster in decoding and serving. If these results hold at scale, attention would no longer force a trade-off between quality and memory efficiency during generation.","feed_headline":"New attention designs cut KV cache in half and double token throughput","feed_subtitle":"GTA matches GQA at half the cache; GLA matches MLA and shards cleanly across GPUs.","key_machinery":"The load-bearing object is the arithmetic intensity of decoding attention, $I \\approx 2 g_q / m_{kv}$ FLOPs per byte loaded, with group size $g_q = h_q/h_{kv}$ (queries per KV head) and $m_{kv}$ the multiplicity of key/value states ($m_{kv}=1$ tied, $m_{kv}=2$ separate). GTA raises $I$ by tying K and V into one shared state and rotating only a partial RoPE slice; GLA raises $I$ while preserving parallel sharding by using $h_c$ latent heads of dimension $2d_h$ that split across tensor-parallel ranks. The zero-redundancy condition $g_q \\le h_q/N$ on $N$ shards is the constraint that keeps high intensity from collapsing into cache duplication.","core_discovery":"The paper's central claim is that the decoding bottleneck is not computation but memory traffic: each token's attention loads a KV cache that grows with batch and context, and standard attention does only about one FLOP per byte fetched. By measuring arithmetic intensity, the paper shows that increasing the group size $g_q$ (query heads per KV head) raises intensity to roughly $2g_q/m_{kv}$ FLOPs per byte, where $m_{kv}=1$ when keys and values are tied and 2 when separate, and that this gain eventually collides with distributed parallelism: with $N$ tensor-parallel shards, the duplication factor is $D=\\lceil N g_q / h_q\\rceil$, and zero duplication requires $g_q \\le h_q/N$. GTA is the tied-state version of GQA—one state serves as both key and value, with only a small separate RoPE head for position—roughly halving the KV cache at equal quality. GLA is a grouped version of MLA: latent heads of dimension $2d_h$ are sharded across devices so each GPU fetches a smaller cache, matching MLA quality at 1.47B parameters while decoding and serving up to 2× faster.","pith_inferences":["Inference: If the quality parity holds at 100B+ scale, GLA could replace MLA in deployed long-context models, where the per-device cache duplication of MLA is currently mitigated by hybrid data parallelism that adds synchronization stalls.","Inference: The paper's framing implies that the next bottleneck after KV traffic will be compute at query length 2 or more; combining GLA with speculative decoding or multi-token prediction could compound the speedup.","Inference: A directly testable extension of the paper's logic is to swap GLA-8 into a Llama-4-style GQA-8 architecture at hundreds of billions of parameters; the paper states this comparison is open.","Inference: The partial-RoPE and tied-KV insight suggests a family of hybrids—tying only a fraction of layers, or combining GTA-style tying with post-hoc KV quantization—that the paper does not evaluate."],"forward_implications":["GTA matches GQA quality with roughly half the KV cache, so switching from GQA to GTA buys longer contexts or larger batches at the same memory cost.","GLA matches MLA quality up to 1.47B parameters while halving the per-device KV cache under tensor parallelism of degree 2 or more, implying longer generations or larger batches before hitting memory limits.","In online serving with 64 concurrent requests on eight H100s, GLA-8 cut median end-to-end latency from 381 to 179 seconds versus MLA and raised throughput by about 70%.","In speculative decoding (query length 2), the GLA kernel is over 2× faster than FlashMLA, and at query length 1 it is about 20% faster.","The distributed offset calculation makes page-size-1 paged KV as fast as page-size-64, enabling prefix caching without a slowdown."],"supporting_citations":[{"why":"Introduces GQA, the grouping baseline against which GTA's quality and cache size are measured.","marker":"Ainslie et al., 2023"},{"why":"Introduces MLA, the latent-attention baseline GLA matches in quality and outperforms in serving.","marker":"DeepSeek-AI, 2024"},{"why":"FlashMLA kernel is the high-performance baseline for the up-to-2x kernel speedup claims.","marker":"Li, 2025"},{"why":"Roofline model and arithmetic intensity, the analysis lens used to derive the design constraints.","marker":"Williams et al., 2009"},{"why":"FlashAttention-3 system techniques (warp specialization, asynchronous pipelining) that the GLA kernel builds on.","marker":"Shah et al., 2024a"},{"why":"Paged KV memory management whose address-calculation overhead the distributed offset calculation removes.","marker":"Kwon et al., 2023"},{"why":"RoPE, whose partial-slice application is central to GTA's tied key/value design.","marker":"Su et al., 2023"},{"why":"Evidence that keys live in a low-rank subspace, motivating the tied and partially rotated key representation.","marker":"Saxena et al., 2024"}],"fun_headline_variants":["Attention redesign cuts KV cache in half, doubles token speed","Half the KV cache, 2x throughput: new attention kernels","GTA and GLA: leaner attention, faster decoding","Memory-efficient attention: same quality, 2x faster","GLA shards cleanly, matches MLA, and runs 2x faster"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the quality parity measured at up to 1.47B parameters on FineWeb-Edu continues to hold for 100B-plus models, the scale where MLA is actually deployed.","fun_headline_variants_meta":{"raw":{"variants":["Attention redesign cuts KV cache in half, doubles token speed","Half the KV cache, 2x throughput: new attention kernels","GTA and GLA: leaner attention, faster decoding","Memory-efficient attention: same quality, 2x faster","GLA shards cleanly, matches MLA, and runs 2x faster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000281,"raw_usage":{"total_tokens":1705,"prompt_tokens":1028,"completion_tokens":677,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":598}},"tokens_in":644,"tokens_out":677,"duration_ms":7278,"temperature":1.0,"reasoning_tokens":598,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:26:21.976027+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a several-hundred-billion-parameter model with GLA and compare it against a matched MLA model on standard downstream benchmarks; a meaningful quality gap would falsify the parity claim, as would serving benchmarks where the GLA kernel is not faster than FlashMLA under the reported query lengths.","supporting_citations":[],"review_version":1}