{"id":"592b7c4d-357b-41d2-8813-2bcdde3aa91e","arxiv_id":"2502.07864","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A framework that converts pretrained GQA models into MLA models using RoRoPE, FreqFold, and balanced KV low-rank compression, regaining baseline performance with only a few billion fine-tuning tokens.","lead":"TransMLA converts existing GQA-based language models into DeepSeek-style MLA models, cutting the KV cache by up to 93% and showing up to a 10.6x inference speedup on one consumer GPU. The paper also argues and proves a representational ordering under equal KV cache size: GQA below MLA below MQA.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Low-loss conversion is not established beyond short, in-domain calibration; RoRoPE's PCA is key-side only, and FreqFold's equal-frequency approximation is untested at long context.","rationale":"The core idea is sound where it is exact: Appendix B correctly proves that applying the same orthogonal U_l to query and key components within each RoPE subspace preserves the RoPE inner product, and the merge-then-absorb construction in Section 4.1 is a faithful rewriting. The reported speedups are credible as engineering measurements, and the public GitHub release is a concrete artifact. My concern is not with the proof of invariance but with the truncation step that follows it. RoRoPE selects U_l from key activations only; after truncation, the removed query components are simply discarded, and the paper provides no bound or measurement of their contribution to attention scores. This is an internal gap between the PCA objective and the quantity being approximated, not a disagreement with external consensus. FreqFold adds a second uncontrolled approximation whose error is position-dependent, yet no long-context evaluation appears anywhere in the paper. The reader's weakest assumption (calibration representativeness) captures part of this; I sharpen it to the missing query-side diagnostic and the missing long-context check. These are addressable with a small number of experiments, which is exactly why the conditional verdict should stand rather than be upgraded or downgraded.","tokens_in":26729,"tokens_out":8248,"duration_ms":78957,"concrete_test":"Run one diagnostic on LLaMA-3-8B with the paper's WikiText-2 calibration: after RoRoPE (before FreqFold), compute the fraction of squared query-norm per RoPE frequency lying outside the first head; then evaluate training-free perplexity on PG-19 at 512 and 16K context. If the residual query fraction exceeds a few percent, or the 16K perplexity degrades substantially more than the 512-token perplexity, the key-only PCA criterion does not control conversion loss and the 'any GQA model' claim is unsubstantiated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—'seamlessly converts any GQA-based pre-trained model' with near-lossless RoPE decoupling—rests on two unvalidated approximations. First, RoRoPE chooses the orthogonal rotations {U_l} by maximizing key variance only (Appendix B, Eq. 32, on WikiText-2 key activations). The same U_l is then applied to queries, and all non-principal query/key components are stripped of RoPE. Since the PCA objective never measures query energy, the discarded query components can be large even when key energy is concentrated; Fig. 3a reports only key norms. Truncation error in the attention inner product is controlled by both query and key residual energy, so the paper's key-norm evidence does not bound the actual loss. Second, FreqFold replaces nearby RoPE frequencies θ_l and θ_{l+1} by a single frequency. The induced phase error grows as (θ_l − θ_{l+1})·(t−j); for LLaMA-style d=128, adjacent θ differ by about 7%, so at 8K–32K context the approximation is not small. Validation is short-context perplexity on WikiText-2 (Fig. 3b) and six short benchmarks (Table 1); no long-context task is reported, and the LLaMA-2-7B row omits the exact FreqFold setting and calibration details. If query-side residual energy or long-range phase error is significant, the training-free conversion degrades and 6B-token fine-tuning may not restore quality on arbitrary GQA models.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents TransMLA, a framework for converting a pretrained GQA-based transformer into the MLA architecture used by DeepSeek, with the stated goal of making existing GQA models directly runnable on DeepSeek's optimized inference stack. The proposed pipeline has four components: an exact merging of all GQA key/value heads into a single latent head; a PCA-based rotation of query and key activations within each RoPE frequency subspace (RoRoPE) that concentrates positional information into the first attention head; a frequency-grouping approximation (FreqFold) that packs more principal components into that head; and a norm-balanced joint low-rank compression of the remaining keys and values (BKV). The paper also claims a theoretical proof that MLA has strictly higher expressive power than GQA under the same KV-cache budget. Experiments on SmolLM-1.7B and LLaMA-2-7B report training-free compression up to 92.97% of the KV cache, recovery of benchmark performance after fine-tuning on 6B tokens, and inference speedups up to 10.6x at 8K context in vLLM. Appendices provide the algebraic proofs, the FreqFold variance argument, and fine-tuning and benchmark details.","tokens_in":27119,"tokens_out":10443,"duration_ms":93452,"significance":"If the conversion were genuinely low-loss, TransMLA would be practically valuable: it would let existing GQA models inherit DeepSeek's MLA inference optimizations, with large KV-cache savings and significant decoding speedups. The RoPE orthogonality invariance in Appendix B is a clean and correctly proven piece of algebra, and the BKV norm-balancing heuristic is well motivated by the observed key/value norm disparity. The reported speedups are large and the fine-tuning token budgets are far below pretraining costs. However, the theoretical expressiveness claim is not established for the actual full MLA with low-rank query, and the low-loss property is validated only on short-context perplexity and six short benchmarks, with no long-context quality evaluation. The 'any GQA model' claim is also broader than the two model families tested. The practical contribution is promising but requires additional validation and a more careful statement of the theoretical and empirical scope.","major_comments":[{"comment":"The strict-expressiveness argument is not valid as written. In the MQA form derived in Eq. (26), the interaction matrix is (W_Q_i)^T W_UK_i W_DKV, whose rank is at most d because W_UK_i is d x 2gd; the claim that MQA allows rank up to 2gd compares against a generic MQA with a larger query projection, which changes the number of query parameters and is not a controlled comparison. Furthermore, A.3 only establishes expressiveness for MLA_Factorized without low-rank query, while the paper's abstract and Section 4 claim that 'MLA consistently offers higher expressive power than GQA' for the full MLA, which in DeepSeek uses a low-rank query (r_q < d). No proof is given for that full variant. The claim should either be proved for the actual low-rank-query MLA or explicitly narrowed.","section":"Appendix A, A.2.3 and A.3"},{"comment":"The PCA objective in Eq. (32) maximizes key-side variance only, but the error introduced by dropping RoPE from non-principal heads depends on both query and key residuals in the discarded components of the attention inner product. Figure 3a reports only key L2 norms, so it does not bound the actual attention-score perturbation. Please report the query-side residual energy across layers, or better, the mean relative error of the attention logits on the calibration set, at the RoPE-removal ratios used in Table 1. Without such evidence, the statement that RoRoPE 'concentrates effectively all positional information' into the first head is not supported.","section":"Section 4.2, Appendix B, Eq. (32), Fig. 3a"},{"comment":"The low-loss validation for RoPE removal and FreqFold consists only of WikiText-2 log-perplexity (Fig. 3b) at an unstated, presumably short, sequence length. FreqFold approximates nearby RoPE frequencies as equal, and the induced phase error grows linearly with the relative position (t-j); the throughput benchmark in Fig. 5 extends to 32K context, but no quality metric is reported at those lengths. Please add long-context quality evaluations (e.g., passkey retrieval, RULER, or LongBench) at 8K and 32K for the converted models, and state the sequence length used in Fig. 3b. This is essential to support the practical claim of seamless conversion for deployed long-context use.","section":"Section 5.2, Fig. 3b, Section 5.4"},{"comment":"The calibration protocol for the main LLaMA-2-7B conversions is under-specified. The paper does not report the FreqFold setting (M) used for LLaMA-2-7B, the number of WikiText-2 samples, the context length used for collecting activations, or how the PCA bases were computed per layer. Since Fig. 3b indicates that the optimal FreqFold setting is model-dependent (4D is best for LLaMA-3-8B), these details are necessary for reproducibility of the headline compression and fine-tuning results, and for qualifying the claim that the method applies to any GQA model.","section":"Table 1, Section 5.2, Appendix E"},{"comment":"The claim of seamless conversion of 'any GQA-based pretrained model' is not supported by the experiments, which cover only SmolLM-1.7B and LLaMA-2-7B, plus an analysis-only study on LLaMA-3-8B. The behavior of RoRoPE and FreqFold may depend on the number of groups, head dimension, and RoPE frequency schedule. Please convert and evaluate at least one additional model family (e.g., Qwen2-7B or Mistral-7B), or explicitly restrict the claim to the tested architectures.","section":"Section 5.1"}],"minor_comments":[{"comment":"The phrase 'the same rotation values ... are applied identically to each dimension' is ambiguous; it should say 'to each head's copy of the l-th RoPE pair' to clarify that the orthogonal transform acts across heads within a fixed frequency subspace.","section":"Section 4.2, text after Eq. (19)"},{"comment":"The row label 'SmolLM-1.7B 1T' is not defined in the caption; please clarify that 1T refers to the original pretraining token count, to avoid confusion with the fine-tuning token counts listed in the same table.","section":"Table 1"},{"comment":"The baseline LLaMA-2-7B is evaluated at context lengths up to 32K even though the model was trained with a 4K context; please state explicitly that Fig. 5 and Table 4 report throughput only and that output quality at extrapolated lengths is not assessed.","section":"Section 5.4 and Table 4"},{"comment":"The symbol W_DKV in Eq. (35) denotes the NoPE-only concatenation while Eq. (11) defines W_DKV as the full 2gd x D matrix; using a different symbol, such as W_DKV^{NoPE}, would avoid confusion.","section":"Appendix D, Eqs. (34)-(36)"},{"comment":"The abstract states '93% of the KV cache' while Table 1 reports 92.97%; please use a consistent number of significant digits.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The practical conversion pipeline is timely and the speedup results are striking, but the theoretical framing overstates the expressiveness result and the empirical support lacks long-context quality evaluation. I would like to see the authors either prove the expressiveness claim for the actual low-rank-query MLA or substantially qualify it, and add long-context benchmarks before this is accepted. The calibration details for LLaMA-2-7B should also be reported, since the headline numbers depend on them."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a well-engineered contribution to KV cache compression, wrapped in a theoretical claim that is partly overreaching. The practical core—RoRoPE, FreqFold, and BKV—is genuinely new and the experiments show that you can convert an MHA/GQA checkpoint into an MLA-compatible model, load it into DeepSeek's stack, and get real speedups with modest quality loss. The 10.6x speedup at 8K context on a 24GB GPU is concrete, and the code is public. That alone makes it worth a serious look.\n\nThe theoretical appendix proves, plausibly, that GQA < MLA_Factorized < MQA for a fixed KV budget. But the step from MLA_Factorized to the full MLA (with low-rank query and decoupled RoPE) is asserted, not proved. The low-rank query constraint could reduce expressiveness, and the paper does not bound that loss. This is a genuine gap in the central narrative.\n\nThe stress-test concern about RoRoPE being key-side only has merit: the PCA objective in Eq. 32 is computed on key activations, and the same rotation is then applied to queries. The paper shows that the key norms concentrate, but it never measures the query residual energy or the induced error in the attention logits. The end-to-end perplexity on WikiText-2 suggests the error is small in-distribution, but that is not evidence for long context. FreqFold's approximation of nearby RoPE frequencies is likewise untested beyond ~4K-token contexts. The paper reports no long-context benchmark, and the LLaMA-2-7B row omits the FreqFold group size and calibration details. That is a reproducibility issue, not just a theoretical one.\n\nThe claim that TransMLA 'seamlessly converts any GQA-based pre-trained model' is too strong: the experiments cover one GQA model (SmolLM 1.7B) and one MHA model (LLaMA-2-7B), not the advertised 'any'. The fine-tuning results are on short benchmarks only. I would also like to see error bars; the speedup numbers are presumably stable, but the benchmark averages are single runs.\n\nNone of these flaws are fatal. The method is useful and the limitations are addressable. The paper deserves peer review, with the expectation of major revisions: a complete expressiveness proof for full MLA, a long-context evaluation, and fuller experimental details. I would bring this to a reading group focused on efficient inference.","headline":"A genuinely useful GQA-to-MLA conversion pipeline with real speedups, but the expressiveness proof is incomplete and long-context behavior is unvalidated.","tokens_in":27620,"tokens_out":5537,"would_cite":true,"duration_ms":47996,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68T50"],"pacs":[],"model":"deepseek-v4-flash","headline":"GQA models can be rewritten as DeepSeek-style MLA with a ~93 percent smaller KV cache and a 10.6x speedup.","keywords":["multi-head latent attention","group-query attention","KV cache compression","RoPE","PCA rotation","model conversion","LLM inference","low-rank factorization"],"falsifier":"Take the converted LLaMA-3-8B model with 90% of RoPE removed and run a needle-in-a-haystack retrieval at 32K context plus a code-completion or math benchmark drawn from a distribution unlike WikiText-2; if log-perplexity rises sharply above the reported ~2 or retrieval accuracy collapses, the RoRoPE concentration assumption fails on out-of-calibration positions.","tokens_in":26571,"feed_emoji":"⚡","tokens_out":7855,"duration_ms":60122,"temperature":0.7,"pith_summary":"The paper claims that Group-Query Attention, the shared-key/value design used by LLaMA and Qwen models, is strictly less expressive than Multi-Head Latent Attention at the same KV-cache budget, and that any pretrained GQA model can be converted into an MLA model rather than retrained from scratch. The conversion works by rotating queries and keys so that almost all positional (RoPE) information lands in a single attention head, dropping RoPE from the other key heads, and then compressing the remaining keys and values jointly by a norm-balanced low-rank projection. On LLaMA-2-7B the paper reports a 92.97% KV-cache reduction, a measured 10.6x inference speedup at 8K context on one consumer GPU, and benchmark recovery to near-original level after just 6B fine-tuning tokens. If the conversion is as lossless as the perplexity curves suggest, existing GQA deployments could inherit the optimized inference stack built for DeepSeek without retraining the model from scratch.","feed_headline":"GQA models become MLA: 93% smaller KV cache, 10.6x speedup","feed_subtitle":"Group-Query Attention models can be rewritten as MLA, regaining quality after only 6B fine-tuning tokens.","key_machinery":"The load-bearing identity is the RoPE rotation invariance: if the same orthogonal matrix $U_l$ is applied to the real and imaginary components of the $l$-th RoPE subspace across all attention heads, the inner product $\\hat q^R_{t,i}^\\top \\hat k^R_j$ is unchanged. RoRoPE uses this identity to run joint PCA on calibration key activations and rotate query/key projections so that principal components from all heads concentrate in the first head's dimensions; FreqFold groups nearby RoPE frequencies as identical so that more than one first-head dimension can carry positional signal. Once RoPE is isolated in one head, the remaining keys and values are compressed by a norm-balanced joint PCA (BKV), and the RoPE-free key up-projection can be absorbed into the query projection exactly as in DeepSeek's MLA, which is what makes the converted model load directly into that inference stack.","core_discovery":"The central claim is that MLA with decoupled RoPE subsumes GQA: any GQA layer can be written as an MLA layer with the same KV cache size by introducing an up-projection matrix, while the reverse is not always possible. The paper proves this by showing GQA's key/value generation is a sparse special case of MLA's low-rank latent key/value mechanism, and that the full MLA, with a shared multi-query RoPE key, is strictly more expressive than GQA. The practical consequence is the TransMLA conversion: merge GQA's key heads, use RoRoPE plus FreqFold to concentrate positional information into the first key head, discard RoPE elsewhere, and compress the remaining keys and values with balanced joint PCA. The paper reports that this conversion, applied to LLaMA-2-7B, shrinks the KV cache to 7.03% of its original size, gives up to a 10.6x throughput gain in vLLM at 8K context, and after 6B fine-tuning tokens matches or nearly matches the original model on six commonsense benchmarks.","pith_inferences":["If the RoRoPE concentration generalizes beyond WikiText-2 calibration, the same recipe should transfer to Qwen, Mistral, and Gemma; a cheap test is to run the conversion on one of those models and compare long-context retrieval over positions beyond the calibration range.","The paper evaluates conversion quality with perplexity and commonsense benchmarks, but not long-context tasks; because RoPE is a positional mechanism, the most sensitive falsifier is long-range needle-in-haystack or code/math inputs whose position matters more.","The 10.6x speedup is a vLLM-specific implementation result that will vary with kernel maturity and hardware; the conversion's architectural benefit (smaller cache) is hardware-independent, so combining it with FP8 quantization or multi-token prediction should compound the gain, as the paper suggests.","The expressiveness proof applies to the attention core, not to downstream training dynamics; converted models still need fine-tuning to recover, so 'seamless' should be read as 'weight-compatible with light adaptation' rather than zero-cost."],"forward_implications":["Any GQA or MHA pretrained model (LLaMA, Qwen, Gemma, Mistral) can in principle be converted to MLA and run on DeepSeek-compatible optimized kernels, inheriting vLLM and SGlang support.","KV-cache memory per token drops with the compression ratio; at 92.97% compression on LLaMA-2-7B the reported throughput jumps from 85.8 to 832.7 output tokens/s at 8K context on a 165 TFLOPS/24GB GPU.","Speedups grow with context length because the saving is proportional to cache size; at 16K context the original model runs out of memory on 24GB while the converted model sustains 414 tokens/s.","Fine-tuning with a few hundred million to 6B tokens restores benchmark averages close to the original, making architecture migration cheaper than retraining an MLA model from scratch.","Training-free conversion is low-loss at moderate compression (-68.75% drops 1.65 points on LLaMA-2) but degrades more at extreme compression (-92.97% drops about 8 points before fine-tuning)."],"supporting_citations":[{"why":"Defines Group-Query Attention, the architecture whose key/value sharing TransMLA converts.","marker":"Ainslie et al. [2023]"},{"why":"Introduces MLA with decoupled RoPE and the Absorb operation, the target format and source of DeepSeek compatibility.","marker":"DeepSeek-AI [2024]"},{"why":"Defines RoPE and its paired real/imaginary rotation structure, which the rotation-invariance proof exploits.","marker":"Su et al. [2024]"},{"why":"MHA2MLA is the concurrent baseline whose conversion loss and fine-tuning results TransMLA compares against.","marker":"Ji et al. [2025]"},{"why":"Palu shows low-rank KV compression without RoPE handling cannot use the Absorb operation, motivating RoRoPE.","marker":"Chang et al. [2024]"},{"why":"WikiText-2 supplies the small calibration set used for PCA and the perplexity evaluation.","marker":"Merity et al. [2016]"},{"why":"The SmolLM corpus provides the fine-tuning data and mixing ratios used to recover benchmark performance.","marker":"Ben Allal et al. [2024]"}],"fun_headline_variants":["Any GQA model becomes MLA: 93% KV cache cut, 10.6x faster","TransMLA: swap GQA for MLA, slash KV cache by 93%, 10.6x speedup","MLA beats GQA: 93% less cache, 10.6x inference speedup","GQA to MLA conversion: 93% KV reduction, 10.6x throughput","Proven: MLA subsumes GQA, 93% KV cut, 10.6x speed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conversion is only near-lossless if a PCA rotation learned on a small WikiText-2 calibration set concentrates essentially all RoPE position information into the first attention head for every input the model will see; any positional signal left in the discarded heads is destroyed.","fun_headline_variants_meta":{"raw":{"variants":["Any GQA model becomes MLA: 93% KV cache cut, 10.6x faster","TransMLA: swap GQA for MLA, slash KV cache by 93%, 10.6x speedup","MLA beats GQA: 93% less cache, 10.6x inference speedup","GQA to MLA conversion: 93% KV reduction, 10.6x throughput","Proven: MLA subsumes GQA, 93% KV cut, 10.6x speed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000646,"raw_usage":{"total_tokens":2960,"prompt_tokens":926,"completion_tokens":2034,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":1920}},"tokens_in":542,"tokens_out":2034,"duration_ms":13680,"temperature":1.0,"reasoning_tokens":1920,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T11:42:03.540188+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the converted LLaMA-3-8B model with 90% of RoPE removed and run a needle-in-a-haystack retrieval at 32K context plus a code-completion or math benchmark drawn from a distribution unlike WikiText-2; if log-perplexity rises sharply above the reported ~2 or retrieval accuracy collapses, the RoRoPE concentration assumption fails on out-of-calibration positions.","supporting_citations":[],"review_version":1}