{"id":"d258edaf-856d-43b6-a0d0-b637af12113c","arxiv_id":"2505.17331","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Shared KV caching across the last 25% to 50% of layers, adapted incrementally from pretrained LLaMA weights, is claimed to improve training speed and inference throughput with little accuracy loss.","lead":"ECHO-LLaMA converts pretrained LLaMA models into an architecture where later layers share one key-value cache, cutting memory and computation. The paper reports large speedups, but the experiments do not control for batch size or training budget, so the headline numbers are not yet supported.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central efficiency claims rest on unverified commensurability: batch sizes differ between ECHO and baseline runs, and MFU uses an undisclosed FLOP update; matched-budget reruns are needed.","rationale":"The paper's central claim is an efficiency/loss advantage for ECHO-LLaMA over LLaMA baselines. What must be true for that claim to hold is that the measurements are apples-to-apples: same hardware, same batch size or batch-size effect controlled, same training budget definition, and same FLOP-counting convention. The paper's own tables and text violate or leave undefined each of these conditions. Table 2 shows ECHO runs with larger batch sizes in the configurations that produce the headline MFU numbers; MFU is known to rise with batch size because of better arithmetic intensity, so the observed 'up to 16%' could be a batch-size effect rather than an architecture effect. The 'equal training budget' caption is not supported by any stated budget; the '4B tokens through 1000 steps' statement conflicts with the reported batch sizes at seq_len=2048, making even the loss comparison's denominator ambiguous. The MFU methodology depends on a private modification of LLaMA-Factory's cal_mfu.py, and the essential change (theoretical FLOPs for shared-KV layers) is not disclosed. These are not mere reporting omissions: they are the load-bearing evidence for every headline number. I do not see an internal contradiction that makes the ECHO architecture impossible; the concern is strictly about verification of the quantitative claims. The reader's weakest-assumption analysis identifies the same point, and I agree with the REJECT verdict: the current evidence does not support the stated magnitude of improvement.","tokens_in":14754,"tokens_out":6949,"duration_ms":53686,"concrete_test":"Rerun the LLaMA-3B and LLaMA-7B comparisons on NPU-910B with identical batch size and total token budget for baseline and ECHO (e.g., Bs=20, T=4B tokens, seq_len=2048), and compute MFU from an explicit per-token FLOP formula for ECHO (self-attn layers ≈12d²; cross-attn layers ≈10d² plus one global KV projection per sequence). Publish the modified cal_mfu.py diff. If the MFU edge (e.g., 48.31 vs 32.21 for 3B at 4 NPUs) shrinks or reverses under matched settings, the central efficiency claim is unsupported.","verdict_should_be":"REJECT","load_bearing_attack":"The abstract's central quantitative claims (77% tokens/sec, 16% MFU, 14% lower loss) presuppose that ECHO and baseline are compared under the same effective training budget and the same FLOP accounting. The paper does not establish this. Table 2 uses systematically larger batch sizes for ECHO (LLaMA-3B: baseline Bs=20 vs ECHO Bs=30 on 4 and 8 NPUs; LLaMA-7B: baseline Bs=10 vs ECHO Bs=18 on 8 NPUs), and MFU is batch-size sensitive. The caption 'final training loss values are reported under an equal training budget' is not operationalized: no budget definition is given, and the §3 statement that each model is 'pretrained on 4B tokens through 1000 training steps' is arithmetically inconsistent with the listed batch sizes at seq_len=2048 (e.g., Bs=50 implies 102M tokens in 1000 steps). MFU is computed with a LLaMA-Factory script 'modified' for devices, with the theoretical-FLOP update unspecified; if the denominator still uses baseline LLaMA FLOPs, part of the ECHO advantage is an artifact of reduced FLOPs rather than higher utilization. Until ECHO and baseline are run with matched batch size, matched token budget, and a published FLOP formula, the claimed gains are unverifiable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ECHO-LLaMA, a modification of LLaMA in which a configurable subset of later layers (cross-decoders) share a single global KV cache computed from the output of an intermediate layer, while the remaining layers keep standard self-attention. Pretrained LLaMA checkpoints are converted into this architecture by an incremental, layer-by-layer adaptation procedure followed by a final fine-tuning phase. The authors report lower training loss, higher tokens-per-second throughput, higher MFU, and reduced memory usage, with headline claims of up to 77% higher training throughput, up to 16% higher MFU, up to 14% lower loss, and about 7% higher inference throughput.","tokens_in":15108,"tokens_out":4462,"duration_ms":35068,"significance":"If the efficiency and quality claims were properly supported, the work would be practically valuable: it offers a way to convert existing pretrained LLaMA checkpoints into cheaper architectures without full retraining, and it relaxes YOCO's rigid half-layer KV sharing. The incremental adaptation idea is coherent, and the paper includes useful ablations (incremental versus full-stage conversion) and generalization experiments to Qwen and LLaMA3.2 models. However, the central quantitative claims are not currently supported by the presented measurements. Batch sizes differ systematically between ECHO and baseline runs, the MFU calculation uses a modified script whose FLOP accounting is unspecified, and multiple table cells are marked 'Loss Scale Error.' These issues affect the headline efficiency numbers directly, so the significance of the contribution cannot be assessed until they are resolved.","major_comments":[{"comment":"The efficiency comparison is not commensurable. In Table 2, ECHO models use systematically larger batch sizes than baselines (e.g., LLaMA-3B: baseline Bs=20 vs ECHO Bs=30 on 4 and 8 NPUs; LLaMA-7B: baseline Bs=10 vs ECHO Bs=18 on 8 NPUs). MFU and tokens/sec are batch-size-sensitive, so the higher MFU and throughput could be due to the larger batch rather than the architecture. Table 3, despite its caption saying 'Batch size (BS) ... are reported,' does not list any batch sizes. The caption 'Final training loss values are reported under an equal training budget' is not operationalized: no definition of the budget is given, and the statement in §3 that each model is pretrained on 4B tokens through 1000 training steps is arithmetically inconsistent with the listed batch sizes at seq_len=2048 (e.g., Bs=50 over 1000 steps is approximately 102M tokens). The authors must run matched-batch-size comparisons, state the exact token budget and step count for every configuration, and justify that any batch-size differences are part of the method's intended benefit rather than an uncontrolled variable.","section":"§3, Tables 2 and 3"},{"comment":"The MFU computation is not reproducible. The paper states that the LLaMA-Factory MFU script was 'modified' and that 'updating the theoretical FLOPs based on your computing devices' was needed, but it does not specify the FLOP formula used for ECHO-LLaMA versus the baseline. If the denominator still counts full self-attention FLOPs for all layers, then ECHO-LLaMA's reduced FLOP count would automatically raise its MFU without any improvement in hardware utilization. The authors should provide the exact FLOP equations for both the baseline and ECHO architectures, including how the shared KV and cross-attention layers are counted.","section":"§3, TinyLLaMA MFU paragraph"},{"comment":"The cross-attention definition is ambiguous or incorrect. Eq. (6) writes Cross-Attn(X) = softmax(X_N W_Q (K_shared)^T / sqrt(d_k)) V_shared, using X_N for the query regardless of the layer index. If this is literal, every cross-decoder layer would attend from the same input X_N, which does not match the description in Figure 1 that each cross-decoder layer computes cross-attention with its own query from the shared KV. If X_N is a typo for X_l, the equation should be corrected. This is central to the architecture definition, so the intended formula must be stated precisely.","section":"§2, Eq. (6)"},{"comment":"The paper conflates two training setups. The scaling-law subsection and Figure 2 caption say 'Each model is pretrained from scratch on 4B tokens through 1000 steps,' but the main method (Section 2 and Table 1) adapts an existing pretrained TinyLLaMA checkpoint, and Appendix B says ECHO was 'pretraining from scratch.' These are different training budgets and different claims. The paper should clearly separate the from-scratch experiments from the adaptation experiments and give the token count, step count, and batch size for each.","section":"§3, Scaling Law Validation and Figure 2"},{"comment":"Several entries are marked 'Loss Scale Error' (ECHO-LLaMA-7B on 4 NPUs in Table 2; TinyLLaMA baseline and ECHO on 8 GPUs in Table 3) with no explanation. These missing values mean the claimed up-to-14% loss reduction is not fully supported by the tables. The authors should either provide the missing loss values (or explain the loss-scale issue and how it was resolved) or explicitly list which configurations are excluded from the loss comparisons.","section":"Tables 2 and 3"}],"minor_comments":[{"comment":"There is an inconsistent use of 'TinyLLaMA' and 'TinyLlama' (including in the reference to the Hugging Face model); please standardize.","section":"Throughout"},{"comment":"The text has several typos and grammatical issues, e.g., 'Let's Xl−1' and 'share KV'; these should be fixed.","section":"§2, text near Eq. (6)"},{"comment":"The last column reports throughput improvement but gives only percentage values, not absolute tokens/sec; reporting absolute throughput for each row would be more informative.","section":"Table 1"},{"comment":"The ablation for selecting S (Figure 4) uses the same loss curves that motivate the final S=150 choice; because the hyperparameter is selected on the basis of the training loss shown, the 'justification' is partly circular. Please state whether this is training loss or held-out loss and acknowledge this selection dependency.","section":"§3, Ablation Study"},{"comment":"Appendix E duplicates the text of Section 3's 'Efficiency Comparison' paragraph almost verbatim; please remove the duplication.","section":"Appendix E"},{"comment":"The conclusion says 'training speeds up to 50% faster,' while the abstract claims up to 77% higher token-per-second throughput; these headline numbers should be reconciled.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The experimental inconsistencies here are substantial enough that I would not accept the current version. The central idea is reasonably motivated, and the incremental adaptation procedure is a plausible contribution, but the efficiency claims rely on unmatched batch sizes and an undisclosed MFU formula. If the authors can provide matched-batch runs, a precise FLOP accounting, and corrected tables, the paper might be salvageable. If they cannot, rejection is appropriate. I would also ask the editor to consider whether the paper's reproducibility expectations are met: no code is provided, and the 'modified' MFU script is not described."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The actual contribution here is the adaptation recipe, not the architecture. YOCO already did shared-KV cross-decoders; ECHO-LLaMA adds a tunable sharing fraction (25% or 50%) and, more importantly, an incremental layer-by-layer procedure that converts a pretrained LLaMA checkpoint instead of training from scratch. That is a legitimate and potentially useful extension, and the algorithm is clearly described. The ablation on training steps per stage is sensible, and the memory-ratio formula is correct. I also credit the authors for trying the recipe on Qwen and LLaMA3.2 models, which suggests the mechanism transfers beyond TinyLLaMA.\n\nThe soft spot is exactly where the reader put it: the efficiency claims are not measured on a commensurable basis. Tables 2 and 3 systematically use larger batch sizes for ECHO models than for baselines, so the throughput, MFU, and even loss comparisons are confounded. The MFU calculation uses a modified LLaMA-Factory script whose FLOP update is unspecified; if the denominator still counts baseline attention FLOPs, part of the ECHO advantage is an accounting artifact. There are also \"Loss Scale Error\" entries, and the scaling-law section says each model was trained on 4B tokens through 1000 steps, which is arithmetically inconsistent with the listed batch sizes at sequence length 2048. The abstract claims up to 77% throughput gain, while the conclusion says up to 50% faster; those should agree.\n\nI do not think the architecture idea is impossible, and the incremental adaptation method is coherent on its own terms. But the central quantitative claims need matched-batch-size reruns with a published FLOP formula before they can be taken seriously. The paper also should compare against a YOCO model trained from scratch at similar scale, or at least explain why that is infeasible beyond citing missing checkpoints.\n\nMy take: this deserves a serious referee, but the referee should demand the controlled experiments. The adaptation recipe is worth keeping; the headline numbers are not. If the authors rerun with matched budgets and release the MFU formula and code, this could become a useful paper for people who want to squeeze efficiency out of existing LLaMA checkpoints.","headline":"The incremental adaptation recipe for converting pretrained LLaMA models into flexible shared-KV architectures is a real extension of YOCO, but the headline efficiency numbers are not backed by the experiments as reported.","tokens_in":15632,"tokens_out":1284,"would_cite":false,"duration_ms":12087,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ECHO-LLaMA claims one shared KV cache lifts LLaMA training throughput by up to 77% while keeping or improving loss.","keywords":["shared KV cache","LLaMA efficiency","cross-attention","incremental adaptation","training throughput","model FLOPs utilization","pretrained model conversion","KV cache memory"],"falsifier":"Rerun the 4-billion-token comparison with identical global batch size, sequence length, device count, and total wall-clock budget for both a LLaMA baseline and its ECHO-LLaMA conversion, and record final validation loss and tokens per second; if ECHO does not show both lower loss at equal tokens and higher throughput, the headline claims are measurement artifacts rather than architectural gains.","tokens_in":14585,"feed_emoji":"⚡","tokens_out":8359,"duration_ms":57167,"temperature":0.7,"pith_summary":"ECHO-LLaMA sets out to show that a pretrained LLaMA model does not need to compute fresh key and value vectors at every layer. It converts a chosen set of upper layers into cross-attention layers that all reuse a single pair of key-value matrices computed once from the middle of the network. The conversion is done gradually, layer by layer on a small token budget, followed by a short fine-tune, so an existing checkpoint is adapted rather than retrained. If the claims hold, the architecture trains substantially faster, with up to 77% higher tokens-per-second, up to 16% higher model FLOPs utilization, and up to 14% lower loss at equal tokens, while matching or slightly beating the baseline on language benchmarks. This matters because most efficiency work targets inference, whereas ECHO-LLaMA targets the much larger cost of training and fine-tuning.","feed_headline":"Shared KV cache lifts LLaMA training throughput up to 77%","feed_subtitle":"One shared key-value memory across upper layers cuts training cost while keeping or improving accuracy.","key_machinery":"The load-bearing object is the shared global KV cache: a single pair of key and value matrices computed once at layer $N$ by global projections followed by RMSNorm, then reused as the attention memory for all converted cross-decoder layers. This replaces per-layer key/value projection and caching in the upper layers with one shared memory, so the KV cache memory ratio versus a standard LLaMA becomes $(1-p) + 1/L$, which approaches $1-p$ for deep models, where $p$ is the fraction of shared layers. The other mechanism is the layer-wise incremental adaptation schedule, which converts layers from the top down with small per-stage token budgets to protect pretrained knowledge; the paper argues that freezing all but the converted layer at each stage mitigates catastrophic forgetting.","core_discovery":"The central claim is that a LLaMA-style transformer can be reorganized so that only the first $N$ layers keep full self-attention, and every layer above $N$ computes cross-attention against one global KV pair derived from the output of layer $N$. The shared pair is obtained as $K_{\\text{shared}} = \\mathrm{RMSNorm}(W^K_{\\text{global}} X_N)$ and $V_{\\text{shared}} = \\mathrm{RMSNorm}(W^V_{\\text{global}} X_N)$, so the upper layers drop their own key and value projections and reuse cached activations. The paper further claims that a pretrained checkpoint can be moved into this shape by an incremental adaptation schedule: convert the last layer first, train about 150 steps on roughly 4 million tokens with other layers frozen, then repeat downward through the chosen range, and finish with one epoch on about 4 billion tokens. The experiments report up to 77% higher training throughput, up to 16% higher model FLOPs utilization, and up to 14% lower loss at equal tokens relative to LLaMA baselines, with about 7% faster test-time generation on the 1.1B model at 50% sharing and roughly unchanged benchmark accuracy. The flexibility of choosing $N$, for example sharing 25% or 50% of layers, is presented as an advantage over the fixed half-layer sharing of prior YOCO-style designs.","pith_inferences":["If the loss improvements are real, the shared KV cache may act as a mild regularizer or parameter-sharing inductive bias; one could test this by checking whether the advantage persists when both models train on far more than 4 billion tokens or on out-of-distribution held-out text.","The paper's own discussion implies the speedup shrinks as models grow because MLP compute dominates, so the practical sweet spot may be mid-size models, and a systematic sweep of speedup versus model size could map where conversion stops paying off.","A direct comparison against YOCO under matched training budgets would isolate whether the gains come from the flexible sharing fraction or from the incremental adaptation recipe, since the paper could not run one for lack of public YOCO checkpoints.","The same incremental conversion recipe could in principle be applied to other transformer families that use RMSNorm and rotary embeddings, which the paper itself tries on Qwen models, so the mechanism is a general efficiency retrofit rather than a LLaMA-specific trick."],"forward_implications":["Existing LLaMA checkpoints can be converted to the shared-KV architecture with roughly 4 billion tokens of adaptation, rather than full pretraining, making efficiency upgrades available to already-trained models.","At 25% sharing the adapted TinyLLaMA matches or slightly exceeds baseline accuracy, while at 50% sharing it stays within about half a point on zero-shot benchmarks and generates about 7% faster, so the same checkpoint can be tuned for speed or accuracy.","KV memory falls by almost half at 50% sharing, with a measured 2.19x reduction at 16k sequence length, which directly extends the feasible context length on fixed hardware.","The claimed gains reach 77% training throughput and 16% model FLOPs utilization on the tested NPU hardware, while on the tested GPUs the gains are smaller, meaning the benefit is hardware-dependent.","Pretraining from scratch with the ECHO structure also yields equal or lower training loss across 125M to 7B parameter scales, suggesting the shared-KV design does not sacrifice optimization quality."],"supporting_citations":[{"why":"Supplies the YOCO shared-KV mechanism that ECHO-LLaMA adapts and the fixed-half-sharing design it argues against.","marker":"(Sun et al., 2024)"},{"why":"EchoAtt, the weight-sharing attention prior that still needs per-layer V caches, motivates sharing the KV pair itself.","marker":"(Rajabzadeh et al., 2024)"},{"why":"The pretrained 1.1B checkpoint whose weights are incrementally converted into ECHO-TinyLLaMA and evaluated.","marker":"TinyLlama/TinyLlama_v1.1"},{"why":"Underpins the claim that stagewise, layer-frozen updates mitigate catastrophic forgetting during conversion.","marker":"(Kirkpatrick et al., 2017)"},{"why":"Provides the MFU calculation script whose modified theoretical FLOP count produces the reported utilization numbers.","marker":"(Zheng et al., 2024)"},{"why":"The evaluation harness used to produce zero-shot and few-shot benchmark accuracies.","marker":"(Gao et al., 2024)"},{"why":"Quantifies the KV-cache memory bottleneck that motivates reducing per-layer caching.","marker":"(Pope et al., 2023)"},{"why":"Defines the LLaMA 2 architecture family used for one of the generalization experiments.","marker":"(Touvron et al., 2023)"}],"fun_headline_variants":["ECHO-LLaMA: shared KV cache speeds training by up to 77%","Shared KV cache lifts LLaMA throughput up to 77%","LLaMA training 77% faster with ECHO-LLaMA's shared KV cache","One global KV cache makes LLaMA training up to 77% faster"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The efficiency figures compare ECHO and baseline on an equal footing, meaning the same effective training budget and comparable hardware settings for both, so the reported throughput, MFU, and loss differences reflect the architecture and not the measurement setup.","fun_headline_variants_meta":{"raw":{"variants":["ECHO-LLaMA: shared KV cache speeds training by up to 77%","Shared KV cache lifts LLaMA throughput up to 77%","LLaMA training 77% faster with ECHO-LLaMA's shared KV cache","One global KV cache makes LLaMA training up to 77% faster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000176,"raw_usage":{"total_tokens":1321,"prompt_tokens":1012,"completion_tokens":309,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":221}},"tokens_in":628,"tokens_out":309,"duration_ms":2731,"temperature":1.0,"reasoning_tokens":221,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:48:08.776871+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the 4-billion-token comparison with identical global batch size, sequence length, device count, and total wall-clock budget for both a LLaMA baseline and its ECHO-LLaMA conversion, and record final validation loss and tokens per second; if ECHO does not show both lower loss at equal tokens and higher throughput, the headline claims are measurement artifacts rather than architectural gains.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"EchoAtt, the weight-sharing attention prior that still needs per-layer V caches, motivates sharing the KV pair itself."}],"review_version":1}