{"id":"09d3552e-0448-4e4c-8c04-0c2583f68551","arxiv_id":"2506.09316","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"On-the-fly distillation of Transformer layers to dual-state linear attention produces about 2.3x faster simulated LLM serving than Llama2-7B with roughly comparable benchmark accuracy.","lead":"The paper introduces an adaptive serving system that replaces selected Transformer attention layers with a cheaper dual-state linear attention, reporting about 2.3x lower latency than Llama2-7B in a simulated trace while keeping most benchmark accuracy. It is worth reading because it addresses the practical tension between long-context accuracy and serving cost, a bottleneck for deployed LLMs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's Llama2-7B baselines are internally implausible (TriviaQA 6.19 vs 86.11 for the same model family in Table 6; GLA-7B scores higher than Llama2), so the 'comparable accuracy' claim is not currently supported.","rationale":"The central claim has two load-bearing components: the 2.29x speedup and the retention of comparable accuracy. The reader's weakest assumption focuses on the generalizability of the dual-state design; I do not dispute that this is a limitation, but the more immediately load-bearing problem is that the accuracy evidence in Table 1 is internally inconsistent and therefore cannot support the 'comparable performance' claim. The Llama2-7B TriviaQA score of 6.19 versus 86.11 for Llama2-7B-chat in Table 6, and the fact that GLA-7B scores far above Llama2-7B on the same metric, suggest transcription or evaluation errors. If the baselines are corrected upward, DSLA's apparent superiority on HotpotQA would likely flip to a large degradation. Similarly, the speedup numbers cannot all be true: if Zamba-7B is 3.0x slower than Llama2-7B and DSLA-Serve is 2.29x faster than Llama2-7B, then DSLA-Serve should be roughly 6.9x faster than Zamba, not 3.0x. These are internal inconsistencies, not mere disagreements with external consensus. The concrete test of reproducing Table 1 with the official harness would settle the accuracy question; a parallel re-measurement of latency ratios would settle the speed question. Because these issues require correction rather than fundamental refutation, the reader's CONDITIONAL verdict remains appropriate. I keep the verdict unchanged but would add the reproduction of Table 1 as an explicit condition.","tokens_in":20159,"tokens_out":13400,"duration_ms":131031,"concrete_test":"Run the official LongBench evaluation harness (the one cited in the paper) on the released Llama2-7B and DSLA 25%/50% checkpoints for TriviaQA, HotpotQA, and 2WikiMQA, using the paper's reported settings. If Llama2-7B's TriviaQA score is ~60 or higher and HotpotQA is much above 5.63, the published Table 1 is inaccurate and the 'comparable accuracy' claim is unsupported; if the reported numbers reproduce exactly, the concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that DSLA-Serve 'retains comparable performance' rests on Tables 1–3, but Table 1 contains internal inconsistencies that make the accuracy evidence unreliable. Llama2-7B is reported at TriviaQA 6.19 and HotpotQA 5.63; the same paper reports Llama2-7B-chat at TriviaQA 86.11 and HotpotQA 29.79 (Table 6). An ~80-point gap between a base model and its instruction-tuned variant on the same task is implausible, and GLA-7B (a 20B-token pretrained model) scoring 57.68 on TriviaQA while Llama2-7B scores 6.19 is a further red flag. If the true Llama2-7B scores are higher—as public LongBench numbers suggest—the DSLA values may represent large drops rather than comparable accuracy. The paper provides no evaluation details (prompt template, metric, decoding parameters) for these rows, and the reader's request for error bars is unaddressed. A related internal inconsistency affects the speedup claim: §5.5 states Zamba-7B is 3.0x slower than Llama2-7B, while §1 claims DSLA-Serve is 3.0x faster than Zamba—numerically incompatible with the reported 2.29x speedup over Llama2-7B. Both issues point to the same need: a reproducible, internally consistent measurement of the central claim's two components.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes dual-state linear attention (DSLA), an extension of gated linear attention with two hidden states intended to preserve historical and recency contexts, and DSLA-Serve, an inference-time framework that progressively converts Transformer self-attention layers to DSLA layers based on sensitivity ordering and chained fine-tuning. The main empirical claims are that DSLA-Serve achieves a 2.29x end-to-end speedup over Llama2-7B and a 3.0x speedup over Zamba-7B while retaining comparable accuracy on long-context QA, commonsense reasoning, and summarization. The paper also reports ablations on the number of states, sensitivity metrics, and applications to different teacher scales (7B, 8B, 1.5B) and includes a limitation discussion of the memory overhead of loading both architectures.","tokens_in":20574,"tokens_out":7453,"duration_ms":72475,"significance":"If the results hold, the central idea is valuable: it offers a dynamic accuracy-efficiency trade-off for LLM serving without retraining the whole model, and the dual-state design with contrastive regularization is simple and interpretable. The paper deserves credit for providing a code link, multiple ablations (number of states, sensitivity metrics, different teachers/scales), and a practical latency-fluctuation analysis (Section 5.3, Appendix E). The chained fine-tuning procedure is a sound approach to train-test consistency. However, the significance is conditional: the internal inconsistencies in the baseline accuracy tables and in the speedup arithmetic currently undermine the two pillars of the paper's central claim.","major_comments":[{"comment":"Table 1 reports Llama2-7B at HotpotQA 5.63, 2WikiMQA 10.24, and TriviaQA 6.19, whereas Table 6 reports Llama2-7B-chat at 29.79, 27.15, and 86.11 on the same tasks. The main-text DSLA[25%] model, distilled from the base model, scores 11.07, 14.20, and 87.46, exceeding its teacher by up to 81.27 accuracy points. No explanation is given for the 80-point gap between base and chat baselines or for a 1.6B-token distilled model surpassing its teacher on multiple QA benchmarks. Because the 'comparable performance' claim rests on these tables, the authors must provide exact evaluation details (prompt template, decoding hyperparameters, metric) and corrected baseline numbers; as reported, Table 1 is internally inconsistent.","section":"Table 1, Table 6, Section 5.1"},{"comment":"The paper claims in the abstract and introduction that DSLA-Serve is 3.0x faster than Zamba-7B and 2.29x faster than Llama2-7B, while Section 5.5 states that Zamba-7B is 3.0x slower than Llama2-7B. With L as Llama2 latency, Z=3L, and D=Z/3=L, these two speedups are mutually exclusive; additionally, the Zamba measurement is on A100 while the 2.29x end-to-end result in Table 4 is on A6000. The authors should report raw latencies for Llama2-7B, Zamba-7B, and DSLA-Serve under identical hardware and decoding settings and reconcile the numbers.","section":"Section 5.5, Introduction/Abstract, Table 4"},{"comment":"The end-to-end trace experiment reports only latency before/after (93.64 ms to 40.83 ms) and the conversion rate schedule; it does not report any accuracy metric on the replayed requests or a link to offline benchmark accuracy at the applied conversion rates. The sentence 'while maintaining performance' is therefore unsupported by the data in that section.","section":"Section 5.4, Table 4"},{"comment":"The paper interprets the learned gamma values and gate similarity as evidence that the two states specialize to history and recency. Because gamma, the gate initializations, and the contrastive penalty are all fitted during the same distillation, Figures 5-6 describe the fit rather than test the specialization claim. An out-of-sample test (e.g., evaluating attention behavior on unseen tasks, or ablating the contrastive term and the initialization while keeping the two-state architecture) would be needed to support the causal claim that the dual-state design is responsible for the observed separation.","section":"Section 4.2, Figures 5-6"}],"minor_comments":[{"comment":"The expression Entropy(A) = sum A_{T,i} log A_{T,i} is written without a minus sign; as written it is negative entropy, so the statement 'lower entropy' should be clarified.","section":"Equation (11)"},{"comment":"The text says 'We report the standard deviation of our measurements for reference,' but only the DSLA row in Table 7 includes error bars; the other baselines do not, so the sentence is misleading.","section":"Table 7, Section 5.5"},{"comment":"Generation hyperparameters (temperature, top-p, max new tokens) for Tables 1, 2, and 6 are not reported; without them the benchmark numbers are not reproducible.","section":"Appendix B"},{"comment":"The phrase 'up to a 72.23% performance improvement compared to Mamba-7B on TriviaQA' appears to describe an absolute accuracy difference, not a relative percentage improvement; please state which one is meant.","section":"Section 5.1"},{"comment":"The conversion rates (12.5%, 25%, 37.5%, 50%) are presented without the rule that maps current load to a specific conversion rate; describe the threshold function used by DSLA-Serve.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The internal inconsistencies in Table 1 and in the speedup arithmetic directly affect the advertised claims and should be resolved with corrected, reproducible measurements before the paper can be considered for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is genuinely worth a serious look. You take a Transformer and, at serving time, progressively convert self-attention layers into a two-state gated linear attention module—one state initialized to retain long history, one to track recency—with a contrastive loss that pushes the gates apart, and you train the conversions in the exact order they'll be applied at inference. That combination is new as far as I know, and the chained fine-tuning is a sensible answer to the train-test mismatch problem for partial conversion. The 1B-scale ablation (Table 5) gives real evidence that two states help over one (81.19 vs 78.97), and the attention-score plots are at least consistent with the specialization story. They also properly credit their earlier layer-wise conversion work (Ro et al. 2024); the novel pieces here are the dual-state mechanism and the dynamic conversion control, and they do ship code.\n\nThe problems are in the evidence for the headline claims. Table 1 lists Llama2-7B at 6.19 on TriviaQA and 5.63 on HotpotQA, while Table 6 lists Llama2-7B-chat at 86.11 and 29.79 on the same tasks. A base model and its instruction-tuned sibling don't differ by 80 points on the same benchmark; public LongBench numbers put Llama2-7B well above 6.19. So the 'comparable accuracy' claim is not supported as written. The speedup claims have a similar issue: the abstract says DSLA-Serve is 3.0x faster than Zamba, and Section 5.5 says Zamba is 3.0x slower than Llama2-7B, which would imply a 6-7x speedup over Llama2-7B, not the 2.29x actually reported. Also, DSLA [25%] scoring 87.46 on TriviaQA against the teacher's 6.19 is impossible unless the teacher number is wrong. These are not minor quibbles; they mean the measurement pipeline needs to be redone or carefully documented.\n\nThe interpretive claim that the two states specialize to history and recency is read off the fitted model, so that's circular in principle, but the controlled ablation gives it some independent support. The bigger gap is that the controlled two-state-vs-one-state comparison is only at 1B scale; the 7B results don't include a single-state GLA distilled under the same budget. The authors also don't report error bars for Tables 1-3, which matters given the swings.\n\nWho should read this: anyone working on efficient inference, linear attention distillation, or serving systems. It deserves a serious referee, but the referee should ask for a corrected, reproducible measurement. I would not cite it until the inconsistencies are resolved and the numbers are verified.","headline":"A promising dual-state linear attention plus adaptive distillation framework, but the accuracy table contains an internal inconsistency that currently undermines the headline 'comparable performance' claim.","tokens_in":21091,"tokens_out":3278,"would_cite":false,"duration_ms":31377,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DSLA-Serve converts Transformer layers to dual-state linear attention on the fly, and on an augmented Azure trace this cuts per-token latency by 2.29x while keeping accuracy near the teacher.","keywords":["dual-state linear attention","linear attention","knowledge distillation","inference serving","long-context LLMs","KV cache","adaptive inference","recency bias"],"falsifier":"Run DSLA-Serve at 50–75% conversion on a long-context retrieval benchmark with 16K+ tokens where evidence sits in the first quarter of the context, and compare accuracy against Llama2-7B. A sharp drop at these lengths would contradict the claim that the history state actually preserves old context, since the paper's evaluations stop near 8K tokens.","tokens_in":19976,"feed_emoji":"⚡","tokens_out":6842,"duration_ms":60996,"temperature":0.7,"pith_summary":"DSLA-Serve claims that a 7B Transformer can be converted, layer by layer, into a linear-attention model during serving, without retraining the teacher, and that the result is both faster and nearly as accurate. The conversion uses dual-state linear attention (DSLA), a module with two gated hidden states: one initialized to favor distant history and one to favor recent tokens, plus a learned per-layer weight balancing them. On an augmented Azure trace, DSLA-Serve cuts per-token end-to-end latency by 2.29x versus Llama2-7B and 3.0x versus Zamba-7B, while retaining comparable scores on long-context QA, commonsense reasoning, and summarization. If correct, this gives serving systems a practical accuracy-memory knob that can be turned mid-request.","feed_headline":"Two-state attention layers cut LLM serving latency 2.29x","feed_subtitle":"Converting the least sensitive layers on the fly keeps accuracy near the teacher while removing KV-cache cost.","key_machinery":"The load-bearing object is the dual-state linear attention module and the serving policy around it. Each DSLA layer keeps two recurrent hidden states with data-dependent forget gates, one biased toward history and one toward recency, and a per-layer learnable coefficient $\\gamma$ that blends their outputs, so a single layer can cover both far and near context at $O(T)$ cost. The other half of the machinery is DSLA-Serve: an offline phase ranks Transformer layers by attention entropy, treating low entropy as low sensitivity to linearization, then distills layers one at a time in that order with chained fine-tuning, committing each converted layer before the next; at runtime, the system converts layers from the least-sensitive end whenever memory pressure rises, stopping when quality is at risk.","core_discovery":"The paper's central claim is that the reason distilled linear-attention models lose accuracy is a single compressed state that forgets older tokens, and that this can be fixed by giving each attention layer two specialized states. DSLA maintains $S^1_t = G^1_t \\odot S^1_{t-1} + k_t^\\top v_t$ and $S^2_t = G^2_t \\odot S^2_{t-1} + k_t^\\top v_t$, with the history gate $G^1$ initialized near the identity and the recency gate $G^2$ initialized randomly; the output blends them as $o_t = q_t(\\gamma S^1_t + (1-\\gamma)S^2_t)$, with $\\gamma$ learned per layer. The paper claims that with a contrastive penalty that pushes the two gates apart and KL distillation that aligns outputs with self-attention, the two states specialize—one holding long-range context and one tracking recent tokens—so that partial or full replacement of Transformer layers preserves quality while removing the KV cache. On an augmented Azure serving trace the adaptive conversion yields a 2.29x per-token latency reduction against Llama2-7B and 3.0x against Zamba-7B, with downstream accuracy comparable to the Transformer teacher at 25% and 50% conversion.","pith_inferences":["A natural generalization the paper leaves implicit is to apply the dual-state design to other single-state sequence models: Mamba, RetNet, or any gated recurrence could gain a history-explicit state initialized toward identity, with the same contrastive penalty.","The runtime conversion policy could be made adaptive at a finer grain than prompt length: instead of fixed conversion caps per length bucket, an online controller could estimate the accuracy-efficiency Pareto frontier from the live request mix and adjust conversion depth continuously.","Because the paper loads both the Transformer and DSLA weight copies simultaneously, the stated memory savings are partially offset by extra weight memory; offloading or prefetching the inactive layer weights would be the direct test of whether the net memory win survives in production.","The reported 2.29x speedup is measured on a trace where Transformer latency suffers intermittent cudaMalloc stalls; a system with preallocated memory pools might see a smaller end-to-end gain, so the architecture's true advantage is best isolated at long prefill lengths."],"forward_implications":["With 25% of layers converted, DSLA matches or beats Llama2-7B on several long-context QA benchmarks while cutting KV cache memory by 1–2 GB per request at 4K–8K context.","Converting 50% of layers still outperforms single-state linear models such as GLA-7B and Mamba-7B on long-context QA, while keeping commonsense-reasoning averages close to the teacher.","Because converted layers drop their KV cache, decoding latency stops growing with generated length and memory-allocation stalls shrink, which the paper cites as the main source of the measured speedup.","The chained fine-tuning procedure means all intermediate conversion depths—25%, 50%, and beyond—share consistent weights, so the runtime can switch conversion levels without retraining.","The attention-entropy ranking gives a deployable ordering rule: in serviced workloads, the method can decide on the fly how far to convert before quality SLOs are threatened."],"supporting_citations":[{"why":"Supplies the gated linear attention recurrence that DSLA extends from a single state to two states.","marker":"Yang et al., 2023"},{"why":"Provides Llama2-7B, the teacher model and the main speed baseline the distillation is measured against.","marker":"Touvron et al., 2023"},{"why":"Provides Zamba-7B, the hybrid baseline whose 3.0x slower end-to-end latency is the comparison point.","marker":"Glorioso et al., 2024"},{"why":"Supplies Mamba-7B, a single-state linear-complexity baseline DSLA is claimed to outperform.","marker":"Gu & Dao, 2023"},{"why":"Supplies the SlimPajama data used for the 1.6B-token distillation of the converted layers.","marker":"Shen et al., 2024"},{"why":"Provides the LongBench-style long-context QA setup used for multi-document QA and code understanding evaluation.","marker":"Bai et al., 2024"},{"why":"Supplies the Azure inference trace that is augmented and replayed for the end-to-end 2.29x latency measurement.","marker":"Patel et al., 2024"},{"why":"Provides the distillation-to-hybrid approach and the observation that full linearization degrades accuracy beyond a conversion threshold.","marker":"Wang et al., 2024a"},{"why":"Earlier layerwise conversion to linear attention that motivates the sensitivity-based ordering used by DSLA-Serve.","marker":"Ro et al., 2024"},{"why":"LoLCATs, the low-rank linearization method DSLA is compared against on lm-eval tasks.","marker":"Zhang et al., 2024a"}],"fun_headline_variants":["Dual-state linear attention slashes serving latency 2.29x","Forgetful linear attention? Two hidden states fix it","On-the-fly layer conversion accelerates LLM inference","LLM serving speedup 2.29x via adaptive distillation","Two-memory attention preserves long-range LLM accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on the empirical assumption that two hidden states—a history gate initialized near the identity and a recency gate initialized randomly—after contrastive regularization and 1.6B-token distillation can cover the attention patterns of all 32 layers well enough that converting up to half of them does not hurt downstream quality.","fun_headline_variants_meta":{"raw":{"variants":["Dual-state linear attention slashes serving latency 2.29x","Forgetful linear attention? Two hidden states fix it","On-the-fly layer conversion accelerates LLM inference","LLM serving speedup 2.29x via adaptive distillation","Two-memory attention preserves long-range LLM accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000367,"raw_usage":{"total_tokens":2043,"prompt_tokens":1090,"completion_tokens":953,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":706,"completion_tokens_details":{"reasoning_tokens":869}},"tokens_in":706,"tokens_out":953,"duration_ms":8377,"temperature":1.0,"reasoning_tokens":869,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:51:48.519646+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DSLA-Serve at 50–75% conversion on a long-context retrieval benchmark with 16K+ tokens where evidence sits in the first quarter of the context, and compare accuracy against Llama2-7B. A sharp drop at these lengths would contradict the claim that the history state actually preserves old context, since the paper's evaluations stop near 8K tokens.","supporting_citations":[{"cited_title":"Splitwise: Efficient generative llm inference using phase splitting","cited_arxiv_id":null,"evidence_quote":"Supplies the Azure inference trace that is augmented and replayed for the end-to-end 2.29x latency measurement."}],"review_version":1}