{"id":"5eb3f99f-0a09-4cf4-a80e-8270c5c0ae20","arxiv_id":"2505.22135","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"RAD selects redundant attention layers via self-speculative decoding throughput, replaces them with SSM blocks, and distills them into hybrid models that beat the original on GSM8K and CRUX.","lead":"Researchers use an inference-speed trick, self-speculative decoding, to find which attention layers of a language model are least valuable, then replace those with faster state-space layers and retrain the model. The resulting hybrid models match or beat the original on math and code reasoning, and can beat distillation from a much larger teacher using a smaller one.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The standard-distillation and self-distillation comparisons confound layer selection with teacher size, initialization, and continued training; without a same-recipe equal-interval and no-replacement control, the central attribution claim is not established.","rationale":"The paper has real strengths: the opt-vs-worse ablation in Tables 6 and 9 shows a clear correlation between the throughput proxy and downstream performance, and Appendix A.2 carefully documents the effect of zero-initialization. However, the central claim that redundancy-aware layer selection is what makes RAD succeed is not cleanly isolated. In the standard-distillation section, the only baselines are the public [43] models, which differ from RAD in three ways at once: teacher size (70B vs 8B), layer schedule (equal vs optimized), and initialization (qkv/out_proj vs zero in_proj). The eql/8B condition appears only in the passkey-retrieval figure, so the headline GSM8K/CRUX comparison in Table 5 cannot attribute the 25-point gap to layer selection. Likewise, the self-distillation Born-Again results compare against the untrained teacher, not against the teacher fine-tuned on the same 20B-token mixture, so continued training alone is a plausible alternative explanation. The opt-vs-worse ablation is suggestive but uses extremes from the same PG-19 objective; it does not include an equal-interval or random selection at the same replacement count, and BO was run on a fixed 10-instance set, so overfitting to those instances is possible. A single controlled experiment (eql/8B with the full RAD recipe) would resolve whether the contribution is the layer selection or simply the recipe. Since this control is absent, the headline claims are conditional on that experiment. I therefore keep the reader's CONDITIONAL verdict.","tokens_in":24577,"tokens_out":11445,"duration_ms":123674,"concrete_test":"Train Mamba2 (eql / 8B) using the exact RAD recipe: zero-initialize in_proj, replace 50% of attention layers at equal intervals, distill from Llama-3.1-8B-Instruct on the same 20B-token mixture for one epoch with LR 2e-5 cosine, and evaluate GSM8K/CRUX under identical ZeroEval settings. If eql/8B reaches roughly 71 GSM8K, layer selection contributes little and the headline claim fails; if it lands near the 46.17-level, the redundancy-aware selection is the driver. This single control isolates the claimed novelty from initialization and teacher-size confounds.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that optimizing self-speculative-decoding throughput (Eq. 1) selects attention layers whose replacement with SSM blocks, followed by redundancy-aware distillation, produces hybrids that beat the teacher (Table 2) and beat a 70B-teacher equal-interval baseline (Tables 3-5). For this claim to hold, the reported gains must be attributable to the layer selection. The experiments do not show this. (i) Standard distillation: RAD (opt intervals, 8B teacher, zero in_proj init) is compared against public [43] baselines (equal intervals, 70B teacher, qkv/out_proj init). Teacher size, layer schedule, and initialization vary simultaneously, so the GSM8K gap (71.27 vs 46.17) cannot be assigned to layer selection; the controlled 'eql / 8B' condition appears only in passkey retrieval (Fig. 8), not in the reasoning tables. (ii) Self-distillation: RAD students are compared to the original teacher, not to the teacher fine-tuned on the same 20B-token mixture, so 'Born-Again' gains could be continued-training effects. (iii) The opt-vs-worse ablation (Tables 6, 9) is the only isolation of layer selection, but it contrasts extremes of the PG-19 objective and lacks an equal-interval or random control at matched replacement counts; moreover, BO was run on a fixed 10-instance PG-19 set, allowing selection to overfit those instances. A controlled eql/8B experiment would settle whether layer selection or the recipe drives the headline results.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RAD, an algorithm that first uses self-speculative decoding to identify attention layers in a pretrained Transformer whose removal maximises measured decoding throughput, then replaces those layers with SSM blocks (Mamba, Mamba2, or Longhorn), initializes the new blocks to mimic skipped layers, and distills the original model into the hybrid. In self-distillation experiments on Llama-3.2-3B-Instruct, replacing 8 selected attention layers and training only the SSM parameters is reported to improve GSM8K and CRUX scores over the unmodified teacher. In standard distillation, an 8B-teacher RAD Mamba2 model is reported to outperform public equal-interval baselines distilled from a 70B teacher on GSM8K (71.27 vs 46.17) and CRUX (28.25 vs 22.75). The paper also includes an ablation contrasting 'opt' and 'worse' layer selections, with layer lists and throughput measurements in Appendix A.2.","tokens_in":24983,"tokens_out":6342,"duration_ms":63661,"significance":"If the causal role of layer selection is confirmed, RAD is a practical recipe: it gives a model-specific, task-agnostic criterion for converting a Transformer into a hybrid without an ad hoc equal-interval schedule, and it shows a converted student can outperform its teacher on reasoning tasks. The paper is transparent about its limitations, and the opt-vs-worse ablation (Tables 6 and 9; Figures 3-7) supplies credible internal evidence that the choice of replaced layers matters. The measured throughput differences in Table 8 are much larger than the reported standard errors, which is a real strength. However, the headline standard-distillation comparison and the Born-Again claim are not yet supported by controlled experiments, so the current evidence does not establish that the layer-selection objective, rather than initialization, teacher size, or continued training, drives the results.","major_comments":[{"comment":"The standard-distillation headline comparison is uncontrolled. 'Mamba2 (opt / 8B)' differs from 'Mamba2 (eql / 70B)' simultaneously in teacher size (8B vs 70B), layer-selection policy (optimized vs equal intervals), initialization (zero in_proj vs qkv/out_proj copied from the original attention), and possibly data/recipe, since the baselines are public models from [43]. The 'eql / 8B' control appears only in Fig. 8 for passkey retrieval, not in the GSM8K/CRUX or LongBench tables. Without a same-teacher, same-initialization, same-data equal-interval (or random-layer) condition, the GSM8K gap 71.27 vs 46.17 cannot be attributed to redundancy-aware selection.","section":"§4.2, Tables 3–5, Fig. 2"},{"comment":"The self-distillation 'Born-Again' result is not separated from continued training on the 20B-token mixture. RAD students are compared only with the original teacher, which has not seen those tokens; the improvement on GSM8K/CRUX could arise from the distillation data and training budget rather than from redundancy-aware replacement. A control that fine-tunes the original teacher on the same mixture, or at least trains an equal-interval replacement with zero-init under an identical recipe, is needed to support the claim that the architectural change plus targeted distillation causes the gain.","section":"§4.2, Table 2"},{"comment":"The redundancy proxy is estimated from only 10 PG-19 summarization instances at 2048-token context, and the same instances are used to drive 2000 BO iterations; there is no held-out estimate of E[τ] or of task transfer. The opt-vs-worse differences in Table 8 are statistically clear, but the selected configuration could still be overfit to those instances. The authors should validate the proxy on held-out PG-19 instances, on other long-context tasks, or against a direct layer-importance metric (e.g., accuracy drop when each layer is removed) to show that the BO objective is not task-specific.","section":"§3.1, Eq. (1); §4.1"},{"comment":"The '~2x faster convergence' claim is not precisely defined. The figure shows KL loss versus steps for RAD versus the [43] baseline, but no convergence threshold, evaluation checkpoint, or matched-compute comparison is supplied. State the threshold and report downstream accuracy at matched steps/tokens to support the convergence claim.","section":"§4.2, Fig. 2"}],"minor_comments":[{"comment":"Model names are inconsistent: 'Llama3.2-3B-Instruct' in §4.1 and 'Llama-3.2-3B-Instruct' elsewhere; please standardize.","section":"Throughout"},{"comment":"Reference [34] is incomplete: 'arXiv preprint arXiv:2406.XXXX' contains a placeholder volume number.","section":"References"},{"comment":"The limitation paragraph says the authors 'primarily tested replacing 50% of attention layers,' but the main self-distillation experiments replace 8 of 28 layers (~29%) and only the standard-distillation experiments replace 50%; reconcile this statement.","section":"§6"},{"comment":"The BO discretization is described inconsistently: D.2 says a 0.5 threshold is used to map z to {0,1}^L, while Algorithm 1 and D.3 use top-k selection; align the description.","section":"Appendix D.2 vs Algorithm 1"},{"comment":"The approximation D_KL ≈ 2 D_TV^2 is presented without qualification; Pinsker's inequality gives only an inequality in one direction, and the quadratic relation holds only in a local regime. Add a citation or state explicit conditions.","section":"Appendix C, Eq. (18)"}],"recommendation":"major_revision","confidential_remarks":"The paper is promising but not yet ready for acceptance. The central risk is that the headline standard-distillation comparisons will be read as evidence for layer selection when they are confounded with teacher size, initialization, and data recipe. A revision with the eql/8B and teacher-finetune controls would either solidify or refute the central claim. I would not reject; the opt-vs-worse ablation and the careful throughput measurements suggest the core idea has substance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is using self-speculative decoding throughput as a layer-redundancy diagnostic, then replacing those layers with SSM blocks and zero-initializing the SSM's in_proj so the student starts from the same state as the skipped configuration in the diagnostic. That is a coherent, non-circular method, and the zero-init choice is well motivated. The opt-vs-worse ablation is a real internal control: for Mamba2, swapping 'opt' for 'worse' layers drops GSM8K from 64.2 to 43.3 (Table 9), so layer selection clearly matters within their framework. The writing is clear and the limitations section is honest.\n\nWhere it falls short is attribution. The headline standard-distillation comparison (Tables 3-5) compares RAD (opt intervals, 8B teacher, zero in_proj init) to public baselines (equal intervals, 70B teacher, qkv/out_proj init). Three things vary at once, so the 71.27 vs 46.17 GSM8K gap cannot be assigned to the interval choice. The controlled same-teacher-size comparison (eql/8B) appears only in passkey retrieval (Fig 8), not in reasoning. Similarly, the self-distillation 'Born-Again' gain compares the RAD student to the original instruct model, not to the base model fine-tuned on the same 20B token mixture, so continued training alone could explain part of the improvement. The speedup claim (Fig 2) has the same confound. Finally, the BO objective is measured on 10 PG-19 summarization instances; the layer selection could be overfitted to that task, though the standard errors on throughput are small.\n\nNone of this kills the idea. The opt-vs-worse ablation shows the diagnostic has teeth, and the framework is practical and reproducible in principle. But the paper overclaims its central attribution without a same-recipe equal-interval control and a continued-training control. I would send this to reviewers and ask for those two baseline conditions (plus code/model release) before accepting the headline results.\n\nThis is a serious, honestly written paper that deserves a referee. I'd bring it to a reading group mostly to discuss how easily the results could flip once the controls are added.","headline":"RAD is a genuinely new recipe for building hybrid models, but its headline numbers are confounded by teacher size, initialization, and training recipe; only the opt-vs-worse ablation cleanly supports the layer-selection claim.","tokens_in":25438,"tokens_out":4154,"would_cite":true,"duration_ms":40512,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A model's own decoding speed can reveal which attention layers are redundant, and replacing those with state-space blocks plus targeted distillation can beat the original teacher.","keywords":["hybrid models","knowledge distillation","self-speculative decoding","attention layer redundancy","state space models","Mamba","Bayesian optimization","layer selection"],"falsifier":"Run the RAD layer-selection on several different proxy tasks (for example, GSM8K itself, CRUX, and a long-context task) and compare the selected skip configurations and downstream scores; if the 'opt' layers chosen on PG-19 do not consistently beat equal-interval or per-task selections on held-out benchmarks, or if the selected sets are unstable across Bayesian-optimization runs and seeds, the claim that throughput identifies intrinsic redundancy would be refuted. A second decisive check is to compute, for every candidate layer, the correlation between spec-decoding throughput gain and the actual task-accuracy drop when that layer is ablated across a diverse task suite: near-zero or negative correlation would falsify the proxy.","tokens_in":24439,"feed_emoji":"⚡","tokens_out":6140,"duration_ms":61341,"temperature":0.7,"pith_summary":"The paper proposes a recipe for turning an existing Transformer into a smaller, faster hybrid model without giving up accuracy: first find which attention layers are functionally redundant, then replace exactly those layers with state-space-model (SSM) blocks, then distill. The redundancy test is self-speculative decoding: the model itself is used as its own draft model with certain attention layers skipped, and the layers whose skipping most speeds up token generation, while keeping proposals acceptable to the full model, are declared redundant. On the Llama-3.2-3B-Instruct base, replacing eight such layers with Mamba, Mamba2, or Longhorn and self-distilling only the new blocks raised GSM8K from 56.25 to 62.77-64.22 and CRUX from 26.12 to 26.62-27.62. In the standard distillation setting, a hybrid distilled from an 8B teacher using RAD's layer choice reached GSM8K 71.27 and CRUX 28.25, beating the equal-interval baseline distilled from a 70B teacher, which scored 46.17 and 22.75. The paper argues that treating layer selection as part of distillation, rather than as a fixed architectural heuristic, is what makes the gain possible.","feed_headline":"Decoding speed finds the attention layers worth replacing","feed_subtitle":"Swapping those layers for SSM blocks and distilling beats the original model on math and code, even with an 8B teacher.","key_machinery":"The load-bearing mechanism is the self-speculative-decoding throughput $E[\\tau(M_p,\\{x_l\\})]$ used as a redundancy score for attention layers. A draft model is formed by skipping a chosen subset of attention layers of the teacher; the expected throughput obeys $E[\\tau] \\approx \\frac{1}{K T(q) + T(p)} \\sum_{k=0}^K \\beta^k$, where $\\beta = 1 - D_{\\mathrm{TV}}(p,q)$ is the token acceptance rate, $T(q)$ is the draft's per-forward time, and $T(p)$ is the teacher's. Maximizing throughput therefore rewards skip sets whose removed layers least disturb the model's output distribution, which is the paper's operational definition of computational redundancy. Bayesian optimization over the binary skip vector $\\{x_l\\}$, with top-$k$ discretization when a fixed number of layers must be replaced, finds the configuration. The replacement step copies 'out_proj' weights and zero-initializes part of 'in_proj' in the SSM block so that the block starts as a no-op mirroring the skipped state, and distillation trains only the newly added SSM parameters with forward KL divergence.","core_discovery":"RAD claims that the throughput of self-speculative decoding, optimized over which attention layers to skip, identifies the attention layers whose computation can be most profitably replaced by an SSM block. The objective is $E[\\tau(M_p,\\{x_l\\})]$, the average tokens per second when a draft model built by skipping layers $\\{x_l\\}$ proposes tokens for the full model $M_p$ to verify; high throughput requires both that the draft be cheap and that its proposals be accepted, so a layer is 'redundant' if skipping it preserves enough agreement with the teacher. The identified layers are swapped for Mamba, Mamba2, or Longhorn blocks, initialized by copying the 'out_proj' weights and zeroing 'in_proj' so the new block initially behaves like a skipped layer, and only the new block parameters are trained under forward KL divergence from the teacher's logits. The central experimental claim is that this procedure produces students that outperform the teacher on GSM8K and CRUX in self-distillation, and that in standard distillation a RAD student distilled from Llama-3.1-8B-Instruct surpasses an equal-interval student distilled from Llama-3.1-70B-Instruct (GSM8K 71.27 vs 46.17; CRUX 28.25 vs 22.75 for Mamba2).","pith_inferences":["Beyond the paper, the throughput diagnostic could be repurposed as a general layer-importance probe for any architecture where layers can be skipped at decoding time, potentially replacing human-designed pruning heuristics with a model-specific measurement.","The redundancy identification uses only 10 PG-19 summarization instances, far narrower than the evaluation suite; a natural extension is to test whether choosing layers on the target benchmarks or on a diverse task mixture changes the selected set and downstream gains, which would quantify how much the proxy's task specificity matters.","Because the zero-initialization trick makes the new block a no-op at initialization, the method resembles a warm start from a pruned model; one could test whether the same gains appear when the 'skipped state' is replicated by another mechanism, such as an identity residual or a gated bypass, which would separate the initialization effect from the layer-selection effect.","The reported gains on GSM8K and CRUX come with much longer generated answers (reasoning length ratio over 2 for the Mamba2 opt/8B model), so part of the improvement may be a length effect; a sharper test would evaluate accuracy under length-controlled decoding or fixed token budgets."],"forward_implications":["If the redundancy score is a valid proxy, hybridizing an existing pretrained Transformer becomes a per-model optimization problem rather than an ad hoc architectural choice: the same base model can be converted with different SSM families by re-running the decoding-speed diagnostic.","A student hybrid can outperform the teacher it was distilled from on reasoning tasks (the reported 'born-again' effect), so layer replacement plus targeted distillation can reallocate capacity toward math and code rather than merely preserve it.","In standard distillation, the quality of the teacher matters less than the match between architecture and layer choice: an 8B-teacher RAD model beat a 70B-teacher equal-interval model on GSM8K and CRUX, implying that where you replace layers can dominate teacher scale.","Training only the replaced SSM blocks converges faster (about 2x faster in the reported loss curves), so the framework lowers the compute cost of hybrid distillation.","The same pipeline appears to generalize across SSM variants (Mamba, Mamba2, Longhorn) with similar gains, suggesting the layer-selection signal, not the specific SSM, carries the effect."],"supporting_citations":[{"why":"Supplies the self-speculative decoding mechanism of skipping attention layers to build the draft model, plus the Bayesian optimization adaptation for discrete search spaces.","marker":"[49]"},{"why":"Provides the speculative decoding acceptance-rate and throughput model that defines the redundancy objective.","marker":"[26]"},{"why":"Provides the equal-interval hybrid baseline, the weight-initialization strategy, the distillation setup, and the public distilled baseline models used for comparison.","marker":"[43]"},{"why":"Defines the Mamba SSM block used as one of the replacement components in hybrid construction.","marker":"[18]"},{"why":"Defines the Mamba2 SSM block and the structured state-space duality theory used for another replacement variant.","marker":"[12]"},{"why":"Defines the Longhorn SSM block and its online-learning interpretation, used as the third replacement variant and for retrieval analysis.","marker":"[29]"},{"why":"Provides the Bayesian optimization library used to maximize the throughput objective over skip configurations.","marker":"[33]"},{"why":"Supplies the PG-19 dataset from which the 10 summarization instances are drawn for throughput measurement in redundancy identification.","marker":"[36]"}],"fun_headline_variants":["Self-speculative decoding finds attention layers to swap for SSM blocks","Redundant attention layers revealed by decoding speed, then replaced","Swap redundant attention for SSMs, distill, and beat a 70B teacher","Using decoding throughput to prune attention layers and distill to SSMs","RAD: self-speculative decoding spots attention layers to replace with SSM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the measured self-speculative-decoding throughput on just ten PG-19 summarization texts is a valid, task-independent signal of which attention layers are functionally redundant, so that layers selected on that proxy are the right ones to replace for math, code, and long-context tasks.","fun_headline_variants_meta":{"raw":{"variants":["Self-speculative decoding finds attention layers to swap for SSM blocks","Redundant attention layers revealed by decoding speed, then replaced","Swap redundant attention for SSMs, distill, and beat a 70B teacher","Using decoding throughput to prune attention layers and distill to SSMs","RAD: self-speculative decoding spots attention layers to replace with SSM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000474,"raw_usage":{"total_tokens":2425,"prompt_tokens":1085,"completion_tokens":1340,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":701,"completion_tokens_details":{"reasoning_tokens":1245}},"tokens_in":701,"tokens_out":1340,"duration_ms":11213,"temperature":1.0,"reasoning_tokens":1245,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:14:28.966106+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the RAD layer-selection on several different proxy tasks (for example, GSM8K itself, CRUX, and a long-context task) and compare the selected skip configurations and downstream scores; if the 'opt' layers chosen on PG-19 do not consistently beat equal-interval or per-task selections on held-out benchmarks, or if the selected sets are unstable across Bayesian-optimization runs and seeds, the claim that throughput identifies intrinsic redundancy would be refuted. A second decisive check is to compute, for every candidate layer, the correlation between spec-decoding throughput gain and the actual task-accuracy drop when that layer is ablated across a diverse task suite: near-zero or negative correlation would falsify the proxy.","supporting_citations":[{"cited_title":"Fast inference from transformers via speculative decoding","cited_arxiv_id":null,"evidence_quote":"Provides the speculative decoding acceptance-rate and throughput model that defines the redundancy objective."},{"cited_title":"Rush, and Tri Dao","cited_arxiv_id":null,"evidence_quote":"Provides the equal-interval hybrid baseline, the weight-initialization strategy, the distillation setup, and the public distilled baseline models used for comparison."},{"cited_title":"Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality","cited_arxiv_id":null,"evidence_quote":"Defines the Mamba2 SSM block and the structured state-space duality theory used for another replacement variant."},{"cited_title":"Bayesian Optimization: Open source constrained global optimization tool for Python, 2014–","cited_arxiv_id":null,"evidence_quote":"Provides the Bayesian optimization library used to maximize the throughput objective over skip configurations."}],"review_version":1}