{"id":"77d35c66-854e-4982-a3bc-e011b017ead0","arxiv_id":"2411.15281","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A post-training routing method that uses self-distillation to let frozen pretrained Transformers process only a subset of parameters and tokens, cutting active compute by 20 to 50 percent.","lead":"ElastiFormer adds tiny trainable routers to a pretrained Transformer, letting it skip parts of its own attention and MLP layers per input. The claim is 20 to 50 percent compute reduction across language, vision, and vision-language models with little quality loss, as measured by matching the original model's outputs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim is unsupported for two of three modalities: LLM and ViT 'performance' is only teacher-matching fidelity (LM loss, cosine similarity), never task accuracy, despite Section 3 defining redundancy via GSM8K/HumanEval tasks.","rationale":"I read the paper as making a general post-training efficiency claim: with tiny routers, a frozen pretrained transformer can skip a substantial fraction of parameters and tokens without users noticing a performance difference. The method is coherent, the self-distillation objective is reasonable, and the scaling experiments are extensive within their chosen proxies. The VLM section does report a genuine downstream benchmark and even surfaces a partial negative result. However, the load-bearing step is the transfer from teacher-matching fidelity to task performance. For the LLM and ViT modalities, the paper never measures task performance, even though Section 3 itself motivates redundancy using GSM8K and HumanEval task metrics. The absent evaluation distribution for the LM-loss curves and the absence of any ViT classification probe make the 'without noticeable impact on performance' claim empirically unverified for two of three modalities. This is exactly the weakness the reader identified, so my stress-test does not change the CONDITIONAL verdict; it sharpens the requested condition: report task-level accuracy at the claimed capacities, or explicitly limit the claim to teacher-output fidelity.","tokens_in":20542,"tokens_out":4487,"duration_ms":47598,"concrete_test":"Re-run the reported Elasti-LLM capacity settings from Fig. 5 (12/32 attention heads, 18/32 experts, 80% MLP token capacity) on the actual GSM8K test set and HumanEval, using greedy decoding and standard evaluation, and compare exact-match accuracy and pass@1 to the frozen teacher model. If task accuracy is preserved within noise, the concern is resolved; if accuracy drops materially despite matching LM loss, the headline claim must be narrowed to teacher-fidelity rather than task performance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Contribution 2 claims a 20-50% reduction in active parameters/tokens 'without noticeable impact on performance' across all modalities. For Elasti-LLM, the only evidence is LM-loss matching to the frozen teacher (Figs. 5-6); for Elasti-ViT, the only evidence is cosine similarity between frozen MAE decoder outputs (Fig. 7). Neither is a task-performance metric. This is not a missing nicety: Section 3 establishes redundancy using GSM8K and HumanEval, i.e., task performance, and yet the Elasti-LLM experiments never report GSM8K accuracy or HumanEval pass@1. The Elasti-LLM models are trained on the GSM8K training set, and the evaluation distribution for the LM-loss curves is not specified; if those curves are measured on the same GSM8K training data, they cannot support a generalization claim. The ViT experiments likewise report no linear-probe or ImageNet classification accuracy, so 'noticeable impact on performance' is untested for image modeling. The VLM experiments do use a genuine benchmark (LLava-Bench, OpenChair), and the paper itself concedes in Section 5.3 that on detail-oriented VLM tasks Elasti-VLM 'offers no clear advantage' - an internal admission that teacher-fidelity transfer can fail exactly when detailed visual information matters. The load-bearing assumption is that matching teacher outputs on a distillation proxy transfers to task performance; that assumption is plausible but unverified in two of the three modalities.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ElastiFormer, a post-training method that inserts lightweight routing modules into pretrained Transformers to select, per input, a subset of attention heads, MLP experts, and tokens. The routers are trained by self-distillation losses so that the elastic model's outputs match the frozen teacher model. The method is applied to causal LLMs (Phi-3.5-mini-instruct, Gemma-2-2b-it), a ViT-MAE encoder, and LLaVA-1.5-7b. The authors report scaling curves showing that roughly 38% of attention heads and 56% of MLP parameters suffice to match teacher LM loss, that 60–80% of tokens suffice in several settings, and that adding rank-1 LoRA to Q/V projections further reduces required capacity. They also report a router-robustness study across ImageNet subsets and VLM evaluations on LLaVA-Bench and OpenChair.","tokens_in":20901,"tokens_out":3967,"duration_ms":38760,"significance":"If the central claim were fully supported, ElastiFormer would be a practically useful and cheap way to obtain input-dependent compute savings from off-the-shelf pretrained models across modalities. The method's strengths are its simplicity, the very small number of added parameters (as low as 0.00006% in the VLM token router), the systematic scaling analysis for several routing schemes, and the explicit robustness study of ViT routers across ImageNet subsets. The paper is also honest about one limitation: on detail-oriented VLM tasks it reports no clear advantage. However, the headline claim that compute can be reduced by 20–50% 'without noticeable impact on performance' is currently supported mainly by teacher-fidelity proxies—LM loss for LLMs and cosine similarity of MAE decoder outputs for ViT—rather than by task-level accuracy. The significance of the contribution therefore depends on whether those proxies are validated against downstream task metrics.","major_comments":[{"comment":"The Elasti-LLM scaling claim is supported only by LM loss, and the evaluation set for those curves is not specified. Since the elastic models are trained on the GSM8K training set via self-distillation, if the LM-loss curves are computed on that same set they would reflect training-set teacher matching rather than generalization. Section 3 motivates redundancy using GSM8K and HumanEval task performance, yet no GSM8K accuracy or HumanEval pass@1 is reported for Elasti-LLM. Without a stated evaluation distribution and task-level metrics, the statement that 38% of heads and 56% of MLP parameters yield no noticeable performance impact is unverified.","section":"§5.1, Figs. 5–6"},{"comment":"Elasti-ViT is evaluated only by cosine similarity between the teacher and student MAE decoder outputs. The 0.95 threshold used in Figure 7 is an arbitrary fidelity cutoff; there is no evidence that exceeding it corresponds to no noticeable impact on an actual vision task such as ImageNet classification or linear probing. Consequently, the claimed 20–50% compute savings for image modeling rest on an unvalidated proxy. Reporting a downstream task metric, or at least a linear-probe accuracy curve, would make the claim load-bearing.","section":"§5.2, Fig. 7"},{"comment":"The paper itself concedes that on detail-oriented VLM tasks (LLaVA-Bench detail and OpenChair), Elasti-VLM offers no clear advantage over the base model. This is an internal admission that teacher-fidelity transfer can fail precisely when detailed visual information is needed, and it directly qualifies the umbrella claim of '20% to 50% reduction for all modalities without noticeable impact on performance.' The authors should either restrict the claim to the settings where task benchmarks support it or analyze why fidelity transfer fails on detail-oriented tasks.","section":"§5.3, Fig. 9"},{"comment":"The compute-savings claims are expressed in terms of nominal capacity factors, but during inference the token router uses a fixed decision threshold of 0.5 rather than the top-k selection used in training. The paper does not report the realized fraction of tokens or experts actually selected at inference. If the threshold-based router selects more than the nominal capacity, the reported savings are overstated. Reporting measured active-token/active-parameter counts at inference would make the compute claims precise.","section":"Appendix B.1, §4.1"}],"minor_comments":[{"comment":"The Abstract states that added parameters can be 'as low as .00006%' while the Introduction states 'as little as 0.0006% additional parameters' (p. 2). These differ by an order of magnitude; please reconcile them.","section":"Abstract vs. §1"},{"comment":"Algorithm 1 multiplies selected sub-network outputs by routing weights, but Appendix B.2 says 'unselected experts simply pass their input through without modification.' Please clarify the exact output aggregation and how the straight-through estimator interacts with the M-normalized softmax weights.","section":"§4.1 and Appendix B.2"},{"comment":"The x-axis label 'train/epoch' is ambiguous; it should be 'epoch' or 'training step.' Also, the 'Top-K' values (5, 20, 50) for the top-K KL variant should be defined in the caption.","section":"Figure 4"},{"comment":"The robustness study trains on 10 subsets of ImageNet based on categories from a GitHub repository [39], but the paper does not specify how the subsets were split from the training set or whether the held-out evaluation images overlap with the training classes. A short description of the split and evaluation protocol would aid reproducibility.","section":"§5.2, Figure 8"},{"comment":"There is a typo in the caption: 'capcaity' should be 'capacity.'","section":"Figure 11 caption"},{"comment":"Reference [39] points to a GitHub repository; a versioned citation or commit identifier would make the data split reproducible.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern is valid and is the main reason for the major-revision recommendation. The paper's central contribution claim—20–50% compute savings without noticeable performance impact—is currently supported by teacher-fidelity proxies for two of the three modalities, and the authors' own VLM results show a failure case on detail-oriented tasks. This is fixable within the manuscript's scope by adding downstream task metrics (GSM8K/HumanEval accuracy for LLMs, linear probing or classification for ViT, and clearer evaluation-set reporting), and by measuring realized inference-time selection rates. The method itself is interesting and the parameter-efficiency numbers are impressive, but the evidence needs to match the strength of the claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read on ElastiFormer. The paper does one thing well: it lays out a clean post-training recipe for adding input-dependent routing to frozen transformers, and it gives scaling curves across three modalities with tiny trainable overhead. The dense-to-MoE conversion is lossless and the even-layer ViT trick is a nice practical touch. The authors are also upfront about the places where routing doesn't pay off (MHA token selection, detail-oriented VLM tasks), which I respect.\n\nThe soft spot is the one the stress-test flags. The headline claim in the intro and contribution 2 is \"20-50% reduction without noticeable impact on performance,\" but for Elasti-LLM and Elasti-ViT, \"performance\" is teacher-matching fidelity: LM loss and cosine similarity to the MAE decoder. Those are legitimate measures of behavioral fidelity, but they are not task accuracy. Section 3 defines redundancy using GSM8K and HumanEval accuracy, and that is the natural yardstick the experiments should have used. They don't report it. The LM-loss curves don't even state which split they are evaluated on, and training was on the GSM8K training set—if those curves are on the training distribution, they generalize poorly. Error bars are absent for these curves, and there is no head-to-head against static pruning or MoD at matched capacity, even though the paper claims to outperform static pruning.\n\nThe VLM experiments are the strongest: they use real benchmarks, and the bootstrap CIs are there. But the authors' own concession that Elasti-VLM gives no clear advantage on detail tasks and OpenChair undercuts the unconditional \"all modalities\" claim. The method works when the task mostly needs gist; it does not yet support the broad claim.\n\nOn novelty: this is an extension/combination of known pieces (MoD, MoE-from-dense, head skipping, self-distillation). That's fine, but it's not a conceptual shift. The scaling curves themselves are a useful empirical contribution.\n\nIn sum: this is a paper worth a serious referee, but it needs task-level evaluations for LLM and ViT, a specified eval set, error bars, and a matched baseline comparison before the central claim is credible. I'd accept it for review with high confidence that it will need revision.","headline":"A clean post-training routing recipe with useful scaling curves, but the headline compute-saving claim rests on teacher-fidelity proxies for two of three modalities, not task accuracy.","tokens_in":21447,"tokens_out":3312,"would_cite":true,"duration_ms":30871,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ElastiFormer adds tiny learned routers to frozen pretrained Transformers that cut active parameters or tokens by 20-50% across language, vision, and multimodal models, trained by self-distillation.","keywords":["transformer efficiency","self-distillation","learned routing","mixture-of-experts","token pruning","attention head selection","variable compute","post-training adaptation"],"falsifier":"Run the exact Elasti-LLM configuration that matches the teacher's LM loss at 38% attention heads and 56% MLP experts on GSM8K and HumanEval and compare exact-match or pass@1 accuracy against the full teacher; if accuracy drops substantially while LM loss stays matched, the fidelity proxy is not sufficient. Similarly, fine-tune a linear probe on top of the Elasti-ViT encoder at 50% head capacity and check ImageNet top-1 accuracy against the teacher encoder.","tokens_in":20334,"feed_emoji":"⚡","tokens_out":6359,"duration_ms":54875,"temperature":0.7,"pith_summary":"ElastiFormer claims that pretrained Transformer models of any modality contain input-dependent redundancy that can be exploited after training, without retraining the backbone. The paper introduces tiny routing modules, as small as 0.00006% of the model's parameters, that learn to skip attention heads, MLP experts, and whole tokens per input. Trained only by self-distillation against the frozen teacher model, these routers preserve teacher output fidelity while cutting active parameters or tokens by 20-50% across language, vision, and vision-language models. If the claim holds, it offers a cheap, post-training path to variable inference-time compute and a tool for studying where redundancy lives in Transformers.","feed_headline":"Learned routers skip 20–50% of Transformer compute","feed_subtitle":"It drops heads, MLP experts, and tokens in language, vision, and multimodal models using under 0.3% new parameters.","key_machinery":"The load-bearing object is the routing module: a linear (or one-hidden-layer MLP) projector that, for each layer and input, produces either a set of top-$k$ scores over sub-networks or a set of top-$k$ scores over tokens. Parameter subset selection picks attention heads, or experts obtained by losslessly factorizing a dense MLP into block matrices; an input subset selection picks which tokens continue through MHA or MLP. The router weights are normalized so that selecting every sub-network with uniform weight reproduces the pretrained model exactly, which anchors the method to the teacher. Training is driven by self-distillation: forward KL divergence over the teacher's top-50 tokens for language and vision-language outputs, and cosine distance between encoder token embeddings for ViT-MAE, plus a load-balancing loss for expert routers and a binary cross-entropy loss that keeps causal-LM token selection consistent during autoregressive inference.","core_discovery":"The central discovery is that a frozen pretrained Transformer can be converted into an 'elastic' model whose per-input compute budget is controlled by learned routers, and that this pays off across modalities. In language modeling, ElastiFormer matches the teacher's LM loss with 38% of attention heads and 56% of MLP parameters active, and with rank-1 LoRA adapters it can also drop 20% of tokens from attention. For a ViT-MAE backbone, routing only even layers lets 50% heads, 31% MLP parameters, 70% MHA tokens, and 10% MLP tokens reproduce the teacher's decoder output with cosine similarity above 0.95. For LLaVA, dropping 40% of image tokens before the language decoder preserves LLava-Bench performance, though the paper notes no clear advantage on detail-oriented benchmarks. The paper frames this as evidence that redundancy in pretrained Transformers is data-dependent and learnable rather than static.","pith_inferences":["Beyond the paper: the fidelity proxies (LM loss, cosine similarity, win rate) are not task accuracy; a natural next test is whether routers trained to match teacher outputs also preserve GSM8K accuracy, ImageNet classification, or code pass rates at the claimed compute reductions.","Beyond the paper: the robustness result suggests routers could serve as lightweight probes for interpretability, e.g., mapping which attention heads or MLP experts are consistently dispensable for a given input class or domain.","Beyond the paper: the method's modality-agnostic formulation implies it could be applied to encoder-decoder transformers, diffusion backbones, or speech models, where the same distillation objective would need a suitable output-space metric."],"forward_implications":["Post-training compute reduction applies to frozen pretrained models without architectural retraining, so existing served models can be made elastic on top of techniques like QLoRA.","A single checkpoint can be deployed at multiple compute budgets: capacity factors set the fraction of heads, experts, or tokens used per input.","The learned routers generalize across data distributions: ViT routers trained on different ImageNet subsets produce highly similar selections on held-out images.","Attention-head and MLP-expert redundancy is substantially larger than static pruning suggests: ElastiFormer skips 640 attention heads across layers in Phi-3.5-mini while matching teacher LM loss.","Very low-rank LoRA adapters (rank 1) can restore performance lost by token-level attention skipping, pointing to a cheap repair for input-selection weaknesses."],"supporting_citations":[{"why":"Supplies the top-k token-selection scheme and the inference-time prediction trick that ElastiFormer adapts for input subset selection.","marker":"[57]"},{"why":"Supplies the pretrained ViT teacher (ViT-MAE) whose decoder output defines fidelity for Elasti-ViT.","marker":"[27]"},{"why":"Supplies the pretrained VLM base and LLava-Bench evaluation for Elasti-VLM.","marker":"[42]"},{"why":"Supplies one of the two pretrained LLM teachers for Elasti-LLM.","marker":"[1]"},{"why":"Supplies the other pretrained LLM teacher and the model used in the redundancy analysis.","marker":"[68]"},{"why":"Supplies the training and evaluation dataset for Elasti-LLM and the toy distillation experiments.","marker":"[14]"},{"why":"Supplies the training data for Elasti-ViT and the domain-robustness study.","marker":"[17]"},{"why":"Supplies the second evaluation task in the redundancy analysis that shows data-dependent redundancy.","marker":"[10]"}],"fun_headline_variants":["Half the compute, same output: learned routers","ElastiFormer: 50% compute cut with tiny routers","Learned routers prune heads, experts, and tokens on the fly","Self-distilled routers skip 50% of Transformer compute"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that matching the frozen teacher's outputs (LM loss, decoder cosine similarity, or benchmark win rate) is a faithful proxy for real-task performance; the paper reports no downstream accuracy numbers, so if teacher-fidelity survives while task accuracy drops, the 'no noticeable impact on performance' claim breaks.","fun_headline_variants_meta":{"raw":{"variants":["Half the compute, same output: learned routers","ElastiFormer: 50% compute cut with tiny routers","Learned routers prune heads, experts, and tokens on the fly","Self-distilled routers skip 50% of Transformer compute"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001051,"raw_usage":{"total_tokens":4409,"prompt_tokens":936,"completion_tokens":3473,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":3405}},"tokens_in":552,"tokens_out":3473,"duration_ms":24187,"temperature":1.0,"reasoning_tokens":3405,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:34:25.073855+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the exact Elasti-LLM configuration that matches the teacher's LM loss at 38% attention heads and 56% MLP experts on GSM8K and HumanEval and compare exact-match or pass@1 accuracy against the full teacher; if accuracy drops substantially while LM loss stays matched, the fidelity proxy is not sufficient. Similarly, fine-tune a linear probe on top of the Elasti-ViT encoder at 50% head capacity and check ImageNet top-1 accuracy against the teacher encoder.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the other pretrained LLM teacher and the model used in the redundancy analysis."}],"review_version":1}