{"id":"2d25c2f2-3b54-43f4-b277-ff736be74956","arxiv_id":"2411.10606","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AmoebaLLM enables instant extraction of arbitrary-shaped subnets from a single fine-tuned LLM, matching or beating dedicated compression methods.","lead":"AmoebaLLM is a method that fine-tunes a large language model once, then instantly extracts smaller sub-models with different depths and widths without retraining each one. If it works as reported, it would make LLM deployment to phones, laptops, and cloud servers much faster and cheaper.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MMLU results are calibration-fitted: the paper calibrates DP depth selection and final subnet search on MMLU, then reports MMLU as a headline metric, so the factual-knowledge/SOTA advantage may be selection bias rather than a property of the method.","rationale":"The reader's weakest_assumption identifies the DP independence hypothesis as the main risk; that is a plausible internal-assumption concern. However, I see the MMLU calibration/evaluation overlap as more immediately load-bearing because it directly threatens the evidence for the central 'accuracy-efficiency frontier' and 'factual knowledge preservation' claims. The paper itself advertises MMLU as the calibration dataset in Sec. 4.4 and uses MMLU samples in the final subnet search, so the reported MMLU numbers are not a fair comparison against baselines that did not receive the same metric-specific selection. This does not by itself disprove the method: the commonsense-reasoning averages in Tables 1-2 and Fig. 3 are not calibrated on those tasks and still show modest gains over baselines. But those gains are small, without error bars, and the final subnet shape was still selected using MMLU samples, which could indirectly affect the reported operating points. Therefore the appropriate verdict remains CONDITIONAL, matching the reader's verdict; my read does not move it to ACCEPT or REJECT. The DP-independence concern is real but secondary: even if optimal substructure fails, the DP may still function as a strong heuristic, as suggested by Table 3, whereas the MMLU leakage cannot be repaired by reinterpreting the same numbers. Hence I partially agree with the reader's weakest_assumption but elevate a different concern as the single most load-bearing issue.","tokens_in":21186,"tokens_out":5294,"duration_ms":62862,"concrete_test":"Hold out MMLU completely: perform DP calibration (Sec. 3.2) and final subnet search (Sec. 3.4) using only a non-MMLU calibration set, such as Wikitext2, BookCorpus, or an unrelated factual benchmark like TriviaQA, then evaluate the resulting subnets on the MMLU test split. If AmoebaLLM's MMLU advantage over FLAP and Shortened LLaMA in Tables 1-2 disappears or falls within noise, the headline factual-knowledge and SOTA claims are calibration artifacts. As a secondary check, also run the same experiment with MMLU train split only and report error bars over at least three seeds to quantify whether the remaining differences are significant.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is the overlap between calibration and evaluation for the paper's headline factual-knowledge metric. Section 4.4 states that 'we adopted MMLU as the calibration dataset throughout the previous experiments,' and Section 3.4's final subnet search also uses 40 MMLU samples to choose the (depth, width) configurations reported in Tables 1 and 2. The MMLU column is therefore not an independent measure of the delivered subnets: the DP layer-selection strategy in Sec. 3.2 was optimized against MMLU, and the specific reported shape was selected on MMLU, while the baselines (LLM-Pruner, FLAP, Shortened LLaMA) are not given the same metric-specific selection. The large MMLU advantages in Tables 1-2 (e.g., 42.40 vs 40.21 at 80% LLaMA2; 48.30 vs 43.50 at 80% Vicuna) could reflect selection bias rather than a genuine property of the one-for-all fine-tuning. Because MMLU is also the primary evidence for 'knowledge preservation' and 'new SOTA compression effectiveness,' the central claim loses its strongest independent support unless this overlap is resolved. The average commonsense-reasoning results in the same tables and in Fig. 3 are not directly calibrated on those tasks, so they provide some independent evidence that the framework works; however, the differences there are small and no error bars are reported, so the strength of the frontier claim depends heavily on the contaminated MMLU comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AmoebaLLM, a framework to fine-tune a given LLM once so that subnets of arbitrary depth/width combinations can be extracted instantly at deployment time. The framework has three components: (1) a subnet selection strategy using DP-based depth shrinking and importance-based width shrinking; (2) a shape-aware mixture of LoRAs (SMoL) that sparsely activates LoRA experts based on the subnet shape; and (3) an in-place distillation objective with loss-magnitude balancing. Experiments on LLaMA2 7B and Vicuna 7B v1.5 compare AmoebaLLM with LLM-Pruner, FLAP, and Shortened LLaMA at 80%/65%/50% remaining ratios, reporting MMLU, seven commonsense reasoning tasks, perplexity, and device latency. The paper claims the resulting subnets achieve state-of-the-art accuracy-efficiency trade-offs after one fine-tuning, and also that the per-subnet fine-tuned variant achieves new SOTA compression effectiveness.","tokens_in":21563,"tokens_out":3575,"duration_ms":36492,"significance":"If validated, the one-for-all paradigm with instant subnet extraction would be practically valuable: a single fine-tuned model could replace per-configuration compression pipelines. The paper makes several concrete, falsifiable design choices and releases code, which is a strength. The profiling across two devices and three deployment flows is a useful empirical contribution, and the SMoL design is a sensible approach to reducing gradient conflicts among subnets. However, the central empirical evidence is currently weakened by the overlap between calibration and evaluation on MMLU, and by the absence of error bars. The commonsense-reasoning averages provide some independent support, but the differences are small and their statistical reliability is not established. The paper therefore contains a promising framework, but its main claims need substantially strengthened evidence before they can be accepted as stated.","major_comments":[{"comment":"The headline MMLU results are compromised by selection-on-evaluation circularity. Section 4.4 states that 'we adopted MMLU as the calibration dataset throughout the previous experiments,' and Section 3.4 uses 40 MMLU samples to select the final (depth, width) subnets reported in Tables 1 and 2. The MMLU column in those tables is therefore not an independent measure of the delivered subnets: the DP layer-selection strategy and the reported subnet shapes were both chosen using MMLU, while the baselines were not given the same metric-specific selection. The large MMLU advantages (e.g., 42.40 vs 40.21 at 80% on LLaMA2 and 48.30 vs 43.50 at 80% on Vicuna) could largely reflect selection bias. This is load-bearing because the abstract and conclusion claim an 'accuracy-efficiency frontier' and 'knowledge preservation' primarily on the basis of MMLU. The authors should either calibrate on a disjoint metric or dataset and evaluate MMLU as a held-out task, or explicitly present the MMLU results as calibration-tuned and support the frontier claim with independent metrics.","section":"Sec. 4.4 and Sec. 3.4, Tables 1-2"},{"comment":"No error bars, confidence intervals, or repeated-seed results are reported, and the checklist explicitly answers 'No' to statistical significance. This matters because several of the claimed advantages over the strongest baselines are small on the commonsense average (e.g., 62.29 vs 60.98 at 80% on LLaMA2; 56.96 vs 56.12 at 65% on LLaMA2), and individual task differences are often within 1-2 points. Without variance estimates, the claims that AmoebaLLM subnets are 'better or comparable' and achieve a 'state-of-the-art trade-off' are not quantitatively supported. The authors should report multiple runs or bootstrap confidence intervals for the main tables, and state which reported differences are outside noise.","section":"Sec. 4.2, Tables 1-2; NeurIPS Checklist item 7"},{"comment":"The DP-based depth shrinking relies on the 'Key hypothesis' that the layer-selection problem can be decomposed into approximately independent subproblems and that the optimal substructure holds. This assumption is central to the correctness of the DP recurrence, but it is not validated. Table 3 compares the DP against two heuristic baselines, which shows that the DP improves on those baselines, but it does not test whether the DP's solution is near-optimal or whether layer interactions are weak. The independence assumption could be tested by comparing the DP-selected layer sets against exhaustive search on a smaller model or a reduced set of layers, or against a randomized/beam-search selection procedure. Without such a validation, the depth-shrinking component's contribution is not established as principled rather than heuristic.","section":"Sec. 3.2, Eq. (1)-(2)"},{"comment":"Table 5 demonstrates that the choice of calibration metric has a very large effect on post-fine-tuning MMLU accuracy: at 20 layers, MMLU-calibrated subnets reach 41.0% MMLU while Wikitext2-calibrated subnets reach 23.6%, and BookCorpus-calibrated subnets reach 27.0%. This sensitivity reinforces the concern that the reported MMLU advantages are driven by calibrating to MMLU itself. The paper should address this directly by reporting at least one set of results where the calibration metric is not MMLU and MMLU is evaluated as an unseen task, to show that the method preserves factual knowledge independently of the calibration choice.","section":"Sec. 4.4, Table 5"}],"minor_comments":[{"comment":"The fine-tuning setting lists the number of Alpaca samples, LoRA rank, learning rate, and iteration count, but omits batch size, sequence length, and the number of GPUs used for the one-for-all fine-tuning. These details should be added for reproducibility.","section":"Sec. 4.1"},{"comment":"The column labeled 'Average' appears to be the average over the seven commonsense reasoning tasks, excluding MMLU. This should be stated explicitly in the table caption or the evaluation setup.","section":"Sec. 4.2, Tables 1-2"},{"comment":"The loss-normalization terms use the norm of cross-entropy losses; the type of norm (L2 over tokens and vocabulary, or something else) and the numerical treatment of zero-norm cases should be clarified.","section":"Eq. (3)"},{"comment":"The checklist's Limitations answer cites 'Sec. 5.5', but the Limitations section is Sec. 4.5 in the manuscript. The cross-reference should be corrected.","section":"NeurIPS Checklist"},{"comment":"The importance-score formula F_i^l is dense and the notation is not fully introduced before the equation; a short verbal description of each symbol (k, t, j, X-bar) in the main text would improve readability.","section":"Sec. 3.2"},{"comment":"The latency-accuracy plot would benefit from labeled markers for each method and from indicating the specific subnet shapes used, since the reader cannot otherwise verify which operating points correspond to the 80%/65%/50% ratios in Tables 1-2.","section":"Fig. 3"}],"recommendation":"major_revision","confidential_remarks":"The calibration-evaluation overlap on MMLU is a serious correctness concern for the central claim, but it is fixable by re-evaluating on held-out benchmarks or by calibrating on a different metric and reporting MMLU as an independent test. The DP independence assumption also needs a direct test. Given the paper's otherwise solid framing and the practical value of the one-for-all idea, I would be open to a revised version that addresses these points, but as it stands the evidence does not support the claimed state-of-the-art frontier."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a serious attempt at a real problem: after one fine-tuning, give an LLM switchable subnets of different depths and widths. The pieces are well chosen. The DP-based layer selection outperforms Shortened LLaMA and the 'unreasonable' baseline on calibration metrics, and Table 3 shows it clearly. The SMoL adapter—shape-gated, with top-k LoRAs that can be merged into weights at deployment—is a nice practical trick, and Table 4 shows it beats both full-model and plain-LoRA fine-tuning. Loss-magnitude balancing also earns its place. The writing is clear, the ablations are informative, and code is promised. Credit where due: this is a thoughtful integration, not a bag of tricks.\n\nThe soft spot is the one the stress-test flagged, and it is real. The paper calibrates its DP depth selection on MMLU (Sec. 4.4 says so explicitly), and the final subnet search in Sec. 3.4 also uses 40 MMLU samples to pick the reported shapes. Then Tables 1 and 2 report MMLU as the headline metric. The baselines are not given the same metric-specific selection. So the large MMLU gaps (e.g., 42.40 vs 40.21 at 80% LLaMA2) are at least partly an artifact of selecting on the test metric. That is a classic selection-on-evaluation circularity, and it directly undermines the 'knowledge preservation' and 'new SOTA' claims.\n\nThe commonsense average numbers are the more believable evidence. They are not calibrated on those tasks, and AmoebaLLM is roughly on par with or a bit better than FLAP across the board, with the instant-subnet benefit. But no error bars are reported (the checklist admits this), so small differences are hard to interpret. The 'key hypothesis' about independent layer contributions is stated and not directly tested; Table 3 suggests the DP works in practice, so I'd treat that as a minor concern, not a flaw. A stronger joint width+depth pruning baseline, like Sheared-LLaMA, is missing; citing it without comparing leaves the frontier claim unsupported against the most relevant competitor.\n\nWho is this for? Anyone working on efficient LLM deployment or one-for-all training. The framework is plausible enough that I'd want to see a revised version with (1) an independent held-out evaluation set or a proper selection/evaluation separation, (2) error bars, and (3) a joint-pruning baseline. With those, the paper could become solid. As is, it deserves a serious referee but not acceptance in its current form.","headline":"A well-built one-for-all fine-tuning framework whose headline MMLU results are partly selected on the test metric; the commonsense and ablation evidence suggest the core mechanism is sound, but the SOTA claims need stronger experimental hygiene.","tokens_in":22134,"tokens_out":2657,"would_cite":true,"duration_ms":24896,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AmoebaLLM claims that one fine-tuned LLM can instantly produce subnets of any shape and still sit on the accuracy-efficiency frontier.","keywords":["large language model compression","one-for-all training","dynamic programming layer pruning","structured pruning","mixture of LoRAs","in-place distillation","any-shape subnet","efficient deployment"],"falsifier":"Take a small transformer (or a smaller random seed of LLaMA), run the DP layer selection, and compare its chosen layer sets against exhaustive search over all subsets for each removal count on the calibration metric; if the DP-selected sets are meaningfully worse than the exhaustive optimum, or if the gains of jointly removed layers diverge from the sum of individual gains, the independence assumption fails.","tokens_in":20980,"feed_emoji":"🪆","tokens_out":5015,"duration_ms":49298,"temperature":0.7,"pith_summary":"The paper proposes AmoebaLLM, a way to turn a pre-trained large language model into a collection of smaller models (subnets) of many different depths and widths after a single fine-tuning pass. The stated goal is that a developer can pick any subnet shape matching a target device or latency budget and extract it immediately, without any per-configuration retraining. The authors argue that three components make this work: a dynamic-programming layer-selection strategy plus an importance-based neuron selection that preserves pre-trained knowledge; a shape-aware mixture of LoRAs that reduces gradient conflicts during joint fine-tuning; and a loss-magnitude-balanced distillation objective. If correct, the approach would replace per-platform compression pipelines with one model that serves many deployment points. The paper reports that on LLaMA2 7B and Vicuna 7B, the extracted subnets match or beat separately fine-tuned compression baselines at 80%, 65%, and 50% of the original size.","feed_headline":"One fine-tuning yields any-shape LLM subnets on demand","feed_subtitle":"A dynamic program picks layers, a shape-aware LoRA mixer avoids conflicts, and every subnet is extractable immediately.","key_machinery":"Three mechanisms carry the argument. The first is a dynamic program over decoder layers, with table $D[n][m]$ storing the best calibration metric when exactly $m$ layers are removed from the first $n$ layers; it assumes that, thanks to residual connections, layer-selection sub-problems are approximately independent, so the optimal solution for removing $m$ layers extends the best solution for $m-1$ layers. The second is the shape-aware mixture of LoRAs (SMoL), a gating function that takes the subnet's shape mask as input and sparsely activates and averages a small set of low-rank adapters, so that subnets sharing similar shapes share similar gradient updates and the adapters can be merged away at deployment. The third is the fine-tuning objective, which samples subnets by sandwich sampling and distills each sampled subnet from the largest subnet while normalizing loss magnitudes so no subnet dominates the update.","core_discovery":"The central claim is that a given LLM can be endowed, through one one-time fine-tuning, with the ability to produce subnets of arbitrary depth and width that lie on the accuracy-efficiency frontier, meaning no separately tuned compressed model is needed for each target configuration. The discovery is framed as a complete workflow: first, a calibration-guided dynamic program selects which layers to keep for every depth remaining ratio, and an importance metric selects which neurons to keep for every width ratio; second, a shape-aware mixture of LoRAs adapts the model while preventing gradient conflicts among subnets; third, an in-place distillation loss with a loss-magnitude balancing term keeps the largest subnet from being dragged down by the higher losses of smaller subnets. The paper's evidence is that instantly extracted subnets outperform or match strong width- and layer-pruning baselines on MMLU and seven commonsense reasoning tasks, and that the per-subnet fine-tuned upper bound is close.","pith_inferences":["The paper does not test the DP independence hypothesis directly, but it could be probed by comparing DP-selected layer sets against exhaustive search on a small transformer, or by checking whether the improvement from removing two layers together equals the sum of the individual improvements.","Because the framework's selection and adapter mechanisms are not tied to LLaMA architecture specifics, the same three-stage recipe could plausibly transfer to other residual transformer LLMs or to multimodal models that use transformer stacks, provided calibration data and LoRA-style adapters are available.","The loss-magnitude balancing insight likely extends beyond AmoebaLLM: any joint training of models at very different capacities (for example, any-precision or any-length variants) faces the same domination problem, so normalizing loss scales before gradient averaging could improve other one-for-all training schemes.","Combining the shape-conditional gating idea with other parameter-efficient methods such as quantization or pruning-aware training could be a natural next step, since SMoL only touches the adapter pathway and leaves the base weight structure fixed."],"forward_implications":["A single fine-tuned AmoebaLLM model can serve many deployment targets: after the one-time fine-tuning, any depth/width combination within the design space can be extracted without additional training, so device-specific or battery-dependent latency budgets can be met on the fly.","Because SMoL's LoRAs are selected by shape rather than by input, they can be merged into the model weights at deployment, so the any-shape capability adds no per-input routing overhead.","The DP depth-shrinking component on its own is claimed to be a new state-of-the-art layer-pruning method, improving on single-layer importance and fixed heuristic pruning rules, which means it can be used as a standalone compression tool.","If factual knowledge is hard to restore after compression, as the calibration-dataset ablations suggest, future compression pipelines should choose calibration metrics and datasets that track the knowledge they care about rather than perplexity alone.","AmoebaLLM† (per-subnet fine-tuning of the selected subnets) forms an upper bound, so the one-for-all subnets are reported to land close to separately tuned models, quantifying the cost of instant extraction."],"supporting_citations":[{"why":"Supplies the LLaMA2 models used for the main experiments and for the depth/width profiling that motivates adaptable shapes.","marker":"[1]"},{"why":"LLM-Pruner is a structured width-pruning baseline that AmoebaLLM must match or beat at comparable model sizes.","marker":"[7]"},{"why":"FLAP is both a baseline and the source of the neuron importance metric adopted for width shrinking.","marker":"[8]"},{"why":"Shortened LLaMA is the layer-pruning baseline and the most direct comparison for the DP-based depth selection.","marker":"[9]"},{"why":"BigNAS provides the sandwich-sampling and in-place-distillation training scheme that the fine-tuning objective builds on.","marker":"[11]"},{"why":"LoRA is the low-rank adaptation basis on which the shape-aware mixture of LoRAs is constructed.","marker":"[18]"},{"why":"The unreasonable layer-removal heuristic is compared against the DP layer selection in the ablation study.","marker":"[33]"},{"why":"MMLU supplies both the calibration metric for layer selection and a key evaluation benchmark for factual knowledge preservation.","marker":"[49]"},{"why":"Vicuna 7B v1.5 serves as the second model on which AmoebaLLM is validated against the same baselines.","marker":"[41]"}],"fun_headline_variants":["One fine-tune, any-shape subnets, instant deploy","Any-shape LLM subnets from a single fine-tune","AmoebaLLM: instant any-depth, any-width subnets","Pull any-shape LLM out of one fine-tuned base","One-time tuning yields all-purpose LLM subnets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole approach rests on the key hypothesis in Section 3.2 that, because transformer layers are residual, choosing which layers to remove can be treated as a set of approximately independent sub-problems; if layer contributions interact strongly, the dynamic program's choices are not near-optimal and every subnet derived from them inherits the error.","fun_headline_variants_meta":{"raw":{"variants":["One fine-tune, any-shape subnets, instant deploy","Any-shape LLM subnets from a single fine-tune","AmoebaLLM: instant any-depth, any-width subnets","Pull any-shape LLM out of one fine-tuned base","One-time tuning yields all-purpose LLM subnets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000261,"raw_usage":{"total_tokens":1626,"prompt_tokens":1010,"completion_tokens":616,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":626,"completion_tokens_details":{"reasoning_tokens":527}},"tokens_in":626,"tokens_out":616,"duration_ms":12680,"temperature":1.0,"reasoning_tokens":527,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:30:30.514959+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a small transformer (or a smaller random seed of LLaMA), run the DP layer selection, and compare its chosen layer sets against exhaustive search over all subsets for each removal count on the calibration metric; if the DP-selected sets are meaningfully worse than the exhaustive optimum, or if the gains of jointly removed layers diverge from the sum of individual gains, the independence assumption fails.","supporting_citations":[{"cited_title":"Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023","cited_arxiv_id":null,"evidence_quote":"Vicuna 7B v1.5 serves as the second model on which AmoebaLLM is validated against the same baselines."}],"review_version":1}