{"id":"7ab9b2a0-48e6-4cc4-bf79-adba755c4292","arxiv_id":"2502.07780","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"DarwinLM uses evolutionary search with training-aware offspring selection to prune LLMs, beating ShearedLlama with 5x less post-training data.","lead":"This paper introduces DarwinLM, a method to shrink large language models using evolutionary search and brief training of candidate pruned models. It reports that a 2.7B-parameter pruned Llama-2-7B beats ShearedLlama's 2.7B model while using 10B training tokens instead of 50B.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training-aware selection is validated only at a 2M-token horizon, not the 10B-token post-training used for the headline result, so the core mechanism's predictive claim is untested at the relevant scale.","rationale":"The reader's weakest assumption—that short-horizon training and KL selection predict full post-training performance—is also the most load-bearing weakness I can find. My reading sharpens it: the only validation (Figure 2) uses 2M tokens as 'full training,' whereas the central comparison uses 10B tokens, a 5000x gap. This is not a matter of disagreement with community consensus; it is an internal gap between the evidence offered and the scale at which the claim is made. The empirical headline itself is also supported by a single 0.2-point margin with no error bars, but that alone is less structurally important than the scale mismatch in the mechanism. Other issues, such as the Table 3 copy error and the 16-vs-2048 calibration-sequence inconsistency, affect reproducibility but do not directly threaten the central claim. I agree with the reader that conditional acceptance is appropriate; the concern does not move the verdict because it is the same concern the reader already identified, but it should be stated precisely and verified before the claim is treated as settled.","tokens_in":17096,"tokens_out":6027,"duration_ms":52380,"concrete_test":"Run the Table 4 ablation at the full 10B-token post-training scale: take the structures selected with and without training-aware selection from the Llama-2-7B target-level-5 search, train each on 10B Fineweb-Edu tokens using the protocol in Table 5, and evaluate the full benchmark suite from Table 1. If the TAS-selected model does not beat the non-TAS model by more than benchmark noise, then the paper has no evidence that the selection mechanism improves the final 10B-token model, and the central claim loses its mechanistic support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim—DarwinLM-2.6B trained on 10B tokens beats ShearedLlama-2.7B trained on 50B tokens—depends on training-aware selection (TAS) actually identifying the offspring that benefits most from the real post-training run. Section 3.5 states that 'training on little data is a good predictor of more large-scale training' and cites Figure 2 as its only demonstration. But Figure 2's y-axis is 'Full Training KL-Div' after only 2M tokens: the text says 'We first use 2M tokens to train all offspring as a “large-scale” full training.' The actual post-training in Table 1 and Section 4.1 is 10B tokens. Thus the paper's only direct evidence for the selection assumption covers a 2M-token horizon, 5000x shorter than the horizon at which the central claim is evaluated. Offspring rankings can plausibly reorder between 2M and 10B tokens, and no experiment in the paper checks this. The TAS ablation in Table 4 trains only to 1B tokens, still 10x short, and reports only PIQA, SciQ, and ArcE without error bars. Absent evidence that 10K–200K-token KL rankings predict 10B-token downstream accuracy, the load-bearing component of the method is unvalidated at the relevant scale.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DarwinLM, a structured pruning method for LLMs that combines second-order layer-wise pruning with an evolutionary search. A sparsity-level database is built with OBS-style structured pruning, and offspring models are generated by sparsity-preserving level-switch mutations. A multi-step training-aware selection (TAS) process trains candidates on progressively larger token budgets (10K-200K) and selects survivors by KL divergence against the dense model. The main empirical claims are: on Llama-2-7B, DarwinLM at 2.6B parameters with 10B post-training tokens reaches 62.8 average downstream accuracy, surpassing ShearedLlama's 62.6 at 2.7B and 50B tokens; on Llama-3.1-8B (4.6B) and Qwen-2.5-14B-Instruct (8.4B) it outperforms ZipLM and matches or exceeds OLMo baselines; and in one-shot settings it beats uniform pruning, ZipLM, EvoPress, ShortGPT, and Shortened-Llama across sparsities. Code and weights are released.","tokens_in":17396,"tokens_out":7901,"duration_ms":62279,"significance":"If the claims hold, the contribution is practically significant: a hardware-agnostic structured pruning pipeline that matches or exceeds ShearedLlama with roughly 5x less post-training data, and a sensible integration of evolutionary search with training-aware selection. The release of code and weights is a clear strength for reproducibility. However, the current manuscript contains unresolved data inconsistencies and validates the core selection mechanism at a token scale far below the scale used in the headline experiments, so the significance cannot be fully assessed without revision.","major_comments":[{"comment":"The assumption that \"training on little data is a good predictor of more large-scale training\" is demonstrated only against a 2M-token \"full training\" in Figure 2 and a 1B-token ablation in Table 4, while the headline results in Tables 1 and 2 use 10B tokens. This is a 5000x gap between the validation and deployment horizons, and no evidence is provided that offspring rankings are stable from 2M to 10B tokens. Since TAS is the paper's core claimed novelty, this gap is load-bearing; please add a validation at the actual post-training scale (e.g., rank correlation between selection-step KL and final 10B-token accuracy, or at least a 2B/5B intermediate checkpoint) or explicitly narrow the claim to short-horizon prediction.","section":"§3.5, Figure 2, Table 4"},{"comment":"The pseudocode does not implement generational evolution as described in the text. The mutation loop runs before the selection loop, and the variable `candidates` is reset to `[parent]` at each iteration of the outer loop, so selection is applied only to the offspring of the last generation. The text states that \"The final surviving candidate is selected as the starting point for the next generation,\" which is not reflected in Algorithm 1. As written, the algorithm is ambiguous and not reproducible; please correct the pseudocode or the text so that the iterative parent-update process is unambiguous.","section":"Algorithm 1 / §3.5"},{"comment":"The Qwen-2.5-14B-Instruct row in Table 3 lists PIQA 69.4, WG 57.3, ArcC 34.2, and HS 44.6, which are exactly the Llama-3.1-8B values from the same table and conflict with the Qwen one-shot row in Table 2 (PIQA 73.9, WG 60.5, ArcC 48.0, HS 53.3). This appears to be a copy-paste error and must be corrected. Additionally, the calibration data size is reported as \"at most 16 sequences\" in §4.1, \"2048 calibration samples\" and \"2K calibration data\" in the Implementation Details, and \"2,048 sequences with 4,096 tokens\" in Appendix A.1. These conflicting numbers need to be reconciled, since the lightweight nature of the search depends on which value is correct.","section":"Table 3, §4.1, Appendix A.1"},{"comment":"For Qwen-2.5-14B-Instruct, the text states that \"After post-compression training with only 2B tokens, the performance of DarwinLM increases to 68.1,\" but Table 2 reports only a 10B-token row with an average of 68.4. The 2B result is neither shown in the table nor otherwise documented, so the claim cannot be verified. In addition, Tables 1 and 2 report single-run evaluations without error bars; given that the headline advantage over ShearedLlama is 0.2 average points (62.8 vs 62.6), variance estimates or multiple evaluation runs would strengthen the comparison.","section":"§4.2 / Table 2"}],"minor_comments":[{"comment":"The caption contains the typo \"Offsping Generation\"; it should be \"Offspring Generation\".","section":"Figure 1 caption"},{"comment":"The token budget \"8196\" should likely be \"8192\"; the number of selection steps and survivor counts are also inconsistent (3 steps with [8,4,1] in §3.5/Figure 2 versus 4 steps with [8,4,2,1] in §4.1 and Appendix A.1). Please make these details consistent across the paper.","section":"§3.5, §4.1, Appendix A.1"},{"comment":"The text refers to \"Qwen-2.5-8B-Instruct\" when the experiments use Qwen-2.5-14B-Instruct; correct the model name.","section":"Appendix A.1"},{"comment":"The conclusion states that prior methods required \"10x more data and training computation,\" while the abstract and main text emphasize a 5x reduction compared to ShearedLlama; please clarify which comparison the 10x figure refers to.","section":"Conclusion"},{"comment":"Calling 2M tokens a \"large-scale full training\" is misleading given that the paper's post-training uses 10B tokens; consider renaming it to \"2M-token training\" or similar to avoid confusion.","section":"§3.5 / Figure 2"},{"comment":"The TAS ablation reports only PIQA, SciQ, and ArcE without error bars; since this table supports a central methodological claim, please report the full benchmark suite or justify the subset and indicate whether the observed gaps are robust.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has real strengths: a well-motivated search pipeline, a concrete evolutionary mechanism, and released code and weights. However, the Table 3 copy-paste error, the calibration-size contradiction, and the Algorithm 1/text mismatch are integrity-level issues that must be fixed before acceptance. The most substantive scientific concern is the mismatch between the 2M-token validation of TAS and the 10B-token evaluation: without evidence of ranking stability at the relevant scale, the central claim that TAS selects the best offspring for real post-training is not established. I also suspect the Qwen \"2B tokens, 68.1\" result in §4.2 is from an earlier experiment that was not included in Table 2; the authors should reconcile the text and table. My recommendation is major_revision, and I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"DarwinLM is a well-engineered structured pruning paper with one genuinely strong result: a 2.6B Llama-2 model that beats ShearedLlama-2.7B on average downstream accuracy after only 10B tokens of post-training versus 50B. The comparison against ShearedLlama retrained on the same 10B tokens is fair and the checkpoints are released, which puts the claim on solid ground. The method itself is a clean combination of second-order OBS-style pruning databases, an evolutionary search with sparsity-preserving level-switch mutations, and a multi-step training-aware selection stage. The level-switch mutation and the staged selection with increasing token budgets are new relative to ZipLM and EvoPress, and Figure 2 gives a clear motivation.\n\nThe soft spots are real but not fatal. The biggest one: the training-aware selection mechanism is argued with 'training on little data is a good predictor of more large-scale training,' but the only direct demonstration, Figure 2, compares selection steps against a 'full training' run of 2M tokens. The actual post-training in the main experiments is 10B tokens. The ablation in Table 4 goes to 1B tokens at most. So the core predictive claim is untested at the scale at which the headline result is evaluated. Rankings of offspring can plausibly reorder between 2M and 10B. I would not block the paper over this, because the main result may not depend critically on TAS, but the paper should either soften the claim or add re-ranking evidence from the 10B runs.\n\nThere are also two clear data errors. Table 3 lists identical numbers for Qwen-2.5-14B-Instruct and Llama-3.1-8B, which is a copy-paste mistake. And the calibration set is described as '16 sequences' in Section 4.1 but '2,048 sequences with 4,096 tokens' in Appendix A.1. Those need fixing. Also no error bars or multiple seeds; for a stochastic search that's a minor concern given the compute, but worth noting.\n\nOverall, the paper is a solid contribution to LLM pruning. The empirical comparisons are mostly fair, the code and weights are released, and the method is clearly specified. It deserves a serious peer review. I'd suggest accepting it conditionally after the data errors are corrected and the TAS validation gap is addressed, either with new experiments or a more modest claim.","headline":"Solid pruning paper with a real headline result, but the training-aware selection at its core is only validated at a 2M-token horizon, not the 10B tokens used in the main experiments.","tokens_in":17929,"tokens_out":2759,"would_cite":true,"duration_ms":23090,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DarwinLM treats structured pruning as an evolutionary search, selects candidates by how well they recover under brief fine-tuning, and produces a 2.6B model that beats a baseline trained with five times more data.","keywords":["structured pruning","evolutionary search","large language models","training-aware selection","non-uniform sparsity allocation","second-order pruning","KL-divergence fitness","post-training recovery"],"falsifier":"Replicate the Llama-2-7B search with a larger offspring population, say 64 per generation, and measure the rank correlation between KL divergence after the 200K-token selection step and average benchmark accuracy after full 10B-token post-training. If the correlation is weak for larger populations, or fails to hold for Llama-3.1-8B or Qwen-2.5-14B-Instruct, the training-aware selection claim is not general. A cheaper check that also isolates the contribution of search: post-train the best one-shot-KL structure and the best training-aware structure under identical 10B-token recipes across several seeds; if the one-shot structure matches the evolved one, the central claim fails. The paper's own Table 4 runs this comparison only at 1B tokens, so the 10B-token version is an open experiment.","tokens_in":16897,"feed_emoji":"🧬","tokens_out":14571,"duration_ms":103295,"temperature":0.7,"pith_summary":"DarwinLM is a method for structured pruning of large language models that treats the search for the best non-uniform sparsity allocation as an evolutionary process. Starting from a uniformly pruned parent, it generates candidate offspring by shifting one sparsity step from one module to another, briefly fine-tunes each candidate on a tiny token budget, and keeps the survivors that look most recoverable. The paper's central claim is that this training-aware selection finds structures that recover better under continued training: a 2.6B-parameter pruned Llama-2-7B averages 62.8 across eight benchmarks, beating ShearedLlama-2.7B (62.6) while using 10B tokens of post-training data instead of 50B. A careful reader should care because structured pruning delivers real speedups on ordinary hardware, and the paper argues that the right search over sparsity structure is worth more than extra recovery data.","feed_headline":"A 2.6B pruned model beats a rival trained on 5x more data","feed_subtitle":"DarwinLM evolves sparsity patterns by how fast they recover from brief fine-tuning, then wins with far less data.","key_machinery":"The load-bearing machinery is a sparsity-level database combined with a sparsity-preserving mutation operator and a training-aware selection loop. The database stores, for every attention and MLP module, a set of pre-pruned versions at discrete sparsity levels, each produced by second-order structured pruning: an OBS-style computation that uses the inverse Hessian of the layer inputs to choose which heads or intermediate columns to remove and to update the remaining weights. A candidate model is stitched together by picking one sparsity level per module, and a level-switch mutation raises the sparsity of one module by one step while lowering another by the same step, so total sparsity is conserved and every offspring automatically satisfies the target constraint. Selection then fine-tunes all candidates on progressively larger small token budgets (10K, 50K, 100K, then 200K tokens) and keeps the top survivors (8, 4, 2, then 1) ranked by KL divergence between the sparse candidate's outputs and the dense model's outputs on calibration data. The claim that rests on this machinery is that KL divergence measured after a brief fine-tune ranks candidates in the same order as their performance after full 10B-token post-training.","core_discovery":"DarwinLM's central claim is that the one-shot best sparse structure and the structure that performs best after continued training are different things, and that the latter can be identified before committing to expensive training. The paper shows that a lightweight multistep fine-tuning of offspring candidates, 10K, 50K, 100K, then 200K tokens, followed by ranking via KL divergence between dense and sparse outputs, picks out the candidates that recover best under full 10B-token post-training (Figure 2). With this selection, the evolved 2.6B Llama-2 model beats ShearedLlama in average downstream accuracy while using five times fewer tokens for recovery; the 4.6B Llama-3.1 model beats OLMO-7B, a dense model trained on roughly 250 times more data; and the 8.4B Qwen-2.5 model reaches 68.4 average after only 10B tokens of continued training. In the one-shot setting, the same search outscores ZipLM on every model tested, on Llama-2 at 2.7B parameters versus ZipLM's 4.0B, on Llama-3.1, and on Qwen-2.5, and far outscores uniform pruning at similar sizes.","pith_inferences":["The selection mechanism is really a predictor of recoverability, so the same multistep fine-tuning and ranking recipe could be reused to choose among other post-training decisions, such as data mixtures, learning-rate schedules, or which structure to keep under a fixed training budget, not just sparsity allocations.","The 5x data-efficiency claim rests on a rank correlation shown in a single 16-offspring experiment (Figure 2); whether the correlation holds at larger population sizes and on Llama-3.1 and Qwen is untested, and is the most direct place to probe the method.","Because the mutation operator conserves total sparsity, the fitness function is swappable: replacing KL divergence with a latency or memory measurement would let the same evolutionary loop optimize sparsity for specific hardware targets rather than for accuracy alone."],"forward_implications":["On Llama-2-7B, the 2.6B DarwinLM model reaches 62.8 average accuracy with 10B recovery tokens, above ShearedLlama's 62.6 with 50B tokens, so a better compressed model can be produced at one fifth of the recovery-training cost.","The 4.6B model pruned from Llama-3.1-8B scores 63.7 average, beating the 7B dense OLMO model (62.9) trained on roughly 2.5T tokens, which indicates that pruning a strong parent plus modest recovery can outperform much larger scratch-trained models.","The method transfers to instruction-tuned models: the 8.4B Qwen-2.5 variant reaches 68.4 average after only 10B tokens, versus 56.9 for uniform pruning at a similar size.","The one-shot results isolate the value of the search itself: non-uniform allocation scores 57.2 on Llama-2 at 2.7B parameters, where uniform pruning collapses to 40.1 at 3.4B parameters."],"supporting_citations":[{"why":"Supplies the ZipLM second-order structured pruning procedure that DarwinLM builds on to create its sparsity-level database.","marker":"Kurtić et al., 2024"},{"why":"EvoPress contributes the evolutionary search framing, the KL-divergence fitness metric, and the multistep selection schedule that DarwinLM adapts.","marker":"Sieberling et al., 2024"},{"why":"ShearedLLaMA is the primary baseline; its 2.7B model, 50B-token post-training, and benchmark protocol define the 5x-data-efficiency claim.","marker":"Xia et al., 2024"},{"why":"Optimal Brain Surgeon provides the second-order weight-update formula underlying the structured pruning masks.","marker":"Hassibi & Stork, 1992"},{"why":"The Optimal BERT Surgeon extends OBS to structured row/column pruning, the technical basis for the per-module masks in the database.","marker":"Kurtic et al., 2022"},{"why":"Llama-2-7B is the testbed model on which the paper's strongest comparative claim against ShearedLlama is made.","marker":"Touvron et al., 2023"}],"fun_headline_variants":["2.6B pruned model beats 5x-data rival via evolution","DarwinLM: evolve pruned LLMs that bounce back fastest","5x less data, same win: DarwinLM evolves best pruned models","Survival of the fittest: DarwinLM picks pruned models that recover"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that after a candidate is fine-tuned on just 10,000 to 200,000 tokens, its KL divergence from the dense model reliably predicts which candidate will be best after full 10B-token post-training; the paper demonstrates this correlation in a single 16-offspring experiment on Llama-2-7B (Figure 2).","fun_headline_variants_meta":{"raw":{"variants":["2.6B pruned model beats 5x-data rival via evolution","DarwinLM: evolve pruned LLMs that bounce back fastest","5x less data, same win: DarwinLM evolves best pruned models","Survival of the fittest: DarwinLM picks pruned models that recover"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000303,"raw_usage":{"total_tokens":1801,"prompt_tokens":1063,"completion_tokens":738,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":679,"completion_tokens_details":{"reasoning_tokens":657}},"tokens_in":679,"tokens_out":738,"duration_ms":7953,"temperature":1.0,"reasoning_tokens":657,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T11:34:18.781094+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replicate the Llama-2-7B search with a larger offspring population, say 64 per generation, and measure the rank correlation between KL divergence after the 200K-token selection step and average benchmark accuracy after full 10B-token post-training. If the correlation is weak for larger populations, or fails to hold for Llama-3.1-8B or Qwen-2.5-14B-Instruct, the training-aware selection claim is not general. A cheaper check that also isolates the contribution of search: post-train the best one-shot-KL structure and the best training-aware structure under identical 10B-token recipes across several seeds; if the one-shot structure matches the evolved one, the central claim fails. The paper's own Table 4 runs this comparison only at 1B tokens, so the 10B-token version is an open experiment.","supporting_citations":[],"review_version":1}