{"id":"bd0f2817-abe2-43d4-9edb-2e9bdcc6a060","arxiv_id":"2510.22228","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Removing just one or two layers from a reasoning-tuned LLM destroys its test-time scaling, and neither LoRA nor full fine-tuning restores it.","lead":"This paper removes one or two layers from reasoning-focused AI models and finds that their ability to improve by thinking longer collapses, even though ordinary knowledge tests still look fine. The result matters because layer pruning is a popular way to make models cheaper, and the damage persists after standard fine-tuning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated LLM-as-a-Judge threatens all reported accuracy curves; exact-match AIME24 check is missing.","rationale":"The reader’s weakest assumption is the unvalidated LLM-as-a-Judge, and I agree that this is the single most load-bearing concern. Every headline accuracy number depends on an automated judge that has not been checked against exact answers or human labels. The paper’s own qualitative examples show pruned models producing repetitive, looping text, which is precisely the kind of output where LLM judges can be unreliable. Since AIME24 has deterministic integer answers, the absence of an exact-match cross-check is a concrete, testable gap. The LoRA learning-rate selection on evaluation datasets is a lesser concern; if anything, it biases toward recovery, so the failure to recover remains meaningful. No error bars are another weakness, but it is secondary to the judge validity. The LaCo parallel-scaling improvement is a nuance that qualifies the abstract’s broad claim, but it does not threaten the main sequential-scaling conclusion if the judge is validated. I therefore see no reason to move the reader’s CONDITIONAL verdict; the paper should be accepted with the condition that the judge be validated, ideally via exact match on AIME24.","tokens_in":30956,"tokens_out":5156,"duration_ms":54383,"concrete_test":"Use the released code to regenerate AIME24 outputs for s1.1-7B and Qwen3-8B at all pruning depths and thinking budgets (512–8192). Parse the final numerical answer from each output (e.g., last \\boxed{} expression or 'the answer is X') and compute exact-match accuracy for the 30 AIME24 problems. Compare these exact-match curves to the LLM-as-a-Judge curves in Figures 2 and 7. If the exact-match curves show the same collapse for 1- and 2-layer pruned models, the judge concern is resolved; if not, the claim needs re-evaluation. As a supplementary check, have a human annotate a stratified sample of 100 outputs (including repetitive traces) to measure judge agreement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"All headline scaling curves (Figures 1, 2, 7) rely on LLM-as-a-Judge via GLM-4.5-Flash with no reported agreement against exact answers or human labels. The paper itself documents that pruned models fall into repetitive loops and self-contradiction (Figure 5, Appendix G). Such outputs are exactly what an LLM judge may systematically under-grade, even when a correct final answer is present. AIME24 has deterministic integer answers and exact-match grading is standard; its omission leaves the central 'collapse' result undisambiguated from judge bias. The 3-seed averages are reported without error bars, so the magnitude of the collapse (e.g., AIME24 from ~0.14 to ~0.04) could also be within noise on only 30 problems, though the qualitative examples suggest a real effect. The judge concern is load-bearing because if exact-match curves do not reproduce the collapse, the central claim weakens substantially. Secondary issue: LaCo's pass@k in Figure 3 actually improves after pruning, so the abstract's unqualified 'pruning harms test-time scaling' is overbroad; but the main sequential-scaling claim still hinges on judge validity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether layer pruning, which is known to preserve performance on knowledge-heavy benchmarks, preserves test-time scaling in long-chain reasoning. The authors prune s1.1-7B and Qwen3-8B with three methods (ShortGPT, Reverse-order, LaCo), removing one or two layers, and measure accuracy as a function of thinking-token budget (512–8192) on MATH500, GPQA Diamond, and AIME24, together with pass@k on AIME24. They report that even one-layer pruning flattens or collapses sequential scaling curves, that two-layer pruning is worse, that LoRA and full fine-tuning do not restore the curves, and that pruned models fall into repetitive reasoning loops. They conclude that layer pruning damages the structural preconditions of test-time scaling and call for pruning methods that preserve reasoning robustness. Code is open-sourced.","tokens_in":31194,"tokens_out":5857,"duration_ms":55730,"significance":"If the reported effect is real, the paper identifies an important failure mode missed by MMLU-style evaluations of pruning: test-time scaling, a central mechanism for modern reasoning models, can break under minimal depth reduction. The experimental matrix (two models, three pruning strategies, three benchmarks, both sequential and parallel scaling) is a genuine strength, as is the open-sourced codebase and the qualitative failure analysis. However, the headline accuracy curves depend on an unvalidated LLM-as-a-Judge and are reported without variance estimates. These two issues affect the empirical core of the paper; the qualitative conclusion is plausible but not yet established at the quantitative level claimed.","major_comments":[{"comment":"All sequential scaling results for MATH500 and AIME24 (Figures 2, 4, 7, and related figures) are obtained with LLM-as-a-Judge using GLM-4.5-Flash, but the paper reports no validation of this judge: no human agreement, no exact-match comparison, and no error analysis. The statement that this is 'a more robust evaluation than exact string matching' is asserted, not demonstrated. This is a load-bearing issue because the paper's own outputs (Section 5.1, Figure 5, Appendix G) show pruned models producing repetitive, self-contradictory text; a judge can systematically under-grade such outputs even when a correct final answer is present. AIME24 has deterministic integer answers and exact-match grading is standard. Please report exact-match AIME24 curves (and ideally a judge/human agreement study on a sample) to show that the collapse is not partly an evaluation artifact.","section":"Section 3, Evaluation datasets and metrics"},{"comment":"The paper states that experiments use three seeds and report average results, but no error bars, standard deviations, or per-seed values are given in Figures 2, 7, or elsewhere. This is important for AIME24, which has only 30 problems: a difference of 0.033 corresponds to one question, and the reported AIME24 collapse (e.g., from ~0.14 to ~0.04 after one layer in Figure 2) is only a few questions wide. With three seeds, the standard error around these points is non-negligible. The qualitative pattern across many curves is reassuring, but the quantitative magnitude of the collapse is not supported without variance reporting or per-seed data.","section":"Section 3, Sequential scaling"},{"comment":"The LoRA fine-tuning procedure selects the learning rate by evaluating on MATH500, GPQA Diamond, and AIME24 and then reports results on 'the corresponding datasets.' As written, this reads like model selection on the test set, which makes the reported LoRA curves optimistically biased. I do not think this threatens the main conclusion, because the conclusion is that SFT does not recover scaling even in the best case, but the methodology needs clarification: was a separate validation split used, or were the test sets used for learning-rate selection? If the latter, the SFT results should be recomputed with a held-out selection criterion or explicitly described as an oracle selection.","section":"Section 4 and Appendix D"}],"minor_comments":[{"comment":"The abstract's unqualified statement that pruning 'even one or two layers can severely impair test-time scaling' is too broad: Figure 3 shows that LaCo with two layers pruned improves pass@k on AIME24 relative to the unpruned model. The body text does qualify this, but the abstract should be adjusted to reflect that the impairment is method- and scaling-mode-dependent.","section":"Abstract and Section 3, Figure 3"},{"comment":"The text says 'ShortGPT and LaCo consistently exhibit markedly higher Self-BLEU scores,' but Table 1 shows Reverse-order pruned models have Self-BLEU scores below the original (e.g., s1.1-7B: 0.685 -> 0.521 and 0.442). Since this table is used to support the reduced-diversity mechanism, the claim should be qualified or the analysis should be aligned with the data.","section":"Table 1"},{"comment":"The self-reflection heuristic counts are produced by gpt-4o-mini as an automated judge, but no validation or agreement measure is reported for this rubric. This is an analysis-level result rather than the central claim, but a brief note on reliability would strengthen it.","section":"Section 5.2, Table 2"},{"comment":"Some table entries appear implausible or contain apparent artifacts (e.g., 'Qwen3-8B LaCo 1 layer' with a value of 0.081 at lr=4e-5 while neighboring rows are ~0.9). Please check the table formatting and ensure the numbers are accurately transcribed.","section":"Appendix D, Table 6"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important question, and the experimental design is broad. The main blocker is the unvalidated LLM-as-a-Judge combined with the absence of variance reporting on small benchmarks; both are fixable within the manuscript's scope. If the authors add exact-match AIME24 results and per-seed/variance information, I would likely be willing to accept after another round. I do not see circularity or invented quantities in the accuracy measurements. The LaCo parallel-scaling improvement is a nuance, not a fatal inconsistency."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper gives the pruning crowd a new measurement lens, and the qualitative finding is probably right. Pruning one or two layers flattens or collapses the accuracy-versus-thinking-tokens curve on MATH500, GPQA, and AIME24, even when knowledge benchmarks look fine. That is a real contribution relative to the prior pruning literature, which mostly reports MMLU/HellaSwag accuracy and leaves reasoning scaling unexamined. The protocol is also honest: two models, three pruning methods, three benchmarks, code released. Where it earns credit: the brute-force layer ablations show the effect is distributed rather than localized to one lucky layer, and the qualitative examples of repetitive loops give a plausible mechanism rather than just a number. The consistency across ShortGPT, Reverse-order, and LaCo makes the central claim solid enough to build on. Soft spots, in proportion. The biggest is the unvalidated LLM-as-a-Judge. All the headline scaling curves go through GLM-4.5-Flash, and the paper never reports agreement with exact answers or human labels. AIME24 has deterministic integer answers; exact-match is standard and cheap. Without that check, the reader cannot fully rule out that the judge systematically underrates long, repetitive pruned-model outputs. The stress-test note has this right, and it is load-bearing only in the sense that the numbers could shift; the qualitative examples suggest the collapse is real, so I would not expect the conclusion to flip. Second, no error bars or variance despite three seeds, which matters on AIME24's 30 problems. Third, LoRA learning rate selection happens on the evaluation datasets, which leaks tuning signal into the reported results, though the full-FT results partially mitigate that. Fourth, the abstract's unqualified 'pruning harms test-time scaling' is overbroad: LaCo's pass@k on AIME24 improves after pruning in Figure 3, so merging-based parallel scaling does not always suffer. The SFT-recovery claim is also limited by the single 1K-sample s1K dataset; calling that 'fail to recover' is too strong for a small-scale study. Bottom line: this paper deserves a serious referee. The soft spots are fixable in revision—add exact-match AIME24, report error bars, change the SFT conclusion's scope, and temper the abstract. The core finding is new and likely correct. I would bring it to a reading group precisely because the methodology conversation about LLM-as-a-Judge is worth having, and I would cite it when writing about pruning or test-time scaling.","headline":"Useful empirical probe showing layer pruning collapses sequential test-time scaling, but the evaluation stack needs validation before the numbers are taken at face value.","tokens_in":733,"tokens_out":791,"would_cite":true,"duration_ms":21055,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Layer pruning that looks harmless on knowledge benchmarks can destroy a language model's ability to reason better with more thinking time, and standard fine-tuning cannot bring that ability back.","keywords":["layer pruning","test-time scaling","long-chain reasoning","sequential scaling","parallel scaling","supervised fine-tuning","reasoning loops","Self-BLEU"],"falsifier":"Run AIME24 after one-layer pruning and score it two ways: exact match of the final integer answer (AIME answers are integers) and the LLM-as-a-Judge method. If exact-match accuracy holds steady while judge scores collapse, the reported collapse is partly a scoring artifact; for MATH500 and GPQA Diamond, human grading of the same outputs would provide the same check.","tokens_in":30847,"feed_emoji":"✂️","tokens_out":3938,"duration_ms":38968,"temperature":0.7,"pith_summary":"This paper argues that test-time scaling — the ability of a reasoning model to improve as it spends more tokens thinking — is the capability most damaged by layer pruning, and that this damage is invisible to knowledge benchmarks. Removing just one or two layers from a 7B or 8B reasoning model flattens or collapses the accuracy-versus-thinking-token curve on math and science reasoning tasks, even when MMLU-style performance barely moves. The paper further shows that neither LoRA nor full-parameter fine-tuning recovers this scaling, and traces the failure to structural damage: pruned models fall into repetitive loops, produce less diverse reasoning trajectories, and exhibit fewer self-correction behaviors. The upshot is a trade-off: efficiency gains from depth pruning come at the cost of the mechanism that makes long reasoning work.","feed_headline":"Pruning one layer can collapse test-time reasoning gains","feed_subtitle":"Accuracy no longer improves with more thinking tokens after minimal pruning, and fine-tuning cannot restore the curve.","key_machinery":"The accuracy-versus-thinking-token curve (sequential test-time scaling) is the probe that exposes the fragility; pruning methods — ShortGPT (Block Influence based), Reverse-order (removing deep layers), and LaCo (layer collapsing) — are the interventions. Self-BLEU and counts of verification, backtracking, and subgoal behaviors provide diagnostics that link the scaling collapse to repetitive loops and lost diversity rather than to simple knowledge erasure.","core_discovery":"The central claim is that sequential test-time scaling is extremely fragile to layer pruning. Across three pruning methods and two reasoning models, pruning one layer substantially impairs the accuracy gains normally obtained by increasing the thinking-token budget, and pruning two layers nearly eliminates them, with AIME24 accuracy approaching zero. Parallel scaling (pass@k) is more mixed: direct removal hurts it, but the merging-based method LaCo sustains or even improves it. Supervised fine-tuning delivers only marginal or negative gains for one-layer pruned models and, while it lifts two-layer pruned models somewhat, it does not restore the original scaling. The paper attributes the brea","pith_inferences":["If the collapse is structural (looping) rather than knowledge loss, other depth-compressing operations — quantization, layer merging at larger scale, or even architectural changes that shorten the residual chain — may show the same flattening of scaling curves; a cheap test is to run the same benchmarks on quantized versions of the same models.","The automated judge used for MATH500, GPQA Diamond, and AIME24 is never validated against ground truth or human labels. Pruned models produce long, repetitive outputs, so a judge trained on fluent text could systematically under-grade them; scoring AIME24 by exact match of its integer answers would settle this.","The qualitative loops suggest pruning may disrupt the internal 'budget forcing' that makes longer thinking productive, not just the knowledge layers. If so, scaling curves could serve as a general diagnostic for whether a compression method preserves reasoning machinery, not only in pruning.","LaCo's preserved parallel scaling hints that merging-based compression might be combined with sampling-based inference search to recover some of what sequential scaling loses, a direction the paper does not explore."],"forward_implications":["Pruned models should not be assumed safe for reasoning workloads: their accuracy will stop improving — or worsen — as inference-time compute increases, even at pruning depths that leave knowledge benchmarks stable.","Reported retention on knowledge benchmarks after pruning is not evidence of reasoning preservation; evaluation of pruning methods should include test-time scaling curves.","Neither LoRA nor full-parameter fine-tuning is a reliable remedy once test-time scaling has been degraded; pruning pipelines for reasoning models need a different recovery mechanism.","Merging-based pruning behaves differently from direct removal, retaining parallel test-time scaling in the tested settings, which singles it out as a more promising direction for efficiency without sacrificing reasoning.","Because most individual layers contribute non-trivially to test-time scaling, the common practice of pruning a few layers by importance scores is not a harmless shortcut for reasoning models."],"fun_headline_variants":["Pruning a single layer breaks LLM reasoning scaling","Even one pruned layer kills LLM reasoning gains","Test-time scaling collapses after minimal layer pruning","Fine-tuning can't restore LLM reasoning after pruning","Why layer pruning ruins LLM test-time scaling"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire collapse story rests on scores assigned by an automated judge model that the paper never validates against ground-truth answers or human labels on the reasoning benchmarks.","fun_headline_variants_meta":{"raw":{"variants":["Pruning a single layer breaks LLM reasoning scaling","Even one pruned layer kills LLM reasoning gains","Test-time scaling collapses after minimal layer pruning","Fine-tuning can't restore LLM reasoning after pruning","Why layer pruning ruins LLM test-time scaling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000232,"raw_usage":{"total_tokens":1334,"prompt_tokens":762,"completion_tokens":572,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":499}},"tokens_in":506,"tokens_out":572,"duration_ms":4985,"temperature":1.0,"reasoning_tokens":499,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T08:08:36.204749+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run AIME24 after one-layer pruning and score it two ways: exact match of the final integer answer (AIME answers are integers) and the LLM-as-a-Judge method. If exact-match accuracy holds steady while judge scores collapse, the reported collapse is partly a scoring artifact; for MATH500 and GPQA Diamond, human grading of the same outputs would provide the same check.","supporting_citations":[],"review_version":1}