{"id":"4e894a5f-9390-4894-a502-9fbd23c96666","arxiv_id":"2412.15077","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"TLC collapses deep networks by linearizing neurons with positive batch normalization shifts and removing those with non-positive shifts, then retraining, achieving up to 70% layer removal on some models with small accuracy loss.","lead":"A new compression method, TLC, removes whole layers from trained neural networks by using batch normalization parameters to decide which neurons can be linearized or dropped. Tests on ResNet, MobileNet, Swin, BERT and RoBERTa show large depth reductions with modest accuracy loss on several benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The per-neuron error metric in Eq. (3) does not bound the actual output perturbation caused by layer removal, because it ignores error magnitude, downstream weight scales, and error propagation through depth; the theoretical justification for TLC's threshold is therefore unverified.","rationale":"The reader identified the same load-bearing assumption: that the per-neuron misclassification probability Phi(-|beta|/gamma) is a valid measure of the output perturbation caused by linearizing or removing a neuron, and that the pre-activations are Gaussian. My concern sharpens this by noting that even if the Gaussian assumption holds, Eq. (3) is a probability, not a magnitude, and it ignores the downstream weight scale and error propagation. This is not a manufactured issue: the paper's abstract makes a general claim about maintaining performance while reducing depth, and Eq. (3) is the only analytical justification for the binary beta-sign rule. The empirical results could still be good, but the presented theory would not explain them. The concrete test I propose would settle whether the predicted per-neuron errors rank layers in the same order as actual output perturbations. If they do not, TLC's success must be attributed to the validation-based selection loop, which is a much weaker claim than the paper makes. The reader's other remarks about pseudocode, threshold, and missing error bars are valid but less central; they are implementation details that could be fixed without changing the method. This concern goes to the core argument, so I agree with the reader's conditional verdict and do not recommend changing it.","tokens_in":16806,"tokens_out":5869,"duration_ms":48016,"concrete_test":"On a fixed pretrained ResNet-18 trained on CIFAR-10, compute for each removable layer l the predicted score S_l = sum_i Phi(-|beta_l,i|/gamma_l,i). Then, for each l, apply TLC's layer-removal procedure and measure the actual output perturbation, e.g., the L2 distance between the original model's logits and the pruned model's logits averaged over the validation set, or the KL divergence of the softmax outputs. Rank layers by S_l and by actual perturbation and compute the Spearman rank correlation. Also compute the empirical fraction of validation samples where sign(z_l,i) differs from sign(beta_l,i) and compare it to Phi(-|beta_l,i|/gamma_l,i) to test the Gaussianity assumption directly. If the rank correlation is low, or if the empirical sign-mismatch rates deviate strongly from the Gaussian prediction, then Eq.","verdict_should_be":"UNCHANGED","load_bearing_attack":"TLC's central claim is that per-neuron ON/OFF decisions based on the sign of beta_l,i remove layers with minimal error. The analytical support is Eq. (3), E_l,i = Phi(-|beta_l,i|/gamma_l,i), which is the probability that a Gaussian pre-activation z_l,i ~ N(beta, gamma^2) lies on the wrong side of the rectifier. This is not a measure of output perturbation. First, it ignores error magnitude: a neuron with beta=0.1, gamma=1 has E about 0.46, but if its output feeds a large downstream weight, the induced change in the next layer can be large; conversely, a neuron with beta=4, gamma=1 has tiny E but a rare large-magnitude error can still matter. Second, the per-neuron decision is based only on the sign of beta, not on E_l,i, so two neurons with very different error likelihoods are treated identically. Third, errors are not independent and they compose through subsequent layers; the paper never verifies Gaussianity, never compares predicted E_l,i with actual changes in the next-layer activations, and never shows that the layer-removal ranking from Eq. (3) correlates with true output perturbation. Because the algorithm selects how many layers to remove using validation accuracy, the reported success could be produced by the selection loop rather than by the batch-norm-based importance metric. This is the load-bearing weakness: if Eq. (3) does not predict actual perturbation, the theoretical motivation collapses and TLC reduces to a validation-guided heuristic with an unsupported per-neuron rule.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TLC (Till the Layers Collapse), a depth-compression method for deep neural networks that uses batch normalization parameters to decide, per neuron, whether to remove the neuron (OFF, β≤0) or linearize its activation and merge it into the next layer (ON, β>0). Whole layers are then removed by this surgical operation. Layer importance is ranked by the validation accuracy after individually removing each layer, and Algorithm 1 iteratively removes the least important layers, retrains, and continues until the validation accuracy falls below a relative threshold θ. The method is evaluated on ResNet-18, Swin-T, MobileNet-V2, VGG-16bn across CIFAR-10, Tiny-ImageNet, ImageNet, PACS, and VLCS, and on BERT/RoBERTa across SST-2, QNLI, and RTE, showing that many layers can be removed with small accuracy degradation and that TLC generally outperforms EGP, LF, EASIER, and magnitude/gradient-based layer removal baselines.","tokens_in":17080,"tokens_out":5742,"duration_ms":50209,"significance":"If the method holds, it provides a practical and computationally efficient way to reduce network depth, which is a well-motivated target because depth largely determines latency on modern hardware. The empirical scope is broad—five image datasets, three NLP tasks, and several architecture families—and the paper reports code release, training details, and wall-clock times, which are strengths. The method's core contribution is a cheap, post-hoc importance signal (batch norm statistics) that avoids expensive per-layer retraining during ranking. However, the analytical justification for the per-neuron ON/OFF rule is heuristic: Eq. (3) measures per-neuron sign-error probability, not actual output perturbation, and the paper does not prove or empirically verify that this metric ranks operations by true perturbation. The method's final layer-selection loop is validation-guided, so the empirical success could be driven by that selection rather than by the batch-norm-based importance signal. These gaps do not invalidate the empirical findings, but they do limit the strength of the theoretical claims.","major_comments":[{"comment":"The importance relation is stated as I(l) < I(l′) ⇔ A(M_rem{l}) < A(M_rem{l′}). This ordering says a layer is less important exactly when removing it produces a smaller validation accuracy (i.e., a larger drop), which is the reverse of what is needed for the 'ascending ranking' described in §3.4 that starts from the 'least important layer.' If taken literally, Algorithm 1 would remove the most damaging layers first, contradicting the reported success. This is likely a typo (the intended relation is probably >), but it must be corrected and the direction of the ranking should be clarified with a concrete example.","section":"§3.3, Eq. (4)"},{"comment":"The per-neuron error likelihood E_l,i is the probability that a Gaussian pre-activation lies on the wrong side of the rectifier when the neuron is linearized (ON) or zeroed (OFF). This is not a measure of the actual output perturbation induced by TLC. It ignores error magnitude (a rare large deviation can dominate), the scaling of the downstream weights, and the propagation of errors through depth. The area-under-the-curve argument in Fig. 2 only compares unweighted per-neuron misclassification probabilities; it does not bound the change in the next layer's activations or the network output. Please either provide an actual perturbation bound that tracks the change in the subsequent layer's pre-activations (accounting for weight norms and merging), or explicitly reframe Eq. (3) as a heuristic motivation and add an empirical check that the ON/OFF decision correlates with the true activation perturbation for the models used.","section":"§3.2, Eq. (3), Fig. 2"},{"comment":"The description of the layer removal and linearization/merge operation assumes a strictly sequential chain of layers (conv/FC → BN → activation → next layer). For residual architectures such as ResNet-18, Swin-T, and MobileNet-V2, the paper does not specify how the removal is performed when the target layer is part of a residual block with a skip connection. It is unclear whether the skip path is folded into the merge, whether the batch norm is absorbed, or how the 'subsequent layer' is defined in the presence of an additive shortcut. Since the reported results on these architectures are central to the paper's claims, the surgical procedure for residual blocks must be detailed (or the release code must be referenced with explicit pointers).","section":"§3.2, §3.4, Fig. 1"},{"comment":"The baselines 'Smallest weights' and 'Smallest gradients' are not precisely defined. It is not stated whether they remove the layer with the smallest sum of absolute weight values (or gradient norms) and whether the same retraining procedure as TLC is applied after each removal. Without this detail, the comparison can be unfair or unreproducible. Please specify the exact criterion and the fine-tuning protocol used for these baselines.","section":"§4.1, Table 1"}],"minor_comments":[{"comment":"The phrase 'bach norm' should be 'batch norm'.","section":"§3.1"},{"comment":"The entry 'Tiny-Inet' should be 'Tiny-ImageNet', and '82,90' uses a comma as a decimal separator; please unify the decimal notation.","section":"Table 1"},{"comment":"The header 'BERT RoBERTatop-1' is missing a space; it should be 'BERT RoBERTa top-1'.","section":"Table 2"},{"comment":"The sentence 'removing the layer with the lowest sum of weights/gradients results performs close to TLC' is grammatically awkward and should be reworded.","section":"§4.2"},{"comment":"The accuracy retention threshold θ is a free parameter but no default value is reported in the experiments. Please state the θ used for Tables 1 and 2.","section":"§3.4"},{"comment":"The green curve labeled 'TLC' is not defined in the caption; please add the formula or description of how it is computed.","section":"Fig. 2"}],"recommendation":"major_revision","confidential_remarks":"The paper has already been accepted at AAAI 2025, so the empirical claims are likely solid, but the current manuscript has a few load-bearing presentation gaps that need to be fixed for a journal publication. The most important are the reversed ordering in Eq. (4), the lack of a concrete description of residual-block handling, and the mismatch between Eq. (3) as a per-neuron sign-error probability and the paper's stronger statement that TLC minimizes the impact on performance. These are fixable without changing the method's empirical conclusions. I would not reject, but the revised version must address at least the ordering and residual-block issues explicitly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"TLC is a heuristic depth-pruning method that works well in many settings, but the paper oversells the theory and glosses over a real accuracy drop on ImageNet MobileNet-V2.\n\nThe genuinely new part is the per-neuron ON/OFF split using the sign of the BN shift beta: ON neurons are linearized and merged into the next layer, OFF neurons are removed outright. That's simple and clean, and it's distinct from entropy-based depth reduction (EGP, EASIER) and from BN filter pruning (Liu et al., Oh et al.). The experimental coverage is broad: five image datasets, three NLP tasks, and architectures including ResNet, Swin, MobileNet, BERT, and RoBERTa. In many cases TLC removes a large fraction of layers with small accuracy loss (12/17 on ResNet-18 CIFAR-10, 6/12 on RoBERTa SST-2, 8/12 on RoBERTa QNLI). It also has a real training-time advantage over EASIER.\n\nThe soft spots are real but mostly fixable. Eq. (3) is a per-neuron probability of being on the wrong side of the rectifier, not an output-perturbation bound; it ignores error magnitude, downstream weight scales, and propagation. The Gaussian assumption is asserted, not verified. The stress-test is right that the validation-driven selection loop does much of the work, but it overstates the consequence: the per-neuron rule still makes the removal safe enough to be empirically effective. The theoretical narrative is weaker than the paper suggests, but the method stands on its experiments.\n\nMechanical issues: Algorithm 1 uses AMtest before it is assigned (line 11); theta is never specified; residual architectures are asserted to work without explanation of how linearization interacts with skip connections.\n\nThe strongest factual concern is ImageNet MobileNet-V2: 59.43 vs 71.87 dense with only 2/35 layers removed, a 12.4-point drop that contradicts the abstract's 'without significantly compromising model performance.' Baselines are single runs without error bars, and the code link has no commit hash.\n\nStill, I agree with the reader's CONDITIONAL verdict. The method is practical, simple, and broadly validated. I'd send it to peer review; it deserves a serious look. The authors should fix the pseudocode, report theta and seeds, validate or de-emphasize the Gaussian assumption, and acknowledge the ImageNet case. I'd cite it as a baseline if I were working on depth pruning. I'd also bring it to a reading group for a discussion of what counts as a justification in pruning papers.","headline":"A simple, empirical depth-pruning method worth engaging, but its theoretical story is thin and one ImageNet result contradicts the abstract.","tokens_in":17703,"tokens_out":8084,"would_cite":true,"duration_ms":66803,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Batch-norm statistics can delete most layers from a deep network while keeping accuracy nearly unchanged.","keywords":["model compression","layer pruning","depth reduction","batch normalization","neural network efficiency","rectified activation","inference latency"],"falsifier":"Compute the actual change in the next layer's activations when a layer is removed via TLC, and compare it with the predicted per-neuron error sum $\\sum_i \\Phi(-|\\beta_{l,i}|/\\gamma_{l,i})$. If the actual perturbation is large when the predicted error is small, or if the predicted ranking of layers disagrees with the ranking by true accuracy loss, the central mechanism is falsified. A second, cheaper check: take a trained ResNet-18 on CIFAR-10, measure the empirical distribution of each pre-activation $z_{l,i}$ over the validation set, and test whether $z_{l,i}$ is close enough to Gaussian that $\\Phi(-|\\beta|/\\gamma)$ matches the observed fraction of wrong-side crossings for the chosen rectifier.","tokens_in":16537,"feed_emoji":"⚡","tokens_out":11475,"duration_ms":91596,"temperature":0.7,"pith_summary":"This paper claims that a deep network can be made much shallower by treating each batch-normalization layer as a set of per-neuron switches. For every neuron, the batch-norm shift $\\beta_{l,i}$ tells which side of its rectifier the pre-activation typically falls on: positive $\\beta$ means the neuron is almost always in its linear region, so the activation can be replaced by the identity and folded into the next layer, while non-positive $\\beta$ means the neuron is almost always silent, so it can be deleted. The paper then removes whole layers, ranked by the accuracy drop each removal causes, and retrains between removals. Across ResNet-18, MobileNet-V2, Swin-T, and VGG-16 on CIFAR-10 it reports removing 12/17, 17/35, 6/12, and 7/15 layers while keeping top-1 accuracy within about one point of the dense model, with comparable results on larger image sets and on the language models BERT and RoBERTa. The reason this matters is that inference latency is set by the serial chain of layers, so deleting layers attacks exactly the cost that parameter-count pruning misses.","feed_headline":"Batch-norm statistics can delete most layers from neural nets","feed_subtitle":"Reading each neuron's batch-norm shift, the method linearizes active units and drops whole layers with minimal accuracy loss.","key_machinery":"The object that carries the argument is the per-neuron error likelihood $E_{l,i} = \\Phi(-|\\beta_{l,i}|/\\gamma_{l,i})$ (Eq. 3), evaluated separately for each neuron in a batch-normalization layer. It converts 'is this layer removable?' into a yes/no decision per neuron: positive $\\beta_{l,i}$ means the neuron is almost always ON (rectifier in its linear region), so the activation function is replaced by the identity and the neuron's weights are folded into the following layer; non-positive $\\beta_{l,i}$ means the neuron is almost always OFF, so the neuron is dropped. Removing a layer is therefore a surgical operation on the BatchNorm parameters rather than a retraining-dependent search. A layer's rank is fixed by measuring validation accuracy after its surgical removal (Eq. 4), and the outer algorithm removes the lowest-ranked layers while the validation accuracy stays above $\\theta \\cdot A_\\mathrm{init}$, retraining after each accepted removal. For transformer models without BatchNorm, the paper attaches the same statistics by computing the mean and standard deviation of the fully-connected layer's output.","core_discovery":"The central discovery is that the boundary between removable and unremovable computation is visible in the batch-norm parameters themselves. Under the paper's Gaussian model, each pre-activation is distributed as $z_{l,i} \\sim \\mathcal{N}(\\beta_{l,i}, \\gamma_{l,i}^2)$, so the probability that the rectifier's choice is wrong if we force the neuron to a single state is $\\Phi(-|\\beta_{l,i}|/\\gamma_{l,i})$. The paper's move is to take the binary sign of $\\beta_{l,i}$ as the state: neurons with $\\beta_{l,i}>0$ are linearized and merged into the next layer via a linear combination, neurons with $\\beta_{l,i}\\le0$ are simply removed, and the whole layer is then gone. TLC evaluates how much accuracy the model loses when each layer is removed in this way, removes the least costly layers one by one while validation accuracy stays above a relative threshold $\\theta$, and retrains after each accepted removal. The paper reports that this procedure removes a large fraction of layers on convolutional, transformer, and language models while keeping validation accuracy near the dense model's, and that it avoids the layer-collapse failure mode that afflicts entropy-based baselines on networks without skip connections.","pith_inferences":["The Gaussian assumption in Eq. (3) is untested; a validation-set histogram of pre-activations would show how far real distributions deviate, and replacing $\\Phi$ with empirical crossing rates would make TLC's error estimate trustworthy.","Nothing in the method prevents using the same ON/OFF score to prune individual channels inside surviving layers, so TLC's neuron-level decisions could be extended into a joint depth-and-width compression algorithm.","The hard $\\beta>0$ threshold suggests a training-time regularizer that pushes pre-activation means away from zero, which would increase the margin $|\\beta|/\\gamma$ and make even more layers removable; the paper does not try this.","The comparison is made in removed-layer counts and FLOPs; actual latency gains will depend on hardware and tensor shapes, and reporting end-to-end inference time on the target device would sharpen the practical claim."],"forward_implications":["Inference latency should drop roughly in proportion to the fraction of removed layers, because the critical serial path through the network, not total parameter count, is what limits GPU latency.","The same BatchNorm-derived ON/OFF slicing can be applied to already-trained models, making depth compression a post-training operation that needs only a validation set and a fine-tuning pass.","Skip-connection architectures tolerate much deeper removal than plain feedforward ones, so TLC's practical ceiling depends on architecture family, not just on dataset.","For transformer models, replacing LayerNorm-based statistics with output mean and standard deviation of the linear layer lets the method transfer to BERT and RoBERTa with comparable layer removals.","The short-finetune variant (TLC-finetuning) suggests the method can scale to larger models, though it removes fewer layers than full retraining between rounds."],"supporting_citations":[{"why":"Supplies the Gaussian pre-activation model for BatchNorm outputs that Eq. (3) builds on.","marker":"(Oh et al. 2022)"},{"why":"Introduces activation linearization and layer folding, the operation TLC adapts to per-neuron decisions.","marker":"(Dror et al. 2021)"},{"why":"Provides the linear-combination merge used to fold ON-state neurons into the following layer.","marker":"(Pilo et al. 2024)"},{"why":"Establishes the use of BatchNorm scaling parameters for filter importance, the prior art TLC contrasts with.","marker":"(Liu et al. 2017)"},{"why":"Defines ResNet-18, whose residual skip connections are the key testbed for deep layer removal.","marker":"(He et al. 2016)"},{"why":"Defines MobileNet-V2, the architecture where TLC removes 17 of 35 layers on CIFAR-10.","marker":"(Howard et al. 2017)"},{"why":"Defines Swin-T, the vision transformer used to show TLC works without explicit BatchNorm.","marker":"(Liu et al. 2021)"},{"why":"Defines RoBERTa, the NLP model used to demonstrate layer removal on transformers.","marker":"(Liu et al. 2019)"},{"why":"Defines EGP, the entropy-based layer-pruning baseline whose layer-collapse failure TLC avoids.","marker":"(Liao et al. 2023)"},{"why":"Defines EASIER, the layer-importance baseline TLC is compared against on image tasks.","marker":"(Qu´etu, Liao, and Tartaglione 2024)"}],"fun_headline_variants":["Batch-norm shifts expose removable neural net layers","TLC: Use batch norm to collapse whole layers","Layer dropout via batch-norm thresholds","Compress deep nets by reading batch-norm stats","Which layers can go? Ask batch normalization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each pre-activation $z_{l,i}$ is Gaussian with mean $\\beta_{l,i}$ and scale $\\gamma_{l,i}$, so the formula $\\Phi(-|\\beta_{l,i}|/\\gamma_{l,i})$ really measures the perturbation caused by linearizing or deleting that neuron, and that the hard sign threshold on $\\beta$ is therefore the right ON/OFF rule.","fun_headline_variants_meta":{"raw":{"variants":["Batch-norm shifts expose removable neural net layers","TLC: Use batch norm to collapse whole layers","Layer dropout via batch-norm thresholds","Compress deep nets by reading batch-norm stats","Which layers can go? Ask batch normalization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1266,"prompt_tokens":941,"completion_tokens":325,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":269}},"tokens_in":557,"tokens_out":325,"duration_ms":3497,"temperature":1.0,"reasoning_tokens":269,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:39:07.511015+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the actual change in the next layer's activations when a layer is removed via TLC, and compare it with the predicted per-neuron error sum $\\sum_i \\Phi(-|\\beta_{l,i}|/\\gamma_{l,i})$. If the actual perturbation is large when the predicted error is small, or if the predicted ranking of layers disagrees with the ranking by true accuracy loss, the central mechanism is falsified. A second, cheaper check: take a trained ResNet-18 on CIFAR-10, measure the empirical distribution of each pre-activation $z_{l,i}$ over the validation set, and test whether $z_{l,i}$ is close enough to Gaussian that $\\Phi(-|\\beta|/\\gamma)$ matches the observed fraction of wrong-side crossings for the chosen rectifier.","supporting_citations":[{"cited_title":"B.; Zehngut, N.; Raviv, A.; Artyomov, E.; Vitek, R.; and Jevnisek, R","cited_arxiv_id":null,"evidence_quote":"Introduces activation linearization and layer folding, the operation TLC adapts to per-neuron decisions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the linear-combination merge used to fold ON-state neurons into the following layer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the use of BatchNorm scaling parameters for filter importance, the prior art TLC contrasts with."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines EGP, the entropy-based layer-pruning baseline whose layer-collapse failure TLC avoids."}],"review_version":1}