{"id":"64326bc2-46c8-4009-b7db-cf65bee365e3","arxiv_id":"2607.03784","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"Heterogeneity-aware depth pruning of attention and activation layers, guided by a polynomial model-accuracy predictor, delivers up to 1.58× speedup on DeiT-B and 5.19× when combined with width pruning.","lead":"HetDPT prunes Vision Transformers by jointly removing attention layers and GELU activations while allocating the budget with a fitted accuracy predictor, then merges adjacent linear layers. This yields higher practical speedups than prior depth or joint width-depth pruning at similar accuracy on ImageNet and transfer tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"MAP ranking fidelity under short finetune is the softest link, but the paper already treats it as a practical surrogate rather than a proven global optimum.","rationale":"The reader correctly isolates the weakest assumption: that a low-degree polynomial fit on cheap short-finetune points yields a maximizer that coincides with the true full-finetune optimum. That is the single most load-bearing soft spot for the “heterogeneity + MAP budget is why it works” claim. Empirically the method still delivers the reported speedups and near-lossless numbers (Tables 1–2, Fig. 7), code is public, and the theory is carefully scoped under Assumption 3.2, so the concern does not overturn the engineering contribution. It does justify keeping CONDITIONAL rather than unconditional ACCEPT until ranking fidelity is checked under full finetune (or multi-seed variance is reported). No stronger internal inconsistency or experimental contradiction is evident; the continuous-relaxation and bias-invariance arguments are standard and do not overclaim. Hence agreement with the reader and no verdict change.","tokens_in":30474,"tokens_out":608,"duration_ms":5822,"concrete_test":"For DeiT-B at the 10-layer budget, take the top-3 MAP-recommended (na, ng) pairs and 2–3 nearby alternatives (including pure-attention and pure-activation extremes). Fully fine-tune each for the same long schedule used in the main tables (not the 10-epoch MAP collection), report final Top-1, and check whether MAP’s argmax remains best or within ~0.2–0.3% of the true best. If a non-MAP allocation wins by a clear margin, the surrogate-ranking assumption fails for that budget.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that heterogeneity-aware mixed budgets are what make depth pruning work rests on MAP choosing near-optimal (attention, activation) counts. Theorems 3.3–3.5 and Lemma 3.4 only guarantee that a constant bias preserves the maximizer and that polynomials can approximate a continuous surface; they do not guarantee that the short-finetune observations (subset data, ~10 epochs, incremental single-layer TE pruning in Algorithms 1–2 / Appendix C) preserve the ranking of full-finetune configurations. If recovery asymmetry (Observation 2 / Fig. 5) interacts with multi-layer interactions so that short-finetune rankings reorder relative to full recovery, the allocated budgets can be systematically suboptimal even while the polynomial fit looks good on the cheap data (Table 10, MAE/RMSE ~0.4–0.5). The paper’s own ablations (Table 7) show that Stage-1 budget allocation matters, so this is load-bearing for the “why prior depth pruning fails” story, not merely an implementation detail.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper argues that prior ViT depth pruning fails mainly because it ignores heterogeneity between attention and GELU activation layers (gradient disparity and recovery asymmetry), and that this, together with dimension mismatch when pruning linear layers, limits both pure depth and joint width–depth methods. It proposes HetDPT: a two-stage pipeline that (i) allocates a mixed pruning budget between attention and activation layers via a polynomial Model Accuracy Predictor (MAP) fitted on lightweight prune–finetune–evaluate data, (ii) selects specific layers within each type using learnable importance scores, and (iii) merges adjacent linear layers after activation removal to restore dimension alignment and gain speedup. Empirically, HetDPT reports 1.58× speedup on DeiT-B at baseline accuracy and 1.39× on DeiT-S with near-lossless accuracy, plus transfer results on CIFAR-100, COCO, and ADE20K; HetDPT+ combined with width pruning improves the Isomorphic-Pruning-2.6G configuration from 4.24× to 5.19× speedup at near-lossless accuracy. Theoretical appendices establish existence of an optimum, polynomial approximability under continuous relaxation, and maximizer preservation under constant bias.","tokens_in":30732,"tokens_out":1231,"duration_ms":10032,"significance":"If the heterogeneity diagnosis and the MAP-guided mixed-budget design hold, the work meaningfully advances structured ViT compression: depth pruning has long been known to offer higher hardware speedup than width pruning but has been hard to recover, and the paper shows that mixed attention/activation removal plus linear merging can deliver lossless or near-lossless depth compression and push extreme joint compression past prior SOTA. Strengths include multi-benchmark evaluation (ImageNet, CIFAR transfer, COCO, ADE20K), ablations isolating budget allocation and within-type selection, orthogonality checks with token pruning, a public code link, and an explicit theoretical appendix (existence, Stone–Weierstrass under continuous relaxation, bias-invariance of maximizers). The free parameters of MAP (degree, coefficients, short-finetune protocol) are acknowledged as practical surrogates rather than first-principles constants, which is appropriate for an empirical compression method.","major_comments":[{"comment":"The central causal claim—that neglect of heterogeneity is why prior depth pruning fails, and that MAP-guided mixed budgets fix it—depends on MAP’s ranking fidelity. Theorems 3.1, 3.3, 3.5 and Lemma 3.4 (and Appendix F) only guarantee existence of an optimum, uniform polynomial approximation under continuous relaxation (Assumption 3.2), and that a constant bias preserves the maximizer. They do not establish that short-finetune observations (subset data, ~10 epochs, incremental single-layer TE pruning in Algorithms 1–2 / Appendix C) preserve the ranking of full-finetune configurations. Recovery asymmetry (Observation 2 / Fig. 5) and multi-layer interactions could reorder budgets relative to full recovery even when the polynomial fit on cheap data looks reasonable (Table 10, MAE/RMSE ~0.4–0.5). Table 7 shows Stage-1 budget allocation is load-bearing; without a direct check that MAP’s recomm","section":null},{"comment":"Figure 7 and the single-type baselines (attention-only vs. activation-only vs. HetDPT) are the main empirical support for the heterogeneity diagnosis, but the paper does not report a controlled comparison against a strong mixed-budget baseline that uses the same total layer budget without MAP (e.g., fixed 50/50 or TE-only mixed selection with full finetuning). Table 7 ablates components of Stage 1 but does not isolate whether MAP’s specific allocation is better than other mixed allocations of the same total k. Without that, it remains possible that any mixed attention+activation pruning plus merging would recover most of the gain, which would weaken the claim that MAP-based heterogeneity-aware allocation is the key contribution rather than activation removal and linear merging alone.","section":null}],"minor_comments":[{"comment":"Notation for retained ratios (a, t) in Appendix C vs. pruning ratios (˜ma, ˜mg) in the main text is easy to confuse; a short explicit conversion box near Eq. (5) would help.","section":null},{"comment":"Figure 3’s dimension-mismatch cartoon is useful but the caption should state more clearly that activation removal is the mechanism that enables safe linear merging, not pruning of Linear1/Linear2 themselves.","section":null},{"comment":"Table 1 and Table 2 report throughput on H800; stating batch size and whether TensorRT/cuDNN settings match prior work would improve reproducibility of the speedup claims.","section":null},{"comment":"Related-work discussion of DepthShrinker is helpful; a short quantitative “DepthShrinker-style on ViT” baseline number (even in the appendix) would make the claimed gap more concrete.","section":null},{"comment":"Typos and minor inconsistencies: “DePTh” capitalization in the method name, occasional “DW Prun.” vs. “WD-Pruning,” and slight baseline accuracy differences between tables (e.g., DeiT-S 79.8 vs. 79.9 in text) should be cleaned.","section":null}],"recommendation":"major_revision","confidential_remarks":"The MAP ranking-fidelity gap is real but fixable with a modest full-finetune validation experiment; I would not reject on that alone. The paper is a solid systems/compression contribution with clear empirical gains; major revision is appropriate so that the causal story matches the evidence. Scope fits a top ML venue that accepts strong empirical compression work with supporting theory."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing worth knowing: treating GELU as a static pruning target and allocating a mixed attention/activation depth budget is what makes aggressive ViT depth pruning practical. HetDPT gets 1.58× on DeiT-B at baseline accuracy and, stacked with width pruning, lifts the Isomorphic-Pruning-2.6G point from 4.24× to 5.19×. That is a real deployment win, not a re-label of prior width work.\n\nWhat is new is the heterogeneity diagnosis (gradient disparity + recovery asymmetry), the first systematic static removal of activation layers in ViTs so adjacent linears can merge without dimension mismatch, and the two-stage recipe: MAP for counts, within-type scores for which layers. DepthShrinker did activation removal for CNNs; this paper correctly argues that attention vs FFN nonlinearity is a different problem and shows single-type pruning collapses under multi-layer budgets (Fig. 7). Empirics are broad—ImageNet, CIFAR transfer, COCO, ADE20K, Swin, DINOv2-Giant—and ablations (Table 7) isolate both stages. Code is public. Theory appendix is honest: existence on a finite set, Stone–Weierstrass under continuous relaxation, constant-bias maximizer preservation. It does not pretend to prove the short-finetune surface equals full recovery.\n\nSoft spot, in proportion: MAP is fitted on cheap incremental ~10-epoch subset runs (Algorithms 1–2, Table 10 MAE/RMSE ~0.4–0.5). Theorems only protect against constant bias and guarantee polynomial approximability; they do not guarantee that short-finetune rankings match full multi-layer recovery when recovery asymmetry interacts across layers. That is load-bearing for the “why prior depth pruning fails” story, not a side detail. Still, the paper treats MAP as a practical surrogate, the final models are fully fine-tuned, and the empirical Pareto is strong enough that the ranking risk looks like a refinement issue, not a fatal one. Multi-seed variance and MAP transfer across families are thinner than I would like; free parameters (degree, subset, finetune length) are real but standard for this genre.\n\nThis is for people who ship ViT compression or care about structured depth pruning. It deserves a serious referee. I would engage with it, cite the recipe and the heterogeneity observations, and ask for fuller ranking-fidelity checks in revision.","headline":"Solid engineering paper: mixed attention/GELU depth pruning with a cheap MAP budget works, and the speedups are real; MAP ranking fidelity is the softest link but not a collapse of the claim.","tokens_in":31366,"tokens_out":631,"would_cite":true,"duration_ms":6616,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Depth pruning of vision transformers fails mainly because attention and activation layers behave differently; allocating a mixed budget by type and then merging linear layers recovers accuracy while raising speedup.","keywords":["vision transformers","depth pruning","structured pruning","heterogeneity","activation-layer removal","model accuracy predictor","linear-layer merging","extreme compression"],"falsifier":"For a fixed total layer budget on DeiT-Base, exhaustively train every feasible (attention-count, activation-count) pair to full convergence and check whether the MAP-selected pair still yields the highest recovered accuracy; a systematic re-ranking would falsify the surrogate.","tokens_in":31371,"feed_emoji":"✂️","tokens_out":694,"duration_ms":6223,"temperature":0.7,"pith_summary":"Vision transformers can be sped up more by removing whole layers (depth pruning) than by shrinking channels or heads inside layers (width pruning), but earlier depth-pruning methods lost too much accuracy. This paper argues that the real problem is heterogeneity: attention layers and GELU activation layers have very different gradient scales and very different recovery curves after pruning. It proposes HetDPT, which first decides how many of each type to remove by fitting a simple model-accuracy predictor on cheap short fine-tunes, then ranks layers only inside each type, prunes, fine-tunes, and merges the now-adjacent linear layers so dimensions stay consistent. On ImageNet the method keeps DeiT-Base accuracy at 1.58× measured speedup and nearly matches DeiT-Small at 1.39×; when stacked with existing width pruning it lifts an extreme compression point from 4.24× to 5.19× with near-lossless accuracy, and the same models transfer to detection and segmentation.","feed_headline":"Mixed attention–activation pruning speeds ViTs 1.58×","feed_subtitle":"Budgeting layers by type and merging linears recovers accuracy that pure depth cuts lose","key_machinery":"HetDPT’s Model Accuracy Predictor (MAP): a low-degree bivariate polynomial fitted on a small set of incremental prune–fast-finetune accuracy points that recommends the optimal split of the total layer budget between attention layers and activation layers, after which importance scores are computed only inside each homogeneous group and adjacent linear layers are merged.","core_discovery":"The accuracy collapse of prior ViT depth pruning is caused less by coarse granularity than by treating attention and activation layers as interchangeable. Because those two layer types exhibit gradient disparity and recovery asymmetry, any metric that compares them directly is biased. Once a mixed pruning budget is chosen by a polynomial accuracy predictor that respects final recovered accuracy, and layers are selected only within type, the resulting network can be fine-tuned and the linear layers that no longer have activations between them can be merged, restoring dimension consistency and delivering large practical speedups.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Layer-type depth pruning lifts DeiT-B to 1.58× speed","Attention-MLP separation restores ViT depth-pruning accuracy","Heterogeneity-aware cuts give near-lossless 1.39× on DeiT-S","Within-type layer budgets enable extreme ViT acceleration","Merging linears after typed depth prune hits 5.19× ratio"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That a cheap polynomial fitted on short, subset-based fine-tunes still ranks pruning budgets the same way a full, expensive fine-tune would, so its recommended mix of attention and activation removals is near-optimal.","fun_headline_variants_meta":{"raw":{"variants":["Layer-type depth pruning lifts DeiT-B to 1.58× speed","Attention-MLP separation restores ViT depth-pruning accuracy","Heterogeneity-aware cuts give near-lossless 1.39× on DeiT-S","Within-type layer budgets enable extreme ViT acceleration","Merging linears after typed depth prune hits 5.19× ratio"]},"model":"grok-4.5","effort":"low","cost_usd":0.005088,"raw_usage":{"total_tokens":1466,"prompt_tokens":831,"num_sources_used":0,"completion_tokens":102,"cost_in_usd_ticks":50880000,"prompt_tokens_details":{"text_tokens":831,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":533,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":831,"tokens_out":102,"duration_ms":4531,"temperature":1.0,"reasoning_tokens":533,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T23:57:52.846769+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"For a fixed total layer budget on DeiT-Base, exhaustively train every feasible (attention-count, activation-count) pair to full convergence and check whether the MAP-selected pair still yields the highest recovered accuracy; a systematic re-ranking would falsify the surrogate.","supporting_citations":[],"review_version":1}