Pith. sign in

REVIEW 3 major objections 4 minor 65 references

Structured LLM pruning should score pairs of units, not single units: the paper derives a Fisher matrix whose off-diagonal entries are the extra damage of removing two units together, and recovers the whole matrix from one forward pass per

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 17:36 UTC pith:S5NYABVG

load-bearing objection CoCurve is a serious, unusually honest training-free structured pruning paper whose cross-module edge matrix is a real idea, but the load-bearing additivity assumption needs a direct pairwise check and the 'label-free' story is a bit stretched. the 3 major comments →

arxiv 2607.17568 v1 pith:S5NYABVG submitted 2026-07-20 cs.LG cs.AI

CoCurve: Cross-Module Co-Pruning Curvature for Training-Free Structured LLM Pruning

Zhiren Gong , Zihao Zeng , Zijie Wang , Tiantong Wang , Chau Yuen , Wei Yang Bryan Lim This is my paper
classification cs.LG cs.AI
keywords structured pruninglarge language modelsco-pruning curvatureFisher information matrixattention headsFFN channel groupstraining-freesecond-order Taylor expansion
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Structured pruning of large language models is usually done by ranking each attention head or FFN channel group independently, implicitly assuming that the damage of removing a set equals the sum of individual damages. This paper argues that this view fails for Transformers, whose sublayers are coupled through a shared residual stream, and that two individually weak units can be jointly indispensable. The central claim is that the missing information is exactly the pairwise co-pruning curvature: the off-diagonal of a Fisher matrix that comes from a second-order expansion of the token-level KL divergence between the frozen model and its masked copy. The paper shows that this full M-by-M interaction matrix is a Gram product of M single-unit ablation features, so modeling the edges costs no more than computing the diagonal, and that pruning then becomes a single budgeted quadratic program solved greedily. If correct, training-free structured pruning can preserve fragile generative capabilities — code and knowledge, where independent scoring collapses — without labels, gradients, or fine-tuning.

Core claim

The paper's central discovery is that the second-order Taylor expansion of a token-level self-distillation KL — the divergence between the frozen full model and its masked copy — yields one Fisher matrix H whose diagonal is exactly the classical node saliency and whose off-diagonal entries are co-pruning curvature edges: the extra distortion of removing two units together. Under an additivity approximation, each entry H_uv equals the Fisher-weighted inner product of the two units' single-ablation logit shifts; after whitening by the teacher probabilities, this becomes a plain Gram product, so the full M×M matrix is obtained from M forward-only ablation passes, with no pairwise sweeps, gradie

What carries the argument

The load-bearing object is the M×M co-pruning curvature (Fisher) matrix H, defined as the Hessian of the pruning risk R(s) = E[ D_KL(p_0 || p_s) ] at s=0. Its diagonal is an Optimal-Brain-Damage style node saliency; its off-diagonal entries are the joint-removal interaction terms. The efficiency trick is the Gram identity: with whitened single-unit logit-ablation features δz̃_u = √p_0 ⊙ (δz_u − E_{p_0}[δz_u]), each entry satisfies H_uv = (1/P) Σ δz̃_u^T δz̃_v, so the entire matrix is a Gram matrix of M single-ablation feature vectors and costs M forward passes instead of O(M²) pairwise ablations. The solver consumes this matrix through the running marginal risk Δ(u|S) = ½ H_uu + Σ_{v∈S} H_uv

Load-bearing premise

The entire edge estimator rests on single-ablation additivity: the logit shift from removing several units equals the sum of each unit's individual shift, so the pairwise entries of H are inner products of single-unit ablation features — if this fails at moderate pruning ratios, H is a phantom objective that the greedy solver optimizes.

What would settle it

Measure additivity directly on a 3B–8B model at 20% pruning: pick 20–30 pairs of attention/FFN units, mask each unit alone and both together, and compare the measured joint token-KL to the value predicted from the Gram H (½(H_uu + H_vv) + H_uv). Alternatively, repeat the same-budget ranking test of the paper — predicted ½ s^T H s versus measured KL over a pool of structured and random mask sets — and check whether the Spearman rank correlation is substantially below the reported 0.81 on a held-out model.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, training-free structured pruning is no longer limited to independent rankings: interaction-aware selection is as cheap as the diagonal, so edge-aware co-pruning is affordable in any deployment setting at the same calibration budget.
  • The Gram reconstruction makes the interaction matrix positive semi-definite by construction, giving a free numerical correctness check that edge-aware pruning ships with.
  • The measured FFN-redundancy statistic |corr|_FFN, computed from calibration H, becomes a pre-hoc architecture-level rule for whether to trust or damp the edge term, making the method predictable across model families without labels.
  • At aggressive ratios, the same edge blocks used for selection can be reused for a closed-form output-rescale compensation, reducing the super-linear collapse region without fine-tuning or gradient updates.
  • Because the λ=0 endpoint of CoCurve is exactly a structured second-order saliency, the method contains a fallback: when edges are not trustworthy on high-redundancy architectures, it degrades gracefully to the classical diagonal selector rather than failing.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the additivity approximation degrades as predicted by the paper's O(∥s∥³) bound, the same Gram machinery could be iterated: re-estimating H on the pruned model in rounds would push the trust region further, turning the one-shot method into an incremental one and potentially extending the usable compression ratio.
  • The locality result (edges concentrated within and between nearby layers) suggests a modular variant that computes only within- or adjacent-layer blocks, cutting the Gram cost further with little accuracy loss; the paper itself notes this as an efficiency opportunity.
  • The paper's observed split between calibration KL and downstream capability — stronger edges can hurt perplexity while saving code generation — implies that any single-metric evaluation protocol for training-free pruning can misorder selectors; reporting both a distributional metric and a generative stress test is a testable improvement for future benchmark practice.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes CoCurve, a training-free structured pruning method for LLMs. It defines pruning risk as the token-level KL divergence between the frozen model and its masked copy, and shows that a second-order expansion yields a Fisher matrix H whose diagonal is node saliency and whose off-diagonal entries are co-pruning curvature edges. Under a single-ablation additivity approximation, H is recovered as a Gram product of M single-unit ablation feature vectors, avoiding pairwise sweeps or gradients. Pruning is then solved as a one-shot budgeted quadratic program with a greedy solver. Experiments on four 3B–8B models and a 24B scaling study compare CoCurve against nine training-free baselines, reporting strong results on perplexity and code generation, with ablations attributing the gain to cross-module edges and a calibration-derived redundancy statistic gating the edge strength.

Significance. If the surrogate is valid, CoCurve is a meaningful advance: it models cross-module, cross-layer interactions at O(M) cost without labels, gradients, or recovery, and the empirical program is unusually thorough. The paper includes honest admission that H is a surrogate (Remark 2), a non-trivial same-budget surrogate discrimination test (Fig. 7, Spearman 0.81 across 26 sets), a matched-pair causal test for bridge units (Table 4), and extensive robustness checks on calibration seed, source, and budget. These are real strengths. However, the central additivity assumption is not directly tested, and the predictive redundancy gate is partly circular because its threshold is extracted from the same measurements used to validate it. The manuscript needs revision before these claims are fully supported.

major comments (3)
  1. [Section 2.2, Eq. (5), Prop. 5; Section R, Figs. 6–7] The load-bearing claim is that off-diagonal entries H_uv are co-pruning curvature edges, recovered by a Gram product of single-unit ablation features. This equality holds only under single-ablation additivity Δz_s ≈ Σ_u s_u δz_u. Proposition 5 bounds the error by O(||s||_1^3) with an unchecked constant, and Remark 2 explicitly concedes that H is a surrogate. The fidelity evidence is set-level: Figure 7 validates ranking of 26 pruning sets at fixed budget, not the individual pairwise entries H_uv, and Figures 6–7 are only on Llama-3.1-8B-Instruct. I request a direct pairwise test: for representative cross-module and same-module pairs (including solver-selected pairs), compare the measured joint logit perturbation or joint KL damage with the additive prediction δz_u + δz_v at moderate ρ, and report whether the Gram ranking of pairs is preserved. Without this, the 'co-pruning curvature edge
  2. [Section T, Table 21] The claim that a single calibration statistic predicts a priori when edges help is weakened by how the gate is set. The threshold |corr|_FFN ≳ 0.12 is introduced after presenting Table 21, and the same table is then used as evidence for the rule's predictive power. This is circular: the boundary is fit to the data used to validate it. The term 'pre-registered' is not supported by any protocol fixed before the measurements. To make the contribution stand, the threshold should be set on a discovery subset (or derived analytically) and then evaluated on held-out families; alternatively, report the continuous edge-benefit-versus-|corr| relationship and avoid a hard threshold. As written, the gate is a post-hoc tuning rule.
  3. [Section 2.4, Prop. 1; Section 3.4] The statement that the edge term 'never' goes below the OBD diagonal floor is too strong. The λ=0 endpoint is exactly the diagonal selector, but for λ>0 the edge term can hurt, as shown by the forced-λ=1 rows in Table 23 and the negative edge benefits in Table 21. The method's actual protection comes from damping λ, not from a guaranteed floor. Relatedly, Proposition 1 assumes as an axiom that the third-order remainder T(s) ≥ 0 on the feasible set ('co-removal is on average super-additively harmful'); this is not derived and may be violated when H has strong negative off-diagonals. The empirical λ sweep is suggestive, but the proposition should be labeled as conditional on that assumption, and Falcon3's λ=0.5 should be acknowledged as a model-specific operating point rather than 'not a free hyperparameter.'
minor comments (4)
  1. [Section 6 vs. Section T] The Limitations section says 'a complete no-recovery comparison at ≥13B is compute-bound and left to future work,' but Section T and Table 14 report a complete 14-task no-recovery protocol on Mistral-Small-24B at all ratios. Please reconcile this inconsistency.
  2. [Figure 6] Spearman ρ=1.00 over five ratio points is not informative by itself; the meaningful same-budget test is Figure 7. Consider reporting per-ratio Spearman or more sampled sets.
  3. [Table 18] The sign-handling ablation reports that signed, clipped, and absolute variants coincide on Llama-3.1-8B-Instruct. This does not demonstrate that signed handling matters; please state this explicitly and, if possible, show a model where signs differ.
  4. [Table 21] The statement that 'not one of the 2.36M off-diagonal correlations exceeds 1' is a tautology for correlation values; remove or replace with a conditioning or numerical-range check.

Circularity Check

1 steps flagged

Core Gram estimator is self-contained; redundancy-gate threshold is fitted in-sample and labeled pre-registered.

specific steps
  1. fitted input called prediction [Section T (Appendix, 'A calibration-only gate, fixed before any benchmark') and Table 21]
    "We use a single pre-registered rule on this statistic: when |corr|FFN is in the high-redundancy band (≳0.12 on our measurements) ... This rule is fixed from calibration alone, before a single benchmark is run ... The three blocks—favorable (<0.10), the ∼0.11 gray zone where edges turn marginally harmful, and the high-redundancy region (>0.15)—trace a monotone favorability-vs-|corr| relationship across ten models and six families, confirming Proposition 8 is predictive a priori (from calibration H, no labels)."

    The band edges (0.10, 0.12, 0.15) are not independent a priori thresholds; they are read off Table 21, which tabulates both |corr|_FFN and measured edge benefit for the same ten models. The same table is then cited as 'confirming ... predictive a priori.' The rule therefore fits the threshold to the outcome it claims to predict, and the word 'pre-registered' is contradicted by this in-sample derivation. This is a fitted-input-as-prediction circularity in the redundancy-gate claim, not in the core H/Gram estimator, whose selection quality is validated on held-out benchmarks.

full rationale

The central derivation is not circular. H is estimated by M single-unit ablations and scored against held-out task benchmarks; the Gram identity (Prop. 4) is a stated finite-difference surrogate (Remark 2), so the fidelity checks in Section R (Figs. 6-7) are genuine empirical validations of a surrogate, not a re-reading of the input. No load-bearing result is imported from author-only citations: the reference list is external (OBD/OBS, SparseGPT, Wanda, etc.) and the uniqueness claims are not used to select the estimator. Diagonal CoCurve being OBD is a mathematical specialization, not a renamed fit. The one real circular step is the redundancy gate. The |corr|_FFN ≳ 0.12 threshold and the 0.10/0.15 bands are described as a 'pre-registered rule' but are obtained from Table 21, the same table that contains the edge-benefit outcomes used to demonstrate the rule's predictive power. Setting a cutpoint on the input using the observed outcomes and then calling the rule 'predictive a priori' is a mild fitted-input-called-prediction circularity. It is secondary: the central claim (CoCurve beats training-free baselines) does not depend on the gate, and the gate is also tested on held-out/high-redundancy models (Qwen2.5-32B), so the circularity is partial, not total. I also note the Falcon3 λ=0.5 headline is chosen from the benchmark λ sweep and disclosed as a 'balanced operating point' rather than a prediction; this is a transparency caveat, not a hidden fit. Overall score 4 reflects the in-sample gate threshold while acknowledging the independent core.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 2 invented entities

The central claim rests on four tuned quantities (λ, the 0.12 gate threshold, compensation γ/τ, anti-collapse config) and two structural premises (logit additivity; calibration-KL-to-benchmark transfer). The math itself uses only standard Fisher/Gram manipulations. No physical entities are invented; the edge matrix and bridge units are internal derived objects validated on held-out tasks.

free parameters (4)
  • λ (interaction strength) = 1.0 (Llama-3.1, Mistral, Llama-3.2), 0.5 (Falcon3-7B), →0 (gated high-redundancy)
    Trust-region optimum of calibration risk for most models, but Falcon3-7B uses a benchmark-informed operating point chosen after observing the perplexity-vs-capability trade-off (Section 3.4); high-redundancy damping depends on a gate threshold fit to Table 21.
  • redundancy gate threshold = |corr|_FFN ≳ 0.12
    Boundary between 'edges help' and 'edges hurt' drawn from the measured edge-benefit column of Table 21 (Mistral-Nemo at 0.111 mildly negative; Yi/Phi/Qwen ≥ 0.169 negative), then described as the pre-registered rule (Section T).
  • compensation hyperparameters γ, τ = γ ∈ {0.05, 0.2, 1.0}×diag(H_KK), τ ∈ {1, 2}
    Selected per model on a held-out calibration split; compensation is 'applied only where it helps' (Sections 2.5, L), making its inclusion outcome-dependent.
  • anti-collapse cap κ and protected layers L_p = Table 8 per model/ratio (e.g., cap 0.30, protect 2/2 for Llama-3.1-8B)
    Per-ratio solver configuration chosen to minimize proxy WikiText perplexity on a calibration holdout (Section D); these change which units are pruned.
axioms (5)
  • domain assumption Local additivity of masked logit perturbations: Δz_s ≈ Σ_u s_u δz_u
    Load-bearing premise for Eq. (5) and the whole Gram estimator (Section 2.2); error bounded in Prop. 5 only under a smoothness assumption that is stated, not verified.
  • domain assumption The logit map is twice differentiable in the mask variables with bounded second derivatives near s=0
    Needed for Prop. 5's O(‖s‖³) surrogate-error bound and the trust-region argument (Section F).
  • ad hoc to paper Co-removal is on average super-additively harmful: the third-order remainder T(s) ≥ 0 on the feasible set
    Assumed to derive λ*(ρ) non-increasing (Prop. 1, Section G.1); no independent justification, though the empirical λ-sweeps are consistent.
  • domain assumption Token-level KL on 128 C4 sequences is a faithful proxy for the 14-task benchmark suite
    All selection, trust-region, and gate decisions are driven by calibration risk; benchmark transfer is validated only indirectly via calibration-source and seed studies (Section R.1).
  • standard math Standard second-order analysis: KL Fisher quadratic expansion (Prop. 3) and Gram/whitening identity (Prop. 4)
    Textbook manipulations; correctly derived.
invented entities (2)
  • 'Co-pruning curvature edge' H_uv no independent evidence
    purpose: Off-diagonal interaction term quantifying the extra KL damage of removing units u and v together; the object the pruning QP optimizes.
    An internal quantity estimated from the model's own ablations, with no falsifiable handle outside the paper's construction; its usefulness is validated on held-out benchmark tasks (Tables 1–3, Fig. 7), not by any independent measurement.
  • 'Bridge units' (low saliency, high edge connectivity) no independent evidence
    purpose: Interpretive label for units that diagonal scoring deletes but CoCurve retains.
    Emergent pattern in the estimated H (Section S); supported by a matched-pair causal test (Table 4), but not an independently measurable entity.

pith-pipeline@v1.3.0-alltime-deepseek · 39757 in / 22741 out tokens · 186137 ms · 2026-08-01T17:36:54.968868+00:00 · methodology

0 comments
read the original abstract

Structured pruning compresses large language models (LLMs) by removing whole computational units, such as attention heads and feed-forward (FFN) channel groups. Most training-free methods, however, rank these units independently, implicitly treating the loss from pruning a set as the sum of its individual losses. This view fails for Transformers, whose sublayers are coupled through a shared residual stream. Two individually weak units can thus be jointly indispensable, yet independent scoring is blind to such dependence and removes them together. We introduce CoCurve (Cross-Module Co-Pruning Curvature), a calibration-only, fine-tuning-free method that prunes attention and FFN units jointly. A second-order Taylor expansion of the token-level KL between the frozen model and its masked copy yields a single Fisher matrix whose diagonal is classical node saliency and whose off-diagonal entries are co-pruning curvature edges: the extra damage of removing two units together. Under a single-ablation additivity approximation this matrix reduces to a Gram product of single-unit ablation features, so the full M x M interaction is recovered from M forward passes, with no pairwise sweeps or gradients. Pruning then reduces to one budgeted quadratic program, solved in a single shot under a shared attention--FFN budget, with no labels, fine-tuning, or recovery.

Figures

Figures reproduced from arXiv: 2607.17568 by Chau Yuen, Tiantong Wang, Wei Yang Bryan Lim, Zhiren Gong, Zihao Zeng, Zijie Wang.

Figure 1
Figure 1. Figure 1: Overview of CoCurve. (a) Transformer sublayers read and write a shared residual stream, so a low-saliency bridge unit u can be jointly indispensable with units many layers away; scoring each unit in isolation misses this. (b) A second-order expansion of the token-level KL yields one Fisher matrix H whose diagonal is classical node saliency and whose off-diagonal is the co-pruning curvature between attentio… view at source ↗
Figure 2
Figure 2. Figure 2: Scaling to Mistral-Small-24B (real measured data). (a) WikiText-2 perplexity (log scale) versus pruning ratio: CoCurve leads at every ratio, and its advantage over the strongest valid baseline widens with compression—4.9× at 30% and 14.5× at 40%, exactly where redundancy is exhausted and co-pruning curvature becomes decisive. (b) Multiple-choice average: CoCurve leads from 20% up, with a near-tie at 10% wh… view at source ↗
Figure 3
Figure 3. Figure 3: CoCurve across pruning ratios. WikiText-2 perplexity (left, log scale) and MMLU (right) vs. ratio for three models. Degradation is graceful through the moderate regime; beyond ∼30% all training-free methods enter the shaded collapse zone. Curves are a single compressed model per point; baselines and the accuracy–efficiency frontier are in Section U. Two controls confirm the mechanism, not a tuned knob (det… view at source ↗
Figure 4
Figure 4. Figure 4: When cross-module edges help. The edge benefit (perplexity gain of the full edge model over the diagonal-only variant) scales inversely with FFN channel redundancy |corr|, measured directly from H before any benchmark: a predictive, calibration-only rule for whether cross-module curvature should be applied at full strength (λ=1) or damped. Low-redundancy models (left) gain; high-redundancy Qwen/Yi (right) … view at source ↗
Figure 5
Figure 5. Figure 5: Edge-structure and cross-layer-reach ablation (Llama-3.1-8B-Instruct, 20%, real data; [PITH_FULL_IMAGE:figures/full_fig_p032_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Surrogate fidelity across the operating range (Llama-3.1-8B-Instruct, real measured data). Left: predicted solver surrogate 1 2 s ⊤Hs vs. measured per-token DKL(p0∥pS) for the pruning sets selected at ρ ∈ {10, . . . , 50}%, colored by ratio: across the full operating range the surro￾gate increases monotonically with the true damage (Spearman ρ=1.00, Pearson r=0.95), confirm￾ing the second-order model track… view at source ↗
Figure 7
Figure 7. Figure 7: Same-budget discrimination (Llama-3.1-8B-Instruct, 20%, real data; from diagnose edge mechanism.py). Predicted surrogate vs. measured KL over 26 pruning sets at the same 20% budget. Structured selections (CoCurve, diagonal, local swaps) cluster at low mea￾sured risk; random sets at the identical budget are ∼3× worse. Rank correlation 0.81 across sets at fixed ρ shows the surrogate discriminates selection q… view at source ↗
Figure 8
Figure 8. Figure 8: Interpretability evidence (Llama-3.1-8B-Instruct), from the real estimated H. (a) the curvature correlation |ϱuv| averaged within (type, layer) blocks: the off-diagonal cross-module block (white dashed) carries substantial coupling (mean 0.086), comparable to the within-module blocks. (b) node saliency vs. edge connectivity: the 87 low-saliency, high-connectivity bridge units (red, shaded region) are exact… view at source ↗
Figure 9
Figure 9. Figure 9: Non-uniform, type-asymmetric per-layer pruning (Llama-3.1-8B-Instruct, 20%), from the real CoCurve solution: the shared attention/FFN budget is spent unevenly across depth—decided by the objective, not a fixed split. attn--attn attn--FFN (cross-module) FFN--FFN 0.00 0.02 0.04 0.06 0.08 0.10 mea n |ϱuv| (off-dia g corr.) Cross-module curvature is non-trivial Llama-3.1-8B-Instruct Mistral-7B Llama-3.2-3B [P… view at source ↗
Figure 10
Figure 10. Figure 10: Block-wise mean |ϱuv| of the estimated H (real data): the cross-module attn–FFN block is non-trivial across models—direct evidence for cross-module co-pruning edges. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

65 extracted references · 10 linked inside Pith

  1. [1]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Attention is All You Need , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  2. [2]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

    Ainslie, Joshua and Lee-Thorp, James and de Jong, Michiel and Zemlyanskiy, Yury and Lebr. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

    2023

  3. [3]

    arXiv preprint arXiv:1503.02531 , year=

    Distilling the Knowledge in a Neural Network , author=. arXiv preprint arXiv:1503.02531 , year=

    Pith/arXiv arXiv

  4. [4]

    Neural Computation , volume=

    Natural Gradient Works Efficiently in Learning , author=. Neural Computation , volume=

  5. [5]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Optimal Brain Damage , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  6. [6]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Second Order Derivatives for Network Pruning: Optimal Brain Surgeon , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  7. [7]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Learning Both Weights and Connections for Efficient Neural Networks , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  8. [8]

    International Conference on Learning Representations (ICLR) , year=

    Pruning Convolutional Neural Networks for Resource Efficient Inference , author=. International Conference on Learning Representations (ICLR) , year=

  9. [9]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=

    Importance Estimation for Neural Network Pruning , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=

  10. [10]

    arXiv preprint arXiv:1801.05787 , year=

    Faster Gaze Prediction with Dense Networks and Fisher Pruning , author=. arXiv preprint arXiv:1801.05787 , year=

    Pith/arXiv arXiv

  11. [11]

    International Conference on Machine Learning (ICML) , year=

    The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networks , author=. International Conference on Machine Learning (ICML) , year=

  12. [12]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  13. [13]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Are Sixteen Heads Really Better than One? , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  14. [14]

    Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL) , year=

    Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned , author=. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL) , year=

  15. [15]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

    Structured Pruning of Large Language Models , author=. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

    2020

  16. [16]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL) , year=

    Structured Pruning Learns Compact and Accurate Models , author=. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL) , year=

  17. [17]

    Frantar, Elias and Alistarh, Dan , booktitle=

  18. [18]

    International Conference on Learning Representations (ICLR) , year=

    A Simple and Effective Pruning Approach for Large Language Models , author=. International Conference on Learning Representations (ICLR) , year=

  19. [19]

    Ma, Xinyin and Fang, Gongfan and Wang, Xinchao , booktitle=

  20. [20]

    Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , year=

    Fluctuation-Based Adaptive Structured Pruning for Large Language Models , author=. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , year=

  21. [21]

    and Nascimento, Marcelo Gennari do and Hoefler, Torsten and Hensman, James , booktitle=

    Ashkboos, Saleh and Croci, Maximilian L. and Nascimento, Marcelo Gennari do and Hoefler, Torsten and Hensman, James , booktitle=

  22. [22]

    Ling, Gui and Wang, Ziyang and Yan, Yuliang and Liu, Qingwen , booktitle=

  23. [23]

    Kurtic, Eldar and Frantar, Elias and Alistarh, Dan , booktitle=

  24. [24]

    van der Ouderaa, Tycho F. A. and Nagel, Markus and van Baalen, Mart and Asano, Yuki M. and Blankevoort, Tijmen , booktitle=. The

  25. [25]

    Tang, Jialong and others , journal=

  26. [26]

    Xiong, Lang and Ren, Ao and Liu, Duo and others , booktitle=

  27. [27]

    Men, Xin and Xu, Mingyu and Zhang, Qingyu and Wang, Bingning and Lin, Hongyu and Lu, Yaojie and Han, Xianpei and Chen, Weipeng , journal=

  28. [28]

    Shortened

    Kim, Bo-Kyeong and Kim, Geonmin and Kim, Tae-Ho and Castells, Thibault and Choi, Shinkook and Shin, Junho and Song, Hyoung-Kyu , journal=. Shortened

  29. [29]

    Yang, Yifei and Cao, Zouying and Zhao, Hai , journal=

  30. [30]

    Wang, Yuxin and others , journal=

  31. [31]

    Everybody Prune Now: Structured Pruning of

    Dery, Lucio and Kolawole, Steven and Kagy, Jean-Fran. Everybody Prune Now: Structured Pruning of. arXiv preprint arXiv:2402.05406 , year=

    arXiv

  32. [32]

    Tang, Shengkun and Sieberling, Oliver and Kurtic, Eldar and Shen, Zhiqiang and Alistarh, Dan , journal=

  33. [33]

    Advances in Neural Information Processing Systems (NeurIPS) , year=

    Compact Language Models via Pruning and Knowledge Distillation , author=. Advances in Neural Information Processing Systems (NeurIPS) , year=

  34. [34]

    Compressing

    Jaiswal, Ajay and Gan, Zhe and Du, Xianzhi and Zhang, Bowen and Wang, Zhangyang and Yang, Yinfei , booktitle=. Compressing

  35. [35]

    arXiv preprint arXiv:2307.09288 , year=

    Llama 2: Open Foundation and Fine-Tuned Chat Models , author=. arXiv preprint arXiv:2307.09288 , year=

    Pith/arXiv arXiv

  36. [36]

    Dubey, Abhimanyu and Jauhri, Abhinav and Pandey, Abhinav and others , journal=. The

  37. [37]

    and Sablayrolles, Alexandre and Mensch, Arthur and others , journal=

    Jiang, Albert Q. and Sablayrolles, Alexandre and Mensch, Arthur and others , journal=

  38. [38]

    arXiv preprint arXiv:2412.15115 , year=

    Qwen2.5 Technical Report , author=. arXiv preprint arXiv:2412.15115 , year=

    Pith/arXiv arXiv

  39. [39]

    Young, Alex and Chen, Bei and Li, Chao and others , journal=

  40. [40]

    2024 , howpublished=

    The Falcon 3 Family of Open Models , author=. 2024 , howpublished=

    2024

  41. [41]

    arXiv preprint arXiv:2408.00118 , year=

    Gemma 2: Improving Open Language Models at a Practical Size , author=. arXiv preprint arXiv:2408.00118 , year=

    Pith/arXiv arXiv

  42. [42]

    International Conference on Learning Representations (ICLR) , year=

    Pointer Sentinel Mixture Models , author=. International Conference on Learning Representations (ICLR) , year=

  43. [43]

    Journal of Machine Learning Research (JMLR) , volume=

    Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , author=. Journal of Machine Learning Research (JMLR) , volume=

  44. [44]

    Computational Linguistics , volume=

    Building a Large Annotated Corpus of English: The Penn Treebank , author=. Computational Linguistics , volume=

  45. [45]

    Think You Have Solved Question Answering? Try

    Clark, Peter and Cowhey, Isaac and Etzioni, Oren and Khot, Tushar and Sabharwal, Ashish and Schoenick, Carissa and Tafjord, Oyvind , journal=. Think You Have Solved Question Answering? Try

  46. [46]

    Zellers, Rowan and Holtzman, Ari and Bisk, Yonatan and Farhadi, Ali and Choi, Yejin , booktitle=

  47. [47]

    Sakaguchi, Keisuke and Bras, Ronan Le and Bhagavatula, Chandra and Choi, Yejin , journal=

  48. [48]

    Bisk, Yonatan and Zellers, Rowan and Gao, Jianfeng and Choi, Yejin and others , booktitle=

  49. [49]

    Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

    Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering , author=. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

    2018

  50. [50]

    Clark, Christopher and Lee, Kenton and Chang, Ming-Wei and Kwiatkowski, Tom and Collins, Michael and Toutanova, Kristina , booktitle=

  51. [51]

    International Conference on Learning Representations (ICLR) , year=

    Measuring Massive Multitask Language Understanding , author=. International Conference on Learning Representations (ICLR) , year=

  52. [52]

    arXiv preprint arXiv:2110.14168 , year=

    Training Verifiers to Solve Math Word Problems , author=. arXiv preprint arXiv:2110.14168 , year=

    Pith/arXiv arXiv

  53. [53]

    arXiv preprint arXiv:2107.03374 , year=

    Evaluating Large Language Models Trained on Code , author=. arXiv preprint arXiv:2107.03374 , year=

    Pith/arXiv arXiv

  54. [54]

    arXiv preprint arXiv:2108.07732 , year=

    Program Synthesis with Large Language Models , author=. arXiv preprint arXiv:2108.07732 , year=

    Pith/arXiv arXiv

  55. [55]

    Zenodo, version v0.4.0 , year=

    A Framework for Few-Shot Language Model Evaluation , author=. Zenodo, version v0.4.0 , year=

  56. [56]

    Frantar, Elias and Ashkboos, Saleh and Hoefler, Torsten and Alistarh, Dan , booktitle=

  57. [57]

    and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , booktitle=

    Hu, Edward J. and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , booktitle=

  58. [58]

    International Conference on Learning Representations (ICLR) , year=

    Scaling Laws for Sparsely-Connected Foundation Models , author=. International Conference on Learning Representations (ICLR) , year=

  59. [59]

    Nonnenmacher, Manuel and Pfeil, Thomas and Steinwart, Ingo and Reeb, David , booktitle=

  60. [60]

    Li, Guanchen and Xu, Yixing and Li, Zeping and Liu, Ji and Yin, Xuanwu and Li, Dong and Barsoum, Emad , journal=

  61. [61]

    arXiv preprint arXiv:2602.00161 , year=

    Jansen, David and Rausch, Roman and Hashemi, Ali and Montero, David and Or. arXiv preprint arXiv:2602.00161 , year=

    Pith/arXiv arXiv

  62. [62]

    Zheng, Chuanyang and Zhang, Kai and Yang, Zhi and Tan, Wenming and Xiao, Jun and Ren, Ye and Pu, Shiliang , booktitle=

  63. [63]

    Efficient

    Mugnaini, Leandro Giusti and Yamamoto, Bruno Lopes and de Alcantara, Lucas Lauton and Zacarias, Victor and Bollis, Edson and Pellicer, Lucas and Costa, Anna Helena Reali and Jordao, Artur , journal=. Efficient

  64. [64]

    Sandri, Fabrizio and Cunegatti, Elia and Iacca, Giovanni , journal=

  65. [65]

    arXiv preprint arXiv:2508.05521 , year=

    Optimal Brain Connection: Towards Efficient Structural Pruning , author=. arXiv preprint arXiv:2508.05521 , year=

    Pith/arXiv arXiv