Pith. sign in

REVIEW 5 major objections 4 minor 27 references

Layer-wise capacity allocation and pruning in LLMs can be reduced to two convex programs with unique closed-form solutions, driven by a curvature-adjusted layer gain.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 02:34 UTC pith:72YYAD5H

load-bearing objection The convex programs are correct and useful, but the experiments feed LayerIF influence scores into Algorithm 1/2 instead of the paper's own ζ_k curvature gain, so the central empirical claim is untested. the 5 major comments →

arxiv 2603.00910 v3 pith:72YYAD5H submitted 2026-03-01 cs.IT cs.AIcs.LGmath.IT

Curvature-Weighted Capacity Allocation: A Minimum Description Length Framework for Layer-Adaptive Large Language Model Optimization

Theophilus Amaefuna , Hitesh Vaidya , Anshuman Chhabra , Ankur Mali This is my paper
classification cs.IT cs.AIcs.LGmath.IT
keywords curvature-adjusted layer gainMinimum Description Lengthcapacity allocationlayer-wise pruningwater-fillingconvex optimizationtransfer regretlarge language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that layer-wise capacity decisions in large language models should be made from a curvature-adjusted layer gain ζ_k², which measures how much empirical risk can be reduced by updating layer k alone. From these gains, it derives two convex programs under a global budget: one allocates expert or adapter capacity to high-gain layers via a water-filling solution; the other assigns pruning ratios that protect high-gain layers. Both admit unique closed-form solutions parameterized by one dual variable and computable by O(K log(1/ε)) bisection. The paper also proves a transfer-regret bound showing that source-domain allocations stay near-optimal when score vectors drift. A sympathetic reader would care because this replaces a heuristic score-to-decision mapping with a provably optimal, budget-feasible procedure that is also computationally cheap.

Core claim

The central claim is that ζ_k² = g_kᵀ H̃_kk⁻¹ g_k equals twice the maximal second-order decrease in the training objective achievable by updating layer k alone, so ranking layers by ζ_k² is consistent with ranking by locally reducible empirical risk up to a controlled approximation gap. After normalizing these gains into scores q_k, the allocation program min Σ [α c_k e_k − γ q_k^β log(1+e_k)] s.t. Σ c_k e_k ≤ B has the closed-form water-filling solution e_k = max(γ q_k^β/((α+λ*)c_k) − 1, 0), and the pruning program with quadratic degradation has the closed form ρ_k = clip((b+λ*)n_k/(2η q_k^κ), 0, 1). Both are unique optimizers found by bisection; a quadratic transfer-regret bound guarantees

What carries the argument

The curvature-adjusted layer gain ζ_k² = g_kᵀ H̃_kk⁻¹ g_k — the layer-restricted Newton decrement with a Tikhonov-regularized Hessian block — is the quantity that carries the argument. It converts raw per-layer gradients into a reducible-risk surrogate, and its normalization q_k drives both programs. The closed-form stationarity conditions then reduce each constrained program to a monotone scalar equation in the single dual variable λ, which is why bisection suffices.

Load-bearing premise

The load-bearing premise is that the influence-function-based layer scores used in the experiments faithfully represent the curvature gain ζ_k² defined in the theory; the paper never specifies the conversion, so if the proxy is unfaithful the empirical validation does not test the central claim.

What would settle it

Compute ζ_k² directly from per-layer Hessian-vector products or a diagonal Fisher approximation on Mistral-7B and compare the resulting allocation and pruning ratios to those from the influence-score proxy; if the rankings differ materially or the accuracy gains reverse, the experimental support for the central claim collapses. A more direct check: update each layer alone by its proposed step, measure the actual loss decrease, and compare that ranking to ζ_k² — the identity says they should match to the stated approximation gap.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Replacing the heuristic knapsack allocation with the water-filling program yields consistent average accuracy improvements on Mistral-7B and a marginal gain on Gemma-7B, with no added compute beyond the shared score calculation.
  • The pruning program recovers the baseline's layer ratios without manual calibration while providing strong convexity, uniqueness, and budget feasibility guarantees.
  • The O(δ²) transfer-regret bound justifies warm-starting allocation and pruning decisions from source-domain curvature estimates.
  • Both algorithms run in O(K log(1/ε)) time, making the theoretically optimal decisions cheap enough for practical LLM pipelines.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the curvature gain is computed with a more faithful Hessian approximation (e.g., Kronecker-factored or diagonal Fisher) rather than influence-score proxies, the allocation gains on knowledge-intensive tasks could be larger or more consistent across models.
  • The quadratic degradation penalty ψ(ρ)=ρ² is identified by the paper as a likely source of the mixed pruning results; richer penalties such as −log(1−ρ) are a natural testable extension.
  • The missing explicit mapping from the influence-score proxies to ζ_k² in the experimental section means the reported results validate the pipeline only if that proxy is faithful; an ablation that computes ζ_k directly would settle the question.
  • The MDL framing suggests one could derive non-vacuous compression-based generalization bounds for the allocated architecture, connecting layer decisions to LLM generalization in a way the paper does not develop.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes a curvature-aware, MDL-inspired framework for layer-wise capacity allocation and pruning in LLMs. The central quantity is the layer gain ζ_k² = g_kᵀ H̃_kk⁻¹ g_k, defined as the maximal second-order decrease in a regularized layer-restricted quadratic model. After normalizing these gains into scores q_k, the authors formulate two convex programs: a capacity-allocation program with a closed-form water-filling solution and a pruning program with a closed-form clipped solution, both computed by O(K log(1/ε)) bisection. They also prove a quadratic transfer-regret bound under score drift. Experiments on Mistral-7B and Gemma-7B compare the resulting decisions with the LayerIF baseline for expert allocation and layer-wise pruning.

Significance. The theoretical core is genuinely useful: if the stated curvature scores were actually computed, the framework would replace heuristic score-to-decision mappings with budget-feasible convex programs that have unique closed-form solutions and a transfer-stability bound. The derivations of the two convex programs and the transfer regret bound are self-contained, and the algorithms are simple and inexpensive. The code is made available. However, the manuscript's experimental validation does not use the proposed ζ_k at all: the q_k values are obtained from LayerIF influence scores with no stated mapping to Eq. (8). As a result, the central empirical claim—that curvature weighting improves allocation—is not supported by the current experiments. There are also technical issues in the bisection brackets and in the treatment of clipped pruning ratios that affect the stated guarantees.

major comments (5)
  1. [§3.3, §2.2 (Eqs. 5, 8, 10)] The experiments substitute LayerIF influence scores for the paper's central curvature gain ζ_k² without any stated equivalence. Section 3.3 says q_k are 'derived from these influence scores as described in Section 2.2,' but Section 2.2 defines ζ_k² in Eq. (8) as a quadratic form in the gradient and the regularized Hessian block, while Eq. (5) defines a per-training-sample influence score I^(k)(x_i). No aggregation, normalization, or approximation connecting I^(k) to ζ_k is given. Consequently, the reported experiments test the MDL programs driven by the LayerIF proxy, not the curvature-weighted claim. The 2.66-point Mistral allocation gain cannot be attributed to curvature weighting. Please either compute ζ_k directly or provide and validate a formal approximation result connecting aggregated influence scores to ζ_k.
  2. [§3.2, Algorithms 1–2] The bisection bracket for Algorithm 2 is incorrect as stated. To ensure ρ_k(λmax)=1 for all k, one needs λmax ≥ max_k(2η q_k^κ/n_k) − b, not the min_k used in the algorithm. Algorithm 1's bracket λmax = γ/(α min_k c_k) also does not guarantee e_k(λmax)=0 for all k, because the condition depends on max_k γq_k^β/c_k − α unless additional assumptions on the costs and scores are imposed. These are load-bearing for the claimed O(K log(1/ε)) guarantee and the 'unique closed-form solution' theorem. The brackets should be corrected or the algorithm restated with valid assumptions.
  3. [§3.4, Tables 1–2] The pruning experiments apply an external post-hoc per-layer sparsity cap of 0.51 (Mistral-7B) and 0.55 (Gemma-7B) after Algorithm 2. This is not part of the theorem's [0,1] projection and may alter the KKT solution, so the reported results do not cleanly validate Theorem 3's budget-feasible solution. The paper should report the actual global sparsity achieved after clipping, the number of layers at the cap, and whether the bisection was run before or after clipping. In addition, the tables report single point estimates with no standard deviations or repetitions; the mixed Gemma pruning gaps (e.g., Wanda 49.47 vs. 52.30) are within plausible noise ranges and should not be interpreted without error bars.
  4. [Appendix C (Eq. 18)] The claimed consistency of the ζ_k ranking with the ranking by locally reducible empirical risk is not proven. Eq. (18) gives a lower bound on the true decrease in terms of ζ_k², the regularization bias, and the Taylor remainder R, but no matching upper bound or monotone relationship is derived. The statement that 'the ranking is preserved whenever τ/λ_min ≪ 1' therefore does not follow. A two-sided approximation gap, or a direct condition under which the ordering is preserved, is needed to support the central conceptual claim.
  5. [§3.4, Figure 3 and Table 5] The Gemma-7B discussion reports that Algorithm 1 'produces identical expert counts to LayerIF under the All variant' yet still shows a small average improvement. This is unexplained: if the counts are identical, the accuracy difference must come from something other than the allocation rule (e.g., rounding, evaluation variance, or a different part of the pipeline). The paper should clarify how identical counts arise from the water-filling formula and why the reported numbers differ, or remove the claim as it currently undermines the empirical argument.
minor comments (4)
  1. [General] There are several typos and inconsistencies: 'expert in mixture-of-expert' (§3.2), 'guaranties' (Abstract), and 'mixutre' in the references. The theorem numbering is also confusing: Theorem 2 and Theorem 3 are stated in §3.1 but the text refers back to §2.3 as if the theorem were introduced there.
  2. [§3.3] The hyperparameters α, γ, β, η, κ, b, and the two σ scaling factors are listed only in Appendix E. Since some of these are effectively free parameters, the paper should state whether they were tuned on a held-out split or fixed a priori. Currently the reader cannot assess the risk of overfitting to the reported benchmarks.
  3. [Figure 2, Figure 3] The figures show only point accuracies, not any measure of variability. Adding error bars or at least reporting the number of seeds would greatly improve interpretability, especially for the small average differences on Gemma-7B.
  4. [Section 2.2] The connection to LayerIF in the background section is insightful, but the statement that ζ_k² 'quantifies exactly how many bits of data-fit codelength can be recovered' is stronger than what Lemma 1 establishes. Lemma 1 concerns a quadratic surrogate of empirical risk, not codelength; the MDL interpretation is motivational rather than derived.

Circularity Check

0 steps flagged

No significant circularity in the theoretical derivation; the experimental substitution of LayerIF scores for the paper's curvature gain q_k is a validation gap, not a constructional circular step.

full rationale

The claimed derivation chain is self-contained and does not reduce to its inputs by construction. Section 2.2 defines zeta_k^2 = g_k^T Htilde_kk^{-1} g_k, and Lemma 1 proves that this equals twice the minimum of the regularized layer-restricted quadratic via explicit first-order stationarity. The allocation program Eq. (11) and pruning program Eq. (14) are stated as convex objectives, and Theorems 2 and 3 derive their closed-form solutions from KKT conditions; the transfer-regret bound in Theorem 4 follows from strong convexity, smoothness, and a score-Lipschitz assumption. None of these results assumes the empirical allocation or pruning outcomes, so they are not fitted predictions. Self-citations to Vitel and Chhabra 2026, Prada et al. 2025, and Askari et al. 2025 are present but not load-bearing: they supply background, motivation, or the LayerIF baseline, and the core lemmas do not depend on them. However, Section 3.3 states that 'The normalized curvature scores q_k entering Algorithm 1 are derived from these influence scores as described in Section 2.2,' while Section 2.2 defines q_k from zeta_k^2 only; no aggregation or transformation connecting the LayerIF score I^(k) in Eq. (5) to zeta_k^2 is given. The experiments therefore compare MDL optimization on a LayerIF influence-score proxy against the LayerIF heuristic using the same inputs, leaving the curvature-specific claim empirically unvalidated. This is a genuine missing-equivalence/correctness risk, but it is not an equivalence-by-construction in the theoretical derivation chain, so the circularity score remains low.

Axiom & Free-Parameter Ledger

10 free parameters · 6 axioms · 0 invented entities

The theory is mathematically self-contained given the modeling choices, but the scores used in experiments come from a different pipeline (LayerIF influence scores), and all utility/penalty forms are postulated without independent evidence. Several hyperparameters and scaling/clipping constants are chosen by hand.

free parameters (10)
  • α (allocation cost penalty) = 0.5
    Scales the linear cost in Eq (11); chosen by hand with no sensitivity analysis.
  • γ (allocation benefit strength) = 0.9
    Scales the log-utility benefit in Eq (11); chosen by hand.
  • β (allocation curvature emphasis) = 1
    Exponent on q_k in Eq (12); paper says β=1 or 2 work well, no systematic selection.
  • η (pruning penalty scale) = 2
    Scales the degradation penalty in Eq (14); chosen by hand.
  • κ (pruning curvature emphasis) = 1
    Exponent on q_k in Eq (15); chosen by hand.
  • b (bits per retained parameter) = 16
    Assumed bits per parameter in the pruning model; no empirical basis given.
  • σ scaling factor (Mistral-7B) = 0.0276
    Sets FLOPs budget B = σ × base model FLOPs; no derivation, likely tuned to make allocation feasible.
  • σ scaling factor (Gemma-7B) = 0.02
    Sets FLOPs budget B = σ × base model FLOPs; no derivation.
  • Per-layer sparsity cap = 0.51 (Mistral), 0.55 (Gemma)
    Post-hoc clipping to ensure the global 50% sparsity target is met while avoiding full ablation of any layer.
  • τ (Tikhonov regularization) = not specified
    H̃_kk = H_kk + τI requires τ>0, but no value is given; it affects ζ_k and the ranking consistency bound.
axioms (6)
  • standard math Taylor's theorem with integral remainder and M-Lipschitz Hessian (Eq 6)
    Foundation of the second-order expansion; assumes ∇²L is locally Lipschitz.
  • domain assumption H_kk + τI is positive definite (Eq 7)
    Needed for invertibility and convexity of the layer-restricted surrogate; neural network Hessians are generally indefinite, and τ is not specified.
  • ad hoc to paper MDL codelength is linear in resources and data-fit reduction is log-concave (Eq 2, Sec 2.3)
    The modeling choice Cl(θ_e)∝αc_k e_k and ΔCl(D|θ)∝γ q^β log(1+e) is postulated, not derived from an actual code.
  • ad hoc to paper Pruning degradation is quadratic in ρ_k weighted by q_k^κ (Eq 14)
    ψ(ρ)=ρ² is a design choice; the paper admits it may underestimate pruning sensitivity (Section 3.4, Appendix F).
  • ad hoc to paper Normalized scores q_k from LayerIF influence scores represent the curvature gain ζ_k (Sec 3.3)
    The mapping is never derived; this is the load-bearing empirical premise and the weakest step in the paper.
  • domain assumption Strong convexity, smoothness, and score-gradient Lipschitz assumptions (Appendix D)
    Standard regularity conditions for the transfer-regret bound; constants L_x, σ, L_xq are not made explicit.

pith-pipeline@v1.3.0-alltime-deepseek · 84 in / 14308 out tokens · 137432 ms · 2026-08-03T02:34:05.015923+00:00 · methodology

0 comments
read the original abstract

Layer-wise capacity in large language models is highly non-uniform: some layers contribute disproportionately to loss reduction, whereas others are nearly redundant. Existing layer-scoring methods provide sensitivity estimates but do not give a principled rule for converting those estimates into allocation or pruning decisions under a global hardware budget. We introduce a curvature-aware, MDL-inspired framework built around the layer gain $\zeta_k^2=g_k^\top\widetilde H_{kk}^{-1}g_k$. This quantity equals twice the maximal decrease predicted by the regularized layer-restricted quadratic model and incorporates inverse local curvature; it is therefore a local surrogate for reducible risk, not a universal dominance claim over gradient-norm scores. After normalizing the gains into scores $q_k$, we formulate two convex programs: one allocates expert slots under diminishing returns, and the other assigns layer-wise pruning ratios while protecting high-score layers. Both continuous programs have unique globally optimal solutions characterized by one dual variable and computable in $O(K\log(1/\varepsilon))$ time by bisection. We also prove a quadratic transfer-regret bound: when source and target score vectors differ by at most $\delta$, the target surrogate cost of the transferred decision is within $O(\delta^2)$ of the target optimum. Experiments on Mistral-7B and Gemma-7B show clear allocation gains in some settings and competitive, though mixed, pruning performance. The framework therefore replaces an empirical score-to-decision heuristic with a budget-feasible optimization procedure whose guarantees apply to the stated continuous surrogates. Code is available on github repo - [TKAI-LAB-Mali/Curvature-Weighted-Capacity-Allocation](https://github.com/TKAI-LAB-Mali/Curvature-Weighted-Capacity-Allocation.git)

Figures

Figures reproduced from arXiv: 2603.00910 by Ankur Mali, Anshuman Chhabra, Hitesh Vaidya, Theophilus Amaefuna.

Figure 1
Figure 1. Figure 1: (a) Layer-wise curvature scores ζ 2 k vary substantially across the transformer stack. High-gain layers (dark bars) hold disproportionate reducible risk and should receive additional capacity; low-gain layers (light bars) are candidates for aggressive pruning. (b) Our framework computes ζ 2 k from per-layer gradients gk and regularized Hessian blocks Hekk, normalizes them to quality scores qk, and solves t… view at source ↗
Figure 2
Figure 2. Figure 2: Expert allocation accuracy (%) on Mistral-7B-v0.1 (5 epochs) [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Expert allocation accuracy (%) on Gemma-7B, +ve variant only (5 epochs) [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 11 linked inside Pith

  1. [1]

    Table 3: Hyperparameter configurations for allocation and pruning

    Layer sizesn k are set to the number of parameters in layerk, and per-unit costsc k are set to the FLOPs of a single LoRA expert at layerk. Table 3: Hyperparameter configurations for allocation and pruning. Program Parameter Mistral-7B Gemma-7B Allocation α0.5 0.5 γ0.9 0.9 budget scaling,σ0.0276 0.02 Pruning b(bits) 16 16 η2 2 κ1 1 Sparsity target,S50% 50...

    2025

  2. [3]

    doi: 10.18653/v1/N19-1300

    Association for Computational Linguistics. doi: 10.18653/v1/N19-1300. URL https://aclanthology.org/N19-1300/. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge,

    doi:10.18653/v1/n19-1300

  3. [8]

    doi: 10.18653/v1/2023.emnlp-main.907

    Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.907. URLhttps: //aclanthology.org/2023.emnlp-main.907/. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, ...

    doi:10.18653/v1/2023.emnlp-main.907 2023

  4. [9]

    Pang Wei Koh and Percy Liang

    URLhttps://arxiv.org/abs/2310.06825. Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. InInternational con- ference on machine learning, pages 1885–1894. PMLR,

    Pith/arXiv arXiv

  5. [14]

    doi: 10.18653/v1/ D18-1260

    Association for Computational Linguistics. doi: 10.18653/v1/ D18-1260. URLhttps://aclanthology.org/D18-1260/. Benjamin Prada, Shion Matsumoto, Abdul Malik Zekri, and Ankur Mali. Bridging predictive coding and mdl: A two- part code framework for deep learning.ArXiv, abs/2505.14635,

    Pith/arXiv arXiv

  6. [15]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J

    URLhttps://arxiv.org/abs/2410.10054. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer,

    Pith/arXiv arXiv

  7. [16]

    Jorma Rissanen

    URL https://arxiv.org/abs/1910.10683. Jorma Rissanen. Modeling by shortest data description*.Autom., 14:465–471,

    Pith/arXiv arXiv 1910

  8. [17]

    doi: 10.1145/3474381

    ISSN 0001-0782. doi: 10.1145/3474381. URLhttps://doi.org/10.1145/3474381. Jürgen Schmidhuber. Discovering neural nets with low kolmogorov complexity and high generalization capability. Neural Networks, 10(5):857–873,

    doi:10.1145/3474381

  9. [19]

    Mingjie Sun, Zhuang Liu, Anna Bair, and J

    URLhttps://arxiv.org/abs/2411.10069. Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models.arXiv preprint arXiv:2306.11695,

    Pith/arXiv arXiv

  10. [20]

    CommonsenseQA: A question answer- ing challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answer- ing challenge targeting commonsense knowledge. In Jill Burstein, Christy Doran, and Thamar Solorio, ed- itors,Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies,...

    2019

  11. [21]

    doi: 10.18653/v1/N19-1421

    Association for Computational Linguistics. doi: 10.18653/v1/N19-1421. URL https://aclanthology.org/N19-1421/. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, ...

    doi:10.18653/v1/n19-1421

  12. [22]

    Leslie G

    URLhttps://arxiv.org/abs/2403.08295. Leslie G. Valiant. A theory of the learnable.Commun. ACM, 27:1134–1142,

    Pith/arXiv arXiv

  13. [23]

    Alex Warstadt, Amanpreet Singh, and Samuel R

    URLhttps://arxiv.org/abs/ 1804.07461. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments.Transactions of the Association for Computational Linguistics, 7:625–641, 09

    Pith/arXiv arXiv

  14. [24]

    doi: 10.1162/tacl_a_00290

    ISSN 2307-387X. doi: 10.1162/tacl_a_00290. URLhttps://doi.org/10.1162/tacl_a_00290. Andrew Gordon Wilson. Position: Deep learning is not so mysterious or different. InForty-second International Conference on Machine Learning Position Paper Track,

    doi:10.1162/tacl_a_00290

  15. [25]

    More recent work scales these ideas to modern architectures using diagonal Fisher approximations Martens and Grosse [2015], Kronecker-factored curvature Botev et al

    frameworks, which identify parameters to remove by computing the Hessian of the training loss. More recent work scales these ideas to modern architectures using diagonal Fisher approximations Martens and Grosse [2015], Kronecker-factored curvature Botev et al. [2017], and randomized sketches. Our work extends this tradition from individual weight pruning ...

    2015

  16. [26]

    address this by introducing compression-based bounds via SubLoRA, a low-dimensional nonlinear parametrization that yields non-vacuous guarantees. Our MDL objective is directly motivated by this line of work: minimizing description length simultaneously controls generalization and penalizes unnecessary model com- plexity, grounding our convex programs in i...

    2017

  17. [1989]

    cc/paper_files/paper/1989/file/6c9882bbac1c7093bd25041881277658-Paper.pdf

    URLhttps://proceedings.neurips. cc/paper_files/paper/1989/file/6c9882bbac1c7093bd25041881277658-Paper.pdf. Hongyang Liu, Sara Elkerdawy, Nilanjan Ray, and Mostafa Elhoushi. Layer importance estimation with imprinting for neural network quantization. In2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 2408–2417,

    1989

  18. [1993]

    Shwai He, Run-Ze Fan, Liang Ding, Li Shen, Tianyi Zhou, and Dacheng Tao

    doi: 10.1109/ICNN.1993.298572. Shwai He, Run-Ze Fan, Liang Ding, Li Shen, Tianyi Zhou, and Dacheng Tao. Merging experts into one: Improving computational efficiency of mixture of experts. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14685–14691, Singapor...

    arXiv 1993

  19. [1997]

    doi: https://doi.org/10.1016/S0893-6080(96)00127-X

    ISSN 0893-6080. doi: https://doi.org/10.1016/S0893-6080(96)00127-X. URLhttps://www.sciencedirect.com/science/article/pii/S089360809600127X. Shai Shalev-Shwartz and Shai Ben-David.Understanding machine learning : from theory to algorithms. Cambridge University Press, Cambridge,

    doi:10.1016/s0893-6080(96)00127-x

  20. [2004]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova

    Available athttps://web.stanford.edu/~boyd/cvxbook/. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio, editors,Proceedings of the 2019 Conference of the North American Chapter o...

    2019

  21. [2015]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors,Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381– 2391, Brussels, Belgium, ...

    2018

  22. [2017]

    Datainf: Efficiently estimating data influence in lora-tuned llms and diffusion models.arXiv preprint arXiv:2310.00902,

    Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in lora-tuned llms and diffusion models.arXiv preprint arXiv:2310.00902,

    Pith/arXiv arXiv

  23. [2018]

    org/abs/1803.05457

    URLhttps://arxiv. org/abs/1803.05457. Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. InThird International Workshop on Paraphrasing (IWP2005). Asia Federation of Natural Language Processing, January

    Pith/arXiv arXiv

  24. [2019]

    doi: 10.1016/j.neunet.2019.04.009

    ISSN 0893-6080. doi: 10.1016/j.neunet.2019.04.009. URLhttps://doi.org/10.1016/j.neunet. 2019.04.009. Avrim Blum and John Langford. Pac-mdl bounds. In Bernhard Schölkopf and Manfred K. Warmuth, editors,Learning Theory and Kernel Machines, pages 344–357, Berlin, Heidelberg,

    doi:10.1016/j.neunet.2019.04.009 2019

  25. [2021]

    Sanae Lotfi, Marc Finzi, Yilun Kuang, Tim G

    doi: 10.1109/CVPRW53098.2021.00273. Sanae Lotfi, Marc Finzi, Yilun Kuang, Tim G. J. Rudner, Micah Goldblum, and Andrew Gordon Wilson. Non-vacuous generalization bounds for large language models. InInternational Conference on Machine Learning,

    arXiv 2021

  26. [2023]

    Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, and VS Subrahmanian

    URLhttps://arxiv.org/abs/2301.00774. Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, and VS Subrahmanian. Higher layers need more lora experts,

    Pith/arXiv arXiv

  27. [2024]

    11 Song Han, Jeff Pool, John Tran, and William J

    URLhttps://arxiv.org/abs/ 2402.08562. 11 Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. InProceedings of the 29th International Conference on Neural Information Processing Systems - Volume 1, NIPS’15, page 1135–1143, Cambridge, MA, USA,

    Pith/arXiv arXiv