Pith. sign in

REVIEW 4 major objections 5 minor 88 references

Forget the Data and Fine-Tuning! Just Fold the Network to Compress

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Model folding compresses neural networks by clustering and merging structurally similar neurons across layers, requiring neither training data nor fine-tuning.

desk verdict Data-free folding via k-means on joint adjacent-layer weights is a real and useful trick on CNNs; the LLaMA 'comparable' claim is overstated and the 70%-sparsity number needs a repair-variant audit. read the letter →

arxiv 2502.10216 v2 pith:L5Y257G2 submitted 2025-02-14 cs.LG cs.AI

Dong Wang , Haris Šikić , Lothar Thiele , Olga Saukh This is my paper · ORCID
classification cs.LGcs.AI MSC 68T0762H30
keywords modelfoldingdata-freecompressionk-meansclusteringvariancecollapserepairstructuredpruninglargelanguagedeepinversion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Model folding is a compression method that shrinks a neural network by grouping and merging structurally similar neurons within each layer, using k-means clustering, and then repairing the damage this merging does to the network's internal activation statistics. The paper argues that this repair can be done without any training data or fine-tuning: Fold-AR estimates the lost variance from weight geometry alone, and Fold-DIR generates a single synthetic batch from the network itself. On ResNet18 trained on CIFAR10, folded models keep over 80% accuracy at 70% sparsity, while structured magnitude pruning collapses to near chance. The authors claim their data-free method is comparable to data-driven compression and substantially better than earlier data-free pruning, and that it extends to LLaMA-7B without post-training.

What carries the argument

The central object is the clustering projection matrix $C = U(U^T U)^{-1}U^T$ induced by k-means, applied not to a single weight matrix but to a concatenated "super weight" that couples adjacent layers and BatchNorm diagonals. K-means on this super weight is the mechanism that simultaneously merges channels and selects the same cluster structure for the next layer's input. The companion mechanism is the variance-ratio repair: Fold-AR rescales each centroid by $N_c / \sqrt{N_c + (N_c^2 - N_c) E[c]}$ using an assumed uncorrelated-input estimate of the intra-cluster correlation $E[c]$; Fold-DIR replaces that estimate by statistics measured on a single synthetic batch. The argument's load is carried by the claim that keeping the variance ratio near one preserves network function.

What would settle it

Measure the layer-wise variance ratio and per-class accuracy for a Fold-AR-compressed ResNet18 on CIFAR10 at 70% sparsity; the method's central claim predicts the ratio stays close to 1 and accuracy stays above 80%, so a ratio far from 1 coupled with a large accuracy drop would falsify the repair mechanism.

Watch

Extended reading notes

Core claim

The central claim is that structured compression can be treated as a clustering problem in weight space: cluster rows of $W_l$ together with the corresponding columns of $W_{l+1}$ (and the BatchNorm scaling and normalization diagonals) into a combined matrix $W_{l,l+1}=[W_l \mid W_{l+1}^T]$, solve k-means, and replace each cluster by its centroid. This minimizes the joint Frobenius reconstruction error, and the paper proves this is the optimal data-free merging rule under that objective. The discovery is that the resulting variance collapse, the reason earlier merging methods fail, can be repaired without data by rescaling each cluster centroid by a factor computed from the cluster size and an estimated intra-cluster correlation (Fold-AR), or by re-estimating BatchNorm statistics on a Deep-Inversion-generated image batch (Fold-DIR). With either repair, the variance ratio of compressed to uncompressed activations stays close to one, and accuracy is preserved at high sparsity levels.

Load-bearing premise

The load-bearing premise is that weight-space similarity, two neurons whose weight vectors are close, is a reliable stand-in for functional similarity on real inputs, so merging close weight vectors and rescaling variances leaves the network's behavior intact.

Editorial extensions

If this is right

  • At 70% sparsity, folded ResNet18 on CIFAR10 stays above 80% accuracy while structured magnitude pruning falls to near random chance.
  • Folding wider variants of VGG11 and ResNet50 yields larger accuracy gains, because wider layers contain more correlated, redundant channels.
  • Fold-DIR closely matches the data-driven REPAIR baseline on ResNet18, and Fold-AR outperforms the earlier data-free IFM method while being cheaper and fully data-free.
  • Model folding compresses LLaMA-7B without fine-tuning, and unlike calibration-based pruning it is robust when the only available calibration data is out of distribution.
  • On edge devices, folded models reduce RAM and flash usage and speed up inference at higher sparsity levels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If weight-space redundancy is the resource, then per-layer correlation statistics could predict how much each layer can be folded; the paper leaves sparsity allocation uniform, so adaptive layer-wise sparsity is a natural extension.
  • Fold-AR's uncorrelated-input assumption is testable: feeding a single minibatch of random or synthetic input through the uncompressed network to measure true intra-cluster correlations would likely tighten the repair, and the gap between Fold-AR and data-driven REPAIR is plausibly the cost of that assumption.
  • A practical implication the authors do not develop is that folding produces a smaller dense network rather than a sparse one, so it can be composed with quantization or further pruning to push edge-device memory even lower.
  • The authors' "comparable" claim for LLMs is strongest when calibration data is scarce or domain-mismatched; when in-distribution calibration data is available, data-driven methods still hold a perplexity edge, so the practical win is for privacy- and regulation-constrained settings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces "model folding," a data-free and fine-tuning-free structured compression method that clusters structurally similar neurons (channels) with k-means on a concatenation of adjacent-layer weights, merges them, and then repairs the resulting activation statistics. Two data-free repair mechanisms are proposed: Fold-AR, which estimates an intra-cluster correlation under an uncorrelated-input assumption and rescales the cluster centroids to preserve variance, and Fold-DIR, which synthesizes a single batch of images via Deep Inversion to update BatchNorm statistics. The authors derive a closed-form Frobenius reconstruction cost for the joint folding of adjacent layers and claim k-means optimally minimizes this cost. Empirically, the paper reports favorable results against IFM, structured magnitude pruning, and knowledge-distillation baselines on ResNet18/VGG11/ResNet50 for CIFAR10, CIFAR100, and ImageNet, plus results on LLaMA-7B and LLaMA2-7B. The central empirical claim is that at 70% sparsity a folded ResNet18 on CIFAR10 retains over 80% accuracy without data or fine-tuning.

Significance. If the claims hold, model folding would be a useful addition to the data-free compression toolbox: it provides a structured way to merge redundant units, the repair coefficients are derived in closed form from weights and cluster sizes rather than fitted to benchmark accuracies, and the authors release code. The supporting lemmas in Appendix B and the Fold-AR variance-correction formula are worked out analytically, which is a genuine strength. However, the significance is tempered by several gaps: the optimization criterion is a weight-space Frobenius proxy with no established link to activation error or task accuracy; the Fold-AR correction relies on an uncorrelated-input assumption and neglects biases without an oracle-based validation; and the headline high-sparsity numbers are presented without consistently specifying which repair variant produced them. The LLM experiments also show a clear gap to data-driven baselines in the standard setting, so the "comparable" wording in the abstract and conclusion overstates the evidence. These issues are fixable in revision, but they currently affect the strength of the central data-free claim.

major comments (4)
  1. [Section 3.1, Eq. (J_{l,l+1})] The theoretical optimality claim concerns the Frobenius reconstruction error of the weight matrices, not the error in activations or final outputs. No bound or experiment connects the minimized quantity J_{l,l+1} to task accuracy, so the statement that k-means is the "theoretically optimal" folding method does not by itself justify the empirical accuracy claims. The authors should either add a perturbation analysis relating the weight-space error to activation error under the network's Lipschitz constants, or provide an ablation showing that activation-level error tracks J_{l,l+1} across sparsity levels and repair variants.
  2. [Section 3.2 and Algorithm 1] Fold-AR estimates the intra-cluster correlation E[c] under the explicit assumption that the previous layer's outputs are uncorrelated, and it corrects only the variance of the merged preactivations; the derivation neglects biases, and Algorithm 1 contains no bias-handling step. Since variance correction is the core mechanism distinguishing Fold-AR from naive merging or IFM, this approximation should be validated against oracle statistics obtained from a forward pass. The authors should report (i) whether biases are fused, averaged, or dropped, and (ii) a comparison of predicted versus oracle E[c] and variance ratios for at least one model and several sparsity levels.
  3. [Section 4, Figures 5-6 and Appendix J] The headline claim that "at 70% sparsity, the folded ResNet18 on CIFAR10 maintains over 80% accuracy" is not attributable to a specific repair variant: Fig. 5 shows Fold-AR below Fold-R at most sparsities, while Fig. 6 and Table 6 use the generic label "model folding" without stating whether the plotted results come from Fold-AR or Fold-DIR. This ambiguity blocks verification of the data-free claim at high sparsity. The authors should label each curve with the exact variant, and if the 70%-sparsity result uses Fold-DIR, state the cost of the Deep Inversion step; if it uses Fold-R, the abstract and conclusion must be re-scoped to distinguish data-driven from data-free results.
  4. [Table 3 and Section 4, "Folding LLMs"] The abstract and Section 4 state that model folding achieves "comparable performance" to data-driven methods on LLaMA-7B, but Table 3 shows WikiText2 perplexity 13.33 versus 6.87 to 10.53 for data-driven baselines, and zero-shot average 50.17 versus 54.27 to 63.00. "Comparable" is only defensible when the data-driven methods are run with out-of-distribution calibration data, as in the Hungarian/Ukrainian example. The claims should be explicitly re-scoped to "competitive in a data-free setting" and the OOD caveat should appear in the abstract and conclusion rather than only in the experimental narrative.
minor comments (5)
  1. [Section 3.3] The block-matrix notation C = [P I] in the K-Means/LAP connection is dimensionally inconsistent as written, since X = [WA; WB] and the subsequent Frobenius expression is not well defined for the stated matrix sizes; the authors should specify the dimensions of each block and write the objective with explicit block matrices.
  2. [Figure 6 caption] The caption should state which dataset and which repair variant each row corresponds to, and it should include numeric labels or points because the right-column ImageNet results are described in the text as collapsing across all methods.
  3. [Section 4, "Folding LLMs"] The LLaMA experiments apply clustering without REPAIR because there is no BatchNorm, but the main text does not state this clearly; the variance-correction theory developed in Section 3.2 therefore does not apply to the LLM results, and the text should present the LLM study as a separate extension rather than as a direct validation of the repair mechanisms.
  4. [Section 3.2] The sentence "Fold-DIR ourperforms Fold-AR" contains a typo and should read "outperforms".
  5. [Appendix J, Table 6] The "Model Folding (ours)" row reports accuracy values without standard deviations or seed information, while the KD baselines have long runtimes; adding error bars or at least multiple seeds would strengthen the comparison.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: k-means objective and Fold-AR variance correction are closed-form derivations, not fitted predictions, and the empirical comparisons use external baselines and public checkpoints.

full rationale

The core derivation chain is self-contained in the relevant sense. The clustering objective J_{l,l+1} = || [W_l | W_{l+1}^T] - C[W_l | W_{l+1}^T] ||_F^2 is a standard k-means / matrix-factorization reconstruction error; minimizing it by k-means is a mathematical fact, not an accuracy-fitted parameter. Fold-AR's variance scaling is derived in closed form from cluster size and cosine similarity of normalized weights under an explicitly stated uncorrelated-input assumption; no benchmark accuracy is used to set the scaling, so the high-sparsity accuracy numbers are an empirical test rather than a construction. Fold-DIR uses the external Deep Inversion technique with statistics stored in BatchNorm layers; it does not depend on this paper's own results. Comparisons against IFM, INN, magnitude pruning, Wanda, FLAP, LLM-Pruner, and REPAIR are run on public or prior checkpoints against external methods, so the headline comparisons are not fitted or defined into existence. The paper does cite prior work by a shared author (Jordan et al. 2022, REPAIR) as a conceptual starting point, but that citation is to an established, published method and is not used to forbid alternatives or to import an unverified uniqueness theorem. The acknowledged limitations -- neglecting biases in Fold-AR, assuming uncorrelated previous-layer outputs, not optimizing per-layer sparsity, and showing that L1-regularized training favors pruning -- are correctness and scope caveats, not circularity. The LLM 'comparable' wording is stronger than Table 3 numerically supports, but overclaiming is not the same as a derivation that reduces to its inputs.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The theory carries no fitted constants: the Fold-AR variance correction is closed-form, and the clustering objective is a standard matrix factorization. The load-bearing premises are the redundancy premise, the weight-space proxy, and the variance-only repair, all empirically motivated but unproven. The main practical knobs are the user-specified sparsity ratio and the unreported k-means and Deep Inversion hyperparameters.

free parameters (3)
  • Per-layer cluster count (sparsity ratio) = User-specified; experiments use 10% to 70-80% sparsity applied uniformly across layers
    The number of clusters k per layer is a hand-set compression target, not fitted to accuracy. The paper applies the same ratio to all layers and explicitly leaves per-layer sparsity optimization to future work (Section 5, Limitations).
  • k-means initialization and restart schedule = Unreported
    k-means is initialization-sensitive and Lloyd's algorithm finds only local optima; the paper does not state the initialization (e.g., k-means++) or the number of restarts, which affects the clusters produced (Section 3.1, Algorithm 1).
  • Deep Inversion regularizer weights and synthesis iterations (Fold-DIR) = Not reported in this paper; inherited from Yin et al. 2020
    The DI objective R(x) in Section 3.2 has weights on the class loss, BN-statistic matching, L2, and total variation terms. The paper does not state these values, yet Fold-DIR's repair quality depends on them.
assumptions (6)
  • domain assumption SGD-trained networks contain many similar or correlated neurons (redundancy premise).
    Section 3.1 and Figure 2 establish correlated hidden units empirically; the whole method exploits this, and the paper concedes that effectiveness is limited in low-redundancy networks (Section 5).
  • domain assumption The joint Frobenius weight-reconstruction cost J is a faithful proxy for preserving network function.
    Section 3.1 defines J as the approximation error and claims k-means minimizes it, but no bound relates J to the network's output error or accuracy; optimality claims concern only the proxy.
  • domain assumption Previous-layer activations are uncorrelated when estimating intra-cluster correlation in Fold-AR.
    Section 3.2: "E[c] can be estimated by assuming that the output values x_l of the previous layer are uncorrelated." If false, the variance repair is miscalibrated.
  • domain assumption Variance-only repair is sufficient; biases, mean shifts, and cross-cluster covariances can be neglected.
    Section 3.2: "we neglect biases for ease of notation." Averaging units with different biases shifts activation means, which the method does not correct; the paper validates this only empirically.
  • standard math k-means as matrix factorization and the projection optimality of C = U(U^T U)^-1 U^T.
    Used throughout Section 3.1 for the centroid formula and the combined-cost minimization; standard linear algebra, cited to Bauckhage 2015.
  • domain assumption Deep Inversion synthesizes images whose statistics match the training distribution well enough to recalibrate BatchNorm.
    Section 3.2, Fold-DIR: relies on the premise from Yin et al. 2020 that BN-statistic matching yields useful synthetic data; the quality depends on DI hyperparameters and the model's stored statistics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Forget the Data and Fine-Tuning! Just Fold the Network to Compress." pith.science (2026). https://pith.science/paper/L5Y257G2

@misc{pith2026250210216,
  author       = {Pith},
  title        = {Pith review of: Forget the Data and Fine-Tuning! Just Fold the Network to Compress},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L5Y257G2}},
  note         = {Machine review of arXiv:2502.10216}
}
read the original abstract

We introduce model folding, a novel data-free model compression technique that merges structurally similar neurons across layers, significantly reducing the model size without the need for fine-tuning or access to training data. Unlike existing methods, model folding preserves data statistics during compression by leveraging k-means clustering, and using novel data-free techniques to prevent variance collapse or explosion. Our theoretical framework and experiments across standard benchmarks, including ResNet18 and LLaMA-7B, demonstrate that model folding achieves comparable performance to data-driven compression techniques and outperforms recently proposed data-free methods, especially at high sparsity levels. This approach is particularly effective for compressing large-scale models, making it suitable for deployment in resource-constrained environments.

Figures

Figures reproduced from arXiv: 2502.10216 by the authors.

Figure 1
Figure 1. Model compression and repair of data statistics. Left: Model folding pipeline is applied layer-wise, consisting of three phases: weight tensor clustering and merging, and data statistics repair. Right: To maintain accuracy, the data variances of compressed and uncompressed models must align (i.e., the variance ratio must be close to 1), as variance collapse or explosion leads to suboptimal performance. Our data-free… view at source ↗
Figure 2
Figure 2. Layer-wise correlation between matched channels in ResNet18 trained on CIFAR10. For each layer, we use activation matching matching with L2 distance measure to greedily pair similar neurons. Each subplot shows the correlation within all matched pairs. assumes disjoint task and worker sets. To overcome the challenge Chen et al. (2023) and He et al. (2018) apply iterative approach greedily merging a pair of the most s… view at source ↗
Figure 3
Figure 3. k-means (KM) outperforms other clustering methods: Spectral Clustering (SC), Agglomer￾ative Clustering (AC) with different linkage criteria and Iterative Greedy (greedy) used to compress ResNet18 trained on CIFAR10. Data-based REPAIR was used to restore data statistics after clustering for all methods. leading to network performance decline [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Variance collapse and overshooting on ResNet18 with CIFAR10. The goal is to align the layer-wise variance in the compressed network to that of the uncompressed model. Naive averaging of statistics (Fold-Naive) leads to variance collapse (Jor￾dan et al., 2022), while IF…
Figure 6
Figure 6. Figure 6: Comparison with IFM (Chen et al., 2023) and structured magnitude pruning (Cai et al., 2020; Yin et al., 2022). Model folding, when tested on ResNet18 (top row) and VGG11-BN (bottom row) trained on CIFAR10 (left column) and ImageNet (right column), outperforms IFM with …
Figure 7
Figure 7. Figure 7: Comparison of model folding with IFM (Chen et al., 2023), and INN (Solodskikh et al., 2023) using ResNet18 on CIFAR10. In the original experiment defined in the IFM and INN papers, where only the last two blocks of a ResNet18 are pruned, folding is significantly better…
Figure 8
Figure 8. Figure 8: Layer-wise correlation among matched channels in VGG11 and its wider variants on CIFAR10. This figure shows correlation matrices for each layer of VGG11 and its 1x and 3x wider variants, derived from activation matching. Opaque black represents the 1x wider model, whil…
Figure 9
Figure 9. Figure 9: Similar patterns in weight map of conv1 layer in ResNet18 pre-trained on Ima￾geNet (Deng et al., 2009). Each small square represents the weights of a single filter in cool-warm color map, where each color of grid corresponds to a weight value. To investigate the filter…
Figure 10
Figure 10. Figure 10: Layer-wise correlation between matched channels in ResNet18 trained on ImageNet. We compute a layer-wise correlation matrix by matching activations between channels, then assign each channel its best match in the same layer using a greedy pairing based on the correlat…
Figure 11
Figure 11. Figure 11: ResNet18 (left column) and VGG11 (right column) models trained with L1 (top row) and L2 (bottom row) regularization. Structured magnitude pruning outperforms model folding only if training explicitly regularizes for model sparsity (L1 norm). REPAIR is hardly beneficia…
Figure 12
Figure 12. Figure 12: Model folding performance improves with increasing model width. The MLP model consists of three stacked mlp blocks (including a fully connected layer, a BN layer, and a ReLU layer), followed by a final classifier. Upscaled versions of MLP (left) and ResNet50 (right) a…
Figure 13
Figure 13. Figure 13: Sample images generated by Deep Inversion (Yin et al., 2020) using ResNet18 trained on CIFAR100. These images are generated from the uncompressed network and used in model folding to restore data statistics in the compressed network. M Further Related Work Model foldi…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

88 extracted references · 30 canonical work pages

  1. [1]

    S. K. Ainsworth, J. Hayase, and S. Srinivasa. Git re-basin: Merging models modulo permutation symmetries, 2023. URL https://arxiv.org/abs/2209.04836

    arXiv 2023

  2. [2]

    Y. An, X. Zhao, T. Yu, M. Tang, and J. Wang. Fluctuation-based adaptive structured pruning for large language models, 2023. URL https://arxiv.org/abs/2312.11983

    arXiv 2023

  3. [3]

    Arduino nano 33 ble documentation

    Arduino . Arduino nano 33 ble documentation. https://docs.arduino.cc/hardware/nano-33-ble/, 2024. Accessed: 2024-11-19

    2024

  4. [4]

    Ashkboos, M

    S. Ashkboos, M. L. Croci, M. G. do Nascimento, T. Hoefler, and J. Hensman. Slicegpt: Compress large language models by deleting rows and columns, 2024. URL https://arxiv.org/abs/2401.15024

    arXiv 2024

  5. [5]

    Bauckhage

    C. Bauckhage. k-means clustering is matrix factorization, 2015. URL https://arxiv.org/abs/1512.07548

    arXiv 2015

  6. [6]

    C. M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag, Berlin, Heidelberg, 2006. ISBN 0387310738

    2006

  7. [7]

    Bommasani, D

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021. URL https://arxiv.org/abs/2108.07258

    arXiv 2021

  8. [8]

    H. Cai, C. Gan, T. Wang, Z. Zhang, and S. Han. Once-for-all: Train one network and specialize it for efficient deployment, 2020

    2020

Show all 88 references
  1. [9]

    Chang, X

    Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15 0 (3): 0 1--45, 2024

    2024

  2. [10]

    H. Chen, Y. Wang, C. Xu, Z. Yang, C. Liu, B. Shi, C. Xu, C. Xu, and Q. Tian. Data-free learning of student networks, 2019. URL https://arxiv.org/abs/1904.01186

    2019 arXiv

  3. [11]

    Y. Chen, B. Zheng, Z. Zhang, Q. Wang, C. Shen, and Q. Zhang. Deep learning on mobile and embedded devices: State-of-the-art, challenges, and future directions. ACM Computing Surveys (CSUR), 53 0 (4): 0 1--37, 2020

    2020

  4. [12]

    Y. Chen, Z. Zhou, and J. Yan. Going beyond neural network feature similarity: The network feature complexity and its interpretation using category theory. arXiv preprint arXiv:2310.06756, 2023

    2023 arXiv

  5. [13]

    Cheng, M

    H. Cheng, M. Zhang, and J. Q. Shi. A survey on deep neural network pruning-taxonomy, comparison, analysis, and recommendations, 2023. URL https://arxiv.org/abs/2308.06767

    2023 arXiv

  6. [14]

    Corti, B

    F. Corti, B. Maag, J. Schauer, U. Pferschy, and O. Saukh. HADS : Hardware-aware deep subnetworks. In 5th Workshop on practical ML for limited/low resource settings, 2024 a . URL https://openreview.net/forum?id=oDacwa4yb2

    2024

  7. [15]

    Corti, B

    F. Corti, B. Maag, J. Schauer, U. Pferschy, and O. Saukh. REDS : Resource-efficient deep subnetworks for dynamic resource constraints, 2024 b . URL https://arxiv.org/abs/2311.13349

    2024 arXiv

  8. [16]

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248--255. Ieee, 2009

    2009

  9. [17]

    Entezari and O

    R. Entezari and O. Saukh. Class-dependent compression of deep neural networks, 2020. URL https://arxiv.org/abs/1909.10364

    2020 arXiv

  10. [18]

    Entezari, H

    R. Entezari, H. Sedghi, O. Saukh, and B. Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks, 2022. URL https://arxiv.org/abs/2110.06296

    2022 arXiv

  11. [19]

    Esp-eye development board - espressif systems

    Espressif Systems . Esp-eye development board - espressif systems. https://www.espressif.com/en/products/devkits/esp-eye/overview, 2024. Accessed: 2024-11-19

    2024

  12. [20]

    G. Fang, J. Song, C. Shen, X. Wang, D. Chen, and M. Song. Data-free adversarial distillation, 2020. URL https://arxiv.org/abs/1912.11006

    2020 arXiv

  13. [21]

    Frankle and M

    J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018. URL https://arxiv.org/abs/1803.03635

    2018 arXiv

  14. [22]

    Frantar and D

    E. Frantar and D. Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 35: 0 4475--4488, 2022

    2022

  15. [23]

    L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac'h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou. A framework ...

    2024

  16. [24]

    Gholami, S

    A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney, and K. Keutzer. A survey of quantization methods for efficient neural network inference, 2021. URL https://arxiv.org/abs/2103.13630

    2021 arXiv

  17. [25]

    J. Gou, B. Yu, S. J. Maybank, and D. Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129 0 (6): 0 1789–1819, Mar. 2021. ISSN 1573-1405. doi:10.1007/s11263-021-01453-z. URL http://dx.doi.org/10.1007/s11263-021-01453-z

    2021 doi

  18. [26]

    Gunasekar, B

    S. Gunasekar, B. Woodworth, S. Bhojanapalli, B. Neyshabur, and N. Srebro. Implicit regularization in matrix factorization, 2017. URL https://arxiv.org/abs/1705.09280

    2017 arXiv

  19. [27]

    Gupta, A

    S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan. Deep learning with limited numerical precision. In International conference on machine learning, pages 1737--1746. PMLR, 2015

    2015

  20. [28]

    S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015

    2015

  21. [29]

    Hassibi, D

    B. Hassibi, D. G. Stork, and G. J. Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks, pages 293--299. IEEE, 1993

    1993

  22. [30]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770--778, 2016

    2016

  23. [31]

    X. He, Z. Zhou, and L. Thiele. Multi-task zipping via layer-wise neuron sharing. Advances in Neural Information Processing Systems, 31, 2018

    2018

  24. [32]

    Y. He, X. Zhang, and J. Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017

    2017

  25. [33]

    Hinton, O

    G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network, 2015. URL https://arxiv.org/abs/1503.02531

    2015 arXiv

  26. [34]

    Horvath, S

    S. Horvath, S. Laskaridis, S. Rajput, and H. Wang. Maestro: Uncovering low-rank structures via trainable decomposition, 2024. URL https://arxiv.org/abs/2308.14929

    2024 arXiv

  27. [35]

    H. Hu, R. Peng, Y.-W. Tai, and C.-K. Tang. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures, 2016. URL https://arxiv.org/abs/1607.03250

    2016 arXiv

  28. [36]

    Ioffe and C

    S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift, 2015. URL https://arxiv.org/abs/1502.03167

    2015 arXiv

  29. [37]

    R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural computation, 3 0 (1): 0 79--87, 1991

    1991

  30. [38]

    X. Jin, X. Ren, D. Preotiuc-Pietro, and P. Cheng. Dataless knowledge fusion by merging weights of language models, 2023. URL https://arxiv.org/abs/2212.09849

    2023 arXiv

  31. [39]

    Jolicoeur-Martineau, E

    A. Jolicoeur-Martineau, E. Gervais, K. Fatras, Y. Zhang, and S. Lacoste-Julien. Population parameter averaging (papa), 2024. URL https://arxiv.org/abs/2304.03094

    2024 arXiv

  32. [40]

    Jordan, H

    K. Jordan, H. Sedghi, O. Saukh, R. Entezari, and B. Neyshabur. Repair: Renormalizing permuted activations for interpolation repair. arXiv preprint arXiv:2211.08403, 2022. URL https://arxiv.org/abs/2211.08403

    2022 arXiv

  33. [41]

    L. V. Kantorovich. On the translocation of masses. Journal of mathematical sciences, 133 0 (4): 0 1381--1382, 2006

    2006

  34. [42]

    Y.-D. Kim, E. Park, S. Yoo, T. Choi, L. Yang, and D. Shin. Compression of deep convolutional neural networks for fast and low power mobile applications, 2016. URL https://arxiv.org/abs/1511.06530

    2016 arXiv

  35. [43]

    Krizhevsky, G

    A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009 a

    2009

  36. [44]

    Krizhevsky, V

    A. Krizhevsky, V. Nair, and G. Hinton. Cifar-100 and cifar-10 (canadian institute for advanced research), 2009 b . URL http://www.cs.toronto.edu/ kriz/cifar.html. MIT License

    2009

  37. [45]

    H. W. Kuhn. The hungarian method for the assignment problem. Naval Research Logistics (NRL), 52, 1955

    1955

  38. [46]

    Kumar, S

    A. Kumar, S. Goyal, and M. Varma. Resource-efficient machine learning in 2 kb ram for the internet of things. In International conference on machine learning, pages 1935--1944. PMLR, 2017

    1935

  39. [47]

    Lebedev, Y

    V. Lebedev, Y. Ganin, M. Rakhuba, I. Oseledets, and V. Lempitsky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition, 2015. URL https://arxiv.org/abs/1412.6553

    2015 arXiv

  40. [48]

    LeCun, J

    Y. LeCun, J. Denker, and S. Solla. Optimal brain damage. In D. Touretzky, editor, Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmann, 1989. URL https://proceedings.neurips.cc/paper_files/paper/1989/file/6c9882bbac1c7093bd25041881277658-Paper.pdf

    1989

  41. [49]

    Leitner, M

    S. Leitner, M. J. Mirza, W. Lin, J. Micorek, M. Masana, M. Kozinski, H. Possegger, and H. Bischof. Sit back and relax: Learning to drive incrementally in all weather conditions, 2023. URL https://arxiv.org/abs/2305.18953

    2023 arXiv

  42. [50]

    F. Li, B. Liu, X. Wang, B. Zhang, and J. Yan. Ternary weight networks. arXiv preprint arXiv:1605.04711, 2016 a . URL https://arxiv.org/abs/1605.04711

    2016 arXiv

  43. [52]

    H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets, 2017. URL https://arxiv.org/abs/1608.08710

    2017 arXiv

  44. [53]

    Y. Li, J. Yosinski, J. Clune, H. Lipson, and J. Hopcroft. Convergent learning: Do different neural networks learn the same representations? arXiv preprint arXiv:1511.07543, 2015. URL https://arxiv.org/abs/1511.07543

    2015 arXiv

  45. [54]

    H.-I. Liu, M. Galindo, H. Xie, L.-K. Wong, H.-H. Shuai, Y.-H. Li, and W.-H. Cheng. Lightweight deep learning for resource-constrained environments: A survey, 2024. URL https://arxiv.org/abs/2404.07236

    2024 arXiv

  46. [55]

    J.-H. Luo, J. Wu, and W. Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017 a

    2017

  47. [56]

    J.-H. Luo, J. Wu, and W. Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pages 5058--5066, 2017 b

    2017

  48. [57]

    X. Ma, G. Fang, and X. Wang. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36: 0 21702--21720, 2023

    2023

  49. [58]

    Matena and C

    M. Matena and C. Raffel. Merging models with fisher-weighted averaging, 2022. URL https://arxiv.org/abs/2111.09832

    2022 arXiv

  50. [59]

    X. Men, M. Xu, Q. Zhang, B. Wang, H. Lin, Y. Lu, X. Han, and W. Chen. Shortgpt: Layers in large language models are more redundant than you expect, 2024. URL https://arxiv.org/abs/2403.03853

    2024 arXiv

  51. [60]

    Merity, C

    S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016. URL https://arxiv.org/abs/1609.07843

    2016 arXiv

  52. [61]

    Micaelli and A

    P. Micaelli and A. Storkey. Zero-shot knowledge transfer via adversarial belief matching, 2019. URL https://arxiv.org/abs/1905.09768

    2019 arXiv

  53. [62]

    G. Monge. M \'e moire sur la th \'e orie des d \'e blais et des remblais. Mem. Math. Phys. Acad. Royale Sci., pages 666--704, 1781

  54. [63]

    Mordvintsev, C

    A. Mordvintsev, C. Olah, and M. Tyka. Inceptionism: Going deeper into neural networks, 2015. URL https://research.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html

    2015

  55. [64]

    Jetson nano - nvidia developer

    NVIDIA . Jetson nano - nvidia developer. https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-nano/product-development/, 2024. Accessed: 2024-11-19

    2024

  56. [65]

    Papst, D

    F. Papst, D. Kraus, M. Rechberger, and O. Saukh. Sensor-guided adaptive machine learning on resource-constrained devices. In Proceedings of the International Conference on the Internet of Things, 2024

    2024

  57. [66]

    Ren and K

    S. Ren and K. Q. Zhu. Low-rank prune-and-factorize for language model compression, 2023. URL https://arxiv.org/abs/2306.14152

    2023 arXiv

  58. [67]

    Rombach, A

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684--10695, 2022

    2022

  59. [68]

    Shazeer, A

    N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538, 2017

    2017 arXiv

  60. [69]

    Simonyan and A

    K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. URL https://arxiv.org/abs/1409.1556

    2014 arXiv

  61. [70]

    S. P. Singh and M. Jaggi. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33: 0 22045--22055, 2020

    2020

  62. [71]

    Solodskikh, A

    K. Solodskikh, A. Kurbanov, R. Aydarkhanov, I. Zhelavskaya, Y. Parfenov, D. Song, and S. Lefkimmiatis. Integral neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16113--16122, June 2023

    2023

  63. [72]

    Stoica, D

    G. Stoica, D. Bolya, J. Bjorner, P. Ramesh, T. Hearn, and J. Hoffman. Zipit! merging models from different tasks without training, 2024. URL https://arxiv.org/abs/2305.03053

    2024 arXiv

  64. [73]

    M. Sun, Z. Liu, A. Bair, and J. Z. Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695, 2023. URL https://arxiv.org/abs/2306.11695

    2023 arXiv

  65. [74]

    Theus, O

    A. Theus, O. Geimer, F. Wicke, T. Hofmann, S. Anagnostidis, and S. P. Singh. Towards meta-pruning via optimal transport. arXiv preprint arXiv:2402.07839, 2024. URL https://arxiv.org/abs/2402.07839

    2024 arXiv

  66. [75]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample. Llama: Open and efficient foundation language models, 2023 a . URL https://arxiv.org/abs/2302.13971

    2023 arXiv

  67. [76]

    Touvron, L

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hoss...

    2023 arXiv

  68. [77]

    S. Wan, L. Qi, X. Xu, C. Tong, and Z. Gu. Deep learning models for real-time human activity recognition with smartphones. Mobile Networks and Applications, 25 0 (2): 0 743--755, 2020

    2020

  69. [78]

    D. Wang, O. Saukh, X. He, and L. Thiele. Subspace-configurable networks, 2024. URL https://arxiv.org/abs/2305.13536

    2024 arXiv

  70. [79]

    Z. Wang, K. Xu, S. Wu, L. Liu, L. Liu, and D. Wang. Sparse-yolo: Hardware/software co-design of an fpga accelerator for yolov2. IEEE Access, 8: 0 116569--116585, 2020

    2020

  71. [80]

    W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li. Learning structured sparsity in deep neural networks. Advances in neural information processing systems, 29, 2016

    2016

  72. [81]

    Wortsman, G

    M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, and L. Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time, 2022. URL https...

    2022 arXiv

  73. [82]

    Yamada, T

    M. Yamada, T. Yamashita, S. Yamaguchi, and D. Chijiwa. Revisiting permutation symmetry for merging models between different datasets, 2023. URL https://arxiv.org/abs/2306.05641

    2023 arXiv

  74. [83]

    H. Yin, P. Molchanov, Z. Li, J. M. Alvarez, A. Mallya, D. Hoiem, N. K. Jha, and J. Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion, 2020. URL https://arxiv.org/abs/1912.08795

    2020 arXiv

  75. [84]

    S. Yin, C. Li, W. Tan, Y. Bao, Y. Liang, and W. Liu. Exploring structural sparsity in neural image compression, 2022. URL https://arxiv.org/abs/2202.04595

    2022 arXiv

  76. [85]

    S. Yu, J. Chen, H. Han, and S. Jiang. Data-free knowledge distillation via feature exchange and activation region constraint. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24266--24275, 2023

    2023

  77. [86]

    A. Zhou, A. Yao, Y. Guo, L. Xu, and Y. Chen. Incremental network quantization: Towards lossless cnns with low-precision weights. arXiv preprint arXiv:1702.03044, 2017. URL https://arxiv.org/abs/1702.03044

    2017 arXiv

  78. [87]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  79. [88]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  80. [89]

    hs @ @ Ծ-GĀgz Z(nuxʙK;]lv9qQǔ1g#DΝ

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    1999

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.