{"id":"476c673f-5757-411f-996a-9b4299b50e7f","arxiv_id":"2502.10216","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Model folding compresses a network by k-means clustering similar neurons across adjacent layers and repairing activation statistics without data (Fold-AR, Fold-DIR), surpassing prior data-free methods at high sparsity.","lead":"This paper introduces \"model folding,\" a way to shrink neural networks by merging similar neurons with k-means clustering, without needing training data or fine-tuning. It reports that folded ResNet and VGG models beat other data-free compression methods at high sparsity, and that a 20% folded LLaMA-7B keeps usable quality.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fold-AR's uncorrelated-input, variance-only repair is never validated against oracle statistics; the 70%-sparsity data-free claim may rest on Fold-R or require re-scoping.","rationale":"The central claim is that data-free, fine-tuning-free folding preserves performance at high sparsity. The k-means clustering step is well motivated and the joint Frobenius cost is correctly minimized, so I do not see a load-bearing error there. The weakest point is the repair step that must preserve the activation distribution: Fold-AR assumes uncorrelated previous-layer activations and repairs only variance, while ignoring biases and mean shifts. Real REPAIR fixes both mean and variance using data, and Fold-DIR partially recovers this by synthesizing data, but Fold-AR is the resource-efficient variant highlighted in the paper. The same-cluster oracle test isolates exactly whether the approximation matters: if true covariance or real-data REPAIR materially improves accuracy over Fold-AR at 70% sparsity, then the data-free method is not preserving data statistics as claimed. I agree with the reader's weakest-assumption analysis. This does not overturn the paper: the k-means framing and Fold-DIR results are still valuable, and a conditional verdict remains appropriate pending this check plus the already-noted reproducibility gaps (seeds, hyperparameters, error bars).","tokens_in":30219,"tokens_out":9946,"duration_ms":113261,"concrete_test":"On the same public ResNet18-CIFAR10 checkpoint, fix one k-means clustering (same seed) and compare three repairs with identical cluster assignments: (a) Fold-AR as published, (b) Fold-AR with E[c] computed from true CIFAR10 activation covariances (oracle correlation for the same clusters), and (c) Fold-R using one forward pass of real data. Report accuracy at 50% and 70% sparsity plus layer-wise mean ratios and variance ratios. If (b) or (c) exceeds (a) by more than about 2-3 points at 70%, the uncorrelated-input, variance-only repair is the limiting assumption and the data-free claim needs re-scoping; if (a) stays within that margin across several k-means seeds, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2's Fold-AR corrects only the variance of each cluster centroid (Algorithm 1, step 10), with the average intra-cluster correlation E[c] estimated from normalized weight dot products. This is justified by assuming the previous layer's activations x_l are uncorrelated, and the derivation explicitly neglects biases. There is no correction for the mean of merged preactivations, and no bound connects the minimized Frobenius proxy J_{l,l+1} to activation error or final accuracy. In a deep residual network, a mean shift or miscalibrated variance correction can compound layer by layer. The paper's own Fig. 5 shows Fold-AR below the data-driven Fold-R at most sparsities, and Fig. 6 / Appendix J report 'model folding' accuracy without specifying which repair variant produced the 70%-sparsity 80%+ number. If that number comes from Fold-R, the data-free claim is overstated; if it comes from Fold-AR, the uncorrelated-input approximation needs to be shown tight enough at high sparsity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces \"model folding,\" a data-free and fine-tuning-free structured compression method that clusters structurally similar neurons (channels) with k-means on a concatenation of adjacent-layer weights, merges them, and then repairs the resulting activation statistics. Two data-free repair mechanisms are proposed: Fold-AR, which estimates an intra-cluster correlation under an uncorrelated-input assumption and rescales the cluster centroids to preserve variance, and Fold-DIR, which synthesizes a single batch of images via Deep Inversion to update BatchNorm statistics. The authors derive a closed-form Frobenius reconstruction cost for the joint folding of adjacent layers and claim k-means optimally minimizes this cost. Empirically, the paper reports favorable results against IFM, structured magnitude pruning, and knowledge-distillation baselines on ResNet18/VGG11/ResNet50 for CIFAR10, CIFAR100, and ImageNet, plus results on LLaMA-7B and LLaMA2-7B. The central empirical claim is that at 70% sparsity a folded ResNet18 on CIFAR10 retains over 80% accuracy without data or fine-tuning.","tokens_in":30293,"tokens_out":6169,"duration_ms":67104,"significance":"If the claims hold, model folding would be a useful addition to the data-free compression toolbox: it provides a structured way to merge redundant units, the repair coefficients are derived in closed form from weights and cluster sizes rather than fitted to benchmark accuracies, and the authors release code. The supporting lemmas in Appendix B and the Fold-AR variance-correction formula are worked out analytically, which is a genuine strength. However, the significance is tempered by several gaps: the optimization criterion is a weight-space Frobenius proxy with no established link to activation error or task accuracy; the Fold-AR correction relies on an uncorrelated-input assumption and neglects biases without an oracle-based validation; and the headline high-sparsity numbers are presented without consistently specifying which repair variant produced them. The LLM experiments also show a clear gap to data-driven baselines in the standard setting, so the \"comparable\" wording in the abstract and conclusion overstates the evidence. These issues are fixable in revision, but they currently affect the strength of the central data-free claim.","major_comments":[{"comment":"The theoretical optimality claim concerns the Frobenius reconstruction error of the weight matrices, not the error in activations or final outputs. No bound or experiment connects the minimized quantity J_{l,l+1} to task accuracy, so the statement that k-means is the \"theoretically optimal\" folding method does not by itself justify the empirical accuracy claims. The authors should either add a perturbation analysis relating the weight-space error to activation error under the network's Lipschitz constants, or provide an ablation showing that activation-level error tracks J_{l,l+1} across sparsity levels and repair variants.","section":"Section 3.1, Eq. (J_{l,l+1})"},{"comment":"Fold-AR estimates the intra-cluster correlation E[c] under the explicit assumption that the previous layer's outputs are uncorrelated, and it corrects only the variance of the merged preactivations; the derivation neglects biases, and Algorithm 1 contains no bias-handling step. Since variance correction is the core mechanism distinguishing Fold-AR from naive merging or IFM, this approximation should be validated against oracle statistics obtained from a forward pass. The authors should report (i) whether biases are fused, averaged, or dropped, and (ii) a comparison of predicted versus oracle E[c] and variance ratios for at least one model and several sparsity levels.","section":"Section 3.2 and Algorithm 1"},{"comment":"The headline claim that \"at 70% sparsity, the folded ResNet18 on CIFAR10 maintains over 80% accuracy\" is not attributable to a specific repair variant: Fig. 5 shows Fold-AR below Fold-R at most sparsities, while Fig. 6 and Table 6 use the generic label \"model folding\" without stating whether the plotted results come from Fold-AR or Fold-DIR. This ambiguity blocks verification of the data-free claim at high sparsity. The authors should label each curve with the exact variant, and if the 70%-sparsity result uses Fold-DIR, state the cost of the Deep Inversion step; if it uses Fold-R, the abstract and conclusion must be re-scoped to distinguish data-driven from data-free results.","section":"Section 4, Figures 5-6 and Appendix J"},{"comment":"The abstract and Section 4 state that model folding achieves \"comparable performance\" to data-driven methods on LLaMA-7B, but Table 3 shows WikiText2 perplexity 13.33 versus 6.87 to 10.53 for data-driven baselines, and zero-shot average 50.17 versus 54.27 to 63.00. \"Comparable\" is only defensible when the data-driven methods are run with out-of-distribution calibration data, as in the Hungarian/Ukrainian example. The claims should be explicitly re-scoped to \"competitive in a data-free setting\" and the OOD caveat should appear in the abstract and conclusion rather than only in the experimental narrative.","section":"Table 3 and Section 4, \"Folding LLMs\""}],"minor_comments":[{"comment":"The block-matrix notation C = [P I] in the K-Means/LAP connection is dimensionally inconsistent as written, since X = [WA; WB] and the subsequent Frobenius expression is not well defined for the stated matrix sizes; the authors should specify the dimensions of each block and write the objective with explicit block matrices.","section":"Section 3.3"},{"comment":"The caption should state which dataset and which repair variant each row corresponds to, and it should include numeric labels or points because the right-column ImageNet results are described in the text as collapsing across all methods.","section":"Figure 6 caption"},{"comment":"The LLaMA experiments apply clustering without REPAIR because there is no BatchNorm, but the main text does not state this clearly; the variance-correction theory developed in Section 3.2 therefore does not apply to the LLM results, and the text should present the LLM study as a separate extension rather than as a direct validation of the repair mechanisms.","section":"Section 4, \"Folding LLMs\""},{"comment":"The sentence \"Fold-DIR ourperforms Fold-AR\" contains a typo and should read \"outperforms\".","section":"Section 3.2"},{"comment":"The \"Model Folding (ours)\" row reports accuracy values without standard deviations or seed information, while the KD baselines have long runtimes; adding error bars or at least multiple seeds would strengthen the comparison.","section":"Appendix J, Table 6"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a solid technical contribution and the code release is a plus. The main risk is overclaiming: the data-free high-sparsity results need to be attributable to a specific repair variant, and the LLM claims need to be re-scoped relative to Table 3. The uncorrelated-input assumption in Fold-AR is the most fragile point and should be validated against oracle statistics before the paper is accepted. I do not have concerns about citation practices or novelty disclosure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one for the CNN results, not the LLaMA headline. The core idea is genuinely new as far as I know: instead of greedily merging pairs of similar neurons, you run k-means on the concatenated weight matrix [W_l | W_{l+1}^T] (plus BN diagonals) and then repair statistics with two data-free schemes. The k-means reformulation of weight matching is elegant and the supporting lemmas B.1-B.6 check out under the stated assumptions. The empirical comparison is honest: they report the IFM protocol where IFM is better at low sparsity, they test on ImageNet and public checkpoints, and they release code. On CNNs the method looks like a genuine improvement over IFM and structured magnitude pruning at high sparsity, and the variance-ratio analysis (collapse/overshoot) is a nice way to diagnose what goes wrong.\n\nThe soft spots are the usual ones for weight-space compression. The Frobenius proxy J is not tied to activation error or accuracy, so the theoretical guarantee is weaker than the framing suggests. Fold-AR's variance correction assumes uncorrelated previous-layer activations and neglects biases; the authors say this explicitly, but they never validate the approximation against the oracle correlation on actual data. That matters because a mean shift or miscalibrated variance can compound through a deep ResNet. I also cannot tell from the paper which repair variant produced the 70% sparsity / 80% accuracy claim in Section 4—if that number comes from Fold-R (data-driven), the data-free claim is overstated. The LLaMA-7B result is weaker than the abstract implies: at 20% sparsity WikiText2 perplexity is 13.33 versus 6.87–10.53 for data-driven baselines, and the generated samples degrade visibly. The OOD calibration experiment is a fair point, but 'comparable to data-driven' only holds when calibration data is out-of-distribution.\n\nReported experimental details are also thin: no seeds, no error bars, no commit hash, and k-means hyperparameters (initialization, restarts) are unstated. These are fixable but should be addressed before acceptance-grade confidence.\n\nOverall: a serious paper with a clean idea and solid, honest CNN experiments. Who gets value: anyone working on data-free compression or edge deployment with restricted data. It deserves a serious referee—the overclaims need re-scoping and the reproducibility gaps need filling, but the core method is worth engaging with.","headline":"Data-free folding via k-means on joint adjacent-layer weights is a real and useful trick on CNNs; the LLaMA 'comparable' claim is overstated and the 70%-sparsity number needs a repair-variant audit.","tokens_in":31013,"tokens_out":2826,"would_cite":true,"duration_ms":45203,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","62H30"],"pacs":[],"model":"deepseek-v4-flash","headline":"Model folding compresses neural networks by clustering and merging structurally similar neurons across layers, requiring neither training data nor fine-tuning.","keywords":["model folding","data-free compression","k-means clustering","variance collapse","variance repair","structured pruning","large language model compression","deep inversion"],"falsifier":"Measure the layer-wise variance ratio and per-class accuracy for a Fold-AR-compressed ResNet18 on CIFAR10 at 70% sparsity; the method's central claim predicts the ratio stays close to 1 and accuracy stays above 80%, so a ratio far from 1 coupled with a large accuracy drop would falsify the repair mechanism.","tokens_in":1639,"feed_emoji":"🧠","tokens_out":1889,"duration_ms":81547,"temperature":0.7,"pith_summary":"Model folding is a compression method that shrinks a neural network by grouping and merging structurally similar neurons within each layer, using k-means clustering, and then repairing the damage this merging does to the network's internal activation statistics. The paper argues that this repair can be done without any training data or fine-tuning: Fold-AR estimates the lost variance from weight geometry alone, and Fold-DIR generates a single synthetic batch from the network itself. On ResNet18 trained on CIFAR10, folded models keep over 80% accuracy at 70% sparsity, while structured magnitude pruning collapses to near chance. The authors claim their data-free method is comparable to data-driven compression and substantially better than earlier data-free pruning, and that it extends to LLaMA-7B without post-training.","feed_headline":"No data, no fine-tuning: fold similar neurons to shrink models","feed_subtitle":"K-means merging plus variance repair keeps ResNet18 over 80% accuracy at 70% sparsity.","key_machinery":"The central object is the clustering projection matrix $C = U(U^T U)^{-1}U^T$ induced by k-means, applied not to a single weight matrix but to a concatenated \"super weight\" that couples adjacent layers and BatchNorm diagonals. K-means on this super weight is the mechanism that simultaneously merges channels and selects the same cluster structure for the next layer's input. The companion mechanism is the variance-ratio repair: Fold-AR rescales each centroid by $N_c / \\sqrt{N_c + (N_c^2 - N_c) E[c]}$ using an assumed uncorrelated-input estimate of the intra-cluster correlation $E[c]$; Fold-DIR replaces that estimate by statistics measured on a single synthetic batch. The argument's load is carried by the claim that keeping the variance ratio near one preserves network function.","core_discovery":"The central claim is that structured compression can be treated as a clustering problem in weight space: cluster rows of $W_l$ together with the corresponding columns of $W_{l+1}$ (and the BatchNorm scaling and normalization diagonals) into a combined matrix $W_{l,l+1}=[W_l \\mid W_{l+1}^T]$, solve k-means, and replace each cluster by its centroid. This minimizes the joint Frobenius reconstruction error, and the paper proves this is the optimal data-free merging rule under that objective. The discovery is that the resulting variance collapse, the reason earlier merging methods fail, can be repaired without data by rescaling each cluster centroid by a factor computed from the cluster size and an estimated intra-cluster correlation (Fold-AR), or by re-estimating BatchNorm statistics on a Deep-Inversion-generated image batch (Fold-DIR). With either repair, the variance ratio of compressed to uncompressed activations stays close to one, and accuracy is preserved at high sparsity levels.","pith_inferences":["If weight-space redundancy is the resource, then per-layer correlation statistics could predict how much each layer can be folded; the paper leaves sparsity allocation uniform, so adaptive layer-wise sparsity is a natural extension.","Fold-AR's uncorrelated-input assumption is testable: feeding a single minibatch of random or synthetic input through the uncompressed network to measure true intra-cluster correlations would likely tighten the repair, and the gap between Fold-AR and data-driven REPAIR is plausibly the cost of that assumption.","A practical implication the authors do not develop is that folding produces a smaller dense network rather than a sparse one, so it can be composed with quantization or further pruning to push edge-device memory even lower.","The authors' \"comparable\" claim for LLMs is strongest when calibration data is scarce or domain-mismatched; when in-distribution calibration data is available, data-driven methods still hold a perplexity edge, so the practical win is for privacy- and regulation-constrained settings."],"forward_implications":["At 70% sparsity, folded ResNet18 on CIFAR10 stays above 80% accuracy while structured magnitude pruning falls to near random chance.","Folding wider variants of VGG11 and ResNet50 yields larger accuracy gains, because wider layers contain more correlated, redundant channels.","Fold-DIR closely matches the data-driven REPAIR baseline on ResNet18, and Fold-AR outperforms the earlier data-free IFM method while being cheaper and fully data-free.","Model folding compresses LLaMA-7B without fine-tuning, and unlike calibration-based pruning it is robust when the only available calibration data is out of distribution.","On edge devices, folded models reduce RAM and flash usage and speed up inference at higher sparsity levels."],"supporting_citations":[{"why":"Supplies the variance-collapse diagnosis and the data-driven REPAIR baseline that Fold-AR and Fold-DIR adapt to be data-free.","marker":"(Jordan et al., 2022)"},{"why":"The main data-free, fine-tuning-free baseline (IFM); the paper shows IFM's variance overshooting and worse accuracy at high sparsity.","marker":"(Chen et al., 2023)"},{"why":"Establishes data-free weight matching that model folding extends from pairwise permutation to k-means clustering of neuron groups.","marker":"(Ainsworth et al., 2023)"},{"why":"Deep Inversion is the generator of the single synthetic batch used by Fold-DIR to re-estimate BatchNorm statistics.","marker":"(Yin et al., 2020)"},{"why":"Justifies k-means as matrix factorization, the basis for minimizing the Frobenius reconstruction error.","marker":"(Bauckhage, 2015)"},{"why":"INN is the state-of-the-art data-free pruning method compared in the last-two-block experiments.","marker":"(Solodskikh et al., 2023)"},{"why":"LLaMA-7B is the large language model on which folding is evaluated against data-driven structured pruning methods.","marker":"(Touvron et al., 2023a)"}],"fun_headline_variants":["Fold neurons to compress without data or fine-tuning","Model folding: cluster weights, merge neurons, shrink models","Data-free compression by folding similar neurons","Merge similar neurons to compress models without data","Forget data and fine-tuning, fold neurons to compress"],"cache_read_input_tokens":33024,"weakest_assumption_plain":"The load-bearing premise is that weight-space similarity, two neurons whose weight vectors are close, is a reliable stand-in for functional similarity on real inputs, so merging close weight vectors and rescaling variances leaves the network's behavior intact.","fun_headline_variants_meta":{"raw":{"variants":["Fold neurons to compress without data or fine-tuning","Model folding: cluster weights, merge neurons, shrink models","Data-free compression by folding similar neurons","Merge similar neurons to compress models without data","Forget data and fine-tuning, fold neurons to compress"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000614,"raw_usage":{"total_tokens":2815,"prompt_tokens":869,"completion_tokens":1946,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":1874}},"tokens_in":485,"tokens_out":1946,"duration_ms":16827,"temperature":1.0,"reasoning_tokens":1874,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T18:58:17.113667+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the layer-wise variance ratio and per-class accuracy for a Fold-AR-compressed ResNet18 on CIFAR10 at 70% sparsity; the method's central claim predicts the ratio stays close to 1 and accuracy stays above 80%, so a ratio far from 1 coupled with a large accuracy drop would falsify the repair mechanism.","supporting_citations":[],"review_version":1}