{"id":"5fc22acc-c0c8-4090-bdea-d0030dc55d8c","arxiv_id":"2502.00744","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CoNNect is a differentiable connectivity regularizer that encourages sparse but connected networks, improving structural pruning accuracy when added to DepGraph and LLM-pruner.","lead":"The authors introduce CoNNect, a regularizer that rewards neural networks for keeping strong paths from inputs to outputs, and show it can be added to existing pruning methods to improve accuracy after pruning. The idea is to make pruning preserve connectivity, and small but consistent gains appear on CNNs and on the LLaMA language model family.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 4.2 replaces the normalized θ of Eq. (2) with θ=|W|; the L0-approximation and stability theorems do not transfer, and the unnormalized objective actually favors dense scaled weights, leaving the headline DepGraph/LLM results outside the paper's guarantees.","rationale":"The paper has two pillars: a theoretical claim about the normalized regularizer and a practical claim about improving one-shot pruners. The theoretical pillar is plausible and Theorem 3.1's proof is provided; the main gap there is the omitted proof of Theorem 3.3, which I flag but do not treat as the primary issue because it concerns stability of the normalized objective that is not used in the large-scale experiments. The practical pillar, which produces the paper's most impressive numbers (Tables 1 and 2), uses a different objective. Because the unnormalized path product is scale-sensitive, the guarantees of Section 3 do not transfer; the regularizer would prefer to grow all weights rather than select a sparse path. This is not an attack on the authors: the mismatch is explicit in the text, but the abstract and conclusions do not carry the caveat. The concern is addressable by either extending the analysis to the unnormalized variant (e.g., showing that the importance-score formula still preserves connectivity relative to the loss) or by clearly labeling Section 4.2 as heuristic and restricting the theoretical claims to Section 4.1. I therefore keep the reader's CONDITIONAL verdict rather than rejecting: the core idea is interesting and the small-scale evidence is consistent with the theory, but the paper as written overclaims what has been established.","tokens_in":21032,"tokens_out":10520,"duration_ms":107441,"concrete_test":"Rerun the Table 2 LLaMA-7B and Table 1 ResNet/VGG CoNNect integrations with θ(W) normalized per Eq. (2) instead of θ(W)=|W| (a per-layer sum of absolute weights, which is cheap for LLM parameter groups). If the CoNNect advantage over LLM-Pruner/DepGraph disappears or becomes unstable, the headline results are due to the unnormalized heuristic and the paper's broad claims need to be scoped; if the gains persist, the theory gap remains but the practical claim is strengthened. A minimal analytical companion: verify that for a 3-layer 2-2-1 net, −log φ_tot with unnormalized θ is lower at the all-ones dense weight matrix than at the sparse path, showing the unnormalized variant does not favor sparsity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims that CoNNect 'proves ... approximates L0 regularization, guaranteeing maximally connected network structures.' Theorems 3.1-3.3 are established only for the layer-normalized θ(W) of Eq. (2). In Section 4.2, the paper explicitly redefines θ(W)_{i,j}=|W_{i,j}| for the DepGraph and LLM-pruner integrations and sets biases to |b|. With unnormalized θ, φ_tot(W) is the sum of path products of raw weights: scaling all weights by c scales every path by c^{K-1}, so −log φ_tot is unbounded below and the minimization in Theorem 3.1 is not well-posed. The L0-type sparsity preference fails qualitatively: for a 3-layer 2-2-1 network, all weights equal to 1 gives φ_tot=2 and R=−log 2≈−0.69, while the sparse single path gives φ_tot=1 and R=0; the unnormalized regularizer prefers the dense network, the opposite of L0. Hence the ResNet-56, VGG-19, and LLaMA-7B results in Tables 1-2 are produced by a heuristic connectivity score, not by the regularizer whose guarantees are advertised. Theorem 3.3's proof is also omitted in Section 3.3.1, a second unverified pillar, but the theory-experiment mismatch is the load-bearing one.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CoNNect, a differentiable regularizer based on the sum of weighted input-to-output paths in a neural network, and claims that it approximates L0 regularization while guaranteeing maximally connected network structures and avoiding layer collapse. Section 3.3.1 defines a layer-normalized connectivity measure θ(W) and proves (or sketches proofs of) three results: Theorem 3.1 bounding the number of nonzero weights at minimizers of -φ_tot, Lemma 3.2 characterizing stationary points, and Theorem 3.3 claiming all stable stationary points are global minimizers. Section 3.3.2 extends the idea to channel-level scaling. Experiments cover unstructured pruning of small MLPs, channel pruning of GNNs, and integration into DepGraph and LLM-pruner for ResNet-56, VGG-19, and LLaMA-7B pruning.","tokens_in":21462,"tokens_out":5360,"duration_ms":54617,"significance":"If the theory and experiments are properly connected, CoNNect would be a useful addition to the pruning toolbox: the regularizer is simple, differentiable, computationally cheap (a single forward pass), and the small-scale experiments show meaningful improvements over L1 and L2 regularization with ablations over initializations and coefficients. The integration with DepGraph and LLM-pruner addresses practically important settings. However, the theoretical guarantees are established only for a normalized version of the connectivity measure, while the large-scale experiments switch to an unnormalized version, and Theorem 3.3 is explicitly left unproved. The paper therefore currently overstates the extent to which its advertised guarantees apply to its headline results.","major_comments":[{"comment":"The theoretical results in Theorems 3.1-3.3 are derived for the layer-normalized θ(W) defined in Eq. (2), where each edge weight is divided by the sum of weights in its layer. In Section 4.2, the paper explicitly redefines (θ(W))_{i,j} = |W_{i,j}| for the DepGraph and LLM-pruner integrations and sets biases to |b|. This unnormalized variant is outside the theory: with raw weights, φ_tot(W) is a sum of products of unscaled weights, so multiplying all weights by a constant c scales every path by c^{K-1}, making -log φ_tot(W) unbounded below and the minimization in Theorem 3.1 not well-posed. Moreover, the L0-type sparsity preference fails qualitatively; for a simple 3-layer 2-2-1 network, the dense all-ones configuration gives φ_tot=2 and R=-log 2≈-0.69, while the sparse single-path configuration gives φ_tot=1 and R=0, so the unnormalized regularizer actually prefers the dense network. Consequently, the ResNet-56, VGG-19, and LLaMA-7B results in Tables 1-2 are produced by a heuristic connectivity score, not by the regularizer whose guarantees are advertised in the abstract. The paper must either extend the theory to the unnormalized objective or explicitly re-scope the claims made for these experiments.","section":"Section 4.2 vs. Eq. (2); Tables 1-2"},{"comment":"Theorem 3.3 is a load-bearing claim for the paper's assertion that CoNNect is a 'well-behaved regularizer' with no stable stationary points except global optima, yet its proof is omitted in the text: the sentence 'we the proof is omitted as it follows directly form the previous observation' is not a proof. In addition, the notion of 'stable stationary point' is never formally defined (e.g., isolated local minimum, infinitesimal stability under gradient flow, or other). Since Lemma 3.2's proof is itself only an induction sketch, the combined chain of reasoning is not sufficiently verifiable. A complete proof of Theorem 3.3 and a precise definition of stability are required.","section":"Section 3.3.1, Theorem 3.3"},{"comment":"The claim that CoNNect 'approximates L0 regularization' rests on Theorem 3.1, but the theorem's bound is |V1|+|VK|+K-3 nonzero weights, whereas an L0-constrained network with connectivity has K-1 nonzero weights. The difference can be large for networks with many input or output nodes, and the paper's justification ('negligible in practice when most input nodes contribute...') is an empirical assumption, not a proven property. Furthermore, the proof in Appendix A.1 argues that a maximizer can be assumed to have a single path with all mass, but this is asserted rather than derived; the subsequent calculation shows only that shifting mass within the first layer preserves φ_tot=1, not that every maximizer has the claimed form. The theorem may be true, but the proof as written is a sketch and should be completed or clearly labeled as such.","section":"Section 3.3.1, Theorem 3.1 and Appendix A.1"}],"minor_comments":[{"comment":"There is a typo in the sentence introducing Theorem 3.3: 'we the proof is omitted as it follows directly form the previous observation' should be 'the proof is omitted as it follows directly from the previous observation.'","section":"Section 3.3.1"},{"comment":"Equation (6) contains an unbalanced parenthesis: '− log( φtot(W )) )' has an extra closing parenthesis.","section":"Section 3.3.1, Eq. (6)"},{"comment":"The caption states that thicker and darker colors correspond to stronger values, but the grayscale rendering makes it difficult to distinguish positive and negative edges; consider adding a legend or using distinct line styles.","section":"Figure 3"},{"comment":"The large-scale pruning results are reported as single runs without variance or multiple seeds; given the stochasticity of calibration sampling and fine-tuning, error bars or repeated runs would strengthen the empirical claims.","section":"Tables 1 and 2"},{"comment":"The description of connectivity input sampling is inconsistent with the main text: Section 4.2 says inputs are 'uniformly sampled between 0 and 1', while Appendix C.4 says 'uniformly sampled between 0 and the vocabulary size' for the LLM experiment. Please clarify which value is used.","section":"Appendix C.4"}],"recommendation":"major_revision","confidential_remarks":"The paper has a clear central idea and the small-scale experiments are well executed, but the gap between the normalized objective that the theory analyzes and the unnormalized objective used in the large-scale integrations is a genuine correctness-risk concern for the headline claims. The omitted proof of Theorem 3.3 compounds this. I believe the issues are fixable within the scope of a revision: either extend the analysis to the unnormalized variant (which may require additional assumptions, e.g., bounded weights or a different normalization) or substantially soften the claims in the abstract and conclusions so they apply only to the normalized setting. The paper should also clarify the formal definition of stability for Theorem 3.3."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read on CoNNect: the layer-normalized path-product regularizer is a genuine idea, and the small-scale evidence backs it. The toy MLP experiment with 100 initializations and the GNN channel-pruning with 10 runs and confidence intervals are honestly done, with real ablations over the regularization coefficient. What is genuinely new is the layer-wise normalization in Eq. (2), the sparsity bound in Theorem 3.1, the stable-stationary-point claim in Theorem 3.3, and the channel-level extension via scaling factors. The intuition behind Theorem 3.1—maximizing normalized path flow concentrates mass on a single path—is clear and the GNN results suggest it works in practice.\n\nThe soft spots are in the theory and in the gap between theory and experiments. Theorem 3.3's proof is omitted with only a sentence saying it follows from Lemma 3.2. The proof of Theorem 3.1 in Appendix A.1 assumes a single-path maximizer rather than showing all maximizers satisfy the bound; Lemma 3.2's induction is sketchy. None of these are necessarily wrong, but they are not yet complete proofs.\n\nMore important is Section 4.2. There the authors redefine θ(W)=|W|, dropping the layer normalization that Theorems 3.1–3.3 rely on. With unnormalized path products, scaling all weights by c scales φ_tot by c^{K−1}, so −log φ_tot is unbounded below, and the regularizer actually prefers dense scaled weights over a sparse single path—the opposite of L0. A 2-2-1 network with all weights 1 gives φ_tot=2; the sparse single path gives φ_tot=1. So the ResNet/VGG/LLaMA results in Tables 1–2 are produced by a heuristic connectivity score, not by the regularizer whose guarantees are advertised. The abstract and conclusion do not flag this.\n\nThis is worth engaging rather than dismissing. The normalized regularizer has a plausible theory and the small-scale experiments support it. The mismatch is fixable: either extend the analysis to the unnormalized case (hard, given the scaling issue) or present the large-scale results as empirical heuristics and add evidence that the heuristic preserves connectivity. I would send it to review with a major-revision recommendation.","headline":"The normalized path-product regularizer is a real idea with credible small-scale evidence, but the headline large-scale results use an unnormalized variant that the theory does not cover, leaving the paper's central claims overbroad.","tokens_in":21880,"tokens_out":4510,"would_cite":false,"duration_ms":46801,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces CoNNect, a differentiable regularizer maximizing normalized input-to-output path weight, claiming it approximates L0 pruning, keeps only sparse connected structures stable, and improves one-shot structural pruning in…","keywords":["neural network pruning","structural pruning","L0 regularization","connectivity regularizer","SynFlow","one-shot pruning","LLM pruning","differentiable regularizer"],"falsifier":"Train a small fully connected network with $K=4$, one input, and one output on a task that needs two separate features and measure the number of non-zero internal weights at convergence under $-\\log(\\phi_{tot}(W))$; if it consistently exceeds the $K-3$ bound, the L0-approximation claim fails, and for the large-scale variant one can check scale-invariance by multiplying all weights by a constant and seeing whether the unnormalized path product changes by that constant raised to $K-1$.","tokens_in":20871,"feed_emoji":"✂️","tokens_out":8936,"duration_ms":76091,"temperature":0.7,"pith_summary":"CoNNect is a differentiable regularizer that replaces the L0 norm in sparse neural-network training with a connectivity objective: it maximizes the total normalized weight of all input-to-output paths. The paper proves that for networks with more than two layers, any global minimizer of this objective has at most $|V_1| + |V_K| + K - 3$ non-zero weights, mirroring the sparsity that L0 regularization would enforce. Because the regularizer is the logarithm of the path sum, a network that is about to disconnect sends the penalty to infinity, which prevents layer collapse during pruning. The authors show that CoNNect improves classical magnitude and SynFlow pruning after training, and that plugging it into DepGraph and LLM-pruner improves one-shot structural pruning of ResNet-56, VGG-19, and LLaMA-7B.","feed_headline":"CoNNect prunes networks to a single input-output path","feed_subtitle":"A differentiable penalty that approximates L0 keeps signal flow intact and boosts one-shot pruning for CNNs and LLMs.","key_machinery":"The central object is the connectivity matrix $\\phi(W)=\\sum_{k=1}^K (\\theta(W))^k$, where $(\\theta(W))_{i,j}$ is the weight $W_{i,j}$ divided by the sum of absolute weights in its layer. The CoNNect regularizer is $R(W)=-\\phi_{tot}(W)$, the negative sum of all entries connecting input layer $V_1$ to output layer $V_K$; equivalently, it is the negative sum of products of normalized weights over all input-to-output paths. This path-product form is what makes the objective prefer one strong path over many weak parallel paths, and the log transform turns imminent disconnection into an infinite penalty.","core_discovery":"On the paper's own terms, the discovery is that a single scalar computed by one forward pass, the sum over all input-to-output paths of products of layer-wise normalized weights $\\phi_{tot}(W)$, acts as an L0 surrogate for pruning. Theorem 3.1 shows that minimizing $-\\phi_{tot}(W)$ forces all but $|V_1|+|V_K|+K-3$ weights to zero, and Theorem 3.3 (proof omitted) asserts that the only stable stationary points are global minimizers, so gradient descent cannot get stuck in a disconnected or overly dense configuration. Using $-\\log(\\phi_{tot}(W))$ makes disconnection infinitely costly, which is what prevents layer collapse. The paper further claims that this connectivity signal transfers to structured pruning through channel scaling factors and improves the importance scores of one-shot pruners.","pith_inferences":["The same connectivity score could be computed at initialization for structured one-shot pruning of untrained models, giving a data-free group-importance signal.","The path-product formulation connects CoNNect to path-norm theory; one could derive closed-form sensitivity of $\\phi_{tot}$ to whole modules such as attention heads or MLP blocks and use that as a group score for transformer pruning.","For tasks requiring genuinely parallel features, a natural extension is a multi-path variant that keeps a small number of top paths rather than a single path, which would trade the L0-style bound against representational capacity."],"forward_implications":["Networks trained with the log-CoNNect penalty can be hard-pruned to very high ratios without layer collapse; in the paper's small MLP experiments this finds the two relevant input features that magnitude-only and L1-trained networks miss.","The same objective extends to channels by inserting learnable scaling factors, so CoNNect gives a structured importance score $I_{k,c}$ for pruning whole channels.","Plugged into DepGraph, CoNNect improves one-shot structural pruning of ResNet-56 and VGG-19, with larger gains at extreme 16x speed-up targets.","Plugged into LLM-pruner, CoNNect improves zero-shot perplexity and classification after pruning 20% and 40% of LLaMA-7B (and LLaMA-13B in the appendix), with or without LoRA fine-tuning."],"supporting_citations":[{"why":"Supplies the centrality-style summed-path measure that the regularizer is built on.","marker":"(Katz, 1953)"},{"why":"Provides the path-product identity used to rewrite $\\phi_{tot}(W)$ as a sum over input-to-output paths.","marker":"(Neyshabur et al., 2015)"},{"why":"SynFlow is the connectivity-preserving pruning principle that CoNNect generalizes, and its saliency formula is used for pruning after CoNNect training.","marker":"(Tanaka et al., 2020)"},{"why":"Optimal Brain Damage is the classic magnitude-based pruning baseline against which CoNNect-regularized training is compared.","marker":"(LeCun et al., 1989)"},{"why":"DepGraph is the structural-pruning framework into which CoNNect is plugged for the CNN experiments.","marker":"(Fang et al., 2023)"},{"why":"LLM-pruner is the one-shot LLM pruning framework and baseline that CoNNect enhances.","marker":"(Ma et al., 2023)"}],"fun_headline_variants":["CoNNect penalty keeps pruned networks fully connected","One forward pass, zero layer collapse in pruning","L0 surrogate that guarantees signal flow after pruning","Boost one-shot pruning with connectivity penalty","Avoid layer collapse in structured pruning with CoNNect"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the layer-wise normalized objective of Eq. (2) is what training actually optimizes, together with the unproved stability theorem; the large-scale experiments instead use unnormalized absolute weights, so it is open whether the L0-approximation and stable-stationary-point guarantees carry over.","fun_headline_variants_meta":{"raw":{"variants":["CoNNect penalty keeps pruned networks fully connected","One forward pass, zero layer collapse in pruning","L0 surrogate that guarantees signal flow after pruning","Boost one-shot pruning with connectivity penalty","Avoid layer collapse in structured pruning with CoNNect"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000547,"raw_usage":{"total_tokens":2558,"prompt_tokens":833,"completion_tokens":1725,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":449,"completion_tokens_details":{"reasoning_tokens":1654}},"tokens_in":449,"tokens_out":1725,"duration_ms":14384,"temperature":1.0,"reasoning_tokens":1654,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T17:52:46.603597+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a small fully connected network with $K=4$, one input, and one output on a task that needs two separate features and measure the number of non-zero internal weights at convergence under $-\\log(\\phi_{tot}(W))$; if it consistently exceeds the $K-3$ bound, the L0-approximation claim fails, and for the large-scale variant one can check scale-invariance by multiplying all weights by a constant and seeing whether the unnormalized path product changes by that constant raised to $K-1$.","supporting_citations":[{"cited_title":"Path-sgd: Path-normalized optimization in deep neural networks","cited_arxiv_id":null,"evidence_quote":"Provides the path-product identity used to rewrite $\\phi_{tot}(W)$ as a sum over input-to-output paths."}],"review_version":1}