{"id":"993fec90-21c2-4be2-a3f5-62f97e420b44","arxiv_id":"2502.02496","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"Factorizing weights into D≥2 multiplicative factors and applying L2 weight decay induces a non-convex sparse L2/D penalty, and with tailored initialization and learning rates, achieves superior sparsity-accuracy tradeoffs.","lead":"Deep weight factorization splits each neural network weight into a product of D factors and trains with ordinary L2 weight decay, which provably acts like a sparse L2/D penalty on the original weights. With a new initialization and large learning rates, the method beats shallow factorization and common pruning baselines at extreme compression ratios.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1 is internally sound, but the paper never proves that SGD trajectories reach the balanced factorization M=0 on which the equivalence relies; the practical sparsity claims therefore depend on hand-tuned LR/init and remain a conditional transfer from theory to algorithm.","rationale":"After checking the proof structure, I find no internal inconsistency in Theorem 1: the two directions of the local-minima correspondence are supported by Lemma 1 (balancedness at minimizers) and Lemma 3 (existence of nearby balanced factorizations). My concern is therefore not about the mathematical equivalence itself, but about the use of that equivalence to justify the training algorithm. The weakest link is dynamical: the paper establishes that balance is absorbing, not that it is attracting, and the method's success depends on large LR and DWF-specific truncation; without those, D>2 fails. The reader's verdict (CONDITIONAL) already captures this. I also note secondary issues raised by the reader - no released code and asymmetric tuning budgets - but they are not central to the theoretical claim. The proposed full-batch vs. SGD test would isolate whether stochastic noise is essential to reach the balanced regime, which is the key unproved condition for transferring Theorem 1 to practice.","tokens_in":46160,"tokens_out":15255,"duration_ms":171352,"concrete_test":"Run LeNet-300-100 on MNIST with D=3, DWF initialization, cosine LR, and the paper's recommended lambda, using both SGD and full-batch GD with the same eta schedule; report final normalized factor misalignment M(omega)/sum_d||omega_d||^2, final sparsity, and test accuracy. If SGD reaches M approx 0 and high sparsity while full-batch GD remains at M>0 with dense weights, the practical success relies on stochastic collapse, which is neither proved nor quantified; if both reach M approx 0, the missing convergence proof is a formal gap only and the conditional verdict can be relaxed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical statement (Theorem 1) is a statement about minimizers: it shows that balanced factorizations of a minimizer of Eq. (4) minimize Eq. (5), and vice versa, via Lemmas 1 and 3. It does not say that SGD converges to those minimizers. Lemma 4 proves only that the balanced set M=0 is absorbing under SGD: once M=0 it stays zero. No result in the paper establishes that the DWF initialization and recommended cosine LR schedule actually drive M to zero. The evidence for this is empirical (Figs. 18-20), and the paper's own ablations (Fig. 5b, Fig. 11) show that the method's sparsity and stability are sharply dependent on large initial learning rates and on the truncation thresholds in Algorithm 1. If those are misspecified, D>2 training yields dense or dead-weight models, i.e., outputs that are not balanced and cannot be identified with minima of the L2/D objective. Consequently, the claim that DWF 'provably induces' sparse regularization in any network is established for the optimization problem, not for the algorithm run in practice. This is the load-bearing gap between the theorem and the method's headline practical claims.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Deep Weight Factorization (DWF), which factorizes each weight of a neural network into D ≥ 2 factors and trains with L2 weight decay on the factors, collapsing them after training. The main theoretical contribution is Theorem 1, which states that the factorized objective in Eq. (5) and the original objective with the non-convex L2/D quasi-norm penalty in Eq. (4) have the same local and global minima, with minimizers related by collapse. This equivalence relies on Lemma 1 (balanced factorizations are necessary for optimality) and Lemma 3 (balanced factorizations exist locally). The authors also analyze initialization, proposing a variance-matching and truncation scheme (Algorithm 1), identify a large-learning-rate requirement for sparsity, describe three training phases, and empirically benchmark DWF against magnitude pruning, SNIP, and SynFlow on several architectures and datasets.","tokens_in":46457,"tokens_out":5084,"duration_ms":46707,"significance":"If the claims held fully, the paper would be significant: it offers a smooth, SGD-compatible surrogate for a family of non-convex sparsity-inducing penalties, with empirical results that are often substantially better than existing pruning methods at high compression ratios. The theoretical proof of Theorem 1 is clean and sound (AM-GM plus a continuity argument), and the empirical study is extensive, covering multiple architectures, datasets, and factorization depths. The proposed truncation-based initialization is a practical contribution. However, the headline claim that DWF 'provably induces' L2/D regularization in trained networks is not supported: Theorem 1 concerns minimizers of the optimization problems, while the algorithm's convergence to balanced factorizations is only demonstrated empirically. The experimental comparisons also use per-depth/per-architecture tuning for DWF while baselines use fixed standard configurations, and many plots rely on only three seeds. These issues make the practical superiority less definitive than claimed.","major_comments":[{"comment":"The central theoretical claim is about the equivalence of minimizers of Eq. (4) and Eq. (5), not about the dynamics of SGD. Lemma 4 shows that the balanced set M(ω)=0 is absorbing under SGD, but it does not establish that the DWF initialization and cosine LR schedule actually drive iterates into this set. The paper's own figures (Figs. 18–20, Appendix F.4) provide empirical evidence of misalignment decay, but the method is sensitive to the initial LR (Fig. 5b, Fig. 11) and to the truncation thresholds in Algorithm 1. Therefore, the statement in the abstract and Section 6 that DWF 'provably induces' L2/D regularization in any network is not justified by the proof; it is a conditional transfer from the optimization problem to the algorithm. I recommend either adding a convergence result (even for a simplified setting) or explicitly qualifying the claim as an empirical finding.","section":"Section 3 / Theorem 1 / Lemma 4 / Appendix C.4"},{"comment":"The benchmark comparisons are not controlled for hyperparameter tuning effort. Table 4 and Appendix G.2 show that DWF learning rates are selected per factorization depth and per architecture/dataset (e.g., initial LRs {0.2, 0.5, 0.7} for ResNet-18 CIFAR10; 'we additionally select the best-performing LR ... for each factorization depth, architecture, and dataset'), while the pruning baselines use fixed configurations from prior work. The large improvements reported in Table 1 and Figs. 8–9 could partly reflect this tuning asymmetry. Please add an ablation where DWF uses the same LR for all depths, or where baselines receive comparable tuning, to make the comparison fair.","section":"Section 5.3 / Table 4 / Section G.2"},{"comment":"Many of the central empirical claims rest on only three random seeds. For example, Fig. 8 reports the 'minimum over three random initializations' and Table 2 reports mean ± std over three seeds; several DWF entries show large variance or collapse, such as LeNet-5 MNIST Depth 2 at 99.5% sparsity (66.88 ± 40.23) and ResNet-18 Tiny ImageNet Depth 4 at 5% tolerance (CR=13, versus CR=67 for Depth 3 in Table 1). These observations indicate that the stability of DWF, especially for D=4, is not yet firmly established. Reporting more seeds (at least 5–10) for the key high-sparsity comparisons, or confidence intervals, would strengthen the claims.","section":"Table 2 / Figs. 8–9"}],"minor_comments":[{"comment":"The axis label contains an encoding artifact ('/uni000000...') that should be fixed.","section":"Fig. 21"},{"comment":"The numerical threshold ε_tiny is set to float32 machine epsilon; the sensitivity of the reported sparsities to this threshold is not discussed, though it directly determines what counts as a 'zero' weight.","section":"Algorithm 2, lines 22–23"},{"comment":"The exact Gaussian product factorization is a nice theoretical aside, but the paper does not compare it to DWF Init in downstream training; a sentence clarifying its role would help.","section":"Appendix H, Lemma 5"},{"comment":"The conclusion repeats 'provably induces' without referencing the qualification needed in Theorem 1; please align the wording with the actual scope of the proof.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is an ICLR 2025 accepted paper; as a journal or second-round review, the main issue is the gap between the minimizer equivalence and the practical algorithm, plus the tuning asymmetry. I would not reject: Theorem 1 is correct and the empirical study is rich. However, the authors should either prove (or rigorously motivate) convergence to balanced factorizations or soften the claims; and they should provide matched hyperparameter comparisons and more seeds for the key results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Deep weight factorization is a solid, useful extension of the shallow factorization trick. Theorem 1 and Lemma 1 are proved correctly via AM-GM and continuity, and the experiments show D=3,4 consistently beat D=2 and standard pruning baselines at high compression. The new practical pieces — DWF initialization with truncation, the learning-rate analysis, and the three-phase dynamics — are real contributions and make the method usable.\n\nThe main soft spot is the gap between the landscape theorem and the algorithm. Theorem 1 equates minimizers of the L2/D problem and the factorized L2 problem; it does not say SGD reaches a balanced factorization. Lemma 4 only proves balance is absorbing, not attracting. The evidence for convergence to balance is empirical (Figs. 18-20), and the ablations show the method is sensitive to hand-set LR and truncation thresholds. So the phrase 'provably induces' in the abstract and conclusion overstates what is proved. This is a moderate concern, not fatal: the empirical case is fairly strong and the paper is reasonably honest about the dynamics being heuristic.\n\nThe comparison protocol is the second soft spot. DWF gets per-depth, per-dataset LR selection while baselines use standard fixed configs. The authors include some fixed-LR experiments in the appendix, which helps, but the headline tradeoff curves still mix tuning budgets. There is also no code or data release, which matters for a method paper. The D=4 collapse on Tiny ImageNet (CR=13 at 5% tolerance) is a small crack, worth a note but not a dealbreaker. The self-citation to Kolb et al. (2023) for D=2 is fine; the D>2 claims stand independently.\n\nWho is this for? People working on sparse training, model compression, or implicit regularization via overparameterization will get real value. It deserves a serious referee and, after revisions, publication. I would ask authors to release code, clarify the theory-algorithm gap in the claims, and standardize the tuning budget.","headline":"DWF is a genuinely useful extension of shallow factorization with a correct equivalence theorem, but the paper overclaims by implying the algorithm provably reaches the balanced states the theory relies on; the empirical work is strong enough to warrant serious engagement.","tokens_in":46957,"tokens_out":3435,"would_cite":true,"duration_ms":32152,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","90C26"],"pacs":[],"model":"deepseek-v4-flash","headline":"Factorizing each weight into $D > 2$ factors lets plain $L_2$ weight decay induce provably equivalent non-smooth $L_{2/D}$ sparse regularization in any neural network.","keywords":["deep weight factorization","sparse regularization","L2/D quasi-norm","weight factorization","artificial symmetry","neural network pruning","factor misalignment","SGD learning dynamics"],"falsifier":"Train a factorized network while tracking $M(\\omega) = D^{-1}\\sum_{d=1}^D \\|\\omega_d\\|_2^2 - \\|\\omega_1 \\odot \\cdots \\odot \\omega_D\\|_{2/D}^{2/D}$; the theory predicts sparsity should appear only as $M(\\omega)$ reaches zero, so a run that achieves high compression with $M(\\omega)$ bounded well above zero, or a value of $\\lambda$ where the collapsed loss decreases while $M(\\omega)$ stays positive, would contradict the claimed equivalence mechanism.","tokens_in":45978,"feed_emoji":"✂️","tokens_out":6402,"duration_ms":58564,"temperature":0.7,"pith_summary":"The paper tries to establish that sparse learning in neural networks can be done without any non-smooth penalty: write every weight as the element-wise product of $D$ factors, train the factors with plain $L_2$ weight decay, then collapse the factors back. It proves that the factorized objective has exactly the same local and global minima as the original network penalized by the non-convex quasi-norm $\\|w\\|_{2/D}^{2/D}$, which for $D>2$ penalizes small weights more aggressively than $L_1$ and approaches $L_0$. That means standard SGD can induce sparsity directly during training, without proximal solvers or post-hoc pruning. The paper also identifies why naive factor training fails, namely that standard initializations concentrate the product near zero and that small learning rates never reach the balanced-factor regime, and it supplies a truncation-based initialization and a large-learning-rate recipe. On the tested vision benchmarks, depth-3 and depth-4 factorizations keep accuracy at compression ratios where magnitude pruning, SNIP, and SynFlow collapse.","feed_headline":"Factorizing weights turns L2 decay into a sparsity penalty","feed_subtitle":"With D factors per weight, plain SGD plus weight decay delivers sparser models than magnitude pruning or SynFlow.","key_machinery":"The load-bearing object is the factor misalignment $M(\\omega) = D^{-1} \\sum_{d=1}^D \\|\\omega_d\\|_2^2 - \\|\\omega_1 \\odot \\cdots \\odot \\omega_D\\|_{2/D}^{2/D}$, which is non-negative and zero exactly at balanced factorizations. The AM-GM inequality gives the lower bound; Lemma 1 shows any local minimizer of the factorized objective must be balanced, and Lemma 4 shows balance is an absorbing state under SGD updates. The equivalence in Theorem 1 then follows by lifting a minimizer of the collapsed problem to a balanced factorization and by constructing balanced factorizations close to any perturbation of a collapsed minimizer. In training dynamics, the paper observes that sparsity emerges precisely as $M(\\omega)$ approaches zero, and that large learning rates are needed to drive SGD into this balanced regime.","core_discovery":"The central discovery is Theorem 1: for any network and any $D \\geq 2$, the smooth problem $\\min_{\\omega} L(\\omega_1 \\odot \\cdots \\odot \\omega_D) + \\frac{\\lambda}{D} \\sum_{d=1}^D \\|\\omega_d\\|_2^2$ and the non-smooth problem $\\min_w L(w) + \\lambda \\|w\\|_{2/D}^{2/D}$ share the same global and local minima, with minimizers related by collapse $\\hat{w} = \\hat{\\omega}_1 \\odot \\cdots \\odot \\hat{\\omega}_D$. The proof rests on the artificial rescaling symmetry of the factorization: for a fixed collapsed weight there are infinitely many factorizations, and the $L_2$ penalty is minimized by balanced factorizations in which all factors share equal magnitude. At balance, the factor penalty equals $\\|w\\|_{2/D}^{2/D}$, so $L_2$ regularization on factors is a differentiable stand-in for a non-convex sparsity penalty, and deeper factorizations give stronger, more $L_0$-like sparsification.","pith_inferences":["The factor-misalignment diagnostic suggests a principled stopping rule that the paper does not propose: monitor $M(\\omega)$ and treat its approach to zero as the signal that sparsification has begun.","Because the equivalence is stated for any differentiable loss, the same factorization trick should extend to structured sparsity by grouping factors, although the paper only treats unstructured sparsity.","The observed coincidence of sparsification with balance connects DWF to grokking and stochastic collapse, so the large-learning-rate requirement may be understood as noise-driven symmetry breaking rather than a mere tuning detail.","A testable prediction: at fixed $\\lambda$, the compression ratio should scale with $D$ through the exponent $2/D$, so measuring sparsity across depths could separate the penalty effect from the initialization effect."],"forward_implications":["Any architecture can be sparsified in principle by training factorized weights with ordinary SGD and weight decay; the collapsed network is a minimizer of an $L_{2/D}$-regularized objective, with no non-smooth solver required.","Deeper factorizations ($D=3,4$) act like a stronger, more $L_0$-like penalty, yielding higher compression ratios at the same accuracy than the shallow $D=2$ factorization.","Standard initializations provably make factorized networks untrainable: the collapsed product concentrates at zero and activation variance vanishes, so the proposed truncation-and-variance-matching initialization is necessary.","Sparsity is not obtained automatically: training must include a large-learning-rate phase, and the paper documents three distinct phases with sparsification onset coinciding with factor balance.","DWF can be combined with post-hoc pruning and fine-tuning, further increasing compression by up to roughly three times at matched accuracy."],"supporting_citations":[{"why":"Introduced the Hadamard-product factorization with $L_2$ regularization as a differentiable formulation of the lasso, which is the starting point for shallow weight factorization.","marker":"Hoff (2017)"},{"why":"Proposed shallow factorization of neural network weights to induce differentiable $L_1$ regularization; this paper extends that construction to depth $D>2$ and uses it as the main baseline.","marker":"Ziyin & Wang (2023)"},{"why":"Shows how gradient noise attracts SGD toward symmetry-induced simpler subnetworks, supporting the claim that SGD dynamics drive factorized networks toward balance.","marker":"Chen et al. (2024)"},{"why":"Demonstrates that large step sizes induce a sparsity-favoring regime in diagonal linear networks, providing the theoretical motivation for the large-learning-rate requirement.","marker":"Nacson et al. (2022)"},{"why":"Discusses $L_q$ penalties for $q<2$ and their closeness to $L_0$, motivating why the induced $L_{2/D}$ penalty is a stronger sparsifier than $L_1$.","marker":"Frank & Friedman (1993)"},{"why":"Provides the Kaiming initialization whose variance analysis is used in Lemma 2 to show why standard initializations fail for factorized networks.","marker":"He et al. (2015)"}],"fun_headline_variants":["Deep weight factorization: L2 decay becomes sparsity","Smooth L2 on deep factors yields L0-style sparsity","Factorize deeper to prune better: theory and practice","Artificial symmetry turns weight decay into a pruning tool","More factors, stronger sparsity: deep weight factorization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's practical success rests on the assumption that training with the proposed initialization and a large enough learning rate actually drives the factors into the balanced state (all $D$ factors of each weight equal in magnitude) where the theory applies; the paper proves balance is absorbing once reached, not that training converges to it.","fun_headline_variants_meta":{"raw":{"variants":["Deep weight factorization: L2 decay becomes sparsity","Smooth L2 on deep factors yields L0-style sparsity","Factorize deeper to prune better: theory and practice","Artificial symmetry turns weight decay into a pruning tool","More factors, stronger sparsity: deep weight factorization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000242,"raw_usage":{"total_tokens":1523,"prompt_tokens":943,"completion_tokens":580,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":502}},"tokens_in":559,"tokens_out":580,"duration_ms":6078,"temperature":1.0,"reasoning_tokens":502,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T11:57:20.154296+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a factorized network while tracking $M(\\omega) = D^{-1}\\sum_{d=1}^D \\|\\omega_d\\|_2^2 - \\|\\omega_1 \\odot \\cdots \\odot \\omega_D\\|_{2/D}^{2/D}$; the theory predicts sparsity should appear only as $M(\\omega)$ reaches zero, so a run that achieves high compression with $M(\\omega)$ bounded well above zero, or a value of $\\lambda$ where the collapsed loss decreases while $M(\\omega)$ stays positive, would contradict the claimed equivalence mechanism.","supporting_citations":[{"cited_title":"Lasso, fractional norm and structured sparse estimation using a hadamard product parametrization","cited_arxiv_id":null,"evidence_quote":"Introduced the Hadamard-product factorization with $L_2$ regularization as a differentiable formulation of the lasso, which is the starting point for shallow weight factorization."},{"cited_title":"Implicit bias of the step size in linear diagonal neural networks","cited_arxiv_id":null,"evidence_quote":"Demonstrates that large step sizes induce a sparsity-favoring regime in diagonal linear networks, providing the theoretical motivation for the large-learning-rate requirement."}],"review_version":1}