pith. sign in

arxiv: 2307.03571 · v4 · submitted 2023-07-07 · 💻 cs.LG · math.OC· stat.ML

Smoothing the Edges: Smooth Optimization for Sparse Regularization using Hadamard Overparametrization

Chris Kolb , Christian L. M\"uller , Bernd Bischl , David R\"ugamer This is my paper

Pith reviewed 2026-05-24 07:40 UTC · model grok-4.3

classification 💻 cs.LG math.OCstat.ML
keywords sparse regularizationsmooth optimizationHadamard overparametrizationlocal minima equivalencegradient descentstructured sparsityneural network pruning
0
0 comments X

The pith

Hadamard overparametrization with smooth surrogates makes sparse regularization fully differentiable while preserving all minima.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows how to turn non-smooth sparse regularization problems into smooth, differentiable ones that standard gradient descent can solve. Selected parameters are re-expressed as Hadamard products of two new vectors, and the original non-smooth penalty is replaced by a smooth surrogate on those vectors. The resulting surrogate objective has exactly the same global minima as the original problem and the same local minima, so no extra spurious solutions appear. The framework covers many existing sparsity-inducing parametrizations and applies to both regression and neural-network training.

Core claim

The surrogate objective obtained via Hadamard overparametrization and smooth surrogate penalties has identical global minima and matching local minima to the original explicitly regularized sparse problem.

What carries the argument

Hadamard overparametrization of chosen parameters together with replacement of the non-smooth penalty by a smooth surrogate that induces the desired sparsity in the base parameters.

If this is right

  • Gradient descent and other smooth optimizers become directly applicable to sparse and structured-sparse problems without custom non-smooth solvers.
  • The same equivalence holds for arbitrary (even unregularized) objectives, giving a general result on preservation of local minima under this reparametrization.
  • The method covers and extends existing sparsity-inducing parametrizations from multiple fields.
  • Experiments confirm the approach works on high-dimensional regression and sparse neural-network training.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same overparametrization device could be tested on other non-smooth penalties such as total variation or rank constraints.
  • Because the theory applies to unregularized objectives, it may simplify analysis of local-minima preservation in plain overparametrized models.
  • Practical implementations could drop the need for specialized sparse optimizers in deep-learning pipelines.

Load-bearing premise

The particular smooth surrogate, once composed with the Hadamard product, produces exactly the non-smooth sparse behavior on the original parameters without moving or changing the character of the minima.

What would settle it

Finding a local minimum of the surrogate objective whose corresponding base-parameter solution is not a local minimum of the original sparse objective would disprove the claimed equivalence of local minima.

Figures

Figures reproduced from arXiv: 2307.03571 by Bernd Bischl, Chris Kolb, Christian L. M\"uller, David R\"ugamer.

Figure 1
Figure 1. Figure 1: Illustration of smooth optimization transfer. Left: univariate lasso problem P(β) = (1 − 3 2 β) 2 + 2|β| (red line indicates the global minimizer βˆ). Middle: contours of the equivalent smooth surrogate Q(u, v) = (1 − 3 2 uv) 2 + u 2 + v 2 using a Hadamard product parametrization (10) with K(u, v) = uv = β. Both global minimizers (dots) map to K(ˆu, vˆ) = βˆ. Right: non-convex surface of higher-dimensional… view at source ↗
Figure 2
Figure 2. Figure 2: Relationship between local minimizer ˆξ of Q, the induced minimizer K( ˆξ) = βˆ of P, and the cont. solution mapping ˆξ(β) of the SVF. Left: solid curves show two fibers K−1 (βˆ) (red) and K−1 (β˜) (blue). The solution map ˆξ(β) (dashed green) maps to minimizers of the SVF for varying β, where Q equals P. Right: concrete example showing scalar parametrization βj = K(uj , vj ) = ujvj , and surrogate ℓ2 regu… view at source ↗
Figure 3
Figure 3. Figure 3: Diagonal linear networks corresponding to different parametrizations of a linear predictor: a) HPP (ℓ1), b) HDP (ℓ1), c) Network corresponding to a structure-inducing parametrization (GHPP for ℓ2,1, cf. 4.1) with grouping layer. Left nodes are inputs and right-most node the output. The HPP β = u⊙v and HDP β = γ⊙γ−δ⊙δ parametrizations reveal close connections to diagonal linear networks and linear regressio… view at source ↗
Figure 4
Figure 4. Figure 4: a) Illustration of ℓ1 optimization transfer using HPP and surrogate ℓ2 regularization on a scalar βj = 10 (lower plane). The hyperbolic paraboloid (blue/green) shows the parametrization K(uj , vj ) = ujvj and the elliptic paraboloid (orange) the ℓ2 surrogate. The fiber K−1 (10) defines a hyperbola (black), whose two vertices achieve minimal a min. ℓ2 penalty of 2|10| = 20 (upper plane) over K−1 (10). b) Ma… view at source ↗
Figure 5
Figure 5. Figure 5: Deep diagonal linear networks corresponding to different parametrizations of a linear predictor set-up. a) HPP (for ℓ2/k), b) GHPPk (for ℓ2,2/k), c) GHPPk1,k1+k2 (for ℓ2/k1,2/(k1+k2)). The depth up to and including the grouping layer is k1, followed by k2 = k − k1 more diagonal layers. Nodes on the left represent input features and the single node on the right the output. Corollary 5.2 The optimization of … view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of regularization paths of (G)HPP-based GD and direct (Sub)GD opti￾mization of the non-smooth ℓ1 regularized lasso (a) and ℓ2,1 regularized group lasso (b) objectives. Dashed lines indicate (optimal) solutions of the non-smooth optimizer. Parameters (groups) with magnitude (ℓ2 norm) below 1 × 10−6 are considered 0. standard GD [PITH_FULL_IMAGE:figures/full_fig_p033_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Stand. estimation error (top row) and test prediction error (bottom row) for two Σ settings (columns) of our approach for depths k ∈ {2, 3, 4, 6}, compared with specialized optimizers for ℓ1 and non-convex SCAD and MCP penalties. that deeper factorizations improve support recovery. Appendix B contains the correspond￾ing results, as well as additional experiments for a low-dimensional (d < n) setting whose … view at source ↗
Figure 8
Figure 8. Figure 8: One-shot pruning curves obtained by overparametrizing the weights and biases of a LeNet-300-100 trained on MNIST using the HPPk. Left: results for unregularized models. Right: adding smooth ℓ2 regularization perhaps counterintuitively produces profound sparsity-inducing effects. Magnitude-based pruning constitutes the baseline and the error bars show standard errors over five random initializations. The re… view at source ↗
Figure 9
Figure 9. Figure 9: Left: regularization paths for (structured) filter sparsity using the GHPowPk for k ∈ {2, 2.5, 3} to overparametrize the filter weights of a small VGG architecture trained on MNIST. Structured magnitude pruning based on filter norms constitutes the baseline. Right: layer-wise sparsity patterns for the GHPowP3. Error bars show standard errors over ten random initializations. 8.5 Computational Complexity An … view at source ↗
Figure 10
Figure 10. Figure 10: Time per sample (training) for different factorization depths and batch sizes. Left: full overparametrization using the HPPk. Right: parameter sharing significantly reduces computational overhead. Averages over four epochs are displayed. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Comparison of parameter norms of (G)HPP-based GD and direct (Sub)GD optimization of the non-smooth ℓ1 regularized lasso (a) and ℓ2,1 regularized group lasso (b) objectives. Dashed lines indicate optimal solutions. B.2 Sparse Linear Regression Detailed Set-Up In our experiments, we simulate 30 data sets D = {(xi , yi)} n i=1, where xi ∈ R d is the feature vector and yi ∈ R the scalar outcome. Each dataset … view at source ↗
Figure 12
Figure 12. Figure 12: Support recovery accuracy (top row) and false positives (bottom row) for different Σ settings (columns) and increasing factorization depths, compared against standard implementations of convex ℓ1 and non-convex SCAD and MCP penalties. Variable selection SGD does not have an in-built mechanism to produce (theoretically) exact zeros, although given a sufficiently large number of iterations, a zero floating-… view at source ↗
Figure 13
Figure 13. Figure 13: Left: standardized estimation error (top row) and (test) support recovery (bottom row) for different Σ settings (columns) of our approach for increasing factorization depths, compared against standard implementations of convex ℓ1 and non-convex SCAD and MCP penalties. Right: prediction error (top row) and false positives (bottom row) for different settings of Σ (columns). B.3 Details on CNN Architecture O… view at source ↗
Figure 14
Figure 14. Figure 14 [PITH_FULL_IMAGE:figures/full_fig_p052_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Left: HPP (blue/green) and the sur￾rogate ℓ2 regularization u 2 j + v 2 j (orange). Right: HDP (blue/green) and surrogate ℓ2 regularization. Both parametrizations define hyperbolic paraboloids for each j = 1, . . . , d that are equivalent bar rotation and scaling. Dif￾ferences in the minimum of the SVF for the HPP (2∥β∥1) and HDP (∥β∥1) arise because the HDP defines hyperbolas that are not only rotated by… view at source ↗
Figure 16
Figure 16. Figure 16: a) Visualization of HPPk, and b) minimum constrained ℓ2 penalty Rβ(u1, u2, u3). c) HPPk with K(u1, u2, u3) = u1u2u3 and minimum constrained ℓ2 regularization term Rβ(u1, u2, u3) = 3 · |u1u2u3| 2/3 . d) fiber of K at β = 5, illustrating the location of the 4 points with minimal distance to the origin at the vertices (ˆu1, uˆ2, uˆ3) of the hyperbolic smooth manifold. The vertices lie tangential to the large… view at source ↗
Figure 17
Figure 17. Figure 17: Visualization of how smooth optimization transfer transforms the loss landscape using Powerprop. (37) on base objectives P(β) ≜ (1 − 1 2 β) 2 + λ|β| 2/k , k ∈ {2, 4, 6}. Appendix D. Derivation of Gradient and Hessian of Smooth Surrogate Q For simplicity, we assume no unregularized parameters ψ so that the objective function Q : R dξ → R + 0 is defined as Q(ξ) = L(K(ξ))+λRξ(ξ), where L : R d → R + 0 and K … view at source ↗
read the original abstract

We present a framework for smooth optimization of explicitly regularized objectives for (structured) sparsity. These non-smooth and possibly non-convex problems typically rely on solvers tailored to specific models and regularizers. In contrast, our method enables fully differentiable and approximation-free optimization and is thus compatible with the ubiquitous gradient descent paradigm in deep learning. The proposed optimization transfer comprises an overparameterization of selected parameters and a change of penalties. In the overparametrized problem, smooth surrogate regularization induces non-smooth, sparse regularization in the base parametrization. We prove that the surrogate objective is equivalent in the sense that it not only has identical global minima but also matching local minima, thereby avoiding the introduction of spurious solutions. Additionally, our theory establishes results of independent interest regarding matching local minima for arbitrary, potentially unregularized, objectives. We comprehensively review sparsity-inducing parametrizations across different fields that are covered by our general theory, extend their scope, and propose improvements in several aspects. Numerical experiments further demonstrate the correctness and effectiveness of our approach on several sparse learning problems ranging from high-dimensional regression to sparse neural network training.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces a framework using Hadamard overparametrization together with smooth surrogate penalties to convert non-smooth sparse regularization problems into fully differentiable objectives amenable to gradient descent. It claims to prove that the resulting surrogate objective possesses identical global minima and matching local minima to the original problem (for arbitrary, even unregularized, base objectives), thereby introducing no spurious critical points; the work also reviews and extends sparsity-inducing parametrizations across fields and reports numerical results on high-dimensional regression and sparse neural-network training.

Significance. If the claimed equivalence of minima holds, the method would allow standard first-order optimizers to be applied directly to a broad class of explicitly regularized sparse problems without custom non-smooth solvers, which is of practical value in deep learning. The general result on matching local minima for arbitrary objectives is stated to be of independent interest, and the systematic review of existing parametrizations adds archival value.

major comments (2)
  1. [Theory / proof sections] The central theoretical claim (identical global minima and matching local minima under the Hadamard-overparametrized smooth surrogate) is load-bearing; the manuscript must supply the key steps or full derivation of the local-minima equivalence (including the case of unregularized base objectives) so that the absence of spurious critical points can be verified.
  2. [Experiments] Experiments section: the abstract states that experiments confirm correctness, yet the manuscript provides neither the precise data-exclusion rules nor the method used to compute error bars; without these details it is impossible to assess whether post-hoc choices affect the reported support for the equivalence claim.
minor comments (2)
  1. [Preliminaries / notation] Notation for the overparametrized variables and the mapping back to the base parameters should be introduced once and used consistently; a short table summarizing the notation would improve readability.
  2. [Introduction] The abstract claims a 'comprehensive review' of sparsity-inducing parametrizations; the introduction or a dedicated section should explicitly delineate the scope of that review relative to prior surveys.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the positive assessment and recommendation for minor revision. We address both major comments by expanding the manuscript with the requested details.

read point-by-point responses

  1. Referee: [Theory / proof sections] The central theoretical claim (identical global minima and matching local minima under the Hadamard-overparametrized smooth surrogate) is load-bearing; the manuscript must supply the key steps or full derivation of the local-minima equivalence (including the case of unregularized base objectives) so that the absence of spurious critical points can be verified.

    Authors: We agree that explicit key steps will improve verifiability. In the revision we will insert a concise proof outline for the local-minima equivalence (covering arbitrary base objectives, including the unregularized case) directly in Section 3, with the complete derivation retained in the appendix. This addition does not alter the stated claims but makes the absence of spurious critical points directly checkable. revision: yes

  2. Referee: [Experiments] Experiments section: the abstract states that experiments confirm correctness, yet the manuscript provides neither the precise data-exclusion rules nor the method used to compute error bars; without these details it is impossible to assess whether post-hoc choices affect the reported support for the equivalence claim.

    Authors: We thank the referee for noting this omission. The revised manuscript will add a new paragraph in Section 5 that specifies (i) the exact data-exclusion criteria applied to each dataset and (ii) the procedure used to obtain error bars (mean and standard deviation over 10 independent random seeds with fixed seeds reported). These details will be provided for all reported tables and figures. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected in derivation chain

full rationale

The central result is a mathematical proof establishing that the Hadamard-overparametrized surrogate objective has identical global minima and matching local minima to the original non-smooth sparse problem (including for arbitrary unregularized base objectives). This equivalence is derived as an independent theoretical statement rather than by fitting parameters, self-definition of quantities, or load-bearing self-citation chains. The review of prior parametrizations is presented as coverage under the new general theory, with numerical experiments serving only as separate validation. No quoted reduction shows any claimed prediction or uniqueness result collapsing to an input by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Abstract-only review; no explicit free parameters, invented entities, or detailed axioms are stated. The central equivalence claim rests on an unstated modeling assumption that the chosen surrogate penalty exactly recovers the target non-smooth regularizer under the overparametrization.

axioms (1)
  • domain assumption The chosen smooth surrogate penalty, under Hadamard overparametrization, induces the exact non-smooth sparse regularization of the base parameters.
    Invoked by the claim that the surrogate objective is equivalent to the original sparse problem.

pith-pipeline@v0.9.0 · 5746 in / 1170 out tokens · 29831 ms · 2026-05-24T07:40:19.347810+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. RobuQ: Pushing DiTs to W1.58A2 via Robust Activation Quantization

    cs.CV 2025-09 conditional novelty 6.0

    RobuQ delivers the first stable DiT image generation at W1.58A2 average bits via Hadamard-based robust activation quantization and layer-wise mixed-precision activations.

Reference graph

Works this paper leans on

13 extracted references · 13 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    Iteratively reweighted algorithms for compressive sensing

    Rick Chartrand and Wotao Yin. Iteratively reweighted algorithms for compressive sensing. In 2008 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 3869–3872. IEEE,

    work page 2008

  2. [2]

    Basis pursuit

    Shaobing Chen and David Donoho. Basis pursuit. In Proceedings of 1994 28th Asilomar Conference on Signals, Systems and Computers , volume 1, pages 41–44. IEEE,

    work page 1994

  3. [3]

    Non-negative least squares via overparametrization

    55 Kolb, M ¨uller, Bischl, R ¨ugamer Hung-Hsu Chou, Johannes Maly, and Claudio Mayrink Verdun. Non-negative least squares via overparametrization. arXiv preprint arXiv:2207.08437 ,

    work page arXiv

  4. [4]

    Path regularization: A convexity and sparsity inducing regularization for parallel relu networks

    Tolga Ergen and Mert Pilanci. Path regularization: A convexity and sparsity inducing regularization for parallel relu networks. arXiv preprint arXiv:2110.09548 , 2021a. Tolga Ergen and Mert Pilanci. Revealing the structure of deep neural networks via convex duality. In International Conference on Machine Learning , pages 3004–3014. PMLR, 2021b. Mathieu Ev...

    work page arXiv

  5. [5]

    Least absolute shrinkage is equivalent to quadratic penalization

    Yves Grandvalet. Least absolute shrinkage is equivalent to quadratic penalization. In ICANN 98: Proceedings of the 8th International Conference on Artificial Neural Net- works, Sk¨ ovde, Sweden, 2–4 September 1998 8, pages 201–206. Springer,

    work page 1998

  6. [6]

    Blessing of nonconvexity in deep linear models: Depth flattens the optimization landscape around the true solution

    Jianhao Ma and Salar Fattahi. Blessing of nonconvexity in deep linear models: Depth flattens the optimization landscape around the true solution. arXiv preprint arXiv:2207.07612,

    work page arXiv

  7. [7]

    Kurdyka-lojasiewicz exponent via hadamard parametrization

    Wenqing Ouyang, Yuncheng Liu, Ting Kei Pong, and Hao Wang. Kurdyka-lojasiewicz exponent via hadamard parametrization. arXiv preprint arXiv:2402.00377 ,

    work page arXiv

  8. [8]

    Deep learning meets sparse regularization: A signal processing perspective

    Rahul Parhi and Robert D Nowak. Deep learning meets sparse regularization: A signal processing perspective. arXiv preprint arXiv:2301.09554 ,

    work page arXiv

  9. [9]

    Dropout: a simple way to prevent neural networks from overfitting

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhut- dinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958,

    work page 1929

  10. [10]

    Implicit bias of sgd in ℓ2-regularized linear dnns: One-way jumps from high to low rank

    Zihan Wang and Arthur Jacot. Implicit bias of sgd in ℓ2-regularized linear dnns: One-way jumps from high to low rank. arXiv preprint arXiv:2305.16038 ,

    work page arXiv

  11. [11]

    Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms

    Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 ,

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    A better way to decay: Proximal gradient training algorithms for neural nets

    Liu Yang, Jifan Zhang, Joseph Shenouda, Dimitris Papailiopoulos, Kangwook Lee, and Robert D Nowak. A better way to decay: Proximal gradient training algorithms for neural nets. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop) ,

    work page 2022

  13. [13]

    Symmetry leads to structured constraint of learning

    Liu Ziyin. Symmetry leads to structured constraint of learning. arXiv preprint arXiv:2309.16932,

    work page arXiv