pith. sign in

arxiv: 2605.25134 · v3 · pith:A22SV6EJnew · submitted 2026-05-24 · 💻 cs.LG · cs.AI

Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Rate

Huangyu Xu , Jingqin Yang , Qianqian Xu , Jiaye Teng This is my paper

Pith reviewed 2026-06-30 11:45 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords sparse optimizationreparameterizationweight decayadaptive learning ratelp regularizationneural network sparsityResNet
0
0 comments X

The pith

ReWA achieves greater sparsity than ℓ1 regularization in ResNets while preserving test accuracy on CIFAR-10 and ImageNet.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces ReWA as a method for sparse optimization that combines reparameterization, weight decay, and adaptive learning rates. This approach is linked to ℓ_p regularization but produces a different optimization landscape intended to avoid instability from unbounded gradients when the exponent falls between zero and one. Experiments with ResNet models on standard image datasets show that ReWA yields sparser solutions than plain ℓ1 regularization without reducing accuracy. A sympathetic reader would care because the method offers a direct way to induce sparsity inside the training loop rather than through separate post-processing steps.

Core claim

ReWA is closely connected to ℓ_p-regularization, yet it unveils a distinct optimization landscape that helps mitigate instability issues. Experiments on CIFAR-10 and ImageNet with ResNets demonstrate that ReWA leads to significant sparsity improvements over the ℓ1-regularization approach while preserving test accuracy.

What carries the argument

ReWA, the combination of reparameterization, weight decay, and adaptive learning rate that produces a distinct optimization landscape connected to but different from ℓ_p regularization.

If this is right

  • ReWA mitigates instability issues of ℓ_p regularization for 0 < p < 1.
  • ReWA produces significant sparsity improvements over ℓ1 regularization.
  • Test accuracy remains comparable to ℓ1 regularization on CIFAR-10 and ImageNet with ResNets.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The method may allow training of models with fewer active parameters from the start, lowering memory use during inference on constrained hardware.
  • It could reduce reliance on separate pruning pipelines that require additional fine-tuning after initial training.
  • The same combination of techniques might be tested on other model families such as transformers to check whether the sparsity gain generalizes.

Load-bearing premise

The distinct optimization landscape created by the combination of reparameterization, weight decay, and adaptive learning rate actually mitigates the instability issues of ℓ_p regularization for 0 < p < 1.

What would settle it

Repeating the CIFAR-10 and ImageNet experiments with ResNets and finding that ReWA produces no measurable increase in sparsity or causes a drop in test accuracy relative to ℓ1 regularization would disprove the central claim.

Figures

Figures reproduced from arXiv: 2605.25134 by Huangyu Xu, Jiaye Teng, Jingqin Yang, Qianqian Xu.

Figure 1
Figure 1. Figure 1: The ablation study of K and M on linear regression models. Blue means the returned test loss could be small, red means the test loss is large, and white means that M > K − 1. (CIFAR-10 and ImageNet in Section 4.2) demonstrate that ReWA consistently returns sparse solutions. Role of sparse optimization. One of the classic applica￾tions revolves around sparse signal recovery, notably preva￾lent in the medica… view at source ↗
Figure 2
Figure 2. Figure 2: Performance of ReWA in linear regression regimes for different hyperparameters. ferently, Theorem 3.14 does not require much problem in￾formation, and only relies on the property that p-norm is naturally close to 0-norm when p is small. Unfortunately, Theorem 3.14 cannot directly recover the compressed sens￾ing results by setting p = 1. 4. Experiments In this section, we conduct experiments on both synthet… view at source ↗
Figure 3
Figure 3. Figure 3: Performance of ReWA in real-world datasets. methods on the same ResNet-18 architecture typically ex￾hibit significant accuracy degradation once the compression ratio reaches 102 (see [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: ℓ1 regularization fails in general cases. The solution is sparse if it falls in the coordinate. Left: ℓ1 regularization returns sparse solutions in linear cases (compressed sensing). Middle: ℓ1 regularization returns non-sparse solutions in general non-linear cases. Right: ℓp regularization (p ∈ (0, 1)) may perform better and return sparse solutions in general non-linear cases. 17 [PITH_FULL_IMAGE:figures… view at source ↗
Figure 5
Figure 5. Figure 5: Clipping for ℓp regularization. The blue lines (solid and dotted) represent the original ℓp regularization, and the orange lines represent the clipped version at some clipping point. C.1. Preliminaries We first present preliminaries before the proof, including additional notations and basic knowledge about compressed sensing. We would start from the notations on ℓ0 minimization. Sparse optimization problem… view at source ↗
read the original abstract

Sparse optimization is a fundamental challenge in various practical applications. A popular approach to sparse optimization is $\ell_p$ regularization. However, it may encounter optimization instability due to the unbounded gradients when $0<p<1$. In this paper, we introduce a novel approach to sparse optimization termed ReWA, based on Reparameterization, Weight decay, and Adaptive learning rate. ReWA is closely connected to $\ell_p$-regularization, yet it unveils a distinct optimization landscape that helps mitigate instability issues. Experiments on CIFAR-10 and ImageNet with ResNets demonstrate that ReWA leads to significant sparsity improvements over the $\ell_1$-regularization approach while preserving test accuracy.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The manuscript introduces ReWA, a sparse optimization method based on reparameterization, weight decay, and adaptive learning rates. It claims a close theoretical connection to ℓ_p regularization (for 0 < p < 1) while asserting a distinct optimization landscape that mitigates the instability caused by unbounded gradients. Experiments on CIFAR-10 and ImageNet using ResNets are reported to yield significant sparsity gains over ℓ1 regularization without loss of test accuracy.

Significance. If the claimed theoretical connection and landscape analysis hold and the empirical gains prove robust, the work could provide a practical route to sparsity in deep networks that avoids known difficulties with non-convex regularizers. The combination of reparameterization with adaptive rates is a potentially useful direction, but its value depends on the rigor of the supporting derivations and controls.

major comments (2)
  1. [Abstract] Abstract: the central claim of a 'distinct optimization landscape that helps mitigate instability issues' is asserted without any derivation, equation, or landscape analysis visible; the connection to ℓ_p regularization therefore cannot be evaluated for correctness or novelty.
  2. [Experiments] Experiments (CIFAR-10 / ImageNet results): the reported 'significant sparsity improvements' are stated without error bars, multiple runs, statistical tests, or ablation controls separating the contributions of reparameterization, weight decay, and the adaptive rate; this leaves the empirical claim load-bearing but unsupported.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We address the two major points below, clarifying the location of the theoretical material and committing to improved experimental reporting.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim of a 'distinct optimization landscape that helps mitigate instability issues' is asserted without any derivation, equation, or landscape analysis visible; the connection to ℓ_p regularization therefore cannot be evaluated for correctness or novelty.

    Authors: The abstract is a high-level summary. The explicit connection to ℓ_p regularization for 0 < p < 1, the reparameterization that produces an equivalent but distinct landscape, and the derivation showing bounded gradients (thereby mitigating the instability of direct ℓ_p penalties) appear in Section 3, with the relevant equations, landscape plots, and proofs. We will add a sentence to the abstract directing readers to Section 3. revision: partial

  2. Referee: [Experiments] Experiments (CIFAR-10 / ImageNet results): the reported 'significant sparsity improvements' are stated without error bars, multiple runs, statistical tests, or ablation controls separating the contributions of reparameterization, weight decay, and the adaptive rate; this leaves the empirical claim load-bearing but unsupported.

    Authors: The current version indeed reports single-run results without error bars, statistical tests, or component-wise ablations. We will rerun the CIFAR-10 and ImageNet experiments with at least three independent seeds, add error bars and significance tests, and include ablations that isolate reparameterization, weight decay, and the adaptive rate schedule. revision: yes

Circularity Check

0 steps flagged

No significant circularity identified

full rationale

The provided abstract and surrounding context present ReWA as a novel combination of reparameterization, weight decay, and adaptive learning rate that is connected to but distinct from ℓ_p regularization, with empirical results on CIFAR-10 and ImageNet. No derivation chain, equations, fitted parameters renamed as predictions, or self-citations are visible in the material. The central claim of a distinct optimization landscape mitigating instability is stated without reduction to prior inputs or self-referential definitions. The derivation is therefore self-contained against external benchmarks, with no load-bearing steps that collapse by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Only abstract available; no free parameters, axioms, or invented entities can be extracted.

pith-pipeline@v0.9.1-grok · 5650 in / 971 out tokens · 26398 ms · 2026-06-30T11:45:04.426830+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

99 extracted references · 9 canonical work pages · 2 internal anchors

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    and Warmuth, M

    Amid, E. and Warmuth, M. K. Winnowing with gradient descent. In Conference on Learning Theory, pp.\ 163--182. PMLR, 2020

    2020

  3. [3]

    Implicit regularization in deep matrix factorization

    Arora, S., Cohen, N., Hu, W., and Luo, Y. Implicit regularization in deep matrix factorization. In Advances in Neural Information Processing Systems, pp.\ 7411--7422, 2019

    2019

  4. [4]

    Bradley, P. S. and Mangasarian, O. L. Feature selection via concave minimization and support vector machines. In International Conference on Machine Learning, pp.\ 82--90. Morgan Kaufmann, 1998

    1998

  5. [5]

    Improving network slimming with nonconvex regularization

    Bui, K., Park, F., Zhang, S., Qi, Y., and Xin, J. Improving network slimming with nonconvex regularization. IEEE Access, 9: 0 107590--107601, 2021

    2021

  6. [6]

    Candes, E. J. The restricted isometry property and its implications for compressed sensing. Comptes Rendus Mathematique, 346 0 (9-10): 0 589--592, 2008

    2008

  7. [7]

    Candes, E. J. and Tao, T. Near-optimal signal recovery from random projections: Universal encoding strategies? IEEE Transactions on Information Theory, 52 0 (12): 0 5406--5425, 2006

    2006

  8. [8]

    Exact reconstruction of sparse signals via nonconvex minimization

    Chartrand, R. Exact reconstruction of sparse signals via nonconvex minimization. IEEE Signal Processing Letters, 14 0 (10): 0 707--710, 2007

    2007

  9. [9]

    Fast algorithms for nonconvex compressive sensing: Mri reconstruction from very few data

    Chartrand, R. Fast algorithms for nonconvex compressive sensing: Mri reconstruction from very few data. In IEEE International Symposium on Biomedical Imaging, pp.\ 262--265. IEEE, 2009

    2009

  10. [10]

    and Staneva, V

    Chartrand, R. and Staneva, V. Restricted isometry properties and nonconvex compressive sensing. Inverse Problems, 24 0 (3): 0 035020, 2008

    2008

  11. [11]

    Strong np-hardness for sparse optimization with concave penalty functions

    Chen, Y., Ge, D., Wang, M., Wang, Z., Ye, Y., and Yin, H. Strong np-hardness for sparse optimization with concave penalty functions. In International Conference on Machine Learning, pp.\ 740--747. PMLR, 2017

    2017

  12. [12]

    On lazy training in differentiable programming

    Chizat, L., Oyallon, E., and Bach, F. On lazy training in differentiable programming. In Advances in Neural Information Processing Systems, volume 32, pp.\ 2933--2943, 2019

    2019

  13. [13]

    More is less: Inducing sparsity via overparameterization

    Chou, H.-H., Maly, J., and Rauhut, H. More is less: Inducing sparsity via overparameterization. arXiv preprint arXiv:2112.11027, 2021

    work page arXiv 2021

  14. [14]

    Representation costs of linear neural networks: Analysis and design

    Dai, Z., Karzand, M., and Srebro, N. Representation costs of linear neural networks: Analysis and design. In Advances in Neural Information Processing Systems, pp.\ 26884--26896, 2021

    2021

  15. [15]

    Imagenet: A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 248--255. IEEE, 2009

    2009

  16. [16]

    Duarte-Carvajalino, J. M. and Sapiro, G. Learning to sense sparse signals: Simultaneous sensing matrix and sparsifying dictionary optimization. IEEE Transactions on Image Processing, 18 0 (7): 0 1395--1408, 2009

    2009

  17. [17]

    and Li, R

    Fan, J. and Li, R. Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association, 96 0 (456): 0 1348--1360, 2001

    2001

  18. [18]

    Understanding implicit regularization in over-parameterized single index model

    Fan, J., Yang, Z., and Yu, M. Understanding implicit regularization in over-parameterized single index model. Journal of the American Statistical Association, 118 0 (544): 0 2315--2328, 2023

    2023

  19. [19]

    E., Pang, J.-S., Shen, X., and W \"a chter, A

    Feng, M., Mitchell, J. E., Pang, J.-S., Shen, X., and W \"a chter, A. Complementarity formulations of l0-norm optimization problems. Technical report, Industrial Engineering and Management Sciences, Northwestern University, 2013

    2013

  20. [20]

    Frank, L. E. and Friedman, J. H. A statistical view of some chemometrics regression tools. Technometrics, 35 0 (2): 0 109--135, 1993

    1993

  21. [21]

    Fu, W. J. Penalized regressions: the bridge versus the lasso. Journal of Computational and Graphical Statistics, 7 0 (3): 0 397--416, 1998

    1998

  22. [22]

    Sign-in to the lottery: Reparameterizing sparse training

    Gadhikar, A., Jacobs, T., Burkholz, R., et al. Sign-in to the lottery: Reparameterizing sparse training. Advances in Neural Information Processing Systems, 38: 0 117941--117971, 2026

    2026

  23. [23]

    George, E. I. The variable selection problem. Journal of the American Statistical Association, 95 0 (452): 0 1304--1308, 2000

    2000

  24. [24]

    The implicit bias of depth: How incremental learning drives generalization

    Gissin, D., Shalev-Shwartz, S., and Daniely, A. The implicit bias of depth: How incremental learning drives generalization. In International Conference on Learning Representations, 2020

    2020

  25. [25]

    Least absolute shrinkage is equivalent to quadratic penalization

    Grandvalet, Y. Least absolute shrinkage is equivalent to quadratic penalization. In International Conference on Artificial Neural Networks, pp.\ 201--206. Springer, 1998

    1998

  26. [26]

    D., Soudry, D., and Srebro, N

    Gunasekar, S., Lee, J. D., Soudry, D., and Srebro, N. Implicit bias of gradient descent on linear convolutional networks. In Advances in Neural Information Processing Systems, pp.\ 9482--9491, 2018

    2018

  27. [27]

    Han, S., Pool, J., Tran, J., and Dally, W. J. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, pp.\ 1135--1143, 2015

    2015

  28. [28]

    Han, S., Mao, H., and Dally, W. J. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In International Conference on Learning Representations, 2016

    2016

  29. [29]

    D., and Zadeh, R

    Hastie, T., Mazumder, R., Lee, J. D., and Zadeh, R. Matrix completion and low-rank svd via fast alternating least squares. Journal of Machine Learning Research, 16: 0 3367--3402, 2015

    2015

  30. [30]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 770--778, 2016

    2016

  31. [31]

    Hoff, P. D. Lasso, fractional norm and structured sparse estimation using a hadamard product parametrization. Computational Statistics & Data Analysis, 115: 0 186--198, 2017

    2017

  32. [32]

    Phase transition and higher order analysis of l_q regularization under dependence

    Huang, H., Zeng, P., and Yan, Q. Phase transition and higher order analysis of l_q regularization under dependence. arXiv preprint arXiv:2211.10541, 2022

    work page arXiv 2022

  33. [33]

    Hunter, D. R. and Li, R. Variable selection using mm algorithms. Annals of Statistics, 33 0 (4): 0 1617--1642, 2005

    2005

  34. [34]

    and Burkholz, R

    Jacobs, T. and Burkholz, R. Mask in the mirror: Implicit sparsification. In International Conference on Learning Representations, volume 2025, pp.\ 36221--36245, 2025

    2025

  35. [35]

    Mirror, mirror of the flow: How does regularization shape implicit bias? In Forty-second International Conference on Machine Learning, 2025

    Jacobs, T., Zhou, C., and Burkholz, R. Mirror, mirror of the flow: How does regularization shape implicit bias? In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=MLiR9LS5PW

    2025

  36. [36]

    Neural tangent kernel: Convergence and generalization in neural networks

    Jacot, A., Gabriel, F., and Hongler, C. Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems, 31: 0 8580--8589, 2018

    2018

  37. [37]

    Feature learning in l_2 -regularized dnns: Attraction/repulsion and sparsity

    Jacot, A., Golikov, E., Hongler, C., and Gabriel, F. Feature learning in l_2 -regularized dnns: Attraction/repulsion and sparsity. arXiv preprint arXiv:2205.15809, 2022

    work page arXiv 2022

  38. [38]

    P., and Gunasekar, S

    Jagadeesan, M., Razenshteyn, I. P., and Gunasekar, S. Inductive bias of multi-channel linear convolutional networks with bounded weight norm. In Conference on Learning Theory, pp.\ 2276--2325. PMLR, 2022

    2022

  39. [39]

    Kang, H., Mina, R. J. L., Madjid, S. R. H., Yoon, J., Hasegawa-Johnson, M., Hwang, S. J., and Yoo, C. D. Forget-free continual learning with winning subnetworks. In International Conference on Machine Learning, pp.\ 10734--10750. PMLR, 2022

    2022

  40. [40]

    Linear convergence of gradient and proximal-gradient methods under the polyak- ojasiewicz condition

    Karimi, H., Nutini, J., and Schmidt, M. Linear convergence of gradient and proximal-gradient methods under the polyak- ojasiewicz condition. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp.\ 795--811. Springer, 2016

    2016

  41. [41]

    Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  42. [42]

    Smoothing the Edges: Smooth Optimization for Sparse Regularization using Hadamard Overparametrization

    Kolb, C., M \"u ller, C. L., Bischl, B., and R \"u gamer, D. Smoothing the edges: A general framework for smooth optimization in sparse regularization using hadamard overparametrization. arXiv preprint arXiv:2307.03571, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  43. [43]

    Learning multiple layers of features from tiny images

    Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

    2009

  44. [44]

    M., Salman, H., and Madry, A

    Leclerc, G., Ilyas, A., Engstrom, L., Park, S. M., Salman, H., and Madry, A. Ffcv: Accelerating training by removing data bottlenecks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 12011--12020, 2023

    2023

  45. [45]

    Sparse signal recovery via tail hadamard product parametrization

    Li, D., Li, G., Ma, C., Yang, A., and Li, S. Sparse signal recovery via tail hadamard product parametrization. In China Automation Congress, pp.\ 7504--7509. IEEE, 2021 a

    2021

  46. [46]

    The tail-hadamard product parametrization algorithm for compressed sensing

    Li, G., Li, S., Li, D., and Ma, C. The tail-hadamard product parametrization algorithm for compressed sensing. Signal Processing, 205: 0 108853, 2023

    2023

  47. [47]

    Li, H., Kadav, A., Durdanovic, I., Samet, H., and Graf, H. P. Pruning filters for efficient convnets. In International Conference on Learning Representations, 2017

    2017

  48. [48]

    V., Hegde, C., and Wong, K

    Li, J., Nguyen, T. V., Hegde, C., and Wong, K. W. Implicit sparse regularization: The impact of depth and early stopping. In Advances in Neural Information Processing Systems, pp.\ 28298--28309, 2021 b

    2021

  49. [49]

    D., and Arora, S

    Li, Z., Wang, T., Lee, J. D., and Arora, S. Implicit bias of gradient descent on reparametrized models: On equivalence to mirror descent. In Advances in Neural Information Processing Systems, 2022

    2022

  50. [50]

    Robust training under label noise by over-parameterization

    Liu, S., Zhu, Z., Qu, Q., and You, C. Robust training under label noise by over-parameterization. In International Conference on Machine Learning, pp.\ 14153--14172. PMLR, 2022

    2022

  51. [51]

    and Wang, Z

    Liu, Z. and Wang, Z. spred: Solving l1 penalty with sgd. In International Conference on Machine Learning, pp.\ 43407--43422. PMLR, 2023

    2023

  52. [52]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019

    2019

  53. [53]

    Louizos, C., Welling, M., and Kingma, D. P. Learning sparse neural networks through l_0 regularization. In International Conference on Learning Representations, 2018

    2018

  54. [54]

    Lustig, M., Donoho, D., and Pauly, J. M. Sparse mri: The application of compressed sensing for rapid mr imaging. Magnetic Resonance in Medicine, 58 0 (6): 0 1182--1195, 2007

    2007

  55. [55]

    L., Santos, J

    Lustig, M., Donoho, D. L., Santos, J. M., and Pauly, J. M. Compressed sensing mri. IEEE Signal Processing Magazine, 25 0 (2): 0 72--82, 2008

    2008

  56. [56]

    and Fattahi, S

    Ma, J. and Fattahi, S. Blessing of nonconvexity in deep linear models: Depth flattens the optimization landscape around the true solution. arXiv preprint arXiv:2207.07612, 2022

    work page arXiv 2022

  57. [57]

    and Solo, V

    Marjanovic, G. and Solo, V. On _q optimization and matrix completion. IEEE Transactions on Signal Processing, 60 0 (11): 0 5714--5724, 2012

    2012

  58. [58]

    and Solo, V

    Marjanovic, G. and Solo, V. On exact _q denoising. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp.\ 6068--6072. IEEE, 2013

    2013

  59. [59]

    H., Aldroubi, A., Sanei, S., and Chambers, J

    Marvasti, F., Amini, A., Haddadi, F., Soltanolkotabi, M., Khalaj, B. H., Aldroubi, A., Sanei, S., and Chambers, J. A unified approach to sparse signal processing. EURASIP Journal on Advances in Signal Processing, 2012 0 (1): 0 1--45, 2012

    2012

  60. [60]

    M., and Ayanzadeh, R

    Mousavi, S., Rezaee Taghiabadi, M. M., and Ayanzadeh, R. A survey on compressive sensing: Classical results and recent advancements. arXiv preprint arXiv:1908.01014, 2019

    work page arXiv 1908

  61. [61]

    Natarajan, B. K. Sparse approximate solutions to linear systems. SIAM Journal on Computing, 24 0 (2): 0 227--234, 1995 a

    1995

  62. [62]

    Natarajan, B. K. Sparse approximate solutions to linear systems. SIAM Journal on Computing, 24 0 (2): 0 227--234, 1995 b

    1995

  63. [63]

    In search of the real inductive bias: On the role of implicit regularization in deep learning

    Neyshabur, B., Tomioka, R., and Srebro, N. In search of the real inductive bias: On the role of implicit regularization in deep learning. In International Conference on Learning Representations Workshop, 2015

    2015

  64. [64]

    A function space view of bounded norm infinite width relu nets: The multivariate case

    Ongie, G., Willett, R., Soudry, D., and Srebro, N. A function space view of bounded norm infinite width relu nets: The multivariate case. In International Conference on Learning Representations, 2020

    2020

  65. [65]

    Task-specific skill localization in fine-tuned language models

    Panigrahi, A., Saunshi, N., Zhao, H., and Arora, S. Task-specific skill localization in fine-tuned language models. In International Conference on Machine Learning, pp.\ 27011--27033. PMLR, 2023

    2023

  66. [66]

    and Nowak, R

    Parhi, R. and Nowak, R. D. Banach space representer theorems for neural networks and ridge splines. Journal of Machine Learning Research, 22 0 (43): 0 1--40, 2021

    2021

  67. [67]

    and Nowak, R

    Parhi, R. and Nowak, R. D. Deep learning meets sparse regularization: A signal processing perspective. IEEE Signal Processing Magazine, 40 0 (6): 0 63--74, 2023

    2023

  68. [68]

    and Ergen, T

    Pilanci, M. and Ergen, T. Neural networks are convex regularizers: Exact polynomial-time convex optimization formulations for two-layer networks. In International Conference on Machine Learning, pp.\ 7695--7705. PMLR, 2020

    2020

  69. [69]

    Label noise (stochastic) gradient descent implicitly solves the lasso for quadratic parametrisation

    Pillaud-Vivien, L., Reygner, J., and Flammarion, N. Label noise (stochastic) gradient descent implicitly solves the lasso for quadratic parametrisation. In Conference on Learning Theory, pp.\ 2127--2159. PMLR, 2022

    2022

  70. [70]

    and Peyr \'e , G

    Poon, C. and Peyr \'e , G. Smooth bilevel programming for sparse regularization. In Advances in Neural Information Processing Systems, pp.\ 1543--1555, 2021

    2021

  71. [71]

    and Peyr \'e , G

    Poon, C. and Peyr \'e , G. Smooth over-parameterized solvers for non-smooth structured optimization. Mathematical Programming, 201 0 (1): 0 897--952, 2023

    2023

  72. [72]

    and Mancera, L

    Portilla, J. and Mancera, L. L0-based sparse approximation: Two alternative methods and some applications. In Wavelets XII, volume 6701, pp.\ 697--711. SPIE, 2007

    2007

  73. [73]

    Rennie, J. D. M. and Srebro, N. Fast maximum margin matrix factorization for collaborative prediction. In International Conference on Machine Learning, pp.\ 713--719. ACM, 2005

    2005

  74. [74]

    Stable sparse approximations via nonconvex optimization

    Saab, R., Chartrand, R., and Yilmaz, O. Stable sparse approximations via nonconvex optimization. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp.\ 3885--3888. IEEE, 2008

    2008

  75. [75]

    How do infinite width bounded norm networks look in function space? In Conference on Learning Theory, pp.\ 2667--2690

    Savarese, P., Evron, I., Soudry, D., and Srebro, N. How do infinite width bounded norm networks look in function space? In Conference on Learning Theory, pp.\ 2667--2690. PMLR, 2019

    2019

  76. [76]

    Winning the lottery with continuous sparsification

    Savarese, P., Silva, H., and Maire, M. Winning the lottery with continuous sparsification. In Advances in Neural Information Processing Systems, 2020

    2020

  77. [77]

    M., Pascanu, R., Latham, P

    Schwarz, J., Jayakumar, S. M., Pascanu, R., Latham, P. E., and Teh, Y. W. Powerpropagation: A sparsity inducing weight reparameterisation. In Advances in Neural Information Processing Systems, pp.\ 28889--28903, 2021

    2021

  78. [78]

    Srebro, N., Rennie, J. D. M., and Jaakkola, T. S. Maximum-margin matrix factorization. In Advances in Neural Information Processing Systems, pp.\ 1329--1336, 2004

    2004

  79. [79]

    A tutorial on sparse signal reconstruction and its applications in signal processing

    Stankovi \'c , L., Sejdi \'c , E., Stankovi \'c , S., Dakovi \'c , M., and Orovi \'c , I. A tutorial on sparse signal reconstruction and its applications in signal processing. Circuits, Systems, and Signal Processing, 38: 0 1206--1263, 2019

    2019

  80. [80]

    and Li, Y

    Sun, Y. and Li, Y. Dice: Leveraging sparsification for out-of-distribution detection. In European Conference on Computer Vision, pp.\ 691--708. Springer, 2022

    2022

Showing first 80 references.