Pith. sign in

REVIEW 3 major objections 7 minor 100 references

Meta-Sparsity: Learning Optimal Sparse Structures in Multi-task Networks through Meta-learning

T0 review · 3 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Meta-learning can learn the sparsity knob itself, not just the weights.

desk verdict The empirical work is substantial, but Algorithm 3's lambda update is not a meta-gradient, so the paper's central claim is not implemented as written. read the letter →

arxiv 2501.12115 v1 pith:WAHLWGFB submitted 2025-01-21 cs.LG cs.CV

Richa Upadhyay , Ronald Phlypo , Rajkumar Saini , Marcus Liwicki This is my paper · ORCID
classification cs.LGcs.CV
keywords meta-learningmulti-tasklearningstructuredsparsitychannel-wisehyperparameterMAMLgrouplassomodelcompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the sparsity-strength parameter in a multi-task network can be learned by meta-learning instead of being tuned by hand. It proposes meta-sparsity, which combines MAML-style bi-level optimization with channel-wise group sparsity on the shared backbone, so that the regularization parameter lambda is updated in the outer loop alongside the model parameters. If this works, sparse multi-task models would no longer require a search over sparsity budgets, and the learned sparse backbone would adapt to unseen tasks by adding new decoders. The paper reports experiments on NYU-v2 and CelebAMask-HQ showing that meta-sparsity matches or outperforms fixed-sparsity and no-sparsity baselines while keeping parameter sparsity around 43-44%.

What carries the argument

The load-bearing mechanism is group lasso regularization with a learnable $\lambda$, applied channel-wise to the shared backbone, combined with MAML's bi-level optimization. The inner loop adapts episode-specific parameters for each task subset, and the outer loop updates both the meta-parameters and $\lambda$ via a proximal operator; a Softplus reparameterization keeps $\lambda$ positive. The proximal operator for the $\ell^1$-$\ell^2$ norm has a closed form, which is what allows entire channels to be zeroed out when their $\ell^2$ norm falls below a threshold.

What would settle it

Compute the outer-loop meta-loss L_meta at the reported learned lambda and compare it with a fixed lambda that produces the same final sparsity; if the algorithm's lambda update does not decrease L_meta relative to a constant or annealed lambda schedule, the claim that lambda is meta-learned fails. Concretely, ablate the lambda update by replacing it with a random walk that keeps sparsity near 44%: if task performance is unchanged, the meta-learning of lambda is not causally responsible for the gains.

Watch

Extended reading notes

Core claim

The central claim is that meta-learning the group-sparsity hyperparameter lambda in a multi-task network produces a channel-sparse shared backbone that improves performance and generalization compared to fixed-sparsity or dense baselines. The paper argues this on its own terms: lambda is treated as a meta-parameter updated in the outer MAML loop, the inner loop adapts episode-specific parameters for single-task and multi-task episodes drawn from the power set of tasks, and the outer loop applies a proximal step that zeroes out entire channels of the shared backbone. Empirically, the paper claims that this yields stable sparsity profiles, better or comparable task metrics across NYU-v2 and CelebAMask-HQ, and successful adaptation when unseen tasks are introduced during meta-testing, with the learned mask transferring across different sparsification schedules.

Load-bearing premise

The training loop updates the sparsity-strength parameter with a value computed from gradients of the task loss with respect to model weights, not with respect to the sparsity parameter itself; the claim that the sparsity level is truly meta-learned depends on that step actually being a valid meta-gradient.

Editorial extensions

If this is right

  • Because lambda is meta-learned, practitioners can skip grid search over sparsity strength: the method yields sparse backbones of roughly 43-44% parameter sparsity without choosing a fixed lambda by hand.
  • The learned sparse backbone transfers to unseen meta-testing tasks: adding a new decoder and fine-tuning either only the new task or all tasks keeps performance stable or improves it relative to the meta-learning baseline without sparsity.
  • Structured l1-l2 sparsity gives stable sparsity profiles during training, whereas unstructured l1 sparsity under meta-learning is more variable; both are learnable with the same outer-loop procedure.
  • Regrowing sparsified channels (with probability rp>0) speeds convergence during meta-training and slightly improves some tasks, at the cost of lower final sparsity.
  • The learned mask outperforms or matches fixed masks from one-shot, iterative, progressive, and sparse-training schedules at the same sparsity budget.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the lambda update in training is not a true meta-gradient of the outer objective, the method may effectively reduce to multi-task learning with an annealed sparsity schedule; a fair comparison should include such a schedule as a baseline.
  • The observation that pixel-level tasks improve image-level tasks in sparse MTL suggests that task-composition effects may be separable from sparsity learning; one testable extension is to hold lambda fixed and vary the task set.
  • The transferability of the learned mask across sparsification schedules suggests the sparsity pattern, rather than the schedule, carries much of the benefit; an extension would be to learn a mask on one dataset and apply it to a different dataset or architecture.
  • Because lambda is initialized randomly in a small range and convergence paths vary, the method's variance may be reducible by learning an initialization for lambda across tasks, which the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes Meta-Sparsity, a training framework for multi-task networks that combines MAML-style meta-learning with channel-wise group sparsity on shared backbone parameters. The method treats the sparsity regularization strength λ as a meta-learned hyperparameter, updated in an outer loop alongside model parameters; inner-loop episodes are single-task and multi-task combinations drawn from the power set of tasks. The authors evaluate on NYU-v2 (segmentation, depth, surface normals, edges) and CelebAMask-HQ (segmentation plus six binary attribute tasks), comparing against single-task and multi-task models with no sparsity and with fixed λ, a dense MTL+meta-learning baseline, and several standard sparsification schedules, and they report performance when novel tasks are added at meta-test time.

Significance. If implemented correctly, a method that learns the sparsity/regularization trade-off across heterogeneous tasks could be a useful step beyond manual λ tuning, and the unseen-task evaluation would be a meaningful generalization check. The paper has genuine strengths: five-seed experiments with reported mean and standard deviation, multiple task combinations, a dense meta-learning baseline, and external unseen-task rows in Tables 3 and 4. However, the central λ-update in Algorithm 3 is not a gradient with respect to λ, so the claimed 'learning to learn sparsity' mechanism is not what is implemented. As written, the contribution is not supported by the derivations; the empirical results would need to be re-derived under a corrected meta-gradient or repositioned as a heuristic update.

major comments (3)
  1. [Algorithm 3 / Section 3.4] The outer-loop λ update is not a meta-gradient. Algorithm 3 computes G_i = ∇_{Θ*_{E_i}} L_meta(Θ*_{E_i}, Θ_meta, D_query), which is a gradient with respect to the adapted model parameters, and then executes 'Update λ*_meta ← prox_αout(λ_meta − α_out G_meta, λ)'. Optimizing Eq. (10) with respect to λ requires ∂L_meta/∂λ plus the regularizer derivative Σ_g √n_g ||θ^g_{bE_i}||_2 and, through the inner loop, dΘ*_{E_i}/dλ; none of these appears anywhere in the algorithm or the surrounding text. The paragraph following Algorithm 3 asserts that 'the gradient of λ in Equation 10' behaves in a certain way, but no such gradient is ever derived. The same issue affects the model-parameter update if Eq. (10) is the objective: G_meta is a first-order approximation and does not include the regularizer's gradient. Since the title, abstract, and Section 1 all claim that λ is meta-learned, this is a load-bearing correctness error. The authors must either derive and implement the correct total derivative with respect to λ (including the inner-loop dependence) or explicitly restate the method as a heuristic λ-schedule and remove the meta-learning claim.
  2. [Algorithm 2 / Section 3.2] Algorithm 2, which is presented as MAML, updates Θ_meta ← Θ_meta − α_out · Average_i [∇_{Θ*_{E_i}} L(...)]. This is first-order MAML (FOMAML), not the full MAML bi-level objective of Eqs. (3)–(4), because the derivative of the inner-loop trajectory dΘ*_{E_i}/dΘ_meta is omitted. The same omission appears in Algorithm 3. If the authors intend first-order MAML, the text should say so explicitly; if they intend full MAML, the derivation is missing. This matters not only for correctness but also for interpreting the comparison: with only first-order updates, the difference between the proposed method and the dense baseline cannot be attributed to exact meta-gradients.
  3. [Tables 3 and 4] The meta-testing results on the same tasks used for meta-training are partly self-confirming: λ is optimized on the same task losses that are later used to report success, so rows such as 'T1,T2,T3 → T1,T2,T3' in Table 3 do not by themselves establish generalization. The genuine external evidence is in the 'Adding unseen tasks' rows, and those rows should be the primary basis for the generalization claim. The authors should present those rows with a direct paired comparison against the dense meta-learning baseline and state whether the improvements are statistically significant over the five seeds. This would also strengthen the paper against the concern that the same-task gains are an artifact of validation-loss early stopping.
minor comments (7)
  1. [Section 4] The GitHub link is a placeholder ('https://github.com/PLACEHOLDER TO THE GIT REPOSITORY'); provide the actual URL or remove the reproducibility claim.
  2. [Section 3.4] There is a typo in the sentence 'It combines MTL’s ability to optimize shared and task-specific parameters with MAML’s meta-optimization framework'; the text 'Algorithm!2' should read 'Algorithm 2'.
  3. [Appendix 7.4] The title of Table 10 contains a typo: 'Compression ration' should be 'Compression ratio'.
  4. [Table 4] The column header 'highcheekbones/no highcheekbones' is formatted inconsistently with the other attribute names; use the same naming convention as in Table 2.
  5. [Section 5 / Appendix 7.2] The initialization of λ from Uniform(0.1, 1) is mentioned in the main text but not listed in the hyperparameter summary in Appendix 7.2; include it for reproducibility.
  6. [Abstract] The clause 'basically learning the parameter that controls the degree of sparsity' is informal for an abstract; consider a more precise formulation.
  7. [Figure 3] The caption acknowledges that a narrow y-axis range may amplify error bars; consider also annotating the plotted values or adding an inset so the reader can recover the scale.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the sparse-backbone generalization results rest on held-out tasks and external datasets; self-citations are background, not load-bearing.

full rationale

The core derivation (Equations 10-12 and Algorithm 3) does not reduce any predicted quantity to a fitted input. The meta-sparsity objective optimizes a penalized outer loss, and the empirical evaluation uses held-out splits of NYU-v2 and CelebAMask-HQ with five seeds, including unseen-task meta-testing in which the sparsity mask is fixed and only a new decoder is fine-tuned. Those external checks are not forced by construction. The authors' earlier work is cited for architectural inspiration and for the notion of multi-task learning episodes (Upadhyay et al. 2023a; 2023b), but the necessary definitions are restated in the paper and no load-bearing result depends on an unverified self-citation. One non-circular but serious correctness gap should be flagged: Algorithm 3 updates lambda with Gmeta = Average of gradients with respect to adapted model parameters Theta*_Ei, not with respect to lambda, and the paper never derives the inner-loop dependence of Theta*_Ei on lambda. This is an omitted derivation or mathematical inconsistency in the implementation of the claimed 'meta-learned lambda', not a case of an output being equal to an input by construction, so it does not raise the circularity score.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a global scalar lambda being meta-learnable through an unproved and under-specified gradient rule. Several hand-set hyperparameters, including sparsity patience, early stopping patience, and unreported inner and outer loop learning rates, influence the final sparsity level and performance. No new physical or model entities are introduced.

free parameters (4)
  • Global sparsity regularization parameter lambda = Unreported; initialized by sampling from U[0.1, 1]
    This is the learned sparsity strength that the entire method is built on. No final value is reported, so the reader cannot assess how strong the learned penalty actually is.
  • Inner and outer loop learning rates alpha_in and alpha_out = Not reported
    MAML-style bilevel training requires these rates, but the paper only lists optimizer learning rates for the backbone and decoders, not the meta-loop rates. This is a key reproducibility gap.
  • Sparsity patience = 30 epochs
    Training stops if sparsity does not increase for 30 epochs. This hand-set rule directly shapes the final sparsity level and therefore the reported parameter sparsity percentages.
  • Early stopping patience on validation loss = 15 epochs
    Early stopping is used to keep lambda meaningful during meta-training, so the chosen patience affects both performance and the learned sparsity profile.
assumptions (4)
  • domain assumption The MAML bilevel objective is differentiable with respect to lambda through the inner-loop optimization.
    Algorithm 3 updates lambda, but no lambda-gradient is derived. The paper assumes such a gradient exists and is computable without presenting it.
  • domain assumption Channel-wise group lasso on the shared backbone produces task-relevant shared features without breaking the network.
    The central claim assumes zeroing entire channels still leaves a functional and generalizable backbone. The paper argues residual connections help, but this is an architectural premise, not a proven fact.
  • domain assumption Episodic training on subsets of the task set transfers to previously unseen tasks.
    Meta-testing with a new task assumes the meta-trained sparse backbone can support a decoder for a task never seen in meta-training. This is the key generalization premise of the paper.
  • ad hoc to paper Validation-loss early stopping and sparsity patience do not unfairly select the reported results.
    Both stopping rules are applied during meta-training and affect lambda and sparsity. The paper does not analyze how sensitive the conclusions are to these hand-set choices.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Meta-Sparsity: Learning Optimal Sparse Structures in Multi-task Networks through Meta-learning." pith.science (2026). https://pith.science/paper/WAHLWGFB

@misc{pith2026250112115,
  author       = {Pith},
  title        = {Pith review of: Meta-Sparsity: Learning Optimal Sparse Structures in Multi-task Networks through Meta-learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WAHLWGFB}},
  note         = {Machine review of arXiv:2501.12115}
}
read the original abstract

This paper presents meta-sparsity, a framework for learning model sparsity, basically learning the parameter that controls the degree of sparsity, that allows deep neural networks (DNNs) to inherently generate optimal sparse shared structures in multi-task learning (MTL) setting. This proposed approach enables the dynamic learning of sparsity patterns across a variety of tasks, unlike traditional sparsity methods that rely heavily on manual hyperparameter tuning. Inspired by Model Agnostic Meta-Learning (MAML), the emphasis is on learning shared and optimally sparse parameters in multi-task scenarios by implementing a penalty-based, channel-wise structured sparsity during the meta-training phase. This method improves the model's efficacy by removing unnecessary parameters and enhances its ability to handle both seen and previously unseen tasks. The effectiveness of meta-sparsity is rigorously evaluated by extensive experiments on two datasets, NYU-v2 and CelebAMask-HQ, covering a broad spectrum of tasks ranging from pixel-level to image-level predictions. The results show that the proposed approach performs well across many tasks, indicating its potential as a versatile tool for creating efficient and adaptable sparse neural networks. This work, therefore, presents an approach towards learning sparsity, contributing to the efforts in the field of sparse neural networks and suggesting new directions for research towards parsimonious models.

Figures

Figures reproduced from arXiv: 2501.12115 by the authors.

Figure 1
Figure 1. This figure illustrates (a-d) a few common approaches to achieving sparse models and (e) the proposed [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. A schematic of the multi-task architecture used in this work (inspired by Upadhyay et al. (2023b)). [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. For NYU dataset, task-wise performance comparison of single-task and multi-task no sparse ( [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: For celebA dataset, task-wise performance comparison of single-task and multi-task no sparse ( [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]
Figure 5
Figure 5. Figure 5: This figure presents a comparison of performance against both the compression ratio (top row) and speed-up [PITH_FULL_IMAGE:figures/full_fig_p022_5.png]
Figure 6
Figure 6. Figure 6: Parameter sparsity patterns while training for (a) single-task fixed sparsity, (b) multi-task fixed sparsity, [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]
Figure 7
Figure 7. Figure 7: Parameter sparsity patterns for structured ( [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]
Figure 8
Figure 8. Figure 8: Comparison of the no-regrow (for rp =0, in black dotted lines) vs regrow (for rp = 0.1, 0.2, 0.3,.., 0.9) on the performance of the tasks during meta-training. the model with regrowth is relatively less sparse than the one without regrowth [PITH_FULL_IMAGE:figures/ful…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

100 extracted references · 45 canonical work pages

  1. [1]

    Learning to learn by gradient descent by gradient descent

    Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016

    2016

  2. [2]

    Multi-task feature learning

    Andreas Argyriou, Theodoros Evgeniou, and Massimiliano Pontil. Multi-task feature learning. Advances in neural information processing systems, 19, 2006

    2006

  3. [3]

    Optimization with sparsity-inducing penalties

    Francis Bach, Rodolphe Jenatton, Julien Mairal, Guillaume Obozinski, et al. Optimization with sparsity-inducing penalties. Foundations and Trends in Machine Learning , 4 0 (1): 0 1--106, 2012

    2012

  4. [4]

    Meta-learning with adaptive hyperparameters

    Sungyong Baik, Myungsub Choi, Janghoon Choi, Heewon Kim, and Kyoung Mu Lee. Meta-learning with adaptive hyperparameters. Advances in neural information processing systems, 33: 0 20755--20765, 2020

    2020

  5. [5]

    Theoretical models of learning to learn

    Jonathan Baxter. Theoretical models of learning to learn. In Learning to learn, pp.\ 71--94. Springer, 1998

    1998

  6. [6]

    Meta learning via learned loss

    Sarah Bechtle, Artem Molchanov, Yevgen Chebotar, Edward Grefenstette, Ludovic Righetti, Gaurav Sukhatme, and Franziska Meier. Meta learning via learned loss. In 2020 25th International Conference on Pattern Recognition (ICPR), pp.\ 4161--4168. IEEE, 2021

    2020

  7. [7]

    Bengio, S

    Y. Bengio, S. Bengio, and J. Cloutier. Learning a synaptic learning rule. In IJCNN-91-Seattle International Joint Conference on Neural Networks, volume ii, pp.\ 969 vol.2--, 1991. doi:10.1109/IJCNN.1991.155621

    work page arXiv 1991

  8. [8]

    What is the state of neural network pruning? Proceedings of machine learning and systems, 2: 0 129--146, 2020

    Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? Proceedings of machine learning and systems, 2: 0 129--146, 2020

    2020

Show all 100 references
  1. [9]

    Evograd: Efficient gradient-based meta-learning and hyperparameter optimization

    Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Evograd: Efficient gradient-based meta-learning and hyperparameter optimization. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume...

    2021

  2. [10]

    Distributed optimization and statistical learning via the alternating direction method of multipliers

    Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine learning , 3 0 (1): 0 1--122, 2011

    2011

  3. [11]

    learning-compression

    Miguel A. Carreira-Perpinan and Yerlan Idelbayev. "learning-compression" algorithms for neural net pruning. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 8532--8541, 2018. doi:10.1109/CVPR.2018.00890

    2018

  4. [12]

    Multitask learning

    Rich Caruana. Multitask learning. Machine learning, 28: 0 41--75, 1997

    1997

  5. [13]

    A convex formulation for learning shared structures from multiple tasks

    Jianhui Chen, Lei Tang, Jun Liu, and Jieping Ye. A convex formulation for learning shared structures from multiple tasks. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, pp.\ 137–144, New York, NY, USA, 2009. Association for Computing ...

    2009

  6. [14]

    Rethinking atrous convolution for semantic image segmentation, 2017

    Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation, 2017. arXiv:1706.05587

    2017 arXiv

  7. [15]

    Metalr: Meta-tuning of learning rates for transfer learning in medical imaging

    Yixiong Chen, Li Liu, Jingxian Li, Hua Jiang, Chris Ding, and Zongwei Zhou. Metalr: Meta-tuning of learning rates for transfer learning in medical imaging. In Hayit Greenspan, Anant Madabhushi, Parvin Mousavi, Septimiu Salcudean, James Duncan, Tanveer Syeda-Mahmood, and Russel...

    2023

  8. [16]

    Signal recovery by proximal forward-backward splitting

    Patrick L Combettes and Val \'e rie R Wajs. Signal recovery by proximal forward-backward splitting. Multiscale modeling & simulation, 4 0 (4): 0 1168--1200, 2005

    2005

  9. [17]

    Multi-task learning with deep neural networks: A survey

    Michael Crawshaw. Multi-task learning with deep neural networks: A survey. arXiv preprint arXiv:2009.09796, 2020

    2009 arXiv

  10. [18]

    Structured sparsity inducing adaptive optimizers for deep learning

    Tristan Deleu and Yoshua Bengio. Structured sparsity inducing adaptive optimizers for deep learning. arXiv preprint arXiv:2102.03869, 2021

    2021 arXiv

  11. [19]

    Model compression and hardware acceleration for neural networks: A comprehensive survey

    Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108 0 (4): 0 485--532, 2020. doi:10.1109/JPROC.2020.2976475

    2020

  12. [20]

    In defense of parameter sharing for model-compression

    Aditya Desai and Anshumali Shrivastava. In defense of parameter sharing for model-compression. arXiv preprint arXiv:2310.11611, 2023

    2023 arXiv

  13. [21]

    Sparse networks from scratch: Faster training without losing performance

    Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.04840, 2019

    1907 arXiv

  14. [22]

    Learning to learn by jointly optimizing neural architecture and weights

    Yadong Ding, Yu Wu, Chengyue Huang, Siliang Tang, Yi Yang, Longhui Wei, Yueting Zhuang, and Qi Tian. Learning to learn by jointly optimizing neural architecture and weights. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 129--...

    2022

  15. [23]

    Neural architecture search: A survey

    Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research, 20 0 (1): 0 1997--2017, 2019

    1997

  16. [24]

    Meta-learning of neural architectures for few-shot learning

    Thomas Elsken, Benedikt Staffler, Jan Hendrik Metzen, and Frank Hutter. Meta-learning of neural architectures for few-shot learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 12365--12375, 2020

    2020

  17. [25]

    Rigging the lottery: Making all tickets winners

    Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp.\ 2943--2952. PMLR, 2020

    2020

  18. [26]

    Model-agnostic meta-learning for fast adaptation of deep networks

    Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Resear...

    2017

  19. [27]

    Bilevel programming for hyperparameter optimization and meta-learning

    Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International conference on machine learning, pp.\ 1568--1577. PMLR, 2018

    2018

  20. [28]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019

    2019

  21. [29]

    The state of sparsity in deep neural networks

    Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574, 2019

    1902 arXiv

  22. [30]

    Searching for robustness: Loss learning for noisy classification tasks

    Boyan Gao, Henry Gouk, and Timothy M Hospedales. Searching for robustness: Loss learning for noisy classification tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 6670--6679, 2021

    2021

  23. [31]

    Meta mirror descent: Optimiser learning for fast convergence

    Boyan Gao, Henry Gouk, Hae Beom Lee, and Timothy M Hospedales. Meta mirror descent: Optimiser learning for fast convergence. arXiv preprint arXiv:2203.02711, 2022 a

    2022 arXiv

  24. [32]

    Loss function learning for domain generalization by implicit gradient

    Boyan Gao, Henry Gouk, Yongxin Yang, and Timothy Hospedales. Loss function learning for domain generalization by implicit gradient. In International Conference on Machine Learning, pp.\ 7002--7016. PMLR, 2022 b

    2022

  25. [33]

    Understanding the difficulty of training deep feedforward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings o...

    2010

  26. [34]

    Gon c alves, Fernando J

    Andr \'e R. Gon c alves, Fernando J. Von Zuben, and Arindam Banerjee. Multi-task sparse structure learning with gaussian copula models. Journal of Machine Learning Research, 17 0 (33): 0 1--30, 2016

    2016

  27. [35]

    Deep Learning

    Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. URL http://www.deeplearningbook.org. Chapter 5: Machine Learning Basics

    2016

  28. [36]

    Learning both weights and connections for efficient neural network

    Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015

    2015

  29. [37]

    A closer look at learned optimization: Stability, robustness, and inductive biases

    James Harrison, Luke Metz, and Jascha Sohl-Dickstein. A closer look at learned optimization: Stability, robustness, and inductive biases. Advances in Neural Information Processing Systems, 35: 0 3758--3773, 2022

    2022

  30. [38]

    Statistical learning with sparsity: the lasso and generalizations

    Trevor Hastie, Robert Tibshirani, and Martin Wainwright. Statistical learning with sparsity: the lasso and generalizations. CRC press, 2015

    2015

  31. [39]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  32. [40]

    Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks

    Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. J. Mach. Learn. Res., 22 0 (1), jan 2021. ISSN 1532-4435

    2021

  33. [41]

    Hospedales, A

    T. Hospedales, A. Antoniou, P. Micaelli, and A. Storkey. Meta-learning in neural networks: A survey. IEEE Transactions on Pattern Analysis & Machine Intelligence, 44 0 (09): 0 5149--5169, sep 2022. ISSN 1939-3539. doi:10.1109/TPAMI.2021.3079209

    2022

  34. [42]

    Revisiting single image depth estimation: Toward higher resolution maps with accurate object boundaries

    Junjie Hu, Mete Ozay, Yan Zhang, and Takayuki Okatani. Revisiting single image depth estimation: Toward higher resolution maps with accurate object boundaries. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), 2019

    2019

  35. [43]

    Neural network pruning

    Nathan Hubens. Neural network pruning. nathanhubens.github.io, 2020

    2020

  36. [44]

    A survey of deep meta-learning

    Mike Huisman, Jan N Van Rijn, and Aske Plaat. A survey of deep meta-learning. Artificial Intelligence Review, 54 0 (6): 0 4483--4541, 2021

    2021

  37. [45]

    Janowsky

    Steven A. Janowsky. Pruning versus clipping in neural networks. Phys. Rev. A, 39: 0 6600--6603, Jun 1989. doi:10.1103/PhysRevA.39.6600. URL https://link.aps.org/doi/10.1103/PhysRevA.39.6600

    1989 doi

  38. [46]

    Multi-task learning using uncertainty to weigh losses for scene geometry and semantics

    Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 7482--7491, 2018

    2018

  39. [47]

    Learning task structure via sparsity grouped multitask learning

    Meghana Kshirsagar, Eunho Yang, and Aur \'e lie C Lozano. Learning task structure via sparsity grouped multitask learning. arXiv preprint arXiv:1705.04886, 2017

    2017 arXiv

  40. [48]

    Soft threshold weight reparameterization for learnable sparsity

    Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine...

    2020

  41. [49]

    Maskgan: Towards diverse and interactive facial image manipulation

    Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020

    2020

  42. [50]

    Layer-adaptive sparsity for the magnitude-based pruning

    Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. Layer-adaptive sparsity for the magnitude-based pruning. In International Conference on Learning Representations, 2021

    2021

  43. [51]

    Learning to optimize neural nets

    Ke Li and Jitendra Malik. Learning to optimize neural nets. arXiv preprint arXiv:1703.00441, 2017

    2017 arXiv

  44. [52]

    Meta-sgd: Learning to learn quickly for few-shot learning

    Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv:1707.09835, 2017

    2017 arXiv

  45. [53]

    Towards fast adaptation of neural architectures with meta learning

    Dongze Lian, Yin Zheng, Yintao Xu, Yanxiong Lu, Leyu Lin, Peilin Zhao, Junzhou Huang, and Shenghua Gao. Towards fast adaptation of neural architectures with meta learning. In International Conference on Learning Representations, 2019

    2019

  46. [54]

    Auxiliary tasks in multi-task learning

    Lukas Liebel and Marco K \"o rner. Auxiliary tasks in multi-task learning. arXiv preprint arXiv:1805.06334, 2018

    2018 arXiv

  47. [55]

    sparseland

    Shiwei Liu and Zhangyang Wang. Ten lessons we have learned in the new "sparseland": A short handbook for sparse neural network researchers, 2023

    2023

  48. [56]

    Learning efficient convolutional networks through network slimming

    Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision, pp.\ 2736--2744, 2017

    2017

  49. [57]

    Rethinking the value of network pruning

    Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint arXiv:1810.05270, 2018

    2018 arXiv

  50. [58]

    Learning gradient descent: Better generalization and longer horizons

    Kaifeng Lv, Shunhua Jiang, and Jian Li. Learning gradient descent: Better generalization and longer horizons. In International Conference on Machine Learning, pp.\ 2247--2255. PMLR, 2017

    2017

  51. [59]

    Pruning filter in filter

    Fanxu Meng, Hao Cheng, Ke Li, Huixiang Luo, Xiaowei Guo, Guangming Lu, and Xing Sun. Pruning filter in filter. Advances in Neural Information Processing Systems, 33: 0 17629--17640, 2020

    2020

  52. [60]

    Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves

    Luke Metz, Niru Maheswaranathan, C Daniel Freeman, Ben Poole, and Jascha Sohl-Dickstein. Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves. arXiv preprint arXiv:2009.11243, 2020

    2009 arXiv

  53. [61]

    Velo: Training versatile learned optimizers by scaling up

    Luke Metz, James Harrison, C Daniel Freeman, Amil Merchant, Lucas Beyer, James Bradbury, Naman Agrawal, Ben Poole, Igor Mordatch, Adam Roberts, et al. Velo: Training versatile learned optimizers by scaling up. arXiv preprint arXiv:2211.09760, 2022

    2022 arXiv

  54. [62]

    Indoor segmentation and support inference from rgbd images

    Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012

    2012

  55. [64]

    On first-order meta-learning algorithms

    Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999, 2018

    2018 arXiv

  56. [65]

    Joint covariate selection and joint subspace selection for multiple classification problems

    Guillaume Obozinski, Ben Taskar, and Michael I Jordan. Joint covariate selection and joint subspace selection for multiple classification problems. Statistics and Computing, 20: 0 231--252, 2010

    2010

  57. [66]

    Proximal algorithms

    Neal Parikh, Stephen Boyd, et al. Proximal algorithms. Foundations and trends in Optimization , 1 0 (3): 0 127--239, 2014

    2014

  58. [67]

    Senthil Kumar

    Sandip Paul, Bhuvan Jhamb, Deepak Mishra, and M. Senthil Kumar. Edge loss functions for deep-learning depth-map. Machine Learning with Applications, 7: 0 100218, 2022. ISSN 2666-8270. doi:https://doi.org/10.1016/j.mlwa.2021.100218

    2022

  59. [68]

    Rapid learning or feature reuse? towards understanding the effectiveness of maml

    Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint arXiv:1909.09157, 2019

    1909 arXiv

  60. [69]

    Learning symbolic model-agnostic loss functions via meta-learning

    Christian Raymond, Qi Chen, and Bing Xue. Learning symbolic model-agnostic loss functions via meta-learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023 a

    2023

  61. [70]

    Online loss function learning

    Christian Raymond, Qi Chen, Bing Xue, and Mengjie Zhang. Online loss function learning. arXiv preprint arXiv:2301.13247, 2023 b

    2023

  62. [71]

    A comprehensive survey of neural architecture search: Challenges and solutions

    Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Computing Surveys (CSUR), 54 0 (4): 0 1--34, 2021

    2021

  63. [72]

    Across-task neural architecture search via meta learning

    Jingtao Rong, Xinyi Yu, Mingyang Zhang, and Linlin Ou. Across-task neural architecture search via meta learning. International Journal of Machine Learning and Cybernetics, 14 0 (3): 0 1003--1019, 2023

    2023

  64. [73]

    Low-rank matrix factorization for deep neural network training with high-dimensional output targets

    Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low-rank matrix factorization for deep neural network training with high-dimensional output targets. In 2013 IEEE international conference on acoustics, speech and signal processing, pp.\ 6...

    2013

  65. [74]

    Towards stochasticity of regularization in deep neural networks

    Ljubinka Sandjakoska and Ana Madevska Bogdanova. Towards stochasticity of regularization in deep neural networks. In 2018 14th Symposium on Neural Networks and Applications (NEUREL), pp.\ 1--4, 2018. doi:10.1109/NEUREL.2018.8587027

    2018

  66. [75]

    Group sparse regularization for deep neural networks

    Simone Scardapane, Danilo Comminiello, Amir Hussain, and Aurelio Uncini. Group sparse regularization for deep neural networks. Neurocomputing, 241: 0 81--89, 2017

    2017

  67. [76]

    Evolutionary principles in self-referential learning

    Jurgen Schmidhuber. Evolutionary principles in self-referential learning. on learning now to learn: The meta-meta-meta...-hook. Diploma thesis, Technische Universitat Munchen, Germany, 14 May 1987

    1987

  68. [77]

    Meta-learning sparse compression networks

    Jonathan Schwarz and Yee Whye Teh. Meta-learning sparse compression networks. Transactions on Machine Learning Research, 2022. ISSN 2835-8856

    2022

  69. [78]

    Meta architecture search

    Albert Shaw, Wei Wei, Weiyang Liu, Le Song, and Bo Dai. Meta architecture search. Advances in Neural Information Processing Systems, 32, 2019

    2019

  70. [79]

    Learning a minimax optimizer: A pilot study

    Jiayi Shen, Xiaohan Chen, Howard Heaton, Tianlong Chen, Jialin Liu, Wotao Yin, and Zhangyang Wang. Learning a minimax optimizer: A pilot study. In International Conference on Learning Representations, 2020

    2020

  71. [80]

    LEARNED LEARNING RATE SCHEDULES FOR DEEP NEURAL NETWORK TRAINING USING REINFORCEMENT LEARNING , 2023

    Shreyas Subramanian, Vignesh Ganapathiraman, and Aly El Gamal. LEARNED LEARNING RATE SCHEDULES FOR DEEP NEURAL NETWORK TRAINING USING REINFORCEMENT LEARNING , 2023. URL https://openreview.net/forum?id=0Zhwu1VaOs

    2023

  72. [81]

    Learning sparse sharing architectures for multiple tasks

    Tianxiang Sun, Yunfan Shao, Xiaonan Li, Pengfei Liu, Hang Yan, Xipeng Qiu, and Xuanjing Huang. Learning sparse sharing architectures for multiple tasks. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp.\ 8936--8943, 2020 a

    2020

  73. [82]

    Adashare: Learning what to share for efficient deep multi-task learning

    Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko. Adashare: Learning what to share for efficient deep multi-task learning. Advances in Neural Information Processing Systems, 33: 0 8728--8740, 2020 b

    2020

  74. [83]

    Xinglong Sun, Maying Shen, Hongxu Yin, Lei Mao, Pavlo Molchanov, and Jose M. Alvarez. Towards dynamic sparsification by iterative prune-grow lookaheads, 2023

    2023

  75. [84]

    Thrun and L.Y

    S. Thrun and L.Y. Pratt (eds.). Learning To Learn. Kluwer Academic Publishers, Boston, MA, 1998

    1998

  76. [85]

    Learning to learn

    Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012

    2012

  77. [86]

    Meta-learning approaches for learning-to-learn in deep learning: A survey

    Yingjie Tian, Xiaoxi Zhao, and Wei Huang. Meta-learning approaches for learning-to-learn in deep learning: A survey. Neurocomputing, 494: 0 203--223, 2022. ISSN 0925-2312

    2022

  78. [87]

    Multi-task meta learning: learn how to adapt to unseen tasks

    Richa Upadhyay, Prakash Chandra Chhipa, Ronald Phlypo, Rajkumar Saini, and Marcus Liwicki. Multi-task meta learning: learn how to adapt to unseen tasks. In 2023 International Joint Conference on Neural Networks (IJCNN), pp.\ 1--10, 2023 a . doi:10.1109/IJCNN54540.2023.10191400

    2023

  79. [88]

    Less is more towards parsimonious multi-task models using structured sparsity

    Richa Upadhyay, Ronald Phlypo, Rajkumar Saini, and Marcus Liwicki. Less is more towards parsimonious multi-task models using structured sparsity. In Conference on Parsimony and Learning (Proceedings Track), 2023 b

    2023

  80. [89]

    Sharing to learn and learning to share; fitting together meta, multi-task, and transfer learning: A meta review

    Richa Upadhyay, Ronald Phlypo, Rajkumar Saini, and Marcus Liwicki. Sharing to learn and learning to share; fitting together meta, multi-task, and transfer learning: A meta review. IEEE Access, 12: 0 148553--148576, 2024. doi:10.1109/ACCESS.2024.3478805

    2024

  81. [90]

    Neural pruning via growing regularization

    Huan Wang, Can Qin, Yulun Zhang, and Yun Fu. Neural pruning via growing regularization. In International Conference on Learning Representations, 2021

    2021

  82. [91]

    Learning structured sparsity in deep neural networks

    Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances in neural information processing systems, 29, 2016

    2016

  83. [92]

    Learned optimizers that scale and generalize

    Olga Wichrowska, Niru Maheswaranathan, Matthew W Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize. In International conference on machine learning, pp.\ 3751--3760. PMLR, 2017

    2017

  84. [93]

    Learning to learn how to learn: Self-adaptive visual navigation using meta-learning

    Mitchell Wortsman, Kiana Ehsani, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Learning to learn how to learn: Self-adaptive visual navigation using meta-learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 6750--6759, 2019

    2019

  85. [94]

    Learning to schedule learning rate with graph neural networks

    Yuanhao Xiong, Li-Cheng Lan, Xiangning Chen, Ruochen Wang, and Cho-Jui Hsieh. Learning to schedule learning rate with graph neural networks. In International Conference on Learning Representation (ICLR), 2022

    2022

  86. [95]

    Dilated residual networks

    Fisher Yu, Vladlen Koltun, and Thomas Funkhouser. Dilated residual networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017

    2017

  87. [96]

    Model Selection and Estimation in Regression with Grouped Variables

    Ming Yuan and Yi Lin. Model Selection and Estimation in Regression with Grouped Variables . Journal of the Royal Statistical Society Series B: Statistical Methodology, 68 0 (1): 0 49--67, 12 2005. ISSN 1369-7412. doi:10.1111/j.1467-9868.2005.00532.x

    2005

  88. [97]

    Two heads are better than one: Boosting graph sparse training via semantic and topological awareness, 2024

    Guibin Zhang, Yanwei Yue, Kun Wang, Junfeng Fang, Yongduo Sui, Kai Wang, Yuxuan Liang, Dawei Cheng, Shirui Pan, and Tianlong Chen. Two heads are better than one: Boosting graph sparse training via semantic and topological awareness, 2024

    2024

  89. [98]

    Learning n: m fine-grained structured sparse neural networks from scratch

    Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning n: m fine-grained structured sparse neural networks from scratch. arXiv preprint arXiv:2102.04010, 2021 a

    2021 arXiv

  90. [99]

    Effective sparsification of neural networks with global sparsity constraint

    Xiao Zhou, Weizhong Zhang, Hang Xu, and Tong Zhang. Effective sparsification of neural networks with global sparsity constraint. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 3599--3608, 2021 b

    2021

  91. [100]

    To prune, or not to prune: exploring the efficacy of pruning for model compression

    Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878, 2017

    2017 arXiv

  92. [101]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.