Pith. sign in

REVIEW 2 major objections 3 minor 21 references

Discrete efficiency choices in deep learning become continuous objectives that train end-to-end.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A thesis that synthesizes the author's published work on continuous approximations to discrete deep learning problems, with experiments showing efficiency gains, but offering little new beyond the author's prior papers.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A transparent, well-run thesis that repackages the author's own strong prior work; the 'principled' quantization claim has a load-bearing assumption that is likely false under the paper's own bit map. the 2 major comments →

arxiv 2509.00174 v2 pith:TYS274IM submitted 2025-08-29 cs.LG cs.AI

Principled Approximation Methods for Efficient and Scalable Deep Learning

Pedro Savarese This is my paper
classification cs.LG cs.AI
keywords deep learning efficiencymodel compressionnetwork sparsificationmixed-precision quantizationneural architecture searchadaptive optimizationcontinuous relaxationℓ0 regularization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This thesis claims that the hard combinatorial decisions behind efficient deep learning—which weights to remove, how many bits each weight needs, and what network topology to use—do not have to be handled by heuristics or post-hoc searches. Each one can be re-framed as a continuous, differentiable objective whose discrete target is recovered in a limit, so the compression scheme is trained jointly with the network by ordinary gradient descent. The thesis develops three such approximations: soft parameter sharing for architecture search, a deterministic homotopy for sparsification (Continuous Sparsification), and perturbation-limit precision learning for mixed-precision quantization (SMOL). It also proposes an adaptive optimizer, AvaGrad, that is cheaper to tune. On image classification, language modeling, and generative modeling tasks, the methods yield leaner models with equal or better accuracy.

Core claim

The central claim is that computationally intractable discrete optimization problems in deep learning can be replaced by principled continuous approximations that remain faithful to the original problem as a parameter approaches a limit. For sparsification, the binary keep/drop mask becomes σ(βs), and annealing β upward drives weights smoothly out of the network, recovering ℓ0 regularization in the limit. For quantization, precision assignment is re-framed as an infinite-constraint problem and then approximated stochastically: each weight learns a perturbation scale that captures how much noise it tolerates, and that scale is converted into a bit precision. For architecture search, each laye

What carries the argument

The unifying mechanism is the continuous relaxation with an annealing or scale parameter. In Continuous Sparsification it is the sigmoidal mask σ(βs) on each weight, with β annealed so the smooth objective converges to the ℓ0-penalized discrete problem. In SMOL it is the perturbation proxy: each precision variable s defines a noise magnitude σ(s), and the loss is computed on w+σ(s)⊙ε, so gradients through the noise magnitude reveal how much precision each weight can give up. In the architecture search chapter it is the template mixing coefficient α, which expresses each layer's effective weight as a linear combination of shared templates and yields a layer-similarity matrix that the method f

Load-bearing premise

The quantization results rest on the assumption that adding precision to a weight never turns an acceptable quantized model into an unacceptable one; if that monotonicity fails, the reformulated objective is not equivalent to the original mixed-precision problem.

What would settle it

Run SMOL's precision training on a small network, then take the assigned per-weight precisions and measure the loss for strictly higher-precision versions of the same weights. If increasing precision ever raises the loss, or if lowering a judged-infeasible precision stays within the δ tolerance, the monotonic-satisfiability and density assumptions that underwrite the method are violated.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Compression need not be a post-hoc step: mask and precision variables receive gradient updates alongside weights, so the compression scheme is learned while the model trains.
  • Per-weight and per-layer precision allocation can be optimized by gradient descent, producing non-uniform bit assignments that reduce total bits while preserving accuracy.
  • Ticket search—finding sparse subnetworks that train from scratch—becomes cheaper than iterative magnitude pruning when runs are parallelized, and the discovered tickets can match or exceed the dense model's accuracy.
  • Architecture search over a topological space can discover recurrent/looped structures from feedforward CNN seeds, and the layer-similarity matrix provides a concrete recipe for folding them into smaller equivalent networks.
  • The adaptive optimizer's design makes learning rate and adaptability near-separable, so hyperparameter optimization reaches a target accuracy in fewer trials than Adam.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The perturbation-limit principle is a general sensitivity measure: it could be used to set tolerances for other discretizations—activation quantization, block-wise formats, or reduced-precision training—without repeating the full derivation.
  • The monotonic satisfiability assumption behind SMOL suggests its precision assignments are conservative: in regimes where extra precision can hurt, an over-precision penalty or a validation-based correction might improve bit savings.
  • The layer-similarity folding analysis could serve as a diagnostic for implicit recurrence in any trained network, independent of whether the network was produced by this search procedure.
  • A joint objective that anneals both the sparsification mask and the quantization scale would combine two compression axes into a single continuous relaxation; the ingredients for it appear in different chapters of the thesis.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 3 minor

Summary. This PhD thesis proposes and evaluates continuous, differentiable approximations to three discrete optimization problems arising in deep learning efficiency, plus a modified adaptive optimizer. Chapter 3 relaxes neural architecture search over a topological search space via soft parameter sharing; Chapter 4 relaxes ℓ0 sparsification using an annealed sigmoid mask, yielding Continuous Sparsification (CS); Chapter 5 derives a perturbation-based method, SMOL, for mixed-precision quantization; Chapter 6 analyzes adaptive methods and proposes AvaGrad. The experimental sections are extensive, covering CIFAR, ImageNet, IWSLT, GANs, and various architectures, and the thesis claims substantial improvements in parameter efficiency, sparsity, precision allocation, and hyperparameter tuning cost.

Significance. If the central claims hold, the thesis is practically significant: CS finds extremely sparse subnetworks with less computation than iterative magnitude pruning; SMOL demonstrates fine-grained precision assignment that can outperform uniform-precision baselines; and AvaGrad shows competitive accuracy with reduced hyperparameter sensitivity. The thesis also contains full proofs for the optimizer convergence results in Chapter 6 and carefully documents ablations and hyperparameter analyses. However, the 'principled approximation' framing is only partially established. For sparsification, the method optimizes a lower bound without a no-gap proof; for quantization, the equivalence with the original mixed-precision problem relies on an unverified monotonicity assumption that is, in fact, false under the paper's own bit-value map. These are load-bearing gaps for the stated theoretical claims, though the empirical methods remain interesting and potentially useful.

major comments (2)
  1. [§5.2.2, Eq. (5.12)]
  2. [§4.2.3, Eq. (4.14)]
minor comments (3)
  1. [§5.2.1]
  2. [Chapter 6, Tables 6.1 and 6.2]
  3. [Throughout]

Circularity Check

0 steps flagged

No significant circularity: the thesis compiles the author's prior peer-reviewed work with self-contained formal derivations and new experiments; no central claim reduces to its own inputs by construction.

full rationale

The thesis is a compilation of four previously published research thrusts (Savarese and Maire 2019; Savarese et al. 2020, 2021, 2022). The self-references are provenance statements, not load-bearing justifications: each chapter re-derives its formal problem, states the discrete objective, introduces a continuous relaxation, and then reports experimental measurements. The approximations are not defined in terms of the target outcomes. For example, Continuous Sparsification instantiates a sigmoid homotopy from a stated ℓ0 objective and explicitly acknowledges the min/limit swap gap (Section 4.2.3, Eq. 4.14); SMOL defines precision via a perturbation proxy and an explicit monotonicity assumption (Eq. 5.12). These are stated assumptions and potential correctness risks, but they are not circular: the re-framed problems are not equivalent to the conclusions by construction, and the paper does not present tuned hyperparameters as predictions. No uniqueness theorem is imported from the authors' prior work, and no benchmark is fitted and then claimed as a forecast. The quantization chapter's reliance on Eq. 5.12 is an unproven and possibly false assumption (the skeptic's counterexample illustrates a robustness issue), but a false or untested assumption is a correctness concern, not circularity. Overall, the derivation chain is self-contained and empirically anchored; the modest self-citation pattern does not constitute circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The thesis relies on several unverified assumptions to make the approximations valid: the lower-bound gap in sparsification, the monotonicity-in-precision assumption in quantization, and the fixed-point binary format. Free parameters such as lambda, beta_final, s_init, and precision regularizers are tuned by hand to achieve target sparsity/performance, which is standard practice but weakens the 'parameter-free' or 'prediction' framing.

free parameters (5)
  • sparsity regularization strength lambda = 10^-8
    Controls trade-off between loss and sparsity in Continuous Sparsification; chosen by hand based on hyperparameter analysis (Section 4.5.1).
  • annealing temperature beta_final = 200
    Final temperature in CS annealing schedule; recommended based on ablation showing larger beta improves accuracy (Section 4.5.1).
  • mask initialization s_init = ranges -0.3 to 0.3
    Initial value of mask parameters; strongly controls final sparsity, tuned per run to achieve desired sparsity (Section 4.5.1).
  • precision regularization lambda = 10^-6, 7e-7, 5e-7, etc.
    Controls trade-off between loss and total bits in SMOL; values chosen per dataset/model (Section 5.3.3).
  • initial precision p_init = 8
    Starting bit width for all weights in SMOL; set to 8 bits (Section 5.3.3).
axioms (4)
  • domain assumption Swapping the order of minimization and limit in the sparsification approximation (Section 4.2.3) yields a valid lower bound; requires Gamma-convergence and equi-coercivity of L_beta to ensure no gap in the limit.
    The paper explicitly states this is an approximation with two sources of error; it does not prove the required conditions.
  • domain assumption Satisfiability is monotone in precision: if (theta,p) is satisfiable then (theta,p') is satisfiable for any p' >= p (Equation 5.12, Section 5.2.2).
    This is the core assumption for the quantization method's re-framing; it is stated but not verified.
  • standard math The bit-value map v_j = 2^{1-j} (fixed-point binary) and reparameterization s = sigma^{-1}(2^{1-p}) restrict the method to uniform fixed-point quantization.
    Standard binary representation, but the choice constrains the quantization format and may not hold for alternative representations.
  • standard math L of f is continuous for the limit-swap justification (Definition 4.4).
    Neural network losses are typically not globally continuous in the discrete mask, but the paper assumes continuity for the theoretical justification.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Principled Approximation Methods for Efficient and Scalable Deep Learning." pith.science (2026). https://pith.science/paper/TYS274IM

@misc{pith2026250900174,
  author       = {Pith},
  title        = {Pith review of: Principled Approximation Methods for Efficient and Scalable Deep Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TYS274IM}},
  note         = {Machine review of arXiv:2509.00174}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Recent progress in deep learning has been driven by increasingly larger models. However, their computational and energy demands have grown proportionally, creating significant barriers to their deployment and to a wider adoption of deep learning technologies. This thesis investigates principled approximation methods for improving the efficiency of deep learning systems, with a particular focus on settings that involve discrete constraints and non-differentiability. We study three main approaches toward improved efficiency: architecture design, model compression, and optimization. For model compression, we propose novel approximations for pruning and quantization that frame the underlying discrete problem as continuous and differentiable, enabling gradient-based training of compression schemes alongside the model's parameters. These approximations allow for fine-grained sparsity and precision configurations, leading to highly compact models without significant fine-tuning. In the context of architecture design, we design an algorithm for neural architecture search that leverages parameter sharing across layers to efficiently explore implicitly recurrent architectures. Finally, we study adaptive optimization, revisiting theoretical properties of widely used methods and proposing an adaptive optimizer that allows for quick hyperparameter tuning. Our contributions center on tackling computationally hard problems via scalable and principled approximations. Experimental results on image classification, language modeling, and generative modeling tasks show that the proposed methods provide significant improvements in terms of training and inference efficiency while maintaining, or even improving, the model's performance.

Figures

Figures reproduced from arXiv: 2509.00174 by Pedro Savarese.

Figure 3.1
Figure 3.1. Figure 3.1: Illustration of our soft parameter sharing scheme. [PITH_FULL_IMAGE:figures/full_fig_p040_3_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: (right) depicts this new view for a two-layer CNN: two template convolutions [PITH_FULL_IMAGE:figures/full_fig_p041_3.png] view at source ↗
Figure 3
Figure 3. Figure 3: shows examples of the LSM [PITH_FULL_IMAGE:figures/full_fig_p042_3.png] view at source ↗
Figure 3.2
Figure 3.2. Figure 3.2: LSM when training with and without parameter sharing, and its connection to the [PITH_FULL_IMAGE:figures/full_fig_p043_3_2.png] view at source ↗
Figure 3.3
Figure 3.3. Figure 3.3: Parameter efficiency for different models on CIFAR-10. [PITH_FULL_IMAGE:figures/full_fig_p051_3_3.png] view at source ↗
Figure 3.4
Figure 3.4. Figure 3.4: Parameter efficiency for different models on CIFAR-100. [PITH_FULL_IMAGE:figures/full_fig_p053_3_4.png] view at source ↗
Figure 3
Figure 3. Figure 3: shows the per-stage LSM for CIFAR-trained SWRN 28-10-4. Here, the six [PITH_FULL_IMAGE:figures/full_fig_p055_3.png] view at source ↗
Figure 3.5
Figure 3.5. Figure 3.5: Folding a SWRN 28-10-4 trained on CIFAR-10. [PITH_FULL_IMAGE:figures/full_fig_p056_3_5.png] view at source ↗
Figure 3
Figure 3. Figure 3: shows a SWRN 40-8-8 (8 parameter templates shared among groups of [PITH_FULL_IMAGE:figures/full_fig_p056_3.png] view at source ↗
Figure 3.6
Figure 3.6. Figure 3.6: 1 Folding a SWRN 40-8-8 trained on CIFAR-10. Red paths are taken before green, [PITH_FULL_IMAGE:figures/full_fig_p057_3_6.png] view at source ↗
Figure 3.7
Figure 3.7. Figure 3.7: Folding a SWRN 40-8-8 trained on CIFAR-10. [PITH_FULL_IMAGE:figures/full_fig_p058_3_7.png] view at source ↗
Figure 3.8
Figure 3.8. Figure 3.8: Example of the synthetic shortest paths task. [PITH_FULL_IMAGE:figures/full_fig_p059_3_8.png] view at source ↗
Figure 3
Figure 3. Figure 3: shows one example from our generated dataset: blue pixels indicate the query [PITH_FULL_IMAGE:figures/full_fig_p059_3.png] view at source ↗
Figure 3.9
Figure 3.9. Figure 3.9: Training curves for the shortest paths task. [PITH_FULL_IMAGE:figures/full_fig_p060_3_9.png] view at source ↗
Figure 3
Figure 3. Figure 3: shows training curves for the 3 trained models: the SCNN not only outperforms [PITH_FULL_IMAGE:figures/full_fig_p060_3.png] view at source ↗
Figure 3.10
Figure 3.10. Figure 3.10: LSMs of a SWRN 40-8-8 over different runs. [PITH_FULL_IMAGE:figures/full_fig_p061_3_10.png] view at source ↗
Figure 3.11
Figure 3.11. Figure 3.11: LSMs of a SWRN 40-8-8 over different epochs of the same run. [PITH_FULL_IMAGE:figures/full_fig_p061_3_11.png] view at source ↗
Figure 3.12
Figure 3.12. Figure 3.12: LSM evolution for 100-layer CNN trained on synthetic task (0, 20, 40 epochs). [PITH_FULL_IMAGE:figures/full_fig_p062_3_12.png] view at source ↗
Figure 4.1
Figure 4.1. Figure 4.1: Performance of different methods when performing one-shot pruning on VGG-16, [PITH_FULL_IMAGE:figures/full_fig_p082_4_1.png] view at source ↗
Figure 4.2
Figure 4.2. Figure 4.2: Performance of different methods when performing one-shot pruning on ResNet-20, [PITH_FULL_IMAGE:figures/full_fig_p083_4_2.png] view at source ↗
Figure 4.3
Figure 4.3. Figure 4.3: VGG-16 on CIFAR-10. Test accuracy and sparsity of subnetworks produced by IMP and [PITH_FULL_IMAGE:figures/full_fig_p084_4_3.png] view at source ↗
Figure 4.4
Figure 4.4. Figure 4.4: ResNet-20 on CIFAR-10. Test accuracy and sparsity of subnetworks produced by [PITH_FULL_IMAGE:figures/full_fig_p085_4_4.png] view at source ↗
Figure 4.5
Figure 4.5. Figure 4.5: Impact on relative test accuracy and sparsity of tickets found by CS in a ResNet-20 [PITH_FULL_IMAGE:figures/full_fig_p090_4_5.png] view at source ↗
Figure 4.6
Figure 4.6. Figure 4.6: Impact on relative test accuracy and sparsity of tickets found by CS in a ResNet-20 [PITH_FULL_IMAGE:figures/full_fig_p091_4_6.png] view at source ↗
Figure 4.7
Figure 4.7. Figure 4.7: Impact on relative test accuracy and sparsity of tickets found by CS in a ResNet-20 [PITH_FULL_IMAGE:figures/full_fig_p092_4_7.png] view at source ↗
Figure 4.8
Figure 4.8. Figure 4.8: Performance of tickets found by Iterative Magnitude Pruning in a ResNet-20 trained on [PITH_FULL_IMAGE:figures/full_fig_p093_4_8.png] view at source ↗
Figure 4
Figure 4. Figure 4: presents results: CS is capable of finding high performing sparse supermasks [PITH_FULL_IMAGE:figures/full_fig_p094_4.png] view at source ↗
Figure 4.9
Figure 4.9. Figure 4.9: Learning a binary mask with weights frozen at initialization with Stochastic Sparsification [PITH_FULL_IMAGE:figures/full_fig_p095_4_9.png] view at source ↗
Figure 4.10
Figure 4.10. Figure 4.10: Learning a binary mask with weights frozen at initialization with Stochastic Sparsifica [PITH_FULL_IMAGE:figures/full_fig_p096_4_10.png] view at source ↗
Figure 4
Figure 4. Figure 4: presents tickets produced by each method, measured by their sparsity and test [PITH_FULL_IMAGE:figures/full_fig_p096_4.png] view at source ↗
Figure 4.11
Figure 4.11. Figure 4.11: Accuracy and sparsity of tickets produced by IMP, ISS and CS after re-training, starting [PITH_FULL_IMAGE:figures/full_fig_p097_4_11.png] view at source ↗
Figure 4
Figure 4. Figure 4: shows the final sparsity of blocks consisting of two consecutive convolutional [PITH_FULL_IMAGE:figures/full_fig_p097_4.png] view at source ↗
Figure 4.12
Figure 4.12. Figure 4.12: Sparsity patterns learned by CS and IMP for VGG-16 trained on CIFAR-10 – each [PITH_FULL_IMAGE:figures/full_fig_p098_4_12.png] view at source ↗
Figure 4.13
Figure 4.13. Figure 4.13: Accuracy and sparsity of tickets produced by IMP and Sequential CS after re-training, [PITH_FULL_IMAGE:figures/full_fig_p099_4_13.png] view at source ↗
Figure 4
Figure 4. Figure 4: shows the performance of tickets produced by Sequential CS and IMP, [PITH_FULL_IMAGE:figures/full_fig_p099_4.png] view at source ↗
Figure 5.1
Figure 5.1. Figure 5.1: Weight noise as a differentiable proxy for precision. A learnable magnitude δ scales uniform random noise added to weight w during training. The width of the basin over which it is possible to perturb w without increasing task loss L drives learning of δ. After training, we reduce the bit precision of the numeric representation of w as much as possible, with the constraint of remaining in the (w − δ, w +… view at source ↗
Figure 5.2
Figure 5.2. Figure 5.2: Multidimensional example where dif￾ferent precisions should be assigned to each pa￾rameter due to their distinct perturbation limits. Note the connection between each parameter’s per￾turbation limit, the width of the level curve in the parameter’s axis, and the allocated precision for each parameter. Once the perturbation limit of each weight has been estimated through optimiza￾tion by our method, we ass… view at source ↗
Figure 5.3
Figure 5.3. Figure 5.3: Performance of SMOL and BSQ when quantizing a ResNet-20 trained on CIFAR-10. [PITH_FULL_IMAGE:figures/full_fig_p123_5_3.png] view at source ↗
Figure 5.4
Figure 5.4. Figure 5.4: Performance of SMOL, BSQ, and LQ-Nets when quantizing a ResNet-50 trained on [PITH_FULL_IMAGE:figures/full_fig_p124_5_4.png] view at source ↗
Figure 5.5
Figure 5.5. Figure 5.5: Image generations with a DCGAN trained on CIFAR-10, quantized with BSQ and [PITH_FULL_IMAGE:figures/full_fig_p130_5_5.png] view at source ↗
Figure 5
Figure 5. Figure 5: shows the precisions allocated to each layer of the ResNet-20 under three [PITH_FULL_IMAGE:figures/full_fig_p131_5.png] view at source ↗
Figure 5.6
Figure 5.6. Figure 5.6: Layer-wise precisions allocated by SMOL-L on ResNet-20 models trained on CIFAR-10. [PITH_FULL_IMAGE:figures/full_fig_p132_5_6.png] view at source ↗
Figure 5.7
Figure 5.7. Figure 5.7: Performance and average precision of ResNet-20 trained on CIFAR-10 with SMOL when [PITH_FULL_IMAGE:figures/full_fig_p133_5_7.png] view at source ↗
Figure 5.8
Figure 5.8. Figure 5.8: Generalization performance when re-training networks, with and without randomly [PITH_FULL_IMAGE:figures/full_fig_p135_5_8.png] view at source ↗
Figure 6.1
Figure 6.1. Figure 6.1: Performance of Adam with different learning rate [PITH_FULL_IMAGE:figures/full_fig_p157_6_1.png] view at source ↗
Figure 6.2
Figure 6.2. Figure 6.2: Performance of AvaGrad with different learning rate [PITH_FULL_IMAGE:figures/full_fig_p157_6_2.png] view at source ↗
Figure 6.3
Figure 6.3. Figure 6.3: Performance of Adam with different learning rate [PITH_FULL_IMAGE:figures/full_fig_p158_6_3.png] view at source ↗
Figure 6.4
Figure 6.4. Figure 6.4: Performance of AvaGrad with different learning rate [PITH_FULL_IMAGE:figures/full_fig_p158_6_4.png] view at source ↗
Figure 6.5
Figure 6.5. Figure 6.5: Iterations to achieve 0.5% suboptimality, measured in terms of validation accuracy on CIFAR-10, for Adam and AvaGrad when tuning α and ϵ with various standard hyperparameter optimizers. it from α. With AvaGrad, α governs optimization dynamics in both regimes: when ϵ is small, changing its value has negligible impact on ηt and ∥ηt∥, hence the updates will be the same, while in the non-adaptive regime we h… view at source ↗
Figure 6.6
Figure 6.6. Figure 6.6: Suboptimality (gap in validation accuracy) when optimizing [PITH_FULL_IMAGE:figures/full_fig_p160_6_6.png] view at source ↗
Figure 6
Figure 6. Figure 6: shows the number of iterations required by different hyperparameter optimizers [PITH_FULL_IMAGE:figures/full_fig_p160_6.png] view at source ↗
Figure 6.7
Figure 6.7. Figure 6.7: The mean gradient norm as function of the iteration [PITH_FULL_IMAGE:figures/full_fig_p162_6_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 5 canonical work pages · 4 internal anchors

  1. [1]

    Estimating or propagating gradients through stochastic neurons for conditional computation.arXiv:1308.3432, 2013a

    Yoshua Bengio, Nicholas L´ eonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation.arXiv:1308.3432, 2013a. Yoshua Bengio, Nicholas L´ eonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation.arXiv:1308.3432, 2013b. Alexandre Boulc...

    Pith/arXiv arXiv

  2. [4]

    Roy, and Michael Carbin

    Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis.arXiv:1903.01611,

    Pith/arXiv arXiv 1903

  3. [5]

    Neural Turing Machines.arXiv:1410.5401,

    Alex Graves, Greg Wayne, and Ivo Danihelka. Neural Turing Machines.arXiv:1410.5401,

    Pith/arXiv arXiv

  4. [11]

    An Analysis of Neural Language Modeling at Multiple Scales.arXiv:1803.08240,

    Stephen Merity, Nitish Shirish Keskar, and Richard Socher. An Analysis of Neural Language Modeling at Multiple Scales.arXiv:1803.08240,

    Pith/arXiv arXiv

  5. [13]

    Regularized evolution for image classifier architecture search.arXiv:1802.01548,

    Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search.arXiv:1802.01548,

    Pith/arXiv arXiv

  6. [16]

    Training Sparse Neural Networks

    Suraj Srinivas, Akshayvarun Subramanya, and R. Venkatesh Babu. Training sparse neural networks.arXiv:1611.06694,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [17]

    ACDC: Weight Sharing in Atom-Coefficient Decomposed Convolution

    Ze Wang, Xiuyuan Cheng, Guillermo Sapiro, and Qiang Qiu. Acdc: Weight sharing in atom-coefficient decomposed convolution.arXiv preprint arXiv:2009.02386,

    work page internal anchor Pith review Pith/arXiv arXiv 2009

  8. [18]

    Mixed precision quantization of ConvNets via differentiable neural architecture search

    Bichen Wu, Yanghan Wang, Peizhao Zhang, Yuandong Tian, Peter Vajda, and Kurt Keutzer. Mixed precision quantization of ConvNets via differentiable neural architecture search. arXiv:1812.00090,

    Pith/arXiv arXiv

  9. [21]

    Deconstructing lottery tickets: Zeros, signs, and the supermask.NeurIPS, 2019a

    Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask.NeurIPS, 2019a. Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. DoReFa- Net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv:1606.06160,

    Pith/arXiv arXiv

  10. [22]

    To prune, or not to prune: exploring the efficacy of pruning for model compression.arXiv:1710.01878,

    Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression.arXiv:1710.01878,

    Pith/arXiv arXiv

  11. [1989]

    J. H. Lim and J. C. Ye. Geometric gan.arXiv:1806.06763,

    Pith/arXiv arXiv

  12. [1994]

    Sparse transfer learning via winning lottery tickets.arXiv:1905.07785,

    Rahul Mehta. Sparse transfer learning via winning lottery tickets.arXiv:1905.07785,

    Pith/arXiv arXiv 1905

  13. [1997]

    MobileNets: Efficient convolutional neural networks for mobile vision applications.arXiv:1704.04861,

    Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient convolutional neural networks for mobile vision applications.arXiv:1704.04861,

    Pith/arXiv arXiv

  14. [2010]

    Lee, and Boris Murmann

    Daisuke Miyashita, Edward H. Lee, and Boris Murmann. Convolutional neural networks using logarithmic data representation.arXiv:1603.01025,

    Pith/arXiv arXiv

  15. [2015]

    BSQ: Exploring bit-level sparsity for mixed-precision neural network quantization.arXiv:2102.10462,

    186 Huanrui Yang, Lin Duan, Yiran Chen, and Hai Li. BSQ: Exploring bit-level sparsity for mixed-precision neural network quantization.arXiv:2102.10462,

    Pith/arXiv arXiv

  16. [2016]

    David Ha, Andrew Dai, and Quoc V. Le. Hypernetworks.arXiv:1609.09106,

    Pith/arXiv arXiv

  17. [2017]

    Iandola, Matthew W

    Forrest N. Iandola, Matthew W. Moskewicz, Khalid Ashraf, Song Han, William J. Dally, and Kurt Keutzer. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size.arXiv:1602.07360,

    Pith/arXiv arXiv

  18. [2018]

    On the Convergence of AdaBound and its Connection to SGD

    Pedro Savarese. On the Convergence of AdaBound and its Connection to SGD. arXiv:1908.04457,

    work page internal anchor Pith review Pith/arXiv arXiv 1908

  19. [2019]

    PACT: Parameterized clipping activation for quantized neural networks.arXiv:1805.06085,

    Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. PACT: Parameterized clipping activation for quantized neural networks.arXiv:1805.06085,

    Pith/arXiv arXiv

  20. [2020]

    SySMOL: Co-designing Algorithms and Hardware for Neural Networks with Heterogeneous Precisions.arXiv:2311.14114,

    Cyrus Zhou, Pedro Savarese, Vaughn Richard, Zack Hassman, Xin Yuan, Michael Maire, Michael DiBrino, and Yanjing Li. SySMOL: Co-designing Algorithms and Hardware for Neural Networks with Heterogeneous Precisions.arXiv:2311.14114,

    work page arXiv

  21. [2023]

    Fine-Tuning Adaptive Stochastic Optimizers: Determining the Optimal Hyperparameter $\epsilon$ via Gradient Magnitude Histogram Analysis

    184 Gustavo Silva and Paul Rodriguez. Fine-Tuning Adaptive Stochastic Optimizers: De- termining the Optimal Hyperparameter ϵ via Gradient Magnitude Histogram Analysis. arXiv:2311.11532,

    work page internal anchor Pith review Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.