Pith. sign in

REVIEW 4 major objections 4 minor 47 references

SPRKD: Effective Knowledge Distillation for Deep Neural Networks via Saddle Region Approximation

T0 review · 4 major / 4 minor · reviewed 2026-07-31 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Distilling a teacher's saddle points, not its logits, removes the teacher-driven accuracy ceiling in knowledge distillation.

desk verdict A fresh reframing and an honest write-up, but the key claim is not supported by the fully tabulated experiment, and the missing no-ASR ablation leaves the mechanism unproven. read the letter →

arxiv 2607.23346 v1 pith:TAWN2OBK submitted 2026-07-25 cs.LG

Aditya Dewan , Arjun Yogeswaran , Benjamin Fedoruk This is my paper
classification cs.LG MSC 68T07
keywords knowledgedistillationsaddlepointslosslandscapeHessianeigenvaluesweakteachermodelcompressioncurvaturetransferSPRKD
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes SPRKD, a distillation method that reframes the teacher as a source of loss-landscape curvature rather than a function to be mimicked. It argues that saddle points encountered by a weak teacher map, via the embedding principle, to promising initialization regions for a smaller student. The method aggregates low-loss teacher saddles into an Approximated Saddle Region (ASR), injects it into the student's parameter space, and then trains the student on true labels with second-order-informed descent. On malaria blood-smear classification, a 6,430-parameter student distilled from a 25,546-parameter teacher trained for only two epochs reaches 94.8% validation accuracy, matching a scratch-trained control and beating logit-matching response distillation by 24.7 percentage points. The paper reports convergence to flatter, wider minima with smaller Hessian trace and spectral radius than controls.

What carries the argument

Saddle points — points where the gradient is zero and the Hessian has both positive and negative eigenvalues — are the central object. The paper detects them during teacher training via Hessian eigenvalue spectral density (estimated by power iteration and Stochastic Lanczos quadrature), aggregates the lowest-loss saddles across a weak-teacher ensemble into an Approximated Saddle Region (ASR), re-parameterizes that region into the student's architecture through Transfer Learning by Injection (TLI), then iteratively biases student weights toward the ASR using an exponentially decaying elementwise transformation, and accelerates descent with Negative Hessian Eigensteps (updates along the most-n

What would settle it

Train the same student architecture under three conditions: an ASR built from actual teacher saddles, an ASR built from randomly selected teacher checkpoints, and pure scratch initialization. If the saddle-built ASR does not produce a significant accuracy advantage over the random-checkpoint ASR, the saddle-selection step is not the active mechanism. Independently, run SPRKD on a teacher/student pair that violates matched depth (e.g., a residual network without careful layer pairing) and check whether the student still beats scratch training.

Watch

Extended reading notes

Core claim

The central claim is that distillation can be re-characterized as curvature transfer: the teacher contributes an Approximated Saddle Region built from low-loss saddle points tracked during its optimization, and the student is iteratively moved toward that region before being trained with Negative Hessian Eigensteps and Gaussian perturbations for accelerated saddle escape. Because the wider teacher's landscape is taken to contain all critical points of the narrower student, teacher saddles are treated as plausible student minima candidates, so the student can exceed the teacher's accuracy rather than being capped by it. The flagship empirical result is the malaria task: 94.80% validation accu

Load-bearing premise

The load-bearing premise is that the loss landscape of the wider teacher contains all critical points of the narrower student, so the teacher's saddles map to promising student minima; the paper concedes this requires matched depth and narrower student width and does not automatically extend to arbitrary deep networks, and if the mapping fails for a specific pair the ASR target is uninformative and the method collapses to scratch training.

Editorial extensions

If this is right

  • Weak teachers trained for a couple epochs become sufficient for distillation, eliminating the need for a strong pretrained teacher and cutting the compute cost of the teacher phase.
  • Student accuracy is no longer bounded by teacher accuracy: students can match or exceed scratch-trained models of their own architecture even when the teacher is far weaker.
  • Distillation no longer requires simultaneous teacher inference during student training; once the ASR is built, the student trains on true labels alone, making training cost comparable to scratch training.
  • Convergence to flatter, wider minima with lower Hessian trace and spectral radius implies SPRKD students should be more robust to perturbation and label noise, a property that can be tested directly.
  • Because the mechanism depends only on the existence of saddle structure in the loss landscape, the method is proposed as general across architectures and data modalities, subject to the depth-matching embedding constraint.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves the marginal contribution of each component unablated; a natural first test is replacing the ASR with a randomly selected teacher checkpoint. If accuracy does not drop, saddle selection itself is not the active ingredient.
  • The embedding principle is stated for width increases, not depth changes; a clean extension is to test SPRKD on a matched-depth teacher/student pair without residual shortcuts to see whether the saddle-to-minimum mapping survives without explicit layer pairing.
  • Since the ASR averages saddles from multiple weak teachers, the method implicitly functions as an ensemble; one could test whether the benefit scales with the number or diversity of weak teachers beyond the single-teacher comparison reported.
  • A falsifiable cross-check: if the embedding principle fails for a given architecture pair, SPRKD should reduce to random initialization plus standard training; measuring the gap between ASR-initialized and randomly initialized students across architecture pairs would map exactly where the principle holds.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes SPRKD, a knowledge-distillation method that replaces teacher logit replication with "saddle region approximation." In Phase 1, weak teachers are trained briefly while Hessian eigenvalue monitoring records low-loss saddle points. Phase 2 averages these saddles into an Approximated Saddle Region (ASR), which is re-parameterized into the student via Transfer Learning by Injection (TLI). Phase 3 biases the student toward the ASR with an exponentially decaying Euclidean transformation, then applies Negative Hessian Eigensteps and Gaussian perturbations. The main empirical claim is that, on malaria blood-smear classification, SPRKD reaches 94.80% accuracy, beating response KD by 24.70 points and matching scratch-trained controls; the paper further claims up to 8-point gains over scratch baselines on MNIST, CIFAR-100, and TinyImageNet, and reports Hessian-ESD and loss-landscape evidence of flatter minima.

Significance. If the central claim were established, the paper would make a useful contribution: weak, cheaply trained teachers could seed students that match or exceed scratch training, and the saddle-region framing would offer a new mechanism for distillation. The paper is also candid in Section 7 about missing ablations and qualitative-only benchmarks, and it releases code. However, the current evidence does not establish the central claim. The one fully tabulated experiment shows SPRKD statistically indistinguishable from scratch training, and the absence of any no-ASR control leaves the teacher signal causally unsupported. The theoretical scaffolding is interesting but is not load-bearing without that experiment. Thus, though the idea has potential, the manuscript in its present form does not meet the evidentiary bar for a journal publication.

major comments (4)
  1. [§5.1, Table 1] The paper's headline claim — that SPRKD "removes the teacher-driven accuracy ceiling of traditional KD" and "exceeds scratch-trained baselines" — is not supported by the tabulated experiment. In Table 1, SPRKD achieves 94.80% vs. Control-S at 94.47%, with McNemar p=1.0, i.e., statistically indistinguishable. The RKD comparison (94.80% vs. 70.10%) is uninformative about the mechanism because RKD is trained to near-zero logit replication of the weak teacher (distillation loss 4.90e-6), so it is expected to plateau at the teacher's accuracy. The only meaningful comparator for the "removes the ceiling" claim is Control-S, and that comparison is a null result. The authors should either report a statistically significant improvement over scratch training or substantially reframe the contribution.
  2. [§3.3/Eq. (4) and §7] The manuscript explicitly states in Section 7: "We do not isolate the marginal contributions of ASR initialization, NHE, and PGD perturbations." This is a load-bearing omission, not a routine limitation. The results in Table 1 are equally consistent with the hypothesis that NHE + Gaussian perturbations alone (Eq. 4 and Phase 3.3) improve optimization starting from random initialization, with the teacher-derived ASR contributing nothing. Without at least the following ablations — (a) Control-S + NHE/PGD with no ASR/TLI, (b) SPRKD with a random or shuffled ASR target, and (c) SPRKD without NHE/PGD — the paper cannot attribute any observed performance to saddle-region information distilled from the teacher. The central mechanism therefore remains unverified.
  3. [§5.2 and Abstract] The abstract and introduction claim that "Across MNIST, CIFAR-100, and TinyImageNet, SPRKD exceeds scratch-trained baselines by up to 8 percentage points," but Section 5.2 states that these benchmarks are "preliminary qualitative findings" with "no dedicated results table." The reported CIFAR-100 figure of 96% validation accuracy at epoch 10, versus 88% for the control, is far outside the range typical for that dataset and is not accompanied by architecture details, hyperparameters, number of trials, or standard deviations. Claims of superiority over scratch training rest entirely on these untabulated numbers. The authors must provide a full results table with trial statistics before these claims can be evaluated.
  4. [§2.2, §3.2, Tenet 2/TLI] The transfer of teacher saddle information to the student depends on the Embedding Principle, under which critical points of a narrower network appear as saddles in a wider network. Section 3.2 concedes that TLI "requires matched computational depth and a strictly narrower student than teacher" and "does not automatically extend to arbitrary deep networks." For the Experiment 1 teacher/student pair, the manuscript does not verify that the depth-matching condition actually holds, nor does it demonstrate that the averaged teacher saddles map to informative locations in the student's landscape. If the mapping is not real for this pair, the ASR target is arbitrary and the method reduces to a particular stochastic optimizer. This concern reinforces the need for the ablations above.
minor comments (4)
  1. [§5.1] The phrase "McNemar p = 1.0" is used to claim statistical equivalence between SPRKD and Control-S. A non-significant McNemar test is not evidence of equivalence; a confidence interval for the accuracy difference or an equivalence test would be more appropriate. This matters because the paper's main claim now rests on a null result.
  2. [§5.3/Figure 4] Figure 4 reports an RKD panel subtitle of 10.86%, whereas Table 1 reports epoch-500 RKD task accuracy of 70.10%. The caption says the subtitle is validation accuracy at the ESD measurement checkpoint; the discrepancy should be explained explicitly.
  3. [§5.2] The text inconsistently calls the MNIST/CIFAR-100/TinyImageNet results "preliminary qualitative findings" while still reporting concrete percentages (96%, 99%, 88%, 97%). If the results are preliminary, they should not be cited in the abstract as quantitative evidence; if they are quantitative, they should be tabulated with full experimental details.
  4. [§3.3/Eq. (4)] Equation (4) uses the notation ∇L ⊙ v ⊙ v with an elementwise product between the gradient and the eigenvector block. The dimensions and the intended meaning of the squared eigenvector (v⊙v) should be clarified, since a negative-curvature step normally uses the eigenvector itself rather than its square.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported accuracies are measured outputs of training, not constructed from the fitted saddle-detection inputs, and the load-bearing theory is cited from non-overlapping prior work.

full rationale

SPRKD's central empirical result (Table 1: 94.80% vs. Control-S 94.47%; SPRKD vs. RKD 94.80% vs. 70.10%) is a post-training validation accuracy; it is not defined in terms of the saddle-detection thresholds (alpha≈0.4, beta=7), the ASR average, or the TLI injection. The hyperparameters are tuned on a small validation sweep, but no reported number is statistically forced by those fitted values. The paper's own Section 7 admits the mechanism is under-identified: 'We do not isolate the marginal contributions of ASR initialization, NHE, and PGD perturbations; reviewers may reasonably ask whether NHE alone explains most of the gain—this remains an open empirical question.' That is a missing ablation / attribution gap, not a definitional reduction: even if NHE+PGD alone explained all gains, the teacher-derived ASR would be inert rather than circularly defined. The TLI/Embedding-Principle apparatus is cited to non-overlapping prior work (Zhang et al. 2021; Czyzewski 2021), not to the present authors, and Section 7 explicitly constrains it ('requires matched depth and narrower student width'), so no uniqueness or ansatz is imported from the authors' own prior work. The RKD comparison is expected from RKD's own objective (matching weak-teacher logits), but that makes RKD a controlled baseline rather than a circular prediction. No equation in the paper equates the reported output to its inputs, and no fitted parameter is renamed as a prediction. The central concern — that the teacher signal may be causally inert — is a correctness/ablation risk, not circularity.

Assumptions & free parameters 6 free parameters · 5 assumptions · 2 invented entities

The paper's claim rests on five literature-sourced tenets about saddles and landscapes, plus several validation-tuned thresholds. The empirical setup never isolates whether the ASR target does work, and no new entity has independent falsifiable evidence outside the paper's own pipeline.

free parameters (6)
  • alpha (saddle strength threshold) = ≈0.4
    Appears in Eq. (1) and is 'tuned on a small validation sweep' (Section 3.1). Affects which teacher states are stored as saddles.
  • beta (negative-eigenvalue density threshold) = 7
    Appears in Eq. (1), tuned on a small validation sweep; controls how sharply negative a saddle must be to qualify.
  • j (stagnation gradient-norm threshold) = 0.02
    Hand-set in Section 3.3; triggers NHE/PGD when the gradient is small enough.
  • epsilon (ASR approach tolerance) = 0.1
    Stopping criterion in Eq. (3); used in all experiments without sensitivity analysis.
  • Gaussian perturbation standard deviation = 0.1
    Chosen in Section 3.3; affects escape behavior during Perturbed Gradient Descent.
  • eigenvalue count z and monitoring interval k = z=4 (student), 2–20 (teacher); k in [1,50]
    Settings in Section 3.1; no ablation shows how sensitive ASR quality is to these choices.
assumptions (5)
  • domain assumption Embedding Principle (Tenet 2): the loss landscape of a wider network contains all critical points of a narrower network, mapping teacher saddles to student minima candidates.
    Load-bearing for TLI cross-architecture transfer; the paper acknowledges it requires matched depth and narrower student width and does not automatically extend to ResNets (Section 7).
  • domain assumption Saddle points lie on the apex of minimum-energy paths connecting distinct minima (Tenet 3, Draxler et al.).
    Used to argue teacher saddles are useful waypoints for student re-exploration; not proven in this paper.
  • domain assumption Saddle points act as basin-fractal decision points separating minima basins (Tenet 4, Liao and Poggio).
    Used to argue that distilling saddles routes the student into promising basins; no direct verification here.
  • domain assumption SGD visits sharp saddles but leaves their negative-curvature descent untapped (Tenet 5, Jastrzebski et al.; Alain et al.).
    Motivates NHE and PGD acceleration; cited from prior work but not re-derived or tested in isolation.
  • domain assumption High-dimensional DNN loss surfaces are dominated by saddle points rather than minima (Tenet 1, Dauphin et al.).
    Sets up the whole saddle-focused framing; taken as established from spin-glass/random-matrix analyses.
invented entities (2)
  • Approximated Saddle Region (ASR)
    purpose: Aggregates lowest-loss saddles from the weak-teacher ensemble as the target region for student parameter approaching; the central object of the distillation step.
    No external falsifiable handle; it is an algorithmic construct whose benefit is not isolated by any ablation or independent prediction.
  • Negative Hessian Eigensteps (NHE)
    purpose: Accelerate saddle escape by stepping along the largest-magnitude negative Hessian eigenvector after ASR approaching.
    A named variant of existing second-order saddle-escape ideas; no standalone evidence in this paper distinguishes it from ordinary negative-curvature descent or PGD.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SPRKD: Effective Knowledge Distillation for Deep Neural Networks via Saddle Region Approximation." pith.science (2026). https://pith.science/paper/TAWN2OBK

@misc{pith2026260723346,
  author       = {Pith},
  title        = {Pith review of: SPRKD: Effective Knowledge Distillation for Deep Neural Networks via Saddle Region Approximation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TAWN2OBK}},
  note         = {Machine review of arXiv:2607.23346}
}
read the original abstract

Modern deep neural networks are potent catalysts for scientific and industrial impact, yet excessive parameter counts impede deployment in low-compute settings such as hospital equipment and energy infrastructure. Predominant knowledge distillation (KD) methods favor replication: smaller students mimic teacher output logits, yet empirically yield low task performance, hamper convergence, and act merely as regularization rather than substantive knowledge transfer. We propose Saddle Point Recruitment for Knowledge Distillation (SPRKD), reframing distillation from replication to employing teachers as optimization-curvature and domain proxies, characterizing saddle points as regions of strong further-descent potential via embedding and basin-fractal properties. Using Hessian eigenvalue spectral density (ESD), SPRKD identifies low-loss saddle regions for student re-exploration; weak-teacher ensembles are aggregated into an Approximated Saddle Region (ASR), re-parameterized into the student via Transfer Learning by Injection, and approached with exponentially decaying Euclidean transformations, Negative Hessian Eigensteps, and Gaussian perturbations. On malaria blood smear classification with a 6,430-parameter CNN distilled from a weak 25,546-parameter teacher, SPRKD reaches 94.8% validation accuracy, outperforming Response KD by 24.70 percentage points (McNemar p = 6.3e-87) and matching scratch-trained baselines of the same architecture to statistical equivalence (p = 1.0). Across MNIST, CIFAR-100, and TinyImageNet, SPRKD exceeds scratch-trained baselines by up to 8 percentage points on preliminary benchmarks. Hessian ESD and 2-D loss landscape analysis show convergence to wider minima with substantially smaller Hessian trace and spectral radius than Response KD and control students, indicating smoother descent and greater noise robustness.

Figures

Figures reproduced from arXiv: 2607.23346 by the authors.

Figure 1
Figure 1. SPRKD high-level algorithm overview and distillation methodology. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Experiment 1 distillation setup. A 25,546- [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Validation loss (left) and accuracy (right) at 175 logged checkpoints over 500 training epochs for SPRKD, Control-S, [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Hessian Eigenvalue Spectral Density (ESD) plots across SPRKD, Control-S, and RKD students upon convergence [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Loss landscapes at convergence (epoch 500), perturbed across the top two Hessian eigenvectors. Each column corre [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

47 extracted references · 1 linked inside Pith

  1. [1]

    L.; and Manzagol, P.-A

    Alain, G.; Roux, N. L.; and Manzagol, P.-A. 2019. Negative Eigenvalues of the H essian in Deep Neural Networks. In International Conference on Learning Representations Workshop

    2019

  2. [2]

    Alkhulaifi, A.; Alsahli, F.; and Ahmad, I. 2021. Knowledge Distillation in Deep Learning and Its Applications. PeerJ Computer Science, 7: e474

    2021

  3. [3]

    Allen-Zhu, Z.; and Li, Y. 2020. Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning. ArXiv:2012.09816

    arXiv 2020

  4. [4]

    Asif, U.; Tang, J.; and Harrer, S. 2020. Ensemble Knowledge Distillation for Learning Improved and Efficient Networks. In 24th European Conference on Artificial Intelligence (ECAI)

    2020

  5. [5]

    Chaudhari, P.; Choromanska, A.; Soatto, S.; LeCun, Y.; Baldassi, C.; Borgs, C.; Chayes, J.; Sagun, L.; and Zecchina, R. 2019. Entropy- SGD : Biasing Gradient Descent into Wide Valleys. Journal of Statistical Mechanics: Theory and Experiment

    2019

  6. [6]

    H.; and Hariharan, B

    Cho, J. H.; and Hariharan, B. 2019. On the Efficacy of Knowledge Distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 4794--4802

    2019

  7. [7]

    B.; and LeCun, Y

    Choromanska, A.; Henaff, M.; Mathieu, M.; Arous, G. B.; and LeCun, Y. 2015. The Loss Surfaces of Multilayer Networks. In Proceedings of the 18th International Conference on Artificial Intelligence and Statistics (AISTATS)

    2015

  8. [8]

    F.; and Verma, A

    Coleman, T. F.; and Verma, A. 1998. Structure and Efficient H essian Calculation. In Advances in Nonlinear Programming, 57--72. Springer

    1998

Show all 47 references
  1. [9]

    Czyzewski, A. M. 2021. Transfer Learning Between Different Architectures via Weights Injection. Pre-print

    2021

  2. [10]

    N.; Pascanu, R.; Gulcehre, C.; Cho, K.; Ganguli, S.; and Bengio, Y

    Dauphin, Y. N.; Pascanu, R.; Gulcehre, C.; Cho, K.; Ganguli, S.; and Bengio, Y. 2014. Identifying and Attacking the Saddle Point Problem in High-Dimensional Non-Convex Optimization. Advances in Neural Information Processing Systems, 27

    2014

  3. [11]

    Domingos, P. 2012. A Few Useful Things to Know About Machine Learning. Communications of the ACM, 55(10): 78--87

    2012

  4. [12]

    Draxler, F.; Veschgini, K.; Salmhofer, M.; and Hamprecht, F. A. 2018. Essentially No Barriers in Neural Network Energy Landscape. In Proceedings of the 35th International Conference on Machine Learning (ICML)

    2018

  5. [13]

    Ericsson . 2022. Ericsson Mobility Report. https://www.ericsson.com/en/reports-and-papers/mobility-report

    2022

  6. [14]

    Fuhad, K. M. F.; Tuba, J. F.; Sarker, M. R. A.; Momen, S.; Mohammed, N.; and Rahman, T. 2020. Deep Learning Based Automatic Malaria Parasite Detection from Blood Smear and Its Smartphone Based Application. Diagnostics, 10(5)

    2020

  7. [15]

    Furlanello, T.; Lipton, Z.; Tschannen, M.; Itti, L.; and Anandkumar, A. 2018. Born Again Neural Networks. In International Conference on Machine Learning (ICML)

    2018

  8. [16]

    Garipov, T.; Izmailov, P.; Podoprikhin, D.; Vetrov, D.; and Wilson, A. G. 2018. Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNN s. In Advances in Neural Information Processing Systems (NeurIPS)

    2018

  9. [17]

    Ghorbani, B.; Krishnan, S.; and Xiao, Y. 2019. An Investigation into Neural Net Optimization via H essian Eigenvalue Density. In Proceedings of the 36th International Conference on Machine Learning (ICML)

    2019

  10. [18]

    W.; and Gonzalez, J

    Golmant, N.; Yao, Z.; Gholami, A.; Mahoney, M. W.; and Gonzalez, J. 2018. pytorch-hessian-eigenthings: Efficient PyTorch H essian Eigendecomposition. https://github.com/noahgolmant/pytorch-hessian-eigenthings

    2018

  11. [19]

    J.; Vinyals, O.; and Saxe, A

    Goodfellow, I. J.; Vinyals, O.; and Saxe, A. M. 2015. Qualitatively Characterizing Neural Network Optimization Problems. In International Conference on Learning Representations (ICLR)

    2015

  12. [20]

    J.; and Tao, D

    Gou, J.; Yu, B.; Maybank, S. J.; and Tao, D. 2021. Knowledge Distillation: A Survey. International Journal of Computer Vision, 129(6): 1789--1819

    2021

  13. [21]

    Hardt, M.; Recht, B.; and Singer, Y. 2016. Train Faster, Generalize Better: Stability of Stochastic Gradient Descent. In International Conference on Machine Learning (ICML)

    2016

  14. [22]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2016

  15. [23]

    Hinton, G.; Vinyals, O.; and Dean, J. 2015. Distilling the Knowledge in a Neural Network. In NIPS Deep Learning and Representation Learning Workshop

    2015

  16. [24]

    Horoi, S.; Huang, J.; Rieck, B.; Lajoie, G.; Wolf, G.; and Krishnaswamy, S. 2022. Exploring the Geometry and Topology of Neural Network Loss Landscapes. In International Symposium on Intelligent Data Analysis

    2022

  17. [25]

    Hutchinson, M. F. 1990. A Stochastic Estimator of the Trace of the Influence Matrix for Laplacian Smoothing Splines. Communications in Statistics -- Simulation and Computation, 19(2): 433--450

    1990

  18. [26]

    Jastrzebski, S.; Kenton, Z.; Ballas, N.; Fischer, A.; Bengio, Y.; and Storkey, A. 2019. On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length. In International Conference on Learning Representations (ICLR)

    2019

  19. [27]

    M.; and Jordan, M

    Jin, C.; Ge, R.; Netrapalli, P.; Kakade, S. M.; and Jordan, M. I. 2017. How to Escape Saddle Points Efficiently. In International Conference on Machine Learning (ICML)

    2017

  20. [28]

    S.; Mudigere, D.; Nocedal, J.; Smelyanskiy, M.; and Tang, P

    Keskar, N. S.; Mudigere, D.; Nocedal, J.; Smelyanskiy, M.; and Tang, P. T. P. 2017. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. In International Conference on Learning Representations (ICLR)

    2017

  21. [29]

    P.; and Ba, J

    Kingma, D. P.; and Ba, J. 2015. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR)

    2015

  22. [30]

    Krizhevsky, A. 2009. Learning Multiple Layers of Features from Tiny Images. Technical report, University of Toronto

    2009

  23. [31]

    Le, Y.; and Yang, X. 2015. Tiny I mage N et Visual Recognition Challenge. CS231N, Stanford University

    2015

  24. [32]

    LeCun, Y.; Bottou, L.; Bengio, Y.; and Haffner, P. 1998. Gradient-Based Learning Applied to Document Recognition. Proceedings of the IEEE, 86(11): 2278--2324

    1998

  25. [33]

    Li, H.; Xu, Z.; Taylor, G.; Studer, C.; and Goldstein, T. 2018. Visualizing the Loss Landscape of Neural Nets. In Advances in Neural Information Processing Systems (NeurIPS)

    2018

  26. [34]

    Liao, Q.; and Poggio, T. 2017. Theory II : Landscape of the Empirical Risk in Deep Learning. Technical report, Center for Brains, Minds and Machines

    2017

  27. [35]

    McNemar, Q. 1947. Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages. Psychometrika, 12(2): 153--157

    1947

  28. [36]

    Park, W.; Kim, D.; Lu, Y.; and Cho, M. 2019. Relational Knowledge Distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 3967--3976

    2019

  29. [37]

    Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. 2019. PyTorch : An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems (NeurIPS)

    2019

  30. [38]

    Pearlmutter, B. A. 1994. Fast Exact Multiplication by the H essian. Neural Computation, 6(1): 147--160

    1994

  31. [39]

    C.; and Fei-Fei, L

    Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; Berg, A. C.; and Fei-Fei, L. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3): 211--252

    2015

  32. [40]

    A.; and Wilson, A

    Stanton, S.; Izmailov, P.; Kirichenko, P.; Alemi, A. A.; and Wilson, A. G. 2021. Does Knowledge Distillation Really Work? Advances in Neural Information Processing Systems, 34: 6906--6919

    2021

  33. [41]

    Strubell, E.; Ganesh, A.; and McCallum, A. 2019. Energy and Policy Considerations for Deep Learning in NLP . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL)

    2019

  34. [42]

    Tung, F.; and Mori, G. 2019. Similarity-Preserving Knowledge Distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 1365--1374

    2019

  35. [43]

    Ubaru, S.; Chen, J.; and Saad, Y. 2017. Fast Estimation of tr(f(A)) via Stochastic Lanczos Quadrature. SIAM Journal on Matrix Analysis and Applications, 38(4): 1075--1099

    2017

  36. [44]

    National Library of Medicine

    U.S. National Library of Medicine . 2018. National Library of Medicine: Malaria Datasets. https://lhncbc.nlm.nih.gov/LHC-research/LHC-projects/image-processing/malaria-datasheet.html

    2018

  37. [45]

    Yao, Z.; Gholami, A.; Keutzer, K.; and Mahoney, M. W. 2020. PyHessian : Neural Networks Through the Lens of the H essian. In 2020 IEEE International Conference on Big Data (Big Data), 581--590

    2020

  38. [46]

    E.; Li, G.; Wang, T.; and Feng, J

    Yuan, L.; Tay, F. E.; Li, G.; Wang, T.; and Feng, J. 2020. Revisiting Knowledge Distillation via Label Smoothing Regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2020

  39. [47]

    Zhang, Y.; Zhang, Z.; Luo, T.; and Xu, Z.-Q. J. 2021. Embedding Principle of Loss Landscape of Deep Neural Networks. In Advances in Neural Information Processing Systems (NeurIPS)

    2021

Pith tools

Reviewed July 31, 2026 · model on record in the stance chip above.