Pith. sign in

REVIEW 4 major objections 5 minor 54 references

A parameter-free rescaling of ReLU networks at initialization can accelerate training by aligning the path kernel with the identity.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 21:30 UTC pith:RGEAIVPQ

load-bearing objection The algorithm as written applies the inverse square of the rescaling that actually minimizes its stated logdet divergence, so the theory does not support the reported experiments. the 4 major comments →

arxiv 2602.19799 v2 pith:RGEAIVPQ submitted 2026-02-23 stat.ML cs.LGmath.OC

Path-conditioned training: a principled way to rescale ReLU neural networks

Arthur Lebeurrier , Titouan Vayer , R\'emi Gribonval This is my paper
classification stat.ML cs.LGmath.OC
keywords path-liftingReLU networksrescaling symmetrypath kernelpreconditioningtraining dynamicsinitializationBregman divergence
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the well-known rescaling symmetries of ReLU networks—scaling a neuron's incoming weights and inversely scaling outgoing weights leaves the function unchanged—can be exploited in a principled, parameter-free way. It introduces PathCond, which rescales the weights once at initialization so that the 'path kernel' governing how gradient descent moves in the lifted path space is as close as possible to the identity. Because that kernel acts as a preconditioner on the loss gradient, aligning it to the identity makes the lifted-space trajectory approximate idealized gradient descent on the loss. The paper shows this reaches the same CIFAR-10 accuracy up to 1.5 times faster in epochs, without degrading (sometimes improving) test accuracy, and predicts which architectures benefit most: unbalanced widths or small-variance initialization.

Core claim

On the paper's own terms, the central discovery is that one can compute an optimal rescaling for any ReLU network parameter vector without ever forming the path kernel, by minimizing a logdet matrix divergence between a scaled version of the Gram matrix and the identity using only its diagonal. That diagonal is obtained in a single backward pass, and the resulting rescaling is parameter-free with cost linear in the number of parameters per iteration. The authors demonstrate that this initialization-only rescaling accelerates training-loss convergence on fully connected CIFAR-10 networks, on a fully convolutional CIFAR-10 architecture, and on MNIST autoencoders, with the strongest gains in th

What carries the argument

The path-lifting map Φ sends a parameter vector to the products of weights along every input-to-output path; the path kernel Pθ = ∂Φ(θ)∂Φ(θ)⊤ is the metric tensor describing how parameter-space gradient flow propagates into the lifted space. Since rescaling changes Pθ while leaving Φ(θ) and the implemented function unchanged, choosing a rescaling is equivalent to choosing a preconditioner. PathCond minimizes the logdet Bregman divergence between α·∂Φ(θ)⊤∂Φ(θ) and the identity, via alternating minimization with closed-form per-neuron updates, requiring only the diagonal of the Gram matrix computed by one backward pass.

Load-bearing premise

The method rises or falls on the assumption, stated in Section 4.2, that aligning the path kernel with the identity is a good proxy for faster convergence of the true loss at finite learning rates; Appendix K concedes that at larger learning rates this proxy breaks down.

What would settle it

Train a multi-layer ReLU network in the favorable regime (varying layer widths or small-variance initialization) at a small learning rate, apply PathCond only at initialization, and measure epochs to a fixed training loss. If it fails to beat the baseline, or if it changes training dynamics on a network whose diagonal diag(G) is provably constant (where the theory says the rescaling should be the identity), the central claim would be falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • At small to moderate learning rates, a single initialization-time rescaling reaches baseline accuracy in up to 1.5 times fewer epochs, with no hyperparameter to tune.
  • The method does not degrade training in the tested settings and can improve test accuracy (80% vs 77% on the convolutional CIFAR-10 architecture).
  • The diagonal-of-G criterion explains when rescaling helps: near constant-width networks with variance-balanced initialization have nearly constant diagonal and need no rescaling, while variable widths or small-variance initialization produce spread that PathCond corrects.
  • The rescaling is computationally cheap, O(niter·p) time and O(p+H) memory with typical niter under 10, making it practical as a one-shot initialization step.
  • Aligning the path kernel with the identity approximates natural-gradient-like dynamics in lifted space without pseudo-inverting a large matrix.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same alignment criterion could be applied periodically during training, not only at initialization; the paper's own idealized algorithm suggests this direction, but the experiments mostly test the one-shot version.
  • Since the path kernel factorizes the neural tangent kernel, PathCond predicts that training speedups concentrate in finite-width networks where the kernel is far from isotropic, with weaker effects in very wide or lazy regimes—a testable extension.
  • The diagonal-only computation is architecture-agnostic for any network with a defined path lifting and rescaling group, so extending PathCond to residual or attention-based networks is natural, though untested here.
  • The paper's own Appendix K notes that larger learning rates erode the benefit; this bounds the practical claim to the small-to-moderate learning rate regime and leaves iterative or adaptive-optimizer variants as open questions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces PathCond, a method that rescales the parameters of a ReLU network at initialization (or during training) to improve training dynamics. The rescaling is chosen to minimize a logdet Bregman divergence between the scaled path kernel and the identity, using the path-lifting framework. The authors reformulate the criterion so that only the diagonal of the Gram matrix G = ∂Φ(θ)ᵀ∂Φ(θ) is needed, derive a coordinate-descent algorithm with closed-form updates, and analyze initialization regimes where the rescaling should be most effective. Experiments on CIFAR-10 MLPs, a CIFAR-10 convolutional network, and MNIST autoencoders report faster training-loss convergence and, in some cases, better test accuracy relative to baseline and Equinormalization. The paper also provides extensive appendices with proofs, complexity analysis, and additional experiments.

Significance. If the claims hold, PathCond would be a valuable parameter-free way to exploit ReLU rescaling symmetry, with potential practical impact on training speed. The paper has genuine strengths: the path-lifting/logdet formulation is elegant, the reduction to a p-dimensional diagonal computation is nontrivial, the coordinate-descent derivation is detailed, and the paper ships code and extensive appendices. The favorable-regime analysis (Prop. 5.1) yields a falsifiable prediction that is tested on MNIST autoencoders. However, the central theoretical connection between the criterion and the implemented algorithm is broken by a sign/exponent mismatch: the algorithm applies D' = diag(e^{Bu}) where the derivation says the applied rescaling should be (D')^{-1/2}. This calls into question whether the reported speedups are produced by the proposed principle. The empirical claims also lack error bars and are overstated relative to the authors' own Appendix K. These issues are load-bearing, so the paper needs major revision.

major comments (4)
  1. [§4.3, Prop. F.1, Algorithms 2 and 3] The implemented rescaling is not the one that minimizes the stated logdet criterion. Prop. F.1 shows that minimizing d_{ζ+}(α P_{Dθ} || I_q) over α>0 and D∈D is equivalent to solving (57) for D', and that the applied rescaling must be D = (D')^{-1/2}. By Lemma F.3, D' = diag(e^{Bu}), so the optimal applied rescaling is diag(e^{-Bu/2}). However, Algorithm 2 outputs θ' = diag(e^{Bu})θ and Algorithm 3 (line 20) returns θ~ = diag(e^v)θ with v=Bu; §4.3 also states that 'D=e^{Bu} describes the overall rescaling.' Thus the published procedure applies D' rather than (D')^{-1/2}. This is not a minor sign issue: the algorithm does not align the path kernel with the identity, and the theoretical motivation (preconditioning to mimic lifted-space gradient flow) does not apply to the published procedure. Since all experiments use this procedure, the empirical speedups are not explained by the paper's
  2. [§1 and §5.1 vs. Appendix K] The paper claims that PathCond 'never degrades' training dynamics (Contributions, §1) and that it 'matches or exceeds baseline' across learning rates (§5.1). This is contradicted by Appendix K, which states that 'performance degrades at larger learning rates' and shows in Figure 6 that at lr=0.1 PathCond is worse than baseline. The claim must be qualified to the small/moderate learning-rate regime where the theoretical assumptions are stated to hold, and the high-learning-rate results should either be shown in the main text or explicitly excluded from the claim. As written, the 'never degrades' assertion is an overclaim.
  3. [§5.1–§5.2] The central quantitative claims — up to 1.5× fewer epochs to reach 99% training accuracy, and 80% vs 77% test accuracy — are based on only 3 runs with no error bars, confidence intervals, or per-run plots (Figures 2, 3, and 5). For a method whose main selling point is a consistent speedup, this is insufficient statistical support. Please provide error bars or at least min/max ranges, state the number of seeds for the CIFAR-NV experiment in §5.2 (which is not specified), and report the variance across runs for the epochs-to-target metric.
  4. [§4.2, §5.3] The method's justification rests on the assumption that aligning the path kernel with the identity improves the actual training dynamics. The paper acknowledges this is an assumption, and Appendix K shows it breaks down at larger learning rates. However, no direct diagnostic is provided that the rescaling actually reduces the logdet divergence (or the conditioning) of G_{Dθ} on the trained architectures, nor that the reduction correlates with faster convergence. Please add such a diagnostic for the corrected algorithm, e.g., report d_{logdet}(αG_{Dθ}||I_p) before and after PathCond on the CIFAR and autoencoder setups. Without this, the 'principled' claim is not empirically anchored even after the sign bug is fixed.
minor comments (5)
  1. [§4.3, Appendix D] The text refers to 'Theorem D.1' and 'Theorem D.3' where the appendix defines Definition D.1 and Lemma D.3. Please correct the cross-references.
  2. [Algorithm 2] The output line uses D for two different objects: the applied rescaling in (8) and the matrix diag(e^{Bu}) that Prop. F.1 calls D'. This overloading is confusing and likely contributed to the sign mismatch. Use a distinct symbol (e.g., D' or S) for the intermediate matrix.
  3. [§5.3 / Algorithm 3] The claim that PathCond is 'parameter-free' should be qualified: Algorithm 3 has a stopping tolerance ε and a maximum iteration count n_iter. Even if these are not tuned, their default values should be stated, and the claim should say 'no data-dependent hyperparameters' if that is the intent.
  4. [§5.2] Please clarify how BatchNorm parameters are treated by PathCond. The ENorm setup states that BatchNorm parameters are excluded 'consistently with how PathCond handles BatchNorm,' but the main text does not describe PathCond's handling of normalization layers.
  5. [Figure 5] The bottom panel reports ‖log(rescaling)‖∞, but it is not clear whether this is the log-rescaling of the applied matrix D or of the intermediate D'. After the sign bug is fixed, this distinction is important.

Circularity Check

0 steps flagged

No significant circularity: the logdet rescaling criterion is derived from first principles and benchmarked externally; the apparent D vs D^{-1/2} mismatch is a correctness bug, not a circular reduction.

full rationale

The paper's central criterion (8)-(9) is derived from a Bregman logdet divergence between the path kernel and the identity, with the optimization solved in closed form in Lemma F.7. There are no fitted constants and no data-dependent parameters in the criterion itself; the algorithm's output is the minimizer of a well-defined convex program. The favorable-regime analysis (Prop. 5.1, Section J) derives expected diagonal entries from initialization statistics and is validated on synthetic and MNIST experiments, rather than being defined in terms of the speedups it is supposed to explain. The self-citations to path-lifting (Stock & Gribonval 2023; Gonon et al. 2023; Marcotte et al. 2025) are load-bearing as tools, but the framework is also attributed to external prior work (Neyshabur et al. 2015; Bona-Pellissier et al. 2022), so the argument does not reduce to an in-house uniqueness theorem. The assumption that gradient descent in lifted space is a suitable idealized trajectory is a stated heuristic (Section 4.2), not a circular definition. One serious issue noted in the review is an apparent inconsistency: Prop. F.1 says the optimal applied rescaling is D = D'^{-1/2} where D' solves (57), while Algorithm 2/3 returns diag(e^{Bu}) = D'. If correct, this means the published algorithm does not implement the criterion's minimizer, so the theoretical justification for the empirical gains would fail. That is a correctness and reproducibility problem, not a circularity: the algorithm's output is not the claimed first-principles result, so the claim does not reduce to its inputs by construction. Under hard rule 1, circularity requires exhibiting a specific reduction (Eq. X = Eq. Y by construction, or fitted parameter renamed as prediction); none is present. The experimental speedups are measured against external CIFAR-10/MNIST benchmarks and are not constructed from the criterion's fit values. I therefore assign a low score reflecting only the mild self-citation reliance, not the sign issue.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 0 invented entities

The central method is derived from standard path-lifting and Bregman geometry and introduces no new physical or mathematical entity. It relies on one explicit modeling assumption (lifted-space GD is a suitable ideal), standard initialization assumptions, and the small-learning-rate gradient-flow approximation. The empirical speedup is not derived from these assumptions alone.

free parameters (1)
  • Algorithm stopping tolerance ε and maximum iterations n_iter = not specified in the paper; described as n_iter ≪ p with convergence in under 10 iterations
    These are user-chosen implementation hyperparameters for Algorithm 3. They are not fitted to data, but their values are not reported for the experiments, which slightly undercuts the 'parameter-free' label.
axioms (5)
  • domain assumption Path-lifting factorization properties: Φ(Dθ)=Φ(θ) and L(θ)=ℓ(Φ(θ)) locally (Eqs. 5-6).
    Taken from the path-lifting literature (Gonon et al.; Stock & Gribonval); used to define the criterion and the lifted-space ODE.
  • domain assumption Small-learning-rate SGD is well approximated by gradient flow (Eq. 1).
    Used to derive the lifted-space flow ẑ=-Pθ∇ℓ(z). Appendix K concedes the approximation breaks down at larger learning rates.
  • ad hoc to paper Gradient descent in lifted space is a suitable ideal trajectory, and aligning Pθ with the identity improves the actual preconditioned flow.
    Assumed in Section 4.2. The paper provides no theorem proving that identity-alignment of the path kernel accelerates the training loss; this is the main conceptual premise.
  • domain assumption At initialization, parameters are independent with layer-wise variance σ_k², and standard initializations satisfy n_k σ_k² = a constant (Prop 5.1 / Section J.3).
    Used for the expected-diagonal formulas and for identifying favorable regimes. ReLU activation patterns are ignored in the expected path-magnitude computation.
  • domain assumption The number of paths q is at least the number of parameters p, and in practice diag(G) is positive.
    Used in Appendix F for the logdet reformulation. The paper notes G is generally only positive semi-definite and proceeds with the same formula.

pith-pipeline@v1.3.0-alltime-deepseek · 29971 in / 13544 out tokens · 126281 ms · 2026-08-02T21:30:57.332904+00:00 · methodology

0 comments
read the original abstract

Despite recent algorithmic advances, we still lack principled ways to leverage the well-documented rescaling symmetries in ReLU neural network parameters. While two properly rescaled weights implement the same function, the training dynamics can be dramatically different. To offer a fresh perspective on exploiting this phenomenon, we build on the recent path-lifting framework, which provides a compact factorization of ReLU networks. We introduce a geometrically motivated criterion to rescale neural network parameters which minimization leads to a conditioning strategy that aligns a kernel in the path-lifting space with a chosen reference. We derive an efficient algorithm to perform this alignment. In the context of random network initialization, we analyze how the architecture and the initialization scale jointly impact the output of the proposed method. Numerical experiments illustrate its potential to speed up training.

Figures

Figures reproduced from arXiv: 2602.19799 by Arthur Lebeurrier, R\'emi Gribonval, Titouan Vayer.

Figure 1
Figure 1. Figure 1: GD for a toy model fθ=(u,v,w)(x) = u ReLU(vx+w) on a loss L(θ) that can be factorized as L(θ) = ℓ(Φ(θ)) (see Section 2). (Left) Loss L(θ) during GD iterations for three different initializations θ0 (three colors). Dashed lines correspond to GD starting at θ0, bold lines to GD starting at rescaled θ (λ) 0 ∼ θ0 using PathCond ; (Middle) Trajectories in lifted space Φ(θ) = (uv, uw) ⊤. Dotted lines are traject… view at source ↗
Figure 2
Figure 2. Figure 2: PathCond performance comparison across network depths on CIFAR-10 with multilayer perceptrons (Left) Number of epochs required to reach 99% training accuracy for networks with 2 to 8 hidden layers (abscissa = number of parameters, which increases with depth). (Middle) Training accuracy curves for the 3-hidden-layer network. (Right) Corresponding training loss curves 0 50 100 Epochs 10 −3 10 −2 10 −1 10 0 T… view at source ↗
Figure 3
Figure 3. Figure 3: Training dynamics on CIFAR-10 with fully convolutional architecture (CIFAR-NV). (Left) Training loss, (Middle) training accuracy, and (Right) test accuracy. Results [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Analysis of the relationship between architectural bal￾ance (controlled by the maximum width ratio max ni nj ) and log￾rescaling magnitude for small and large variance regimes. 0.20 0.25 0.30 0.35 0.40 Final Training Loss Baseline Pathcond 1.0 2.0 4.0 7.0 Compression Factor 0 2 ‖log(rescaling)‖∞ [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Effect of compression on MNIST autoencoder train￾ing. (Top) Final training loss for different compression factors. (Bottom) Maximum absolute value of the log rescaling at initial￾ization. Proposition 5.1 (Expected diagonal under standard initial￾ization). For any edge parameter i at layer k ∈ {0, . . . , L− 1}, the expected ith coefficient of diag(G) is given by E[Gii] = nL nk+1 a L−1 + nL nk+1 k X−1 j=0 a… view at source ↗
Figure 6
Figure 6. Figure 6: Convergence speed across different learning rates and network sizes. L. Additional Experiments for Section 5.2 We provide complementary experiments for Section 5.2, exploring the impact of learning rate on the generalization benefits of PathCond across a wider range of values [PITH_FULL_IMAGE:figures/full_fig_p029_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: presents training and test dynamics for learning rates ranging from 10−2 to 10−4 . Consistent with our theoretical predictions, PathCond demonstrates improved convergence and generalization performance at small to moderate learning rates, where proper initialization plays a critical role in training stability. For the smallest learning rate (lr = 10−4 ), we extend training to 500 epochs to observe converge… view at source ↗
Figure 8
Figure 8. Figure 8: Final training loss (top) and rescaling magnitude (bottom) across compression factors for different learning rates. N. ENorm : Experimental Setup We describe the ENorm hyperparameter choices used in our comparisons. ENorm (Stock & Gribonval, 2023) rescales network weights at regular intervals to minimize a weighted sum of per-layer norms, X ℓ cℓ∥Wℓ∥ p p where the depth-dependent coefficient cℓ affects each… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 8 linked inside Pith

  1. [1]

    Natural gradient works efficiently in learning

    Amari, S.-I. Natural gradient works efficiently in learning. Neural computation, 10 0 (2): 0 251--276, 1998

    1998

  2. [2]

    Neural teleportation

    Armenta, M., Judge, T., Painchaud, N., Skandarani, Y., Lemaire, C., Gibeau Sanchez, G., Spino, P., and Jodoin, P.-M. Neural teleportation. Mathematics, 11 0 (2): 0 480, 2023

    2023

  3. [3]

    Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks

    Arora, S., Du, S., Hu, W., Li, Z., and Wang, R. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International conference on Machine Learning (ICML), pp.\ 322--332. PMLR, 2019

    2019

  4. [4]

    S., Woodworth, B

    Azulay, S., Moroshko, E., Nacson, M. S., Woodworth, B. E., Srebro, N., Globerson, A., and Soudry, D. On the implicit bias of initialization shape: Beyond infinitesimal mirror descent. In International Conference on Machine Learning (ICML), volume 139. PMLR, 2021

    2021

  5. [5]

    Symmetry-invariant optimization in deep networks

    Badrinarayanan, V., Mishra, B., and Cipolla, R. Symmetry-invariant optimization in deep networks. arXiv preprint arXiv:1511.01754, 2015

    Pith/arXiv arXiv 2015

  6. [6]

    Bock, A. A. and Andersen, M. S. Connecting kaporin's condition number and the bregman log determinant divergence, 2025

    2025

  7. [7]

    Local identifiability of deep relu neural networks: the theory

    Bona-Pellissier, J., Malgouyres, F., and Bachoc, F. Local identifiability of deep relu neural networks: the theory. volume 35, 2022

    2022

  8. [8]

    A brief introduction to the neural tangent kernel

    Bowman, B. A brief introduction to the neural tangent kernel. 2023

    2023

  9. [9]

    On lazy training in differentiable programming

    Chizat, L., Oyallon, E., and Bach, F. On lazy training in differentiable programming. volume 32, 2019

    2019

  10. [10]

    H., Leiserson, C

    Cormen, T. H., Leiserson, C. E., Rivest, R. L., and Stein, C. Introduction to algorithms. MIT press, 2022

    2022

  11. [11]

    Natural neural networks

    Desjardins, G., Simonyan, K., Pascanu, R., et al. Natural neural networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 28, 2015

    2015

  12. [12]

    Dhillon, I. S. The log-determinant divergence and its applications. In Householder Symposium XVII, Zeuthen, Germany, 2008

    2008

  13. [13]

    Sharp minima can generalize for deep nets

    Dinh, L., Pascanu, R., Bengio, S., and Bengio, Y. Sharp minima can generalize for deep nets. In International Conference on Machine Learning (ICML). PMLR, 2017

    2017

  14. [14]

    Domin \'e , C. C. J., Anguita, N., Proca, A. M., Braun, L., Kunin, D., Mediano, P. A. M., and Saxe, A. M. From lazy to rich: Exact learning dynamics in deep linear networks. In International Conference on Learning Representations (ICLR), 2025

    2025

  15. [15]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021

    2021

  16. [16]

    S., Hu, W., and Lee, J

    Du, S. S., Hu, W., and Lee, J. D. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, 2018

    2018

  17. [17]

    and Pavel, L

    Gao, B. and Pavel, L. On the properties of the softmax function with application in game theory and reinforcement learning. arXiv preprint arXiv:1704.00805, 2017

    Pith/arXiv arXiv 2017

  18. [18]

    A unified paths perspective for pruning at initialization

    Gebhart, T., Saxena, U., and Schrater, P. A unified paths perspective for pruning at initialization. arXiv preprint arXiv:2101.10552, 2021

    Pith/arXiv arXiv 2021

  19. [19]

    and Ginsburg, B

    Gitman, I. and Ginsburg, B. Comparison of batch normalization and weight normalization algorithms for the large-scale image classification. arXiv preprint arXiv:1709.08145, 2017

    Pith/arXiv arXiv 2017

  20. [20]

    and Bengio, Y

    Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp.\ 249--256. JMLR Workshop and Conference Proceedings, 2010

    2010

  21. [21]

    Harnessing symmetries for modern deep learning challenges: a path-lifting perspective

    Gonon, A. Harnessing symmetries for modern deep learning challenges: a path-lifting perspective. PhD thesis, Ecole normale sup \'e rieure de lyon-ENS LYON, 2024

    2024

  22. [22]

    A path-norm toolkit for modern networks: consequences, promises and challenges

    Gonon, A., Brisebarre, N., Riccietti, E., and Gribonval, R. A path-norm toolkit for modern networks: consequences, promises and challenges. arXiv preprint arXiv:2310.01225, 2023

    Pith/arXiv arXiv 2023

  23. [23]

    Characterizing implicit bias in terms of optimization geometry

    Gunasekar, S., Lee, J., Soudry, D., and Srebro, N. Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning (ICML). PMLR, 2018

    2018

  24. [24]

    Harris, C. R. et al. Array programming with NumPy . Nature, 585: 0 357--362, 2020

    2020

  25. [25]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

    He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp.\ 1026--1034, 2015

    2015

  26. [26]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  27. [27]

    G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H

    Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017

    Pith/arXiv arXiv 2017

  28. [28]

    Neural tangent kernel: Convergence and generalization in neural networks

    Jacot, A., Gabriel, F., and Hongler, C. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, 2018

    2018

  29. [29]

    Estimation with quadratic loss

    James, W., Stein, C., et al. Estimation with quadratic loss. In Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, volume 1, pp.\ 361--379. University of California Press, 1961

    1961

  30. [30]

    Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 25, 2012

    2012

  31. [31]

    A., and Dhillon, I

    Kulis, B., Sustik, M. A., and Dhillon, I. S. Low-rank kernel learning with bregman matrix divergences. Journal of Machine Learning Research (JMLR), 10 0 (2), 2009

    2009

  32. [32]

    L., and Tanaka, H

    Kunin, D., Sagastuy-Brena, J., Ganguli, S., Yamins, D. L., and Tanaka, H. Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics. In International Conference on Learning Representations (ICLR), 2021

    2021

  33. [33]

    Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning

    Kunin, D., Ravent \'o s, A., Domin \'e , C., Chen, F., Klindt, D., Saxe, A., and Ganguli, S. Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 2024

    2024

  34. [34]

    Abide by the law and follow the flow: Conservation laws for gradient flows

    Marcotte, S., Gribonval, R., and Peyr \'e , G. Abide by the law and follow the flow: Conservation laws for gradient flows. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, 2023

    2023

  35. [35]

    Intrinsic training dynamics of deep neural networks

    Marcotte, S., Peyr \'e , G., and Gribonval, R. Intrinsic training dynamics of deep neural networks. arXiv preprint arXiv:2508.07370, 2025

    arXiv 2025

  36. [36]

    New insights and perspectives on the natural gradient method

    Martens, J. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research (JMLR), 21 0 (146): 0 1--76, 2020

    2020

  37. [37]

    G- SGD : Optimizing re LU neural networks in its positively scale-invariant space

    Meng, Q., Zheng, S., Zhang, H., Chen, W., Ma, Z.-M., and Liu, T.-Y. G- SGD : Optimizing re LU neural networks in its positively scale-invariant space. In International Conference on Learning Representations (ICLR), 2019

    2019

  38. [38]

    and Burkholz, R

    Mustafa, N. and Burkholz, R. Dynamic rescaling for training GNN s. In Advances in Neural Information Processing Systems (NeurIPS), 2024

    2024

  39. [39]

    R., and Srebro, N

    Neyshabur, B., Salakhutdinov, R. R., and Srebro, N. Path-sgd: Path-normalized optimization in deep neural networks. volume 28, 2015

    2015

  40. [40]

    Pytorch: An imperative style, high-performance deep learning library

    Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32. 2019

    2019

  41. [41]

    Patil, S. M. and Dovrolis, C. Phew: Constructing sparse networks that learn fast and generalize well without training data. In International Conference on Machine Learning (ICML), pp.\ 8432--8442. PMLR, 2021

    2021

  42. [42]

    and Corvellec, M

    Quemener, E. and Corvellec, M. Sidus—the solution for extreme deduplication of an operating system. Linux Journal, 2013 0 (235): 0 3, 2013

    2013

  43. [43]

    K., Ngo, R., Pilz, K., et al

    Sastry, G., Heim, L., Belfield, H., Anderljung, M., Brundage, M., Hazell, J., O'keefe, C., Hadfield, G. K., Ngo, R., Pilz, K., et al. Computing power and the governance of artificial intelligence. arXiv preprint arXiv:2402.08797, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Saul, L. K. Weight-balancing fixes and flows for deep learning. Transactions on Machine Learning Research (TMLR), 2023 a

    2023

  45. [45]

    Saul, L. K. Weight-balancing fixes and flows for deep learning. Transactions on Machine Learning Research (TMLR), 2023 b . ISSN 2835-8856

    2023

  46. [46]

    J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al

    Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. Mastering the game of go with deep neural networks and tree search. Nature, 529 0 (7587): 0 484--489, 2016

    2016

  47. [47]

    and Zisserman, A

    Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014

    Pith/arXiv arXiv 2014

  48. [48]

    and Gribonval, R

    Stock, P. and Gribonval, R. An embedding of relu networks and an analysis of their identifiability. Constructive Approximation, 57 0 (2): 0 853--899, 2023

    2023

  49. [49]

    Equi-normalization of neural networks, 2019

    Stock, P., Graham, B., Gribonval, R., and Jégou, H. Equi-normalization of neural networks, 2019

    2019

  50. [50]

    Going deeper with convolutions

    Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 1--9, 2015

    2015

  51. [51]

    Growing tiny networks: Spotting expressivity bottlenecks and fixing them optimally

    Verbockhaven, M., Rudkiewicz, T., Chevallier, S., and Charpiat, G. Growing tiny networks: Spotting expressivity bottlenecks and fixing them optimally. Transactions on Machine Learning Research (TMLR), 2024. ISSN 2835-8856

    2024

  52. [52]

    Thermodynamik quantenmechanischer gesamtheiten

    Von Neumann, J. Thermodynamik quantenmechanischer gesamtheiten. Nachrichten von der Gesellschaft der Wissenschaften zu G \"o ttingen, Mathematisch-Physikalische Klasse , 1927: 0 273--291, 1927

    1927

  53. [53]

    Symmetry teleportation for accelerated optimization

    Zhao, B., Dehmamy, N., Walters, R., and Yu, R. Symmetry teleportation for accelerated optimization. volume 35, 2022

    2022

  54. [54]

    Symmetries, flat minima, and the conserved quantities of gradient flow

    Zhao, B., Ganev, I., Walters, R., Yu, R., and Dehmamy, N. Symmetries, flat minima, and the conserved quantities of gradient flow. In International Conference on Learning Representations (ICLR), 2023

    2023