Pith. sign in

REVIEW 3 major objections 4 minor 99 references

This paper makes the case that Lipschitz continuity is the fundamental principle for building trustworthy neural networks, and supports it with a new sum-over-paths Lipschitz bound for DAG networks and corrected constants for sigmoid and so

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A systematic survey of Lipschitz continuity in deep learning that corrects sigmoid (1/4) and softmax (1/2) Lipschitz constants and proves a sum-over-paths Lipschitz bound for additively-evaluated DAG networks.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection A solid, genuinely useful survey whose new DAG Lipschitz bound is correct but narrower than its headline claim; referee it with a request to fix the framing. the 3 major comments →

arxiv 2607.16329 v2 pith:FI5A65XO submitted 2026-07-15 stat.ML cs.LG

Lipschitz Continuity in Deep Learning: A Systematic Review of Theoretical Foundations, Estimation Methods, Regularization Approaches, and Certifiable Robustness

R\'ois\'in Luo , James McDermott , Colm O'Riordan This is my paper
classification stat.ML cs.LG MSC 68T0726A16
keywords Lipschitz continuitydeep learningcertifiable robustnessDAG networksactivation functionsspectral normregularizationgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Lipschitz continuity — the requirement that small input changes produce proportionally small output changes — is presented as the unifying lens for robustness, generalization, regularization, and certification in deep learning. Beyond surveying existing work, the paper contributes new theory: for a feedforward network represented as a directed acyclic graph, the Lipschitz constant is bounded by the sum, over all computational paths, of the product of edge-wise Lipschitz constants. It also derives exact Lipschitz constants for common activations, correcting the literature: sigmoid is 1/4-Lipschitz and softmax is 1/2-Lipschitz, not 1. If correct, these results tighten standard spectral-product bounds and enlarge the class of architectures for which certified robustness radii can be computed.

Core claim

Theorem 2.21 establishes that for a finite DAG network with unique input node s and output node t, where each node v applies a Lipschitz module h_v to each predecessor output and sums the results, the overall Lipschitz constant satisfies Lip[f] ≤ Σ_{p∈P} ∏_{i} Lip[h_{v_i}], with the sum taken over all computational paths from s to t. The proof proceeds by induction along a topological order, using the Minkowski inequality and a path-expansion lemma. The paper also proves that Lip[Sigmoid] = 1/4 and Lip[Softmax] = 1/2, with numerical validation, and derives a p-norm margin robustness radius via Hölder's inequality, generalizing the standard 2-norm result to all ℓ_p perturbations.

What carries the argument

The central object is the path constant C_p = ∏_{i} Lip[h_{v_i}], the product of module Lipschitz constants along a computational path, combined with the additive evaluation rule x_v(x) = Σ_{(u→v)∈E} h_v(x_u(x)). The theorem converts a global Lipschitz bound into a sum over paths: S(v) is defined recursively as the sum over incoming edges of C_{(u→v)} S(u), and Lemma 2.22 shows this equals the path sum. This mechanism recovers the residual network bound Lip[m_res] ≤ 1 + Lip[φ] as a special case. For activations, the key identity is K = sup_x ||∇f(x)||, with Clarke sub-differentials handling non-differentiable functions like ReLU.

Load-bearing premise

The DAG bound in Section 2.7.1 applies only when every node is evaluated by summing the outputs of all incoming edges (equation 36); the theorem's claim to cover 'arbitrary' DAG networks therefore fails for graphs whose nodes concatenate outputs or apply joint multi-input functions.

What would settle it

Construct a two-input DAG where a node v concatenates h_u1(x) and h_u2(x) instead of summing them. With simple maps, e.g., identity and zero, the true Lipschitz constant can exceed the path-sum bound because the concatenation norm combines path contributions in ℓ_2; numerically computing sup_{a≠b} ||f(a)−f(b)||/||a−b|| over random points would violate equation 39.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Spectral-product bounds for feedforward networks can be replaced by the tighter path-sum bound for networks with skip connections, reducing overestimation in global Lipschitz certificates.
  • The corrected sigmoid and softmax constants tighten layer-wise Lipschitz factors in product bounds for networks using these activations, directly improving certified radii.
  • The residual network bound Lip[m_res] ≤ 1 + Lip[φ] follows as a special case of the DAG theorem, unifying two previously separate results.
  • The p-norm margin radius m(x) / (2^{1−1/p} Lip_p[f]) generalizes existing 2-norm robustness certificates to all ℓ_p perturbations.
  • The survey's synthesis implies that Lipschitz control is a common mechanism underlying adversarial robustness, generalization bounds, and stable optimization.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The DAG theorem's additive evaluation rule is restrictive: architectures such as DenseNet-style concatenations or joint multi-input operations do not fit the proof, and extending the bound to them would require a different inequality than the Minkowski step used here.
  • If the softmax 1/2 constant holds across all ℓ_p norms as concurrent work suggests, attention-layer Lipschitz bounds in transformers could shrink by a constant factor, improving certified radii for language models.
  • The path-sum bound's looseness grows with the number of paths; the paper's non-biconnected decomposition suggests a practical direction: recursively decompose computation graphs into biconnected components to obtain tighter bounds.
  • A testable extension would be to numerically maximize gradient norms on common architectures to see how often the true Lipschitz constant approaches the DAG bound; if it is rarely tight, the bound is safe but loose.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper is a systematic survey of Lipschitz continuity in deep learning, organized into theoretical foundations, estimation methods, regularization approaches, and certifiable robustness. Beyond the survey material, it presents several new or corrected results: exact Lipschitz constants for common activation functions (sigmoid 1/4, softmax 1/2, swish ≈1.0998, GELU ≈1.1289), a Lipschitz upper bound for networks represented as directed acyclic graphs (Theorem 2.21), a bound for non-biconnected DAGs (Theorem 2.23), and a p-norm margin robustness radius (Theorem 5.1). The overall thesis is that Lipschitz continuity is a unifying principle for analyzing robustness, generalization, and optimization dynamics in neural networks.

Significance. If the claims hold, the survey would serve as a useful reference, and the corrected activation constants would tighten spectral-product bounds and certified-radius computations in practice. The Appendix A derivations of the activation constants are correct, and the DAG theorem's proof is valid for the additive evaluation rule stated in Eq. (36). The public code release and the numerical sanity checks on the constants are commendable. However, the DAG theorem is advertised as applying to arbitrary neural network DAGs, while the proof only covers a restricted additively evaluated class; this overclaim weakens the paper's main new theoretical contribution. The survey's breadth and organization remain valuable, but the scope of the headline result needs to be corrected.

major comments (3)
  1. [§1.2, §2.7.1 (Eq. 36)] Contribution 3 and Theorem 2.21 claim a Lipschitz bound for 'arbitrary neural networks represented as directed acyclic graphs (DAGs).' The proof, however, relies on the additive evaluation rule of Eq. (36), x_v(x) = Σ_{(u→v)∈E} h_v(x_u(x)), where the same module h_v is applied to every incoming edge. This is a restricted computation class: it excludes standard sum-then-activate layers with per-edge affine maps, DenseNet-style concatenations, and joint multi-input modules such as attention. The Minkowski step and the recursion for S(v) both depend on the additive form, so the theorem cannot be read as a general DAG bound as stated. Please restrict the theorem statement and all downstream claims (including §2.7.3) to additively evaluated DAGs, or explain how arbitrary DAGs are transformed into this form without changing the bound.
  2. [§5.1 (Theorem 5.1)] The proof of Theorem 5.1 is not rigorous as written. It writes m(x) = (e_c - e_j)^T [f_c(x); max_{j≠c} f_j(x)] and then applies a Hölder/conjugacy argument as though max_j f_j were a fixed coordinate. This is not a valid linear-functional representation. The theorem itself is correct, but a proper proof should define g_j := f_c - f_j for each j, note that |g_j(y) - g_j(z)| ≤ 2^{1/q} Lip_p[f] ‖y−z‖_p, and then use the fact that m = min_j g_j has the same Lipschitz constant. The same notational issue appears in Eq. (127) of §4.4.
  3. [§2.8 (Eq. 46)] The displayed derivation of the vector-contraction Rademacher bound contains a dimensional error: the first factor is written as √(2K), but a Lipschitz constant should enter linearly. The vector contraction inequality yields a constant times K, not √K. As written, the chain implies a bound of order √K, which is inconsistent with the immediately following conclusion R(h)−R_S(h) ≤ O(K). This is likely a typographical slip, but it must be corrected because the section supports the survey's claim that Lipschitz constants control generalization.
minor comments (4)
  1. [Remark 2.6] The claim that neural-network domains can be assumed convex is only valid for standard box input domains. For data manifolds, discrete inputs, or other non-convex geometries it is unjustified, and Lemma 2.5's equality and the numerical gradient-maximization procedure depend on this assumption. Please qualify the remark.
  2. [§2.9 / Abstract] The abstract states that Lipschitz continuity 'governs ... optimization dynamics.' Section 2.9 is a summary of the authors' own framework and explicitly notes that only a few studies have explored this direction. Please add a caveat in the abstract or introduction so that the survey does not present a single recent line of work as established consensus.
  3. [§2.7.1, paragraph 1] The statement that 'the existing literature does not present an explicit Lipschitz bound for a general feedforward network with skip connections' is too strong given the path-norm and path-metric bounds cited immediately afterward (Neyshabur et al., 2015; Gonon et al., 2025). Please rephrase to say that the present bound has a different, explicitly graph-theoretic form.
  4. [Table 1 / Appendix A] The text says the corrected activation constants are 'validated by our numerical experiment' but gives no experimental details. Since code is provided, a short description of the optimization setup (domain, initialization, number of restarts) would make the validation reproducible and the claim more precise.

Circularity Check

0 steps flagged

No significant circularity: the new bounds and corrected constants are derived from stated definitions, and the self-citations in the survey sections are not load-bearing for the new results.

full rationale

The paper's central new claims—Theorem 2.21 (DAG Lipschitz bound), the corrected activation-function Lipschitz constants in Appendix A, and Theorem 5.1 (Hölder-based margin radius)—are each derived from stated definitions rather than assumed. Theorem 2.21 proves an upper bound on Lip[f] by induction and path expansion from the additive evaluation rule in eq. (36); the bound is not the definition of Lip[f]. The activation constants are computed as suprema of derivative norms, not fitted and then relabeled as predictions; the numerical experiments are sanity checks. Theorem 5.1 follows from Hölder's inequality applied to the Lipschitz property and the margin definition. Section 2.9 does summarize the authors' prior work (Luo et al., 2025a,b), but it is presented as a literature review and does not substitute for the derivations of the paper's new results; self-citation in a survey is expected and is not load-bearing for the novel claims. The DAG theorem's scope is narrower than the headline phrase 'arbitrary neural networks represented as DAGs' because the proof relies on the specific additive evaluation rule of eq. (36); however, that is a scope/framing limitation, not circularity, since within that stated rule the proof is self-contained. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors, and no known result is presented as new purely by renaming.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The paper's new results rest on standard analysis (listed as axioms), one convexity domain assumption, and one restricted modeling assumption (additive DAG evaluation). No free parameters are fitted: the activation constants are derived in closed form and the numerical checks are sanity validations of those closed forms, not fits. No invented entities.

axioms (4)
  • domain assumption Neural network input domains are convex, so the tight Lipschitz constant equals sup‖∇f‖₂ (Lemma 2.5 equality).
    Remark 2.6 asserts "the domain of a neural network f: R^m → R can be assumed to be a convex set on R^m. Consequently, the tight Lipschitz bound is attained." Box-like input spaces are convex, but this is an unstated modeling assumption for other data geometries.
  • ad hoc to paper DAG networks evaluate additively: x_v(x) = Σ_{(u→v)∈E} h_v(x_u(x)).
    Eq. (36), the premise of Theorem 2.21. Each node applies the same module h_v to every predecessor and sums; this excludes concatenation-style or jointly-computed multi-input nodes, so the theorem's "arbitrary DAG" framing (Section 2.7.1) is narrower than claimed.
  • domain assumption Non-biconnected DAG factorizes through articulation points as f = S_L ∘ a_{L-1} ∘ ⋯ ∘ a_1 ∘ S_1.
    Theorem 2.23 assumes this factorization (Figure 2); the bound then follows by repeated composition (Prop 2.10). Whether a given architecture decomposes this way must be checked per architecture.
  • standard math Standard analysis results: MVT-based gradient-norm characterization of Lipschitz constants, Minkowski/triangle inequality, Hölder's inequality, Popoviciu's variance bound, Rademacher contraction lemmas (Talagrand, Maurer), Clarke subdifferential calculus, power iteration convergence.
    Invoked in Lemmas 2.5/2.15, Propositions 2.10-2.12, Lemma 2.22, Theorem 5.1, Appendix A.6, and Section 2.8.

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Lipschitz Continuity in Deep Learning: A Systematic Review of Theoretical Foundations, Estimation Methods, Regularization Approaches, and Certifiable Robustness." pith.science (2026). https://pith.science/paper/FI5A65XO

@misc{pith2026260716329,
  author       = {Pith},
  title        = {Pith review of: Lipschitz Continuity in Deep Learning: A Systematic Review of Theoretical Foundations, Estimation Methods, Regularization Approaches, and Certifiable Robustness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FI5A65XO}},
  note         = {Machine review of arXiv:2607.16329}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Lipschitz continuity is a fundamental property of neural networks that characterizes their sensitivity to input perturbations. It plays a pivotal role in deep learning, governing \textbf{robustness}, \textbf{generalization} and \textbf{optimization dynamics}. Despite its importance, research on Lipschitz continuity is scattered across various domains, lacking a unified perspective. This paper addresses this gap by providing a systematic review of Lipschitz continuity in deep learning. We explore its \textbf{theoretical foundations}, \textbf{estimation methods}, \textbf{regularization approaches}, and \textbf{certifiable robustness}. By reviewing existing research through the lens of Lipschitz continuity, this survey serves as a comprehensive reference for researchers and practitioners seeking a deeper understanding of Lipschitz continuity and its implications in deep learning.

Figures

Figures reproduced from arXiv: 2607.16329 by Colm O'Riordan, James McDermott, R\'ois\'in Luo.

Figure 1
Figure 1. Figure 1: Example of Feedforward DAG Network. There are four computational paths: [PITH_FULL_IMAGE:figures/full_fig_p012_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Topology of non-biconnected DAG network. If a DAG is separated into two sub-DAGs by the [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: A residual module m(x) = x + ϕ(x) consists of a non-identity unit ϕ : x 7→ ϕ(x) and an identity skip connection unit s : x 7→ x. 2.8 Complexity-Theoretic Generalization Bound Notations. Let H be a hypothesis family and H ∋ h : X → Y be a hypothesis. Let ℓ : Y × Y → R be a loss function. For each ℓ(h(x), y), we can associate ℓ and h with a map G ∋ g : X × Y → R, where G is the family of loss functions assoc… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

99 extracted references · 2 canonical work pages

  1. [1]

    Detecting brittle decisions for free: leveraging margin consistency in deep robust classifiers , year =

    Ngnaw\'. Detecting brittle decisions for free: leveraging margin consistency in deep robust classifiers , year =. Proceedings of the 38th International Conference on Neural Information Processing Systems , articleno =

  2. [2]

    Proceedings of the 32nd International Conference on Neural Information Processing Systems , pages =

    Tsuzuku, Yusuke and Sato, Issei and Sugiyama, Masashi , title =. Proceedings of the 32nd International Conference on Neural Information Processing Systems , pages =. 2018 , publisher =

    2018

  3. [3]

    Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation , url =

    Hein, Matthias and Andriushchenko, Maksym , booktitle =. Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulation , url =

  4. [4]

    ImageNet Classification with Deep Convolutional Neural Networks , url =

    Krizhevsky, Alex and Sutskever, Ilya and Hinton, Geoffrey E , booktitle =. ImageNet Classification with Deep Convolutional Neural Networks , url =

  5. [5]

    Attention is All you Need , url =

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, ukasz and Polosukhin, Illia , booktitle =. Attention is All you Need , url =

  6. [6]

    Language Models are Few-Shot Learners , url =

    Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom and Child, Rewon and Ramesh, Aditya and Ziegler, Daniel and Wu, Jeffrey and Winte...

  7. [7]

    International Conference on Learning Representations , year=

    Semi-Supervised Classification with Graph Convolutional Networks , author=. International Conference on Learning Representations , year=

  8. [8]

    Aakanksha Chowdhery and Sharan Narang and Jacob Devlin and Maarten Bosma and Gaurav Mishra and Adam Roberts and Paul Barham and Hyung Won Chung and Charles Sutton and Sebastian Gehrmann and Parker Schuh and Kensen Shi and Sasha Tsvyashchenko and Joshua Maynez and Abhishek Rao and Parker Barnes and Yi Tay and Noam Shazeer and Vinodkumar Prabhakaran and Emi...

  9. [9]

    2025 , eprint=

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning , author=. 2025 , eprint=

    2025

  10. [10]

    2024 , eprint=

    GPT-4 Technical Report , author=. 2024 , eprint=

    2024

  11. [11]

    2023 , eprint=

    LLaMA: Open and Efficient Foundation Language Models , author=. 2023 , eprint=

    2023

  12. [12]

    2016 , journal=

    Concrete Problems in AI Safety , author=. 2016 , journal=

    2016

  13. [13]

    International Conference on Learning Representations , year=

    Towards Deep Learning Models Resistant to Adversarial Attacks , author=. International Conference on Learning Representations , year=

  14. [14]

    International Conference on Learning Representations , year=

    Benchmarking Neural Network Robustness to Common Corruptions and Perturbations , author=. International Conference on Learning Representations , year=

  15. [15]

    International Conference on Learning Representations , year=

    Robustness May Be at Odds with Accuracy , author=. International Conference on Learning Representations , year=

  16. [16]

    International Conference on Learning Representations (ICLR) , year=

    Explaining and harnessing adversarial examples , author=. International Conference on Learning Representations (ICLR) , year=

  17. [17]

    IEEE Symposium on Security and Privacy (SP) , pages=

    Towards evaluating the robustness of neural networks , author=. IEEE Symposium on Security and Privacy (SP) , pages=. 2017 , organization=

    2017

  18. [18]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    The many faces of robustness: A critical analysis of out-of-distribution generalization , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

  19. [19]

    Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings , pages =

    Maurer, Andreas , title =. Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings , pages =. 2016 , isbn =. doi:10.1007/978-3-319-46379-7_1 , abstract =

    work page doi:10.1007/978-3-319-46379-7_1 2016

  20. [20]

    and Foster, Dylan J

    Bartlett, Peter L. and Foster, Dylan J. and Telgarsky, Matus , title =. Proceedings of the 31st International Conference on Neural Information Processing Systems , pages =. 2017 , isbn =

    2017

  21. [21]

    , date =

    Gouk, Henry and Frank, Eibe and Pfahringer, Bernhard and Cree, Michael J. , date =. Regularisation of neural networks by enforcing. Machine Learning , number =. 2021 , bdsk-url-1 =. doi:10.1007/s10994-020-05929-w , id =

    doi:10.1007/s10994-020-05929-w 2021

  22. [22]

    LipsFormer: Introducing

    Xianbiao Qi and Jianan Wang and Yihao Chen and Yukai Shi and Lei Zhang , booktitle=. LipsFormer: Introducing. 2023 , url=

    2023

  23. [23]

    Pay Attention to Attention Distribution: A New Local

    Yudin, Nikolay and Gaponov, Alexander and Kudriashov, Sergei and Rakhuba, Maxim , journal=. Pay Attention to Attention Distribution: A New Local

  24. [24]

    Proceedings of the Thirtieth International Conference on Very Large Data Bases - Volume 30 , pages =

    Chen, Lei and Ng, Raymond , title =. Proceedings of the Thirtieth International Conference on Very Large Data Bases - Volume 30 , pages =. 2004 , isbn =

    2004

  25. [25]

    , title =

    Levenshtein, Vladimir I. , title =. Soviet Physics Doklady , volume =. 1966 , publisher =

    1966

  26. [26]

    Certified Robustness in

    Elias Abad Rocamora and Grigorios Chrysos and Volkan Cevher , booktitle=. Certified Robustness in. 2024 , url=

    2024

  27. [27]

    2016 , eprint=

    Layer Normalization , author=. 2016 , eprint=

    2016

  28. [28]

    2024 , isbn =

    Hu, Xixu and Zheng, Runkai and Wang, Jindong and Leung, Cheuk Hang and Wu, Qi and Xie, Xing , title =. 2024 , isbn =. doi:10.1007/978-3-031-72949-2_20 , booktitle =

    work page doi:10.1007/978-3-031-72949-2_20 2024

  29. [29]

    Exploring Generalization in Deep Learning , url =

    Neyshabur, Behnam and Bhojanapalli, Srinadh and Mcallester, David and Srebro, Nati , booktitle =. Exploring Generalization in Deep Learning , url =

  30. [30]

    Sokolić, Jure and Giryes, Raja and Sapiro, Guillermo and Rodrigues, Miguel R. D. , journal=. Robust Large Margin Deep Neural Networks , year=

  31. [31]

    Proceedings of the 41st International Conference on Machine Learning , pages =

    How Smooth Is Attention? , author =. Proceedings of the 41st International Conference on Machine Learning , pages =. 2024 , editor =

    2024

  32. [32]

    Berg, Rianne van den and Hasenclever, Leonard and Tomczak, Jakub M and Welling, Max , booktitle=

  33. [33]

    Proceedings of the 32nd International Conference on Machine Learning , pages =

    Variational Inference with Normalizing Flows , author =. Proceedings of the 32nd International Conference on Machine Learning , pages =. 2015 , editor =

    2015

  34. [34]

    Density estimation using Real

    Laurent Dinh and Jascha Sohl-Dickstein and Samy Bengio , booktitle=. Density estimation using Real. 2017 , url=

    2017

  35. [35]

    Invertible DenseNets with Concatenated LipSwish , url =

    Perugachi-Diaz, Yura and Tomczak, Jakub and Bhulai, Sandjai , booktitle =. Invertible DenseNets with Concatenated LipSwish , url =

  36. [36]

    Proceedings of the 34th International Conference on Machine Learning , pages =

    Martin Arjovsky and Soumith Chintala and L. Proceedings of the 34th International Conference on Machine Learning , pages =. 2017 , editor =

    2017

  37. [37]

    2008 , publisher=

    Optimal transport: old and new , author=. 2008 , publisher=

    2008

  38. [38]

    Proceedings of the 31st International Conference on Neural Information Processing Systems , pages =

    Gulrajani, Ishaan and Ahmed, Faruk and Arjovsky, Martin and Dumoulin, Vincent and Courville, Aaron , title =. Proceedings of the 31st International Conference on Neural Information Processing Systems , pages =. 2017 , isbn =

    2017

  39. [39]

    International Conference on Learning Representations , year=

    Towards Principled Methods for Training Generative Adversarial Networks , author=. International Conference on Learning Representations , year=

  40. [40]

    International Conference on Learning Representations , year=

    Spectral Normalization for Generative Adversarial Networks , author=. International Conference on Learning Representations , year=

  41. [41]

    International Conference on Learning Representations , year=

    Neural Photo Editing with Introspective Adversarial Networks , author=. International Conference on Learning Representations , year=

  42. [42]

    Goodfellow and Rob Fergus , editor =

    Christian Szegedy and Wojciech Zaremba and Ilya Sutskever and Joan Bruna and Dumitru Erhan and Ian J. Goodfellow and Rob Fergus , editor =. Intriguing properties of neural networks , booktitle =. 2014 , url =

    2014

  43. [43]

    International Conference on Learning Representations , year=

    Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach , author=. International Conference on Learning Representations , year=

  44. [44]

    1976 , publisher=

    Principles of Mathematical Analysis , author=. 1976 , publisher=

    1976

  45. [45]

    1991 , publisher=

    Functional Analysis , author=. 1991 , publisher=

    1991

  46. [46]

    1987 , publisher=

    Real and complex analysis , author=. 1987 , publisher=

    1987

  47. [47]

    , title =

    Nair, Vinod and Hinton, Geoffrey E. , title =. Proceedings of the 27th International Conference on International Conference on Machine Learning , pages =. 2010 , isbn =

    2010

  48. [48]

    2022 , note =

    Boyd, Stephen and Duchi, John and Pilanci, Mert and Vandenberghe, Lieven , title =. 2022 , note =

    2022

  49. [49]

    1990 , publisher=

    Optimization and nonsmooth analysis , author=. 1990 , publisher=

    1990

  50. [50]

    Transactions of the American Mathematical Society , volume=

    Generalized gradients and applications , author=. Transactions of the American Mathematical Society , volume=

  51. [51]

    , title =

    Jordan, Matt and Dimakis, Alexandros G. , title =. Proceedings of the 34th International Conference on Neural Information Processing Systems , articleno =. 2020 , isbn =

    2020

  52. [52]

    ICML Workshop on Deep Learning for Audio, Speech and Language Processing , year=

    Rectifier nonlinearities improve neural network acoustic models , author=. ICML Workshop on Deep Learning for Audio, Speech and Language Processing , year=

  53. [53]

    Mathematics of Control, Signals and Systems , volume=

    Approximation by superpositions of a sigmoidal function , author=. Mathematics of Control, Signals and Systems , volume=

  54. [54]

    Incorporating Second-Order Functional Knowledge for Better Option Pricing , url =

    Dugas, Charles and Bengio, Yoshua and B\'. Incorporating Second-Order Functional Knowledge for Better Option Pricing , url =. Advances in Neural Information Processing Systems , editor =

  55. [55]

    ICLR , year=

    Fast and accurate deep network learning by exponential linear units (ELUs) , author=. ICLR , year=

  56. [56]

    2018 , url=

    Searching for Activation Functions , author=. 2018 , url=

    2018

  57. [57]

    Hendrycks, Dan and Gimpel, Kevin , journal=

  58. [58]

    International Conference on Machine Learning (ICML) , pages =

    Cisse, Moustapha and Bojanowski, Piotr and Grave, Edouard and Dauphin, Yann and Usunier, Nicolas , title =. International Conference on Machine Learning (ICML) , pages =. 2017 , publisher =

    2017

  59. [59]

    2025 , issn =

    Higher-order singular-value derivatives of real rectangular matrices , journal =. 2025 , issn =. doi:https://doi.org/10.1016/j.jmaa.2025.130236 , url =

    arXiv 2025

  60. [60]

    Optimization-Induced Dynamics of

    R. Optimization-Induced Dynamics of. arXiv preprint arXiv:2506.18588 , year=. 2506.18588 , archivePrefix=

    arXiv

  61. [61]

    Proceedings of the 36th International Conference on Machine Learning , pages =

    Invertible Residual Networks , author =. Proceedings of the 36th International Conference on Machine Learning , pages =. 2019 , editor =

    2019

  62. [62]

    Kim, Hyunjik and Papamakarios, George and Mnih, Andriy , booktitle =. The. 2021 , editor =

    2021

  63. [63]

    arXiv preprint arXiv:2007.02876 , year=

    A mathematical theory of attention , author=. arXiv preprint arXiv:2007.02876 , year=

    Pith/arXiv arXiv 2007

  64. [64]

    International Conference on Learning Representations (ICLR) , year=

    The Singular Values of Convolutional Layers , author=. International Conference on Learning Representations (ICLR) , year=

  65. [65]

    2012 , publisher=

    Matrix analysis , author=. 2012 , publisher=

    2012

  66. [66]

    arXiv preprint arXiv:1705.10941 , year=

    Spectral norm regularization for improving the generalizability of deep learning , author=. arXiv preprint arXiv:1705.10941 , year=

    Pith/arXiv arXiv

  67. [67]

    Praktische verfahren der gleichungsaufl

    Mises, RV and Pollaczek-Geiringer, Hilda , journal=. Praktische verfahren der gleichungsaufl. 1929 , publisher=

    1929

  68. [68]

    Virmaux, Aladin and Scaman, Kevin , booktitle =

  69. [69]

    2013 , publisher=

    Matrix computations , author=. 2013 , publisher=

    2013

  70. [70]

    Proceedings of The 28th Conference on Learning Theory , pages =

    Norm-Based Capacity Control in Neural Networks , author =. Proceedings of The 28th Conference on Learning Theory , pages =. 2015 , editor =

    2015

  71. [71]

    A Rescaling-Invariant

    Antoine Gonon and Nicolas Brisebarre and Elisa Riccietti and R. A Rescaling-Invariant. Forty-second International Conference on Machine Learning , year=

  72. [72]

    2018 , publisher=

    Foundations of machine learning , author=. 2018 , publisher=

    2018

  73. [73]

    2014 , publisher =

    Understanding Machine Learning: From Theory to Algorithms , author =. 2014 , publisher =

    2014

  74. [74]

    arXiv preprint arXiv:1412.6980 , year=

    Adam: A method for stochastic optimization , author=. arXiv preprint arXiv:1412.6980 , year=

    Pith/arXiv arXiv

  75. [75]

    Efficient and Accurate Estimation of

    Fazlyab, Mahyar and Robey, Alexander and Hassani, Hamed and Morari, Manfred and Pappas, George , booktitle =. Efficient and Accurate Estimation of

  76. [76]

    Towards Fast Computation of Certified Robustness for

    Weng, Lily and Zhang, Huan and Chen, Hongge and Song, Zhao and Hsieh, Cho-Jui and Daniel, Luca and Boning, Duane and Dhillon, Inderjit , booktitle =. Towards Fast Computation of Certified Robustness for. 2018 , editor =

    2018

  77. [77]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    Goodfellow, Ian and Pouget-Abadie, Jean and Mirza, Mehdi and Xu, Bing and Warde-Farley, David and Ozair, Sherjil and Courville, Aaron and Bengio, Yoshua , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  78. [78]

    An Iterative Algorithm for Computing the Best Estimate of an Orthogonal Matrix , journal =

    Bj\". An Iterative Algorithm for Computing the Best Estimate of an Orthogonal Matrix , journal =. 1971 , doi =. https://doi.org/10.1137/0708036 , abstract =

    doi:10.1137/0708036 1971

  79. [79]

    Chen, Ricky T. Q. and Behrmann, Jens and Duvenaud, David K and Jacobsen, Joern-Henrik , booktitle =. Residual Flows for Invertible Generative Modeling , url =

  80. [80]

    Preventing Gradient Attenuation in

    Li, Qiyang and Haque, Saminul and Anil, Cem and Lucas, James and Grosse, Roger B and Jacobsen, Joern-Henrik , booktitle =. Preventing Gradient Attenuation in

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.