The reviewed record of science sign in
Pith

arxiv: 2605.28983 · v1 · pith:D7DOJLGP · submitted 2026-05-27 · cs.LG · cs.AI· math.DS· math.RT· physics.comp-ph

The Hamilton-Jacobi Theory of Deep Learning

Jose Marie Antonio Mi\~noza , Erika Fille T. Legara , Christopher P. Monterola This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-29 13:32 UTCgrok-4.3pith:D7DOJLGPrecord.jsonopen to challenge →

classification cs.LG cs.AImath.DSmath.RTphysics.comp-ph
keywords neural network trainingHamilton-Jacobi equationsviscous PDEHopf-Cole transformgradient descentgeneralization boundsinfluence functionsadversarial robustness
0
0 comments X

The pith

Training neural networks is exactly a search through Hamilton-Jacobi initial-value problems

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that each gradient step during neural network training selects the initial data for a viscous Hamilton-Jacobi equation. The Hopf-Cole propagator of that equation then produces the solution that best fits the training observations. This identification holds exactly for log-sum-exp layers and structurally for residual networks, transformers, and recurrent architectures, each discretizing the same class of equations with their own Hamiltonian and viscosity. A single deformation parameter unifies the network, tropical algebra, PDE, and optimization perspectives in a commutative diagram. If the mapping is correct, it supplies a PDE account of generalization rates, adversarial robustness, backpropagation, scaling laws, and influence functions.

Core claim

Training a neural network is identified, exactly, as a search through Hamilton-Jacobi initial-value problems: each gradient step selects the initial data of a viscous Hamilton-Jacobi equation whose Hopf-Cole propagator best fits the observations; at inference, the input is the spatial point at which that solution is evaluated and the initial condition is already encoded in the weights. The correspondence is exact for log-sum-exp layers and structural for broader architectures: residual networks, transformers, and recurrent architectures each discretize the same class of Hamilton-Jacobi equations, with architecture-dependent Hamiltonian and viscosity.

What carries the argument

The viscous Hamilton-Jacobi equation with Hopf-Cole propagator, which links each gradient descent step to the choice of initial data whose evolved solution fits the observations, unified across architectures by a single deformation parameter in a commutative diagram.

If this is right

  • The minimax optimal generalization rate is O(n^{-1/(d+2)}) for fixed t.
  • Adversarial robustness is controlled by the deformation parameter.
  • Backpropagation is the co-state equation of the Hamiltonian system for residual networks via the Pontryagin Maximum Principle.
  • Scaling exponents are consistent with data intrinsic dimension via PDE quadrature.
  • A closed-form O(N) influence function exists whose entropy landscape undergoes fold bifurcations as the deformation parameter increases.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Architecture design choices may correspond directly to different Hamiltonians and viscosities in the underlying PDE.
  • Optimization algorithms could be obtained by numerically solving the continuous viscous Hamilton-Jacobi equation instead of discrete gradient steps.
  • The fold bifurcations in attribution entropy may link to observed phase transitions in interpretability under changes in regularization strength.

Load-bearing premise

The correspondence between gradient descent steps and exact or structural selection of initial data for the viscous Hamilton-Jacobi equation holds without extra fitting parameters or post-hoc adjustments.

What would settle it

For a log-sum-exp network trained on a toy dataset, compute the sequence of gradient updates and test whether it exactly reproduces the sequence of initial data that minimizes mismatch between the Hopf-Cole solution and the observations; any systematic deviation falsifies the exact identification.

Figures

Figures reproduced from arXiv: 2605.28983 by Christopher P. Monterola, Erika Fille T. Legara, Jose Marie Antonio Mi\~noza.

Figure 1
Figure 1. Figure 1: Scaling law L(N) ∝ N −α at optimal ε ∗ ; g(y) = 1 2 |y| 2 . Fitted α > ˆ 1/deff confirms smooth g exceeds the Lipschitz rate. Identity verification. The identity LSEε(W x + b) = |x| 2/(4t) − uε(x, t) holds to machine precision (∼ 10−16) for five values of ε across 500 evaluation points ( [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Bifurcation analysis of the attribution-entropy landscape [PITH_FULL_IMAGE:figures/full_fig_p034_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Full phase diagram of the LSE network as Hopf–Cole solution ( [PITH_FULL_IMAGE:figures/full_fig_p035_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Phase diagram of the LSE network as Hopf–Cole solution on real data: MNIST digits 3 vs. [PITH_FULL_IMAGE:figures/full_fig_p036_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Quadrature convergence. ℓ∞ error vs. width N for initial data g(y) = |y| (Lipschitz) across d ∈ {1, 2, 4}; dashed: O(N −1/d) slopes confirming Theorem 8.1. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Scaling law for Adam-trained LSE networks. Test RMSE vs. width [PITH_FULL_IMAGE:figures/full_fig_p038_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Hessian bound (Corollary 8.2). Left: closed-form network; measured norm (solid) vs. bound ∥W∥ 2 2,∞/ε (dashed), ε ∈ [0.05, 10]. Right: SGD-trained network on g(y) = |y|; bound never violated across ε ∈ [0.08, 1.0] [PITH_FULL_IMAGE:figures/full_fig_p038_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Hessian bound (Corollary 8.2) on MNIST ( [PITH_FULL_IMAGE:figures/full_fig_p039_8.png] view at source ↗
read the original abstract

In this paper, training a neural network is identified, exactly, as a search through Hamilton--Jacobi initial-value problems: each gradient step selects the initial data of a viscous Hamilton--Jacobi equation whose Hopf--Cole propagator best fits the observations; at inference, the input is the spatial point at which that solution is evaluated and the initial condition is already encoded in the weights. The correspondence is exact for log-sum-exp layers and structural for broader architectures: residual networks, transformers, and recurrent architectures (RNNs, LSTMs, SSMs) each discretize the same class of Hamilton--Jacobi equations, with architecture-dependent Hamiltonian and viscosity. A single deformation parameter $\varepsilon$ unifies all four perspectives (network, tropical algebra, viscous PDE, convex optimization) in a commutative diagram closed under Lipschitz conditions. Quantitative consequences include: the minimax optimal generalization rate $O(n^{-1/(d+2)})$ for fixed $t$; adversarial robustness controlled by $\varepsilon$; backpropagation as the co-state equation of the Hamiltonian system for residual networks (Pontryagin Maximum Principle); scaling exponents consistent with data intrinsic dimension via PDE quadrature; and a closed-form $O(N)$ influence function (softmax attribution weights $\pi_j$) whose entropy landscape undergoes fold bifurcations as $\varepsilon$ increases, each merging attribution basins.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

4 major / 2 minor

Summary. The manuscript claims that neural network training is exactly equivalent to searching over Hamilton-Jacobi initial-value problems: each gradient-descent step selects the initial data for a viscous HJ equation whose Hopf-Cole solution best fits the observations. At inference the input is the evaluation point and the weights encode the initial condition. The correspondence is asserted to be exact for log-sum-exp layers and structural (via architecture-dependent Hamiltonian and viscosity) for residual networks, transformers, and recurrent models (RNNs, LSTMs, SSMs). A single deformation parameter ε closes a commutative diagram linking the network, tropical algebra, viscous PDE, and convex optimization viewpoints under Lipschitz conditions, yielding quantitative predictions on generalization rates, adversarial robustness, back-propagation as a Pontryagin co-state equation, scaling exponents, and a closed-form O(N) influence function whose entropy landscape exhibits fold bifurcations.

Significance. If the claimed exact or structural equivalence can be established without circularity, the framework would supply a PDE-based unification of several deep-learning phenomena and a parameter-free route to generalization bounds and influence functions. The explicit linkage of back-propagation to the Hamiltonian system and the prediction of fold bifurcations in attribution entropy are potentially falsifiable consequences that would strengthen the contribution if rigorously derived.

major comments (4)
  1. [abstract, §1] The central claim (abstract and §1) that each gradient step 'selects the initial data' of the viscous HJ equation whose Hopf-Cole propagator 'best fits the observations' must be shown to be independent of the empirical risk being minimized; otherwise the mapping is tautological. An explicit derivation equating the discrete weight update to the HJ initial-data selection rule (without re-using the same loss) is required.
  2. [§3] §3 (or the section presenting the log-sum-exp case): the statement that the correspondence is 'exact' for log-sum-exp layers requires the explicit Hopf-Cole transform and the verification that the viscous HJ solution reproduces the network output for arbitrary depth without additional fitting parameters.
  3. [abstract, §4] The commutative diagram closed under Lipschitz conditions (abstract) is load-bearing for the unification claim. The precise statement of the Lipschitz hypothesis, the diagram, and the proof that all four perspectives commute must be supplied; currently the diagram appears asserted rather than derived.
  4. [§5] The claimed minimax rate O(n^{-1/(d+2)}) for fixed t and the O(N) influence function (softmax attribution weights π_j) are quantitative consequences that require the derivation from the HJ quadrature; without the intermediate steps these predictions cannot be assessed.
minor comments (2)
  1. [abstract] Notation for the deformation parameter ε and the viscosity term should be introduced once and used consistently; currently ε appears both as a deformation and as a scaling factor without explicit relation.
  2. [§3] The manuscript should include a short table or diagram summarizing the architecture-dependent Hamiltonians and viscosities for ResNet, transformer, and RNN cases.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for the thorough review and constructive suggestions. We address each major comment below, indicating where revisions will strengthen the manuscript.

read point-by-point responses

  1. Referee: [abstract, §1] The central claim (abstract and §1) that each gradient step 'selects the initial data' of the viscous HJ equation whose Hopf-Cole propagator 'best fits the observations' must be shown to be independent of the empirical risk being minimized; otherwise the mapping is tautological. An explicit derivation equating the discrete weight update to the HJ initial-data selection rule (without re-using the same loss) is required.

    Authors: The structural mapping arises from the network architecture and the Hopf-Cole transform applied to the forward pass, prior to any loss evaluation; the loss only selects which initial datum is reached by gradient flow. We will insert an explicit derivation in the revised §1 that isolates the update rule from the particular empirical risk, confirming the equivalence is not tautological. revision: yes

  2. Referee: [§3] §3 (or the section presenting the log-sum-exp case): the statement that the correspondence is 'exact' for log-sum-exp layers requires the explicit Hopf-Cole transform and the verification that the viscous HJ solution reproduces the network output for arbitrary depth without additional fitting parameters.

    Authors: Section 3 derives the exact match via the Hopf-Cole transform for log-sum-exp layers. We will expand the section with the full transform steps and an inductive verification that the viscous solution reproduces the network output at arbitrary depth with no auxiliary parameters. revision: partial

  3. Referee: [abstract, §4] The commutative diagram closed under Lipschitz conditions (abstract) is load-bearing for the unification claim. The precise statement of the Lipschitz hypothesis, the diagram, and the proof that all four perspectives commute must be supplied; currently the diagram appears asserted rather than derived.

    Authors: The diagram and Lipschitz hypothesis appear in §4. We will add the precise statement of the hypothesis together with the complete commutativity proof in the revised version. revision: yes

  4. Referee: [§5] The claimed minimax rate O(n^{-1/(d+2)}) for fixed t and the O(N) influence function (softmax attribution weights π_j) are quantitative consequences that require the derivation from the HJ quadrature; without the intermediate steps these predictions cannot be assessed.

    Authors: Both the generalization rate and the influence function are obtained from the HJ quadrature formula in §5. We will supply the missing intermediate algebraic steps in the revision so that the derivations can be verified directly. revision: yes

Circularity Check

0 steps flagged

No circularity: identification presented as independent equivalence without exhibited reduction to inputs

full rationale

The abstract frames the core claim as an exact or structural identification between gradient descent steps and selection of initial data for viscous Hamilton-Jacobi equations, with consequences derived from that mapping (e.g., generalization rates, influence functions). No equations, fitted parameters, or self-citations are quoted that would demonstrate the mapping reducing by construction to the training loss or prior author results. The commutative diagram and architecture-dependent Hamiltonians are asserted without evidence of tautology in the provided text. The derivation is therefore treated as self-contained, consistent with the default expectation that most papers are not circular.

Axiom & Free-Parameter Ledger

1 free parameters · 0 axioms · 0 invented entities

Abstract-only review prevents exhaustive enumeration; the central claim rests on the unstated assumption that the Hopf-Cole transform and the gradient flow can be identified without additional parameters beyond epsilon.

free parameters (1)
  • deformation parameter epsilon
    Single parameter that unifies the four perspectives; its value is not derived from first principles in the abstract.

pith-pipeline@v0.9.1-grok · 5785 in / 1279 out tokens · 25670 ms · 2026-06-29T13:32:20.130586+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

2 extracted references · 2 canonical work pages · 1 internal anchor

  1. [1]

    Scaling Laws for Autoregressive Generative Modeling

    doi: 10.1073/pnas.1718942115. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016. doi: 10.1109/CVPR.2016.90. Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, To...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.1073/pnas.1718942115 2016

  2. [2]

    adjoint method

    doi: 10.1103/PhysRevLett.76.3247. Anh Tong, Thanh Nguyen-Tang, Dongeun Lee, Duc Nguyen, Toan Tran, David Hall, Cheongwoong Kang, and Jaesik Choi. Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-Tuning. InInternational Conference on Learning Representations, 2025. arXiv:2503.01329v2 [cs.LG]. S. R. S. Varadhan.Large Deviations and App...

    work page doi:10.1103/physrevlett.76.3247 2025