An optimal control approach for neural network architecture adaptation with a posteriori error estimation

C G Krishnanunni; Tan Bui-Thanh; Thomas Scott

REVIEW 3 major objections 7 minor 27 references

An optimal control approach for neural network architecture adaptation with a posteriori error estimation

T0 review · 3 major / 7 minor · reviewed 2026-07-09 · glm-5.2

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Error estimates from optimal control guide where to add neural network layers

desk verdict DWR-based a posteriori error estimation for neural network depth adaptation: novel synthesis with a real theory-practice gap read the letter →

arxiv 2607.07637 v1 pith:4AS2ZKMO submitted 2026-07-08 cs.LG cs.NAmath.NAmath.OC

C G Krishnanunni , Thomas Scott , Tan Bui-Thanh This is my paper · ORCID

classification cs.LGcs.NAmath.NAmath.OC

keywords neuralarchitectureadaptationaposteriorierrorestimationoptimalcontroldualweightedresidualdepthgrowingfiniteelementmethodadjointequations

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that neural network training can be formulated as a continuous-time optimal control problem in which weights and biases are functions of a continuous depth variable, and that the error introduced by discretizing these functions into piecewise linear segments between layers can be exactly expressed and bounded in a way that decomposes additively across each layer-to-layer interval. The central object is an a posteriori error estimator derived via the dual weighted residual method from finite element analysis: it takes a trained network and produces, for each interval between consecutive layers, a computable quantity measuring how much that interval contributes to the total approximation error. The paper shows that this error decomposition directly answers the three questions any depth-growing scheme must resolve — where to insert a new layer (at the interval of maximum estimated error), when to stop (when validation loss stops improving), and how to initialize the new layer (by interpolating its neighbors). The authors prove Theorem 1, which gives the exact error expression, and Corollary 1, which converts it into a fully computable upper bound using interpolation error estimates and supremum norms of adjoint variables. Algorithm 1 implements the full cycle: train, estimate per-interval errors, insert a layer at the worst interval, retrain, and repeat. On a synthetic regression problem and a severely ill-posed inverse problem for the Navier-Stokes equations, the resulting networks generalize better than those produced by Net2Net, Forward Thinking, and random layer insertion, at the cost of longer training times due to a sub-discretization parameter needed to accurately solve the forward and adjoint ODEs.

What carries the argument

The dual weighted residual (DWR) method applied to the optimal control formulation of neural network training. The method forms a Lagrangian combining the loss functional and the ODE constraint (forward propagation as a dynamical system), derives adjoint equations solved backward in time, and uses the difference between the true continuous controls and their piecewise linear discretization — weighted by adjoint sensitivities — to produce local, computable error indicators. Galerkin orthogonality (from the first-order optimality condition of the discrete problem) allows replacing the unknown true discretization error with arbitrary finite element functions, making the bound computable.

What would settle it

If one trained networks to approximate training-loss stationarity (e.g., to very small gradient norms) before computing the error estimator, and the resulting layer insertion decisions performed no better than random insertion, the core claim that the error decomposition identifies the right place to add depth would be undermined.

Watch

Extended reading notes

Core claim

The paper's central result is that when a residual network's weights and biases are viewed as piecewise linear approximations to continuous control functions over depth, the functional error of the network decomposes into a sum of interval-wise contributions, each of which can be bounded computationally from the trained network's own parameters and adjoint variables. This decomposition is derived by forming the Lagrangian of the continuous optimal control problem, applying a Taylor expansion with Galerkin orthogonality, and bounding the resulting residual terms with Cauchy-Schwarz and interpolation estimates. The interval-wise bound has two factors per interval: one measuring how far the离散化（

Load-bearing premise

The error estimator in Theorem 1 assumes the network parameters satisfy first-order optimality conditions — i.e., the network is at a local minimum of the training loss. But Algorithm 1 selects networks based on validation loss rather than training-loss convergence, so the parameters are generally not at a local minimum of the training objective when the error estimate is computed. If the parameters are far from stationary, the error bound may be inaccurate, weakening thelink

Editorial extensions

If this is right

If the error decomposition accurately reflects where a network's capacity is strained, depth adaptation becomes a guided refinement process rather than a search, potentially reducing the number of candidate architectures that must be trained.
The framework could extend to width adaptation: if weights are piecewise linear in depth, neuron-level error indicators might analogously identify which neurons in a layer contribute most to approximation error, guiding pruning or growth in width.
The adjoint-based error indicators connect neural network architecture design to mesh adaptivity in PDE solvers, suggesting that other numerical analysis tools (anisotropic refinement, hp-adaptivity) could transfer to network design.
For scientific computing applications where training data is expensive and model accuracy is critical (e.g., surrogate models for inverse problems), the higher training cost of the sub-discretization may be justified by the improved generalization the method demonstrates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The error estimator's reliance on first-order optimality conditions suggests it would be most reliable in regimes where training converges close to a local minimum — e.g., fine-tuning or transfer learning — and less reliable during early training when parameters are far from stationarity. A staged approach where the estimator is applied only after preliminary convergence might improve its practica
The sub-discretization parameter K introduces a trade-off between estimator accuracy and training speed. If K could be adapted dynamically — small during early iterations, larger as the network approaches convergence — the computational overhead might be reduced without sacrificing the quality of the final error estimates.
The piecewise linear assumption on weight variation between layers is a modeling choice; if the true optimal weight function has sharp transitions or discontinuities, higher-order finite element spaces (e.g., piecewise quadratic) might capture the variation with fewer layers, suggesting an alternative architecture where the polynomial degree, not just the mesh, is adapted.
The connection to goal-oriented error estimation in PDEs implies that different loss functions (classification vs. regression vs. physics-informed losses) would produce different adjoint variables and thus different layer insertion decisions, meaning the optimal architecture is inherently task-dependent in a way the framework makes explicit.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this paper applies dual weighted residual (DWR) methodology from finite element analysis to neural network depth adaptation, and the core idea is genuinely new. The theory is clean, the numerical results are promising but small-scale, and there is an unquantified gap between the assumptions behind the error bound and what the algorithm actually does. It deserves a serious referee, but the gap needs to be addressed before publication. Here is the longer take. What is new and good: The synthesis of DWR with neural network training-as-optimal-control is non-trivial and well-executed. Theorem 1 and Corollary 1 give an exact error representation and a computable upper bound that decomposes additively across layer intervals. The proof structure is standard for the DWR literature but the adaptation to the neural network ODE setting is careful. The two-level discretization (coarse mesh for parameters, fine sub-mesh for state/adjoint propagation) is a sensible engineering choice. Numerically, the method beats Net2Net, Forward Thinking, and random insertion on a synthetic regression problem and a Navier-Stokes inverse problem. The error decomposition plots (Figure 4, Figure 9) do show the expected behavior: error drops at the interval where a layer was inserted. That is the right qualitative check. The stress-test concern about optimality conditions is real but somewhat overstated. The concern is that Algorithm 1 uses Adam with validation-based early stopping, so the discrete solution may not satisfy the first-order optimality condition (13) that Galerkin orthogonality requires. This is a legitimate issue. But in practice, Adam with enough epochs gets reasonably close to a local minimum on these small problems, and the DWR estimator is known to be fairly robust to small optimality residuals in the FEM literature. The bigger problem is that the paper does not quantify this. Remark 6 acknowledges the gap and then moves on. A sensitivity analysis showing how the estimator degrades as the optimality residual grows would close the loop. Without it, we cannot tell whether the estimator is reliable or just heuristically useful. A secondary concern: the 'true error' in Figure 3 (right) is computed under the assumption that the continuous optimal control solution achieves near-zero loss, which is unverified. So the agreement between bound and 'true error' is not as strong evidence as it appears. The circularity burden is moderate. The interpolation estimates for the omega terms require second derivatives of the true solution, approximated by finite differences of discrete weights. This is standard in a posteriori estimation but introduces an uncontrolled approximation. The paper is honest about this. No code is released, which limits reproducibility. The computational overhead from the sub-discretization parameter K is significant (46 minutes vs. 5 minutes for random insertion on the synthetic problem), and the experiments are limited to small networks (5-20 neurons per layer). Scalability is an open question. Overall: the paper is for researchers at the intersection of numerical analysis and scientific machine learning. The idea is good, the execution is mostly careful, and the gap between theory and practice is the main thing a referee should push on. I would recommend a conditional accept pending (1) a sensitivity analysis or at least a quantitative discussion of the optimality gap's impact, (2) code release, and (3) at least one experiment at larger scale to show the approach is not limited to toy problems.

Referee Report

3 major / 7 minor

Summary. This paper formulates neural network training as a continuous-time optimal control problem and uses dual weighted residual (DWR) methodology to derive an a posteriori error estimate (Theorem 1, Corollary 1) that decomposes the functional error additively across layer-to-layer intervals. This decomposition drives a depth adaptation algorithm (Algorithm 1) that inserts new layers at intervals of maximum estimated error. The approach is demonstrated on a 2D regression problem and a Navier-Stokes inverse problem, showing improved generalization over Net2Net, Forward Thinking, and random layer insertion.

Significance. The application of DWR-based a posteriori error estimation to neural network depth adaptation is a novel and well-motivated contribution. The piecewise linear weight representation (Section 3) and the two-level discretization scheme (Section 5.2) are clean ideas that bridge the optimal control and deep learning literatures. Theorem 1 and Corollary 1 provide a principled, interval-wise error decomposition that directly addresses the three design questions of depth growing (where, when, how to initialize). The numerical results on the Navier-Stokes inverse problem are a good test of the method on a scientifically relevant task.

major comments (3)

Theorem 1's proof critically depends on Galerkin orthogonality (eq. 28), which requires the discrete first-order optimality condition (13) to hold exactly. In practice, Algorithm 1 trains with Adam and uses validation-based early stopping (lines 2, 9), so the network parameters are not at a local minimum. Remark 6 acknowledges this gap but does not quantify its effect on the estimator. A sensitivity analysis showing how the error bound degrades as the optimality residual grows—or at minimum a discussion of why the estimator remains a useful heuristic when (13) is only approximately satisfied—would substantially strengthen the link between theory and practice. Without this, the central claim that the interval-wise decomposition (22) reliably identifies error concentration is not fully established.
Section 6.1.2 validates the estimated upper bound against the 'true error' |J̃({x_s(t_T)}) - J̃({x^h_s(t_T)})|, but computes the true error under the assumption J̃({x_s(t_T)}) ≈ 0 (i.e., the continuous problem achieves near-zero loss). This assumption is unverified and, if violated, renders the comparison in Figure 3 (right) uninformative. The authors should either verify this assumption (e.g., by training a very deep/wide network to near-zero loss and using it as a reference) or use a different validation strategy that does not depend on it.
The practical error estimator (Section 5.2.3) substitutes the forward Euler discretization (42) for the exact ODE solutions (18) and (21), controlled by the sub-discretization parameter K. The mesh convergence study in Section 6.1.1 chooses K=4 for the initial 3-layer network, but it is unclear whether K=4 remains adequate as the network grows to 17 layers (where interval sizes shrink and stiffness may increase). The paper should justify or verify that the chosen K remains sufficient throughout the adaptation process, not just for the initial network.

minor comments (7)

Table 1: the 'Baseline network' achieves a test loss of 3.82e-5, which is better than the intermediate loss of the proposed approach (2.5e-5 is lower, but the baseline is lower than some intermediate results). The relationship between the proposed approach and the baseline at comparable computational budgets deserves clarification.
Table 2: the improvement over random layer insertion (0.161 vs 0.170) is modest. A discussion of whether this difference is statistically significant (e.g., across multiple random seeds) would strengthen the claim of consistent outperformance.
Section 5.1, eq. (39): the interpolation error bound uses centered differences of discrete weights to approximate second derivatives. For networks with few layers and large intervals, this approximation may be crude. A brief discussion of its reliability would be helpful.
Algorithm 1, line 6: the notation 'arg min' for random layer insertion in Appendix A appears to be a typo; it should likely be 'arg max' with random selection, or the description should be clarified.
The paper states (Section 6.1) that 20 random initializations are tried and the best is retained. It would be useful to report variance across these initializations to assess robustness.
Figure 4: the y-axis scales change across subfigures (from 1e-3 to 1e-5), making cross-iteration comparison difficult. Using a consistent scale or clearly labeling the scale factor would improve readability.
The abstract states the method 'consistently outperforms existing architecture adaptation methods,' but the computational cost is 5-10x higher (Tables 1-2). This trade-off should be acknowledged in the abstract.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for a careful and constructive report. The referee raises three major comments concerning (1) the gap between the exact first-order optimality condition required by Theorem 1 and the approximate optimality achieved by Adam-based training in practice, (2) the unverified assumption that the continuous problem achieves near-zero loss in the validation of the error estimator, and (3) whether the sub-discretization parameter K=4 remains adequate as the network grows to 17 layers. We address each comment below and describe the revisions we will make.

read point-by-point responses

Referee: Theorem 1's proof critically depends on Galerkin orthogonality (eq. 28), which requires the discrete first-order optimality condition (13) to hold exactly. In practice, Algorithm 1 trains with Adam and uses validation-based early stopping, so the network parameters are not at a local minimum. Remark 6 acknowledges this gap but does not quantify its effect on the estimator. A sensitivity analysis or at minimum a discussion of why the estimator remains a useful heuristic when (13) is only approximately satisfied would substantially strengthen the link between theory and practice.

Authors: The referee correctly identifies a genuine gap between the theoretical assumption (exact first-order optimality) and the practical algorithm (Adam with early stopping). We agree that this gap is not currently quantified and that addressing it strengthens the paper. We will make two specific revisions. First, we will add a numerical sensitivity study: after training with Adam to the validation-best checkpoint, we will measure the norm of the Lagrangian gradient (the optimality residual) and report how the estimated error bound changes as we continue training toward a closer approximation of a local minimum (e.g., by switching to L-BFGS or running Adam to convergence). This will show empirically whether the estimator's interval-wise ranking is stable under approximate optimality. Second, we will expand Remark 6 with a discussion of why the estimator remains useful as a heuristic: the DWR estimator's primary role in Algorithm 1 is to rank intervals by relative error magnitude, and this ranking is more robust to optimality residual perturbations than the absolute bound value, because the Galerkin orthogonality term enters additively and uniformly across intervals. We acknowledge that a full theoretical analysis of the approximate-optimality regime is beyond the scope of this paper, and we will state this limitation explicitly. revision: partial
Referee: Section 6.1.2 validates the estimated upper bound against the 'true error' under the assumption that the continuous problem achieves near-zero loss. This assumption is unverified and, if violated, renders the comparison in Figure 3 (right) uninformative. The authors should either verify this assumption or use a different validation strategy.

Authors: The referee is correct that the assumption J_tilde({x_s(t_T)}) ~ 0 is unverified in the current manuscript, and we agree that this weakens the validation in Figure 3 (right). We will address this by adopting a reference-solution strategy that does not depend on the near-zero-loss assumption. Specifically, we will train a very deep and wide network (significantly larger than any network produced during adaptation) to serve as a reference solution approximating the continuous problem (8). The 'true error' will then be computed as the difference between the loss of this reference network and the loss of the adapted network at each iteration. This removes the near-zero-loss assumption entirely. We will also report the reference network's training and validation losses to confirm that it provides a meaningful approximation of the continuous solution. The revised Figure 3 (right) will use this reference-based true error. revision: yes
Referee: The mesh convergence study in Section 6.1.1 chooses K=4 for the initial 3-layer network, but it is unclear whether K=4 remains adequate as the network grows to 17 layers (where interval sizes shrink and stiffness may increase). The paper should justify or verify that the chosen K remains sufficient throughout the adaptation process.

Authors: This is a valid concern. The mesh convergence study in Section 6.1.1 is currently performed only for the initial 3-layer network, and the referee is right that interval sizes shrink and dynamics may stiffen as layers are added. We will address this by repeating the mesh convergence study (computing the relative error indicator I_epsilon for K = 2, 4, 8, 16, 32, and a reference K=1000) at two additional points during the adaptation process: at an intermediate depth (e.g., 10 layers) and at the final depth (17 layers). This will verify whether K=4 remains adequate or whether K should be increased as the network grows. If the results show that K=4 is insufficient at greater depths, we will report the required K values and adjust the algorithm accordingly. We will add these results to Section 6.1.1 and discuss the relationship between network depth, interval size, and the required sub-discretization resolution. revision: yes

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity found; derivation follows standard DWR methodology with non-load-bearing self-citations.

full rationale

The paper's central derivation chain is not circular. Theorem 1 derives an exact error representation (22) by applying a Taylor expansion of the Lagrangian (eq. 24), invoking first-order optimality conditions (13)-(14) for Galerkin orthogonality (eq. 28), and using the trapezoidal rule — all standard steps from the dual weighted residual (DWR) methodology externally cited to Becker-Rannacher [21] and Kraft-Larsson [22]. Corollary 1 then applies triangle inequality and Cauchy-Schwarz to obtain the computable bound (29). The practical computation (Section 5.2.3) substitutes discrete approximations for the true continuous quantities and uses classical FEM interpolation estimates (eq. 39, from external theory) with second derivatives approximated by centered differences of discrete weights — this is standard a posteriori practice, not circular definition. The self-citations [12, 13] appear only in the related-work paragraph (Section 1) as context for the authors' prior depth-growing strategies; they are not invoked anywhere in the derivation of Theorem 1, Corollary 1, or Algorithm 1. The reader's concerns about optimality conditions not being exactly satisfied in practice (Remark 6) and the assumption J̃({x_s(t_T)}) ≈ 0 for computing 'true error' in Figure 3 are correctness and validation concerns, not circularity: the theoretical bound is derived from independently stated assumptions, and the practical algorithm is an approximation whose gap from theory is explicitly acknowledged. The score of 1 reflects the presence of non-load-bearing self-citations in the introduction only.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical entities or particles. It relies on standard mathematical constructs from optimal control and finite element analysis. The free parameters are standard hyperparameters chosen empirically.

free parameters (3)

K (sub-discretization parameter) = 4
Chosen via mesh convergence study in Section 6.1.1 to control state/adjoint discretization error.
n1 (neurons per hidden layer) = 5, 20
Set manually for the proof-of-concept and Navier-Stokes problems respectively.
T (initial hidden layers) = 3
Initial depth of the network, chosen manually.

assumptions (3)

domain assumption The state and adjoint equations (18) and (21) are satisfied exactly for the theoretical derivation.
Invoked in Remark 3 and Remark 5 to isolate parameter discretization error, though in practice a forward Euler scheme is used.
domain assumption The network parameters are at a local minimum satisfying first-order optimality conditions.
Required for Theorem 1, but Remark 6 notes the practical algorithm uses validation loss, meaning this may not hold.
domain assumption The true optimal weights and biases W(t) and b(t) are twice continuously differentiable.
Invoked in Section 3 (Eq. 5) to define the true spaces U_w, U_b and apply interpolation estimates in Section 5.1.

how reviews work

0 comments

Cite this review

Pith. "Pith review of An optimal control approach for neural network architecture adaptation with a posteriori error estimation." pith.science (2026). https://pith.science/paper/4AS2ZKMO

@misc{pith2026260707637,
  author       = {Pith},
  title        = {Pith review of: An optimal control approach for neural network architecture adaptation with a posteriori error estimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4AS2ZKMO}},
  note         = {Machine review of arXiv:2607.07637}
}

read the original abstract

This work presents a novel approach for adapting neural network architecture along the depth based on a posteriori error estimation. By formulating neural network training as a continuous-time optimal control problem, we derive rigorous error estimates that quantify how approximation error distributes across network layers. This error decomposition enables a principled depth adaptation strategy: new layers are inserted at locations of maximum estimated error, allowing the network to efficiently capture complex, nonlinear variations in the underlying problem. Our framework introduces a novel network architecture that treats weights and biases as piecewise linear functions varying across layers, with the error estimator bounding the discrepancy between this discrete representation and the true continuous optimal control solution. The approach leverages dual weighted residual methodology from finite element analysis to derive computable upper bounds on the functional error. A key theoretical contribution is the derivation of explicit error bounds that decompose the total approximation error into interval-wise contributions, providing a rigorous basis for targeted architecture refinement. We demonstrate the effectiveness of our method on scientific datasets, including learning the observable-to-parameter map for the Navier-Stokes equation. Numerical results reveal that our approach consistently outperforms existing architecture adaptation methods in terms of generalization performance.

Figures

Figures reproduced from arXiv: 2607.07637 by the authors.

**Figure 1.** Left to Right: The true space Uw, Ub and the finite element space (Uw)h ,(Ub)h for weights and biases; Decomposition of error (10) across intervals and the adaptation procedure. Remark 2 The goal of our approach is to characterize the error arising from the piecewise linear representation of weights and biases in the neural network. In [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

**Figure 2.** illustrates the two-level discretization scheme: the coarse mesh {t1, . . . , tT } defines the network layers where parameters are stored, while the fine sub-mesh (controlled by parameter K) is used to accurately propagate states and adjoints between layers. Adjoint propagation discretization. Similarly, the adjoint equation (21) is discretized backward in time using an analogous backward Euler scheme to estimate z˜… view at source ↗

**Figure 3.** Left to Right: Mesh convergence study for estimating discretization parameter [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

Figures from the paper (9 more)

**Figure 4.** Figure 4: Decomposition of the a-posteriori error En in (29) at different iterations of the algorithm (x-axis denotes interval number and y-axis denotes the magnitude of error). Block in red denotes the interval of maximum error where a new layer is added for the next training p…

**Figure 5.** Figure 5: Evolution of contours of the solution upon adding new hidden layers in our proposed [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

**Figure 6.** Figure 6: Training loss and testing loss achieved at different iterations of the algorithm. [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

**Figure 7.** Figure 7: Predicted contours by different approaches. Our approach outperformed all the other [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

**Figure 8.** Figure 8: Left to right: Initial vorticity field; Vorticity field at time [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

**Figure 9.** Figure 9: shows the decomposition of error En computed using (29) for the first few iterations of Algorithm 1. We clearly see that upon adding a new hidden layer at the location of maximum error (indicated as a red block in [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

**Figure 10.** Figure 10: Average relative error (48) achieved on the test dataset at different iterations of the algorithm. of our Algorithm 1 [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

**Figure 11.** Figure 11: Left to Right: Predicted vorticity field; and true vorticity field for a randomly chosen test [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]

**Figure 12.** Figure 12: Pointwise average relative error (49) achieved by different approaches on the spatial domain. Our approach outperformed all the other adaptation strategies. 7 Conclusion This work presents a rigorous mathematical framework for neural network architecture adaptation ba…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 27 canonical work pages

[1]

Learning multiple layers of representation.Trends in cognitive sciences, 11(10):428–434, 2007

Geoffrey E Hinton. Learning multiple layers of representation.Trends in cognitive sciences, 11(10):428–434, 2007

work page 2007
[2]

Visualizing and understanding convolutional networks

Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818–833. Springer, 2014

work page 2014
[3]

Very Deep Convolutional Networks for Large-Scale Image Recognition

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition.arXiv preprint arXiv:1409.1556, 2014

work page Pith review arXiv 2014
[4]

Going deeper with convolutions

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015

work page 2015
[5]

A genetic programming approach to designing convolutional neural network architectures

Masanori Suganuma, Shinichi Shirakawa, and Tomoharu Nagao. A genetic programming approach to designing convolutional neural network architectures. InProceedings of the genetic and evolutionary computation conference, pages 497–504, 2017

work page 2017
[6]

A survey on evolutionary neural architecture search.IEEE transactions on neural networks and learning systems, 2021

Yuqiao Liu, Yanan Sun, Bing Xue, Mengjie Zhang, Gary G Yen, and Kay Chen Tan. A survey on evolutionary neural architecture search.IEEE transactions on neural networks and learning systems, 2021

work page 2021
[7]

Neural Architecture Search with Reinforcement Learning

Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning.arXiv preprint arXiv:1611.01578, 2016

work page Pith review arXiv 2016
[8]

Random search and reproducibility for neural architecture search

Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. InUncertainty in artificial intelligence, pages 367–377. PMLR, 2020

work page 2020

Show all 27 references

[9]

Node splitting: A constructive algorithm for feed-forward neural networks

Mike Wynne-Jones. Node splitting: A constructive algorithm for feed-forward neural networks. Advances in neural information processing systems, 4, 1991

1991
[10]

Gradmax: Growing neural networks using gradient information.arXiv preprint arXiv:2201.05125, 2022

Utku Evci, Bart van Merrienboer, Thomas Unterthiner, Max Vladymyrov, and Fabian Pedregosa. Gradmax: Growing neural networks using gradient information.arXiv preprint arXiv:2201.05125, 2022

2022 arXiv
[11]

Firefly neural architecture descent: a general approach for growing neural networks.Advances in neural information processing systems, 33:22373–22383, 2020

Lemeng Wu, Bo Liu, Peter Stone, and Qiang Liu. Firefly neural architecture descent: a general approach for growing neural networks.Advances in neural information processing systems, 33:22373–22383, 2020

2020
[12]

Topological derivative approach for deep neural network architecture adaptation.arXiv preprint arXiv:2502.06885, 2025

CG Krishnanunni, Tan Bui-Thanh, and Clint Dawson. Topological derivative approach for deep neural network architecture adaptation.arXiv preprint arXiv:2502.06885, 2025

2025
[13]

An adaptive and stability-promoting layerwise training approach for sparse deep neural network architecture.Computer Methods in Applied Mechanics and Engineering, 441:117938, 2025

CG Krishnanunni and Tan Bui-Thanh. An adaptive and stability-promoting layerwise training approach for sparse deep neural network architecture.Computer Methods in Applied Mechanics and Engineering, 441:117938, 2025

2025
[14]

Splitting steepest descent for growing neural architectures.Advances in neural information processing systems, 32, 2019

Lemeng Wu, Dilin Wang, and Qiang Liu. Splitting steepest descent for growing neural architectures.Advances in neural information processing systems, 32, 2019

2019
[15]

Greedy layer-wise training of deep networks.Advances in neural information processing systems, 19, 2006

Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks.Advances in neural information processing systems, 19, 2006

2006
[16]

Forward thinking: Building and training neural networks one layer at a time.arXiv preprint arXiv:1706.02480, 2017

Chris Hettinger, Tanner Christensen, Ben Ehlert, Jeffrey Humpherys, Tyler Jarvis, and Sean Wade. Forward thinking: Building and training neural networks one layer at a time.arXiv preprint arXiv:1706.02480, 2017. 21 IOP PublishingJournal(2025) Krishnanunniet al

2017 arXiv
[17]

Net2net: Accelerating learning via knowledge transfer.arXiv preprint arXiv:1511.05641, 2015

Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer.arXiv preprint arXiv:1511.05641, 2015

2015 arXiv
[18]

Sensli: Sensitivity-based layer insertion for neural networks.arXiv preprint arXiv:2311.15995, 2023

Leonie Kreis, Evelyn Herberg, Frederik Köhne, Anton Schiela, and Roland Herzog. Sensli: Sensitivity-based layer insertion for neural networks.arXiv preprint arXiv:2311.15995, 2023

2023 arXiv
[19]

An optimal control approach to deep learning and applications to discrete-weight neural networks

Qianxiao Li and Shuji Hao. An optimal control approach to deep learning and applications to discrete-weight neural networks. InInternational Conference on Machine Learning, pages 2985–2994. PMLR, 2018

2018
[20]

Neural ordinary differential equations.Advances in neural information processing systems, 31, 2018

Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations.Advances in neural information processing systems, 31, 2018

2018
[21]

An optimal control approach to a posteriori error estimation in finite element methods.Acta numerica, 10:1–102, 2001

Roland Becker and Rolf Rannacher. An optimal control approach to a posteriori error estimation in finite element methods.Acta numerica, 10:1–102, 2001

2001
[22]

The dual weighted residuals approach to optimal control of ordinary differential equations.BIT Numerical Mathematics, 50(3):587–607, 2010

Karin Kraft and Stig Larsson. The dual weighted residuals approach to optimal control of ordinary differential equations.BIT Numerical Mathematics, 50(3):587–607, 2010

2010
[23]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

2016
[24]

Deep learning as optimal control problems: Models and numerical methods.arXiv preprint arXiv:1904.05657, 2019

Martin Benning, Elena Celledoni, Matthias J Ehrhardt, Brynjulf Owren, and Carola-Bibiane Schönlieb. Deep learning as optimal control problems: Models and numerical methods.arXiv preprint arXiv:1904.05657, 2019

1904 arXiv
[25]

Explicit-in-time goal-oriented adaptivity.Computer Methods in Applied Mechanics and Engineering, 347:176–200, 2019

Judit Munoz-Matute, Victor M Calo, David Pardo, Elisabete Alberdi, and Kristoffer G van der Zee. Explicit-in-time goal-oriented adaptivity.Computer Methods in Applied Mechanics and Engineering, 347:176–200, 2019

2019
[26]

Numerical solution of the navier-stokes equations.Mathematics of computation, 22(104):745–762, 1968

Alexandre Joel Chorin. Numerical solution of the navier-stokes equations.Mathematics of computation, 22(104):745–762, 1968

1968
[27]

Springer, 2006

Claudio Canuto, M Youssuff Hussaini, Alfio Quarteroni, and Thomas A Zang.Spectral methods: fundamentals in single domains. Springer, 2006. 22

2006

Pith tools

Reviewed July 9, 2026 · model on record in the stance chip above.

[1] [1]

Learning multiple layers of representation.Trends in cognitive sciences, 11(10):428–434, 2007

Geoffrey E Hinton. Learning multiple layers of representation.Trends in cognitive sciences, 11(10):428–434, 2007

work page 2007

[2] [2]

Visualizing and understanding convolutional networks

Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818–833. Springer, 2014

work page 2014

[3] [3]

Very Deep Convolutional Networks for Large-Scale Image Recognition

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition.arXiv preprint arXiv:1409.1556, 2014

work page Pith review arXiv 2014

[4] [4]

Going deeper with convolutions

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015

work page 2015

[5] [5]

A genetic programming approach to designing convolutional neural network architectures

Masanori Suganuma, Shinichi Shirakawa, and Tomoharu Nagao. A genetic programming approach to designing convolutional neural network architectures. InProceedings of the genetic and evolutionary computation conference, pages 497–504, 2017

work page 2017

[6] [6]

A survey on evolutionary neural architecture search.IEEE transactions on neural networks and learning systems, 2021

Yuqiao Liu, Yanan Sun, Bing Xue, Mengjie Zhang, Gary G Yen, and Kay Chen Tan. A survey on evolutionary neural architecture search.IEEE transactions on neural networks and learning systems, 2021

work page 2021

[7] [7]

Neural Architecture Search with Reinforcement Learning

Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning.arXiv preprint arXiv:1611.01578, 2016

work page Pith review arXiv 2016

[8] [8]

Random search and reproducibility for neural architecture search

Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. InUncertainty in artificial intelligence, pages 367–377. PMLR, 2020

work page 2020