{"id":"6f38b71c-9105-4586-a193-4e5877e46595","arxiv_id":"2607.07637","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper derives a posteriori error estimates for neural network depth adaptation by formulating training as an optimal control problem and using dual weighted residuals to insert layers where error is highest.","lead":"This paper uses finite element error estimation to decide where to add new layers to a neural network during training. It offers a principled, math-backed alternative to the trial-and-error usually used to set network depth.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The error bound's validity hinges on exact first-order optimality and exact ODE satisfaction, but Algorithm 1 uses Adam with validation-based early stopping—so the estimator's reliability as an upper bound is unquantified in practice.","rationale":"The reader correctly identifies the most load-bearing concern: the gap between the theoretical assumptions (exact optimality, exact ODE satisfaction) and the practical algorithm (Adam, validation stopping, Euler discretization). This is the right concern, and the authors themselves flag it in Remark 6 without quantifying it. I agree with the reader's assessment. However, I do not think this moves the verdict below CONDITIONAL. The theoretical framework is sound under its stated assumptions, the DWR methodology is well-established in the FEM literature, and the numerical results—while limited in scale—show the estimator tracking the true error and the adaptation strategy outperforming baselines. The concern is about the unquantified gap between theory and practice, which is exactly what a CONDITIONAL verdict captures. The paper would be substantially strengthened by the sensitivity analysis described in the concrete test, but the current evidence is sufficient to support a conditional acceptance. The lack of released code and small-scale experiments are additional limitations, but they are secondary to the main theoretical-practical gap. One additional note: the comparison in Tables 1-2 is not perfectly fair because the proposed architecture (with K substeps) has more effective expressivity per layer than the conventional feedforward baselines, but this does not undermine the central claim about the error estimator's utility for depth adaptation.","tokens_in":21829,"tokens_out":2974,"duration_ms":210004,"concrete_test":"For the 2D regression problem (section 6.1), train the initial 3-layer network to three different levels of optimality: (a) full convergence (gradient norm < 1e-6), (b) typical early stopping as in Algorithm 1, and (c) deliberately undertrained (10% of typical epochs). At each level, compute the error estimator E_n from Corollary 1 and compare against the actual discretization error obtained by solving the continuous problem on a very fine mesh. If the correlation between E_n and the true error degrades significantly from (a) to (c), the practical reliability of the estimator depends on near-optimality, and the gap between theory and algorithm is load-bearing. If the correlation remains strong even at (c), the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Theorem 1's proof requires two sets of first-order optimality conditions to hold exactly: (13) for the discrete problem and (14) for the continuous problem. Galerkin orthogonality (the step replacing W^h with arbitrary Ŵ^h in eq. 28) depends critically on (13) holding exactly. Additionally, the state equation (18) and adjoint equation (21) must be satisfied exactly for the cancellation of residual terms in the proof. In practice, Algorithm 1 trains with Adam, stops based on validation loss (lines 2, 9), and discretizes the ODEs with forward Euler (section 5.2). Remark 6 acknowledges the optimality gap but does not quantify its effect on the estimator. The concern is that if the discrete solution is far from satisfying (13), Galerkin orthogonality fails, the replacement in (28) is invalid, and the interval-wise decomposition in (22) no longer holds—so the layer insertion criterion (argmax of E_n) may not correspond to actual error concentration. The numerical results (Figure 3, right) show the estimated bound tracking the 'true error,' but this true error is computed under the assumption J({x_s(t_T)}) ≈ 0, which is itself unverified. Without a sensitivity analysis showing the estimator degrades gracefully as optimality residual grows, the link between the theoretical bound and the practical algorithm remains unestablished.","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"This paper formulates neural network training as a continuous-time optimal control problem and uses dual weighted residual (DWR) methodology to derive an a posteriori error estimate (Theorem 1, Corollary 1) that decomposes the functional error additively across layer-to-layer intervals. This decomposition drives a depth adaptation algorithm (Algorithm 1) that inserts new layers at intervals of maximum estimated error. The approach is demonstrated on a 2D regression problem and a Navier-Stokes inverse problem, showing improved generalization over Net2Net, Forward Thinking, and random layer insertion.","tokens_in":22052,"tokens_out":2573,"duration_ms":190805,"significance":"The application of DWR-based a posteriori error estimation to neural network depth adaptation is a novel and well-motivated contribution. The piecewise linear weight representation (Section 3) and the two-level discretization scheme (Section 5.2) are clean ideas that bridge the optimal control and deep learning literatures. Theorem 1 and Corollary 1 provide a principled, interval-wise error decomposition that directly addresses the three design questions of depth growing (where, when, how to initialize). The numerical results on the Navier-Stokes inverse problem are a good test of the method on a scientifically relevant task.","major_comments":[{"comment":"Theorem 1's proof critically depends on Galerkin orthogonality (eq. 28), which requires the discrete first-order optimality condition (13) to hold exactly. In practice, Algorithm 1 trains with Adam and uses validation-based early stopping (lines 2, 9), so the network parameters are not at a local minimum. Remark 6 acknowledges this gap but does not quantify its effect on the estimator. A sensitivity analysis showing how the error bound degrades as the optimality residual grows—or at minimum a discussion of why the estimator remains a useful heuristic when (13) is only approximately satisfied—would substantially strengthen the link between theory and practice. Without this, the central claim that the interval-wise decomposition (22) reliably identifies error concentration is not fully established.","section":null},{"comment":"Section 6.1.2 validates the estimated upper bound against the 'true error' |J̃({x_s(t_T)}) - J̃({x^h_s(t_T)})|, but computes the true error under the assumption J̃({x_s(t_T)}) ≈ 0 (i.e., the continuous problem achieves near-zero loss). This assumption is unverified and, if violated, renders the comparison in Figure 3 (right) uninformative. The authors should either verify this assumption (e.g., by training a very deep/wide network to near-zero loss and using it as a reference) or use a different validation strategy that does not depend on it.","section":null},{"comment":"The practical error estimator (Section 5.2.3) substitutes the forward Euler discretization (42) for the exact ODE solutions (18) and (21), controlled by the sub-discretization parameter K. The mesh convergence study in Section 6.1.1 chooses K=4 for the initial 3-layer network, but it is unclear whether K=4 remains adequate as the network grows to 17 layers (where interval sizes shrink and stiffness may increase). The paper should justify or verify that the chosen K remains sufficient throughout the adaptation process, not just for the initial network.","section":null}],"minor_comments":[{"comment":"Table 1: the 'Baseline network' achieves a test loss of 3.82e-5, which is better than the intermediate loss of the proposed approach (2.5e-5 is lower, but the baseline is lower than some intermediate results). The relationship between the proposed approach and the baseline at comparable computational budgets deserves clarification.","section":null},{"comment":"Table 2: the improvement over random layer insertion (0.161 vs 0.170) is modest. A discussion of whether this difference is statistically significant (e.g., across multiple random seeds) would strengthen the claim of consistent outperformance.","section":null},{"comment":"Section 5.1, eq. (39): the interpolation error bound uses centered differences of discrete weights to approximate second derivatives. For networks with few layers and large intervals, this approximation may be crude. A brief discussion of its reliability would be helpful.","section":null},{"comment":"Algorithm 1, line 6: the notation 'arg min' for random layer insertion in Appendix A appears to be a typo; it should likely be 'arg max' with random selection, or the description should be clarified.","section":null},{"comment":"The paper states (Section 6.1) that 20 random initializations are tried and the best is retained. It would be useful to report variance across these initializations to assess robustness.","section":null},{"comment":"Figure 4: the y-axis scales change across subfigures (from 1e-3 to 1e-5), making cross-iteration comparison difficult. Using a consistent scale or clearly labeling the scale factor would improve readability.","section":null},{"comment":"The abstract states the method 'consistently outperforms existing architecture adaptation methods,' but the computational cost is 5-10x higher (Tables 1-2). This trade-off should be acknowledged in the abstract.","section":null}],"recommendation":"major_revision","confidential_remarks":"The reader's concern about the optimality gap is the most substantive issue. The theory is clean under its assumptions, but the paper currently leaves the theory-practice gap as an acknowledged but unquantified limitation. If the authors can provide even a modest sensitivity analysis or a more careful validation of the estimator's reliability under approximate optimality, this would likely warrant acceptance. The concern about the J≈0 assumption for computing 'true error' is also important and should be straightforward to address. The paper is a good fit for the journal's scope at the intersection of numerical methods and machine learning."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive report. The referee raises three major comments concerning (1) the gap between the exact first-order optimality condition required by Theorem 1 and the approximate optimality achieved by Adam-based training in practice, (2) the unverified assumption that the continuous problem achieves near-zero loss in the validation of the error estimator, and (3) whether the sub-discretization parameter K=4 remains adequate as the network grows to 17 layers. We address each comment below and describe the revisions we will make.","responses":[{"response":"The referee correctly identifies a genuine gap between the theoretical assumption (exact first-order optimality) and the practical algorithm (Adam with early stopping). We agree that this gap is not currently quantified and that addressing it strengthens the paper. We will make two specific revisions. First, we will add a numerical sensitivity study: after training with Adam to the validation-best checkpoint, we will measure the norm of the Lagrangian gradient (the optimality residual) and report how the estimated error bound changes as we continue training toward a closer approximation of a local minimum (e.g., by switching to L-BFGS or running Adam to convergence). This will show empirically whether the estimator's interval-wise ranking is stable under approximate optimality. Second, we will expand Remark 6 with a discussion of why the estimator remains useful as a heuristic: the DWR estimator's primary role in Algorithm 1 is to rank intervals by relative error magnitude, and this ranking is more robust to optimality residual perturbations than the absolute bound value, because the Galerkin orthogonality term enters additively and uniformly across intervals. We acknowledge that a full theoretical analysis of the approximate-optimality regime is beyond the scope of this paper, and we will state this limitation explicitly.","revision_made":"partial","referee_comment":"Theorem 1's proof critically depends on Galerkin orthogonality (eq. 28), which requires the discrete first-order optimality condition (13) to hold exactly. In practice, Algorithm 1 trains with Adam and uses validation-based early stopping, so the network parameters are not at a local minimum. Remark 6 acknowledges this gap but does not quantify its effect on the estimator. A sensitivity analysis or at minimum a discussion of why the estimator remains a useful heuristic when (13) is only approximately satisfied would substantially strengthen the link between theory and practice."},{"response":"The referee is correct that the assumption J_tilde({x_s(t_T)}) ~ 0 is unverified in the current manuscript, and we agree that this weakens the validation in Figure 3 (right). We will address this by adopting a reference-solution strategy that does not depend on the near-zero-loss assumption. Specifically, we will train a very deep and wide network (significantly larger than any network produced during adaptation) to serve as a reference solution approximating the continuous problem (8). The 'true error' will then be computed as the difference between the loss of this reference network and the loss of the adapted network at each iteration. This removes the near-zero-loss assumption entirely. We will also report the reference network's training and validation losses to confirm that it provides a meaningful approximation of the continuous solution. The revised Figure 3 (right) will use this reference-based true error.","revision_made":"yes","referee_comment":"Section 6.1.2 validates the estimated upper bound against the 'true error' under the assumption that the continuous problem achieves near-zero loss. This assumption is unverified and, if violated, renders the comparison in Figure 3 (right) uninformative. The authors should either verify this assumption or use a different validation strategy."},{"response":"This is a valid concern. The mesh convergence study in Section 6.1.1 is currently performed only for the initial 3-layer network, and the referee is right that interval sizes shrink and dynamics may stiffen as layers are added. We will address this by repeating the mesh convergence study (computing the relative error indicator I_epsilon for K = 2, 4, 8, 16, 32, and a reference K=1000) at two additional points during the adaptation process: at an intermediate depth (e.g., 10 layers) and at the final depth (17 layers). This will verify whether K=4 remains adequate or whether K should be increased as the network grows. If the results show that K=4 is insufficient at greater depths, we will report the required K values and adjust the algorithm accordingly. We will add these results to Section 6.1.1 and discuss the relationship between network depth, interval size, and the required sub-discretization resolution.","revision_made":"yes","referee_comment":"The mesh convergence study in Section 6.1.1 chooses K=4 for the initial 3-layer network, but it is unclear whether K=4 remains adequate as the network grows to 17 layers (where interval sizes shrink and stiffness may increase). The paper should justify or verify that the chosen K remains sufficient throughout the adaptation process."}],"tokens_in":21882,"tokens_out":1070,"duration_ms":173313,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"Short version: this paper applies dual weighted residual (DWR) methodology from finite element analysis to neural network depth adaptation, and the core idea is genuinely new. The theory is clean, the numerical results are promising but small-scale, and there is an unquantified gap between the assumptions behind the error bound and what the algorithm actually does. It deserves a serious referee, but the gap needs to be addressed before publication. Here is the longer take. What is new and good: The synthesis of DWR with neural network training-as-optimal-control is non-trivial and well-executed. Theorem 1 and Corollary 1 give an exact error representation and a computable upper bound that decomposes additively across layer intervals. The proof structure is standard for the DWR literature but the adaptation to the neural network ODE setting is careful. The two-level discretization (coarse mesh for parameters, fine sub-mesh for state/adjoint propagation) is a sensible engineering choice. Numerically, the method beats Net2Net, Forward Thinking, and random insertion on a synthetic regression problem and a Navier-Stokes inverse problem. The error decomposition plots (Figure 4, Figure 9) do show the expected behavior: error drops at the interval where a layer was inserted. That is the right qualitative check. The stress-test concern about optimality conditions is real but somewhat overstated. The concern is that Algorithm 1 uses Adam with validation-based early stopping, so the discrete solution may not satisfy the first-order optimality condition (13) that Galerkin orthogonality requires. This is a legitimate issue. But in practice, Adam with enough epochs gets reasonably close to a local minimum on these small problems, and the DWR estimator is known to be fairly robust to small optimality residuals in the FEM literature. The bigger problem is that the paper does not quantify this. Remark 6 acknowledges the gap and then moves on. A sensitivity analysis showing how the estimator degrades as the optimality residual grows would close the loop. Without it, we cannot tell whether the estimator is reliable or just heuristically useful. A secondary concern: the 'true error' in Figure 3 (right) is computed under the assumption that the continuous optimal control solution achieves near-zero loss, which is unverified. So the agreement between bound and 'true error' is not as strong evidence as it appears. The circularity burden is moderate. The interpolation estimates for the omega terms require second derivatives of the true solution, approximated by finite differences of discrete weights. This is standard in a posteriori estimation but introduces an uncontrolled approximation. The paper is honest about this. No code is released, which limits reproducibility. The computational overhead from the sub-discretization parameter K is significant (46 minutes vs. 5 minutes for random insertion on the synthetic problem), and the experiments are limited to small networks (5-20 neurons per layer). Scalability is an open question. Overall: the paper is for researchers at the intersection of numerical analysis and scientific machine learning. The idea is good, the execution is mostly careful, and the gap between theory and practice is the main thing a referee should push on. I would recommend a conditional accept pending (1) a sensitivity analysis or at least a quantitative discussion of the optimality gap's impact, (2) code release, and (3) at least one experiment at larger scale to show the approach is not limited to toy problems.","headline":"DWR-based a posteriori error estimation for neural network depth adaptation: novel synthesis with a real theory-practice gap","tokens_in":22761,"tokens_out":768,"would_cite":true,"duration_ms":292043,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Error estimates from optimal control guide where to add neural network layers","keywords":["neural architecture adaptation","a posteriori error estimation","optimal control","dual weighted residual","depth growing","finite element method","adjoint equations"],"falsifier":"If one trained networks to approximate training-loss stationarity (e.g., to very small gradient norms) before computing the error estimator, and the resulting layer insertion decisions performed no better than random insertion, the core claim that the error decomposition identifies the right place to add depth would be undermined.","tokens_in":21941,"feed_emoji":"📐","tokens_out":1680,"duration_ms":109621,"temperature":0.7,"pith_summary":"This paper claims that neural network training can be formulated as a continuous-time optimal control problem in which weights and biases are functions of a continuous depth variable, and that the error introduced by discretizing these functions into piecewise linear segments between layers can be exactly expressed and bounded in a way that decomposes additively across each layer-to-layer interval. The central object is an a posteriori error estimator derived via the dual weighted residual method from finite element analysis: it takes a trained network and produces, for each interval between consecutive layers, a computable quantity measuring how much that interval contributes to the total approximation error. The paper shows that this error decomposition directly answers the three questions any depth-growing scheme must resolve — where to insert a new layer (at the interval of maximum estimated error), when to stop (when validation loss stops improving), and how to initialize the new layer (by interpolating its neighbors). The authors prove Theorem 1, which gives the exact error expression, and Corollary 1, which converts it into a fully computable upper bound using interpolation error estimates and supremum norms of adjoint variables. Algorithm 1 implements the full cycle: train, estimate per-interval errors, insert a layer at the worst interval, retrain, and repeat. On a synthetic regression problem and a severely ill-posed inverse problem for the Navier-Stokes equations, the resulting networks generalize better than those produced by Net2Net, Forward Thinking, and random layer insertion, at the cost of longer training times due to a sub-discretization parameter needed to accurately solve the forward and adjoint ODEs.","feed_headline":"Optimal control error estimates tell neural networks where to add depth","feed_subtitle":"By treating layer weights as piecewise linear functions and applying finite element error bounds, the method identifies which layer gaps add","key_machinery":"The dual weighted residual (DWR) method applied to the optimal control formulation of neural network training. The method forms a Lagrangian combining the loss functional and the ODE constraint (forward propagation as a dynamical system), derives adjoint equations solved backward in time, and uses the difference between the true continuous controls and their piecewise linear discretization — weighted by adjoint sensitivities — to produce local, computable error indicators. Galerkin orthogonality (from the first-order optimality condition of the discrete problem) allows replacing the unknown true discretization error with arbitrary finite element functions, making the bound computable.","core_discovery":"The paper's central result is that when a residual network's weights and biases are viewed as piecewise linear approximations to continuous control functions over depth, the functional error of the network decomposes into a sum of interval-wise contributions, each of which can be bounded computationally from the trained network's own parameters and adjoint variables. This decomposition is derived by forming the Lagrangian of the continuous optimal control problem, applying a Taylor expansion with Galerkin orthogonality, and bounding the resulting residual terms with Cauchy-Schwarz and interpolation estimates. The interval-wise bound has two factors per interval: one measuring how far the离散化（","pith_inferences":["The error estimator's reliance on first-order optimality conditions suggests it would be most reliable in regimes where training converges close to a local minimum — e.g., fine-tuning or transfer learning — and less reliable during early training when parameters are far from stationarity. A staged approach where the estimator is applied only after preliminary convergence might improve its practica","The sub-discretization parameter K introduces a trade-off between estimator accuracy and training speed. If K could be adapted dynamically — small during early iterations, larger as the network approaches convergence — the computational overhead might be reduced without sacrificing the quality of the final error estimates.","The piecewise linear assumption on weight variation between layers is a modeling choice; if the true optimal weight function has sharp transitions or discontinuities, higher-order finite element spaces (e.g., piecewise quadratic) might capture the variation with fewer layers, suggesting an alternative architecture where the polynomial degree, not just the mesh, is adapted.","The connection to goal-oriented error estimation in PDEs implies that different loss functions (classification vs. regression vs. physics-informed losses) would produce different adjoint variables and thus different layer insertion decisions, meaning the optimal architecture is inherently task-dependent in a way the framework makes explicit."],"forward_implications":["If the error decomposition accurately reflects where a network's capacity is strained, depth adaptation becomes a guided refinement process rather than a search, potentially reducing the number of candidate architectures that must be trained.","The framework could extend to width adaptation: if weights are piecewise linear in depth, neuron-level error indicators might analogously identify which neurons in a layer contribute most to approximation error, guiding pruning or growth in width.","The adjoint-based error indicators connect neural network architecture design to mesh adaptivity in PDE solvers, suggesting that other numerical analysis tools (anisotropic refinement, hp-adaptivity) could transfer to network design.","For scientific computing applications where training data is expensive and model accuracy is critical (e.g., surrogate models for inverse problems), the higher training cost of the sub-discretization may be justified by the improved generalization the method demonstrates."],"fun_headline_variants":["Error bounds guide targeted layer insertion in neural networks","Optimal control pinpoints where neural networks need more depth","Dual weighted residuals show neural networks where to add layers","A posteriori error estimates direct neural network depth adaptation","Bounded error decomposition guides neural network architecture growth"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The error estimator in Theorem 1 assumes the network parameters satisfy first-order optimality conditions — i.e., the network is at a local minimum of the training loss. But Algorithm 1 selects networks based on validation loss rather than training-loss convergence, so the parameters are generally not at a local minimum of the training objective when the error estimate is computed. If the parameters are far from stationary, the error bound may be inaccurate, weakening thelink","fun_headline_variants_meta":{"raw":{"variants":["Error bounds guide targeted layer insertion in neural networks","Optimal control pinpoints where neural networks need more depth","Dual weighted residuals show neural networks where to add layers","A posteriori error estimates direct neural network depth adaptation","Bounded error decomposition guides neural network architecture growth","Finite element error bounds steer neural network depth expansion"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":907,"prompt_tokens":521,"completion_tokens":386,"prompt_tokens_details":null},"tokens_in":521,"tokens_out":386,"duration_ms":26315,"temperature":1.0,"reasoning_tokens":384,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T04:12:31.204871+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If one trained networks to approximate training-loss stationarity (e.g., to very small gradient norms) before computing the error estimator, and the resulting layer insertion decisions performed no better than random insertion, the core claim that the error decomposition identifies the right place to add depth would be undermined.","supporting_citations":[],"review_version":1}