REVIEW 5 major objections 5 minor 27 references
State-dependent preconditioning for the inner-loop in Variational Data Assimilation using Machine Learning
T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A neural network learns approximate eigenvectors of the 4D-Var Hessian and uses them to precondition conjugate-gradient solves; in a shallow-water test this cuts inner-loop iterations by roughly 30 percent.
desk verdict A sound new idea for state-dependent ML preconditioning in 4D-Var, but the numerical evidence is too thin to support the 30% claim until held-out and baseline comparisons are added. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the spectral (limited-memory) preconditioner $P_\alpha = \beta I_n + U_r(\mu\Lambda_r^\alpha - \beta I_r)U_r^T$, used with $\alpha=-1/2$ and $\beta=1$ so that the preconditioned matrix has $r$ eigenvalues near $\mu$ and $n-r$ eigenvalues effectively unchanged. The DNN supplies the approximate eigenpairs: a U-Net with transformer blocks maps the state reshaped as a $64\times64\times3$ image to raw vectors and values, QR enforces orthonormality, and a scaled sigmoid enforces positivity. Training minimizes a randomized estimator of the Frobenius norm of the reconstruction error, so only applications of $A_x$ to Gaussian random vectors are required, making the dataset cheap to construct online. The identity carrying the argument is the Eckart\,--\,Young\,--\,Mirsky theorem: the rank-$r$ SVD is the best Frobenius-norm low-rank approximation, so minimizing the reconstruction error is the right training objective.
What would settle it
Take a set of linearization states disjoint from the training set; at each, compare the CG iteration count with and without the DNN preconditioner. If the roughly 30% iteration reduction does not appear at those held-out states, or if comparing the learned eigenpairs with a direct iterative eigensolver shows the preconditioned condition number is no better than the original, the central claim is falsified.
Extended reading notes
Core claim
The central claim is that a state-dependent spectral preconditioner can be learned from data and used inside the inner loop of incremental 4D-Var. For a linearization state $x$, the network outputs a rank-$r$ approximation $A_\theta(x)=U_\theta\Lambda_\theta U_\theta^T$ of $A_x=G_x^T R^{-1}G_x+B^{-1}$, and the split preconditioner $L_\theta(x)=I+U_\theta(\mu\Lambda_\theta^{-1/2}-I)U_\theta^T$ clusters the $r$ leading eigenvalues around $\mu$. Because the network is trained by minimizing the Frobenius norm of $A_\theta(x)-A_x$ estimated via random projections, neither the matrix nor its SVD ever has to be formed. In the paper's shallow-water numerical experiment, the best combinations of rank $r$ and shift $\mu$ reduce the CG iterations to a residual threshold by about 30% compared with unpreconditioned CG; retaining $r=2000$ approximate eigenpairs, where the smallest learned eigenvalues are poorly represented, can instead degrade performance. The paper attributes this degradation to errors in small eigenpairs being amplified by the inverse eigenvalues in the preconditioner.
Load-bearing premise
The load-bearing premise is that the trained network's approximate eigenvectors and eigenvalues stay accurate enough on new linearization states to improve the conditioning of $A_x$; the paper does not quantify this generalization error, and its own $r=2000$ results show that when small learned eigenvalues are inaccurate the preconditioner degrades performance.
Editorial extensions
If this is right
- In the tested shallow-water 4D-Var system, good choices of $r$ and $\mu$ reduce the number of CG iterations needed to reach the residual threshold by roughly 30% compared with unpreconditioned CG.
- The method needs only matrix-vector products with $A_x$ for training and with $L_\theta$ for application, so it remains usable when the Gauss-Newton matrix is available only as an operator.
- Because the network is trained once and then evaluated at any linearization state, the learned preconditioner can serve as a first-level preconditioner and be combined with second-level randomized preconditioners.
- Retaining too many approximate eigenpairs can worsen convergence, so the rank $r$ and shift $\mu$ must be chosen conservatively rather than set to the full learned spectrum.
- The training loss can be normalized by the estimated norm of $A_x$, giving an 'unexplained variance' interpretation that parallels principal component analysis.
Reading between the lines
- Beyond the paper, the reported 30% gain comes from a single moderate-size test case; transfer to larger or operational systems would require a measure of how the learned eigenpairs generalize to linearization states not seen during training, which the paper does not quantify.
- Beyond the paper, the observed sensitivity to small learned eigenvalues suggests a natural extension: weight the training loss toward relative eigenvalue accuracy, or adaptively adjust the lower bound on $\mu$, instead of relying on one scalar shift parameter.
- Beyond the paper, the online dataset construction opens the possibility of a self-tuning loop that periodically retrains or fine-tunes the network on newly encountered linearization states, allowing the preconditioner to track slow changes in the model or observation operator.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a machine-learned, state-dependent spectral preconditioner for the inner-loop conjugate gradient (CG) solve in incremental 4D-Var data assimilation. The method trains a deep neural network to map the linearization state x to approximate leading eigenpairs of the Gauss-Newton matrix A_x, using a Hutchinson-type Frobenius-norm reconstruction loss. A spectral preconditioner L_theta(x) is then built from these approximate eigenpairs via Eq. (2.4). The method is tested on a 64x64 shallow-water system (state dimension 12160), and the paper reports that for suitable choices of rank r and shift parameter mu, the preconditioner reduces the number of CG iterations to reach a residual threshold of 1e-7 by roughly 30% relative to unpreconditioned CG (Section 3.4, Figure 5). The paper also discusses sensitivity to r and mu, noting that retaining too many poorly estimated eigenpairs degrades performance.
Significance. If the empirical claim is robust, the paper offers a practical, non-intrusive approach to preconditioning non-sparse, state-dependent linear systems in variational data assimilation, where traditional preconditioners rely on sparsity or explicit spectral information. The use of a randomized Frobenius-norm loss and an online dataset construction is sensible and potentially useful. However, the current evidence is too thin to support the central claim: it rests on a single run with no error bars, no held-out evaluation of the DNN's generalization, hyperparameters selected after observing the test result, and no comparison to standard DA preconditioners. These gaps are load-bearing because the method's value is entirely empirical. The theoretical derivation also contains a misstatement about the square root of the spectral preconditioner that should be corrected.
major comments (5)
- [Section 3.4, Figure 5] The central claim of roughly 30% iteration reduction is based on a single assimilation experiment. There are no repeated trials, error bars, or statistical significance assessment. Given the randomized elements in the experiment (truth perturbation, observation noise, initial state, and the random vectors used in the loss estimate), the observed improvement could be within run-to-run variability. Please run multiple independent experiments with different random seeds and report the mean/median and spread of iteration counts for both the preconditioned and unpreconditioned cases.
- [Sections 3.3 and 3.4, Algorithm 3.1] The manuscript does not provide any held-out evaluation of the DNN's generalization. The network is trained on N_training=1000 states from model trajectories, but the preconditioner is applied to states encountered along the assimilation path, which are not guaranteed to lie in the training distribution and which evolve as the outer loop updates x_i. No reconstruction loss, eigenpair error, or conditioning improvement is reported on such held-out states. This is a load-bearing gap because the method's success depends on accurate approximate eigenpairs at new linearization states; the paper itself shows (r=2000 case) that inaccurate small eigenvalues degrade performance.
- [Section 3.4] The hyperparameters r and mu are selected after observing the test result, as indicated by the phrase 'For good combinations of the parameters r and µ'. This risks overfitting to the single test case. The authors should specify a model-selection procedure using a separate validation set (or cross-validation) and then report the performance of the chosen configuration on one or more independent test experiments.
- [Sections 1.5 and 3.4] The comparison is only against unpreconditioned CG. In variational data assimilation, standard baselines include B^{-1/2} preconditioning (Eq. 1.17) and limited-memory spectral preconditioners of the type studied in Tshimanga et al. (2008), which are directly related to the proposed P_alpha. Without a comparison to at least one such existing method, the practical significance of the 30% improvement over unpreconditioned CG is unclear.
- [Section 1.6 and Eq. (2.4)] The statement that 'P_{\alpha/2} is a matrix square root of P_\alpha' is incorrect for the parameterization in Eq. (1.21) when \mu is not equal to 1. With L defined in Eq. (2.4) as P_{-1/2}, one obtains L^T A L = A P_{-1} with parameter \mu^2 (assuming A and P share eigenvectors), not A P_{-1/2}. The effective exponent in the numerical experiments is therefore -1, not -1/2 as claimed in the text preceding Eq. (2.4). This misstatement affects the theoretical motivation and should be corrected; the preconditioner formula itself can be kept, but the derivation and the description of the effective spectrum need revision.
minor comments (5)
- [Algorithm 1.1] The inner-loop condition 'while j ≤ ninner or ∥rj∥2 < ϵ' appears to be intended as 'while j ≤ ninner and ∥rj∥2 > ϵ'; as written, the loop would continue even after the residual is below the threshold, contradicting the stopping criterion described in Section 3.4.
- [Section 3.2 and 3.3] The description of the U-Net/transformer architecture and the training setup is too sparse for reproducibility. Please report the number of layers, parameter counts, optimizer, learning rate, batch size, number of epochs, and the way the training states are generated (e.g., distribution of lead times and perturbation amplitudes).
- [Section 1.2, near Eq. (1.9)] The definition 'Gx = HMxMx' is garbled; it should read G_x = H_x M_x (or similar). Please clarify the notation for the Jacobian of the generalized observation operator.
- [Section 3.4, Figure 5] The y-axis label 'Number of iterations needed to reach the norm threshold' should explicitly state that the norm is the Euclidean norm of the residual and that the stopping criterion is the threshold 10^{-7} or 2000 iterations, whichever comes first; this is clear in the text but should also be in the figure caption.
- [References] Some references lack complete bibliographic information (e.g., Häusner et al. 2023, Diouane et al. 2024, and several arXiv preprints). Please provide arXiv identifiers or journal/venue details where available.
Circularity Check
No significant circularity: the reported iteration reduction is an empirical measurement and is not enforced by the training objective or by a self-citation chain.
full rationale
The paper's central claim is experimental: a DNN trained to approximate the leading eigenpairs of the Gauss-Newton matrix A_x is used to build a spectral preconditioner, and the number of conjugate-gradient iterations to reach a residual threshold is measured against unpreconditioned CG. The training loss in Eq. (2.10) minimizes a Monte-Carlo estimate of the Frobenius reconstruction error between A_theta(x) and A_x; it contains no term involving the CG iteration count or the final residual threshold. Therefore the roughly 30% reduction reported in Section 3.4 is not forced by construction: the network could have minimized the reconstruction objective while failing to reduce iterations, as the paper itself observes for r=2000, where poorly approximated small eigenvalues degrade performance. The preconditioner construction in Eqs. (1.21) and (2.4) follows standard spectral and limited-memory preconditioning mathematics, and the convergence bound in Eq. (1.14) is the classical CG bound. No load-bearing argument relies on the authors' own prior work: the cited spectral preconditioners, matrix-norm estimators, and Eckart-Young-Mirsky theorem are external results, and the authors do not cite themselves to justify the method. The absence of a train/test split and the selection of favorable values of r and mu are legitimate statistical-generalization concerns, but they are not circularity: the paper does not rename a fitted parameter as a prediction, nor does any equation reduce to its own input. The finding is no significant circularity.
Assumptions & free parameters
free parameters (4)
- DNN parameters theta =
trained on 1000 states
- shift parameter mu =
chosen in [min_i lambda_i, max_i lambda_i]
- retained rank r =
1000, 1900, 2000 tested
- eigenvalue bound M (sigmoid scale) =
rough upper bound of singular values
assumptions (5)
- standard math Eckart-Young-Mirsky theorem characterizes optimal low-rank approximation in Frobenius norm
- domain assumption Gaussian assumptions for observation and background errors (Eq. 1.5-1.6)
- domain assumption Tangent linear model and adjoint are available
- domain assumption The spectrum of A_x decays fast enough for low-rank approximation
- ad hoc to paper The DNN generalizes from training states to assimilation states
Cite this review
Pith. "Pith review of State-dependent preconditioning for the inner-loop in Variational Data Assimilation using Machine Learning." pith.science (2026). https://pith.science/paper/YIYJ5SZ4
@misc{pith2026250104369,
author = {Pith},
title = {Pith review of: State-dependent preconditioning for the inner-loop in Variational Data Assimilation using Machine Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/YIYJ5SZ4}},
note = {Machine review of arXiv:2501.04369}
}
read the original abstract
Data Assimilation is the process in which we improve the representation of the state of a physical system by combining information coming from a numerical model, real-world observations, and some prior modelling. It is widely used to model and to improve forecast systems in Earth science fields such as meteorology, oceanography and environmental sciences. One key aspect of Data assimilation is the analysis step, where the output of the numerical model is adjusted in order to account for the observational data. In Variational Data Assimilation and under Gaussian assumptions, the analysis step comes down to solving a high-dimensional non-linear least-square problem. In practice, this minimization involves successive inversions of large, and possibly ill-conditioned matrices constructed using linearizations of the forward model. In order to improve the convergence rate of these methods, and thus reduce the computational burden, preconditioning techniques are often used to get better-conditioned matrices, but require either the sparsity pattern of the matrix to inverse, or some spectral information. We propose to use Deep Neural Networks in order to construct a preconditioner. This surrogate is trained using some properties of the singular value decomposition, and is based on a dataset which can be constructed online to reduce the storage requirements.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Ackmann, J., D \"u ben, P., Palmer, T., and Smolarkiewicz, P. (2021). Machine- Learned Preconditioners for Linear Solvers in Geophysical Fluid Flows . In EGU General Assembly Conference , pages EGU21--5507
work page 2021
-
[2]
Arcucci, R., Zhu, J., Hu, S., and Guo, Y.-K. (2021). Deep Data Assimilation : Integrating Deep Learning with Data Assimilation . Applied Sciences , 11(3):1114
work page 2021
-
[3]
Benner, P., Qiu, Y., and Stoll, M. (2018). Low-rank computation of posterior covariance matrices in Bayesian inverse problems. SIAM/ASA Journal on Uncertainty Quantification , 6(2):965--989
work page 2018
-
[4]
Bonavita, M., Lean, P., and Holm, E. (2018). Nonlinear effects in 4D-Var . Nonlinear Processes in Geophysics , 25(3):713--729
work page 2018
-
[5]
Boudier, P., Fillion, A., Gratton, S., and G \"u rol, S. (2020). DAN -- An optimal Data Assimilation framework based on machine learning Recurrent Networks . arXiv:2010.09694 [cs, eess]
work page Pith review arXiv 2020
-
[6]
Cheng, S., Quilodran-Casas , C., Ouala, S., Farchi, A., Liu, C., Tandeo, P., Fablet, R., Lucor, D., Iooss, B., Brajard, J., Xiao, D., Janjic, T., Ding, W., Guo, Y., Carrassi, A., Bocquet, M., and Arcucci, R. (2023). Machine learning with data assimilation and uncertainty quantification for dynamical systems: A review
work page 2023
-
[7]
Dau z ickait \.e , I., Lawless, A. S., Scott, J. A., and van Leeuwen , P. J. (2021). Randomised preconditioning for the forcing formulation of weak constraint 4D-Var . Quarterly Journal of the Royal Meteorological Society , 147(740):3719--3734
work page 2021
-
[8]
Diouane, Y., G \"u rol, S., Mouhtal, O., and Orban, D. (2024). An Efficient Scaled spectral preconditioner for sequences of symmetric positive definite linear systems. (arXiv:2410.02204)
work page Pith review arXiv 2024
Show all 27 references
-
[9]
Dubois, P., Gomez, T., Planckaert, L., and Perret, L. (2020). Data-driven predictions of the Lorenz system. Physica D: Nonlinear Phenomena , 408:132495
2020
-
[10]
Freitag, M. A. (2020). Numerical linear algebra in data assimilation. GAMM-Mitteilungen , 43(3):e202000014
2020
-
[11]
Gottwald, G. A. and Reich, S. (2021). Combining machine learning and data assimilation to forecast dynamical systems from noisy partial observations. Chaos: An Interdisciplinary Journal of Nonlinear Science , 31(10):101103
2021
-
[12]
S., and Nichols, N
Gratton, S., Lawless, A. S., and Nichols, N. K. (2007). Approximate Gauss -- Newton Methods for Nonlinear Least Squares Problems . SIAM Journal on Optimization , 18(1):106--132
2007
-
[13]
and Titley-Peloquin , D
Gratton, S. and Titley-Peloquin , D. (2018). Improved Bounds for Small-Sample Estimation . SIAM Journal on Matrix Analysis and Applications , 39(2):922--931
2018
-
[14]
Grote, M. J. and Huckle, T. (1997). Parallel Preconditioning with Sparse Approximate Inverses . SIAM Journal on Scientific Computing , 18(3):838--853
1997
-
[15]
S., and Laub, A
Gudmundsson, T., Kenney, C. S., and Laub, A. J. (1995). Small- Sample Statistical Estimates for Matrix Norms . SIAM Journal on Matrix Analysis and Applications , 16(3):17
1995
-
[16]
T., Moore, A
G \"u rol, S., Weaver, A. T., Moore, A. M., Piacentini, A., Arango, H. G., and Gratton, S. (2014). B -preconditioned minimization algorithms for variational data assimilation with the dual formulation: B -preconditioned minimization algorithms. Quarterly Journal of the Royal M...
2014
-
[17]
Haben, S., Lawless, A., and Nichols, N. (2011). Conditioning and preconditioning of the variational data assimilation problem. Computers & Fluids , 46(1):252--256
2011
-
[18]
a usner, P., \
H \"a usner, P., \"O ktem, O., and Sj \"o lund, J. (2023). Neural incomplete factorization: Learning preconditioners for the conjugate gradient method
2023
-
[19]
Indyk, P., Vakilian, A., and Yuan, Y. (2019). Learning- Based Low-Rank Approximations
2019
-
[20]
Luna, K., Klymko, K., and Blaschke, J. P. (2021). Accelerating GMRES with Deep Learning in Real-Time
2021
-
[21]
and Aulign \'e , T
M \'e n \'e trier, B. and Aulign \'e , T. (2015). An Overlooked Issue of Variational Data Assimilation . Monthly Weather Review , 143(10):3925--3930
2015
-
[22]
Peyron, M., Fillion, A., G \"u rol, S., Marchais, V., Gratton, S., Boudier, P., and Goret, G. (2021). Latent Space Data Assimilation by using Deep Learning . arXiv:2104.00430 [cs, math]
2021 arXiv
-
[23]
Sappl, J., Seiler, L., Harders, M., and Rauch, W. (2019). Deep Learning of Preconditioners for Conjugate Gradient Solvers in Urban Water Related Problems
2019
-
[24]
Spantini, A., Solonen, A., Cui, T., Martin, J., Tenorio, L., and Marzouk, Y. (2015). Optimal low-rank approximations of Bayesian linear inverse problems. arXiv:1407.3463 [math, stat]
2015 arXiv
-
[25]
M., Dance, S
Tabeart, J. M., Dance, S. L., Lawless, A. S., Nichols, N. K., and Waller, J. A. (2021). New bounds on the condition number of the Hessian of the preconditioned variational data assimilation problem
2021
-
[26]
Tang, Z., Zhang, H., and Chen, J. (2022). Graph Neural Networks for Selection of Preconditioners and Krylov Solvers . In NeurIPS 2022 Workshop : New Frontiers in Graph Learning
2022
-
[27]
T., and Sartenaer, A
Tshimanga, J., Gratton, S., Weaver, A. T., and Sartenaer, A. (2008). Limited-memory preconditioners, with application to incremental four-dimensional variational data assimilation. Quarterly Journal of the Royal Meteorological Society , 134(632):751--769
2008
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.