Pith. sign in

REVIEW 4 major objections 4 minor 10 references

Jacobian-Enforced Neural Networks (JENN) for Improved Data Assimilation Consistency in Dynamical Models

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that adding tangent-linear and adjoint loss terms to neural-network training makes learned emulators keep their forecast skill while reproducing the true model's sensitivities, which is the structure variational data…

desk verdict A sensible Jacobian-enforcement recipe for NN emulators, but the central DA claim is asserted rather than tested, and the manuscript is too unfinished to accept. read the letter →

arxiv 2412.01013 v1 pith:4GYBTSD3 submitted 2024-12-02 cs.LG physics.ao-ph

Xiaoxu Tian This is my paper · ORCID
classification cs.LGphysics.ao-ph
keywords JacobianenforcementdataassimilationneuralnetworkemulatortangentlinearmodeladjointLorenz96variationaltwo-steptraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Jacobian-Enforced Neural Networks (JENN), a two-phase training procedure that teaches a neural network emulator not just to predict the next state of a dynamical system but to reproduce the system's tangent-linear and adjoint sensitivities. The claim is that adding RMSE losses on perturbation propagation to the forecast loss enforces Jacobian consistency without harming nonlinear forecast skill. Using Lorenz 96 with 40 variables as the testbed, the paper reports that the trained network's tangent-linear and adjoint responses align much more closely with the true model's, with noise reduced in sharp-gradient regions. The motivation is that ML weather models forecast well but lack the sensitivity structures variational data assimilation methods need, so JENN is presented as a route to make them DA-ready with minimal reconfiguration.

What carries the argument

The machinery is the composite loss function that couples the nonlinear forward map to its linear sensitivities. During the second training phase, random perturbations are fed through the network to produce a tangent-linear response $\delta y$ and an adjoint response $\hat{y}$; these are compared by RMSE against the true Lorenz 96 tangent-linear and adjoint outputs. The gradients of these extra loss terms propagate back through the same weights, so the network's Jacobian is shaped without ever forming the full 40 by 40 Jacobian explicitly during training. The two-step schedule (forecast-only first, Jacobian-enforced second) is what protects the already-learned forecast skill.

What would settle it

A concrete test: integrate the JENN-trained and the standard network into the same 4DVar or ensemble Kalman filter on Lorenz 96, cycle assimilation with identical observations, and compare analysis RMSE; if JENN's analysis errors are not smaller, the bridge from Jacobian consistency to DA skill fails.

Watch

Extended reading notes

Core claim

The central discovery is that enforcing Jacobian relationships during a second, fine-tuning training phase yields a neural network whose Jacobian matrix, tangent-linear propagator, and adjoint operator approximate those of the true Lorenz 96 dynamics far better than a standard forecast-trained network. The two-step scheme first trains on state-to-state predictions until the forecast loss converges, then continues with the composite loss $L_{\text{total}} = \alpha L_{\text{forecast}} + \beta L_{\text{TLM}} + \gamma L_{\text{ADJ}}$, where $L_{\text{TLM}}$ and $L_{\text{ADJ}}$ compare the network's responses to random perturbations against labels produced by the physical model's tangent-linear and adjoint code. The reported results show the JENN forecast tracks the true trajectory as closely as the standard network, while absolute deviations in tangent-linear and adjoint responses shrink markedly, especially where gradients are sharp. The paper interprets this as evidence that neural emulators can carry the sensitivity infrastructure required by 4DVar-style assimilation.

Load-bearing premise

The load-bearing assumption is that making a neural network's tangent-linear and adjoint responses look like the physical model's actually improves data assimilation performance, since the paper demonstrates the sensitivity improvements but never runs a data assimilation experiment.

Editorial extensions

If this is right

  • A JENN-trained emulator retains the nonlinear forecast accuracy of the standard network while its tangent-linear and adjoint responses move closer to the physical model's, so the sensitivity information DA systems rely on is no longer buried in NN noise.
  • Because the method adds only a training-phase loss term and does not alter the network architecture, it can be applied on top of pretrained forecast models such as GraphCast, Pangu, or FuXi with minimal reconfiguration.
  • With accurate TL and AD representations, ML-based weather models become more plausible candidates for integration into 4DVar and ensemble-based assimilation systems rather than remaining forecast-only tools.
  • The reduction in Jacobian noise, particularly in sharp-gradient regions, suggests the emulator's local linearizations are more trustworthy for computing analysis increments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper stops at Jacobian agreement; it does not run a data assimilation experiment. A natural next step would be to insert the JENN and standard NN emulators into the same 4DVar or ensemble filter and compare analysis RMSE, which would directly test whether cleaner Jacobians translate into better analyses.
  • The TL and AD labels come from the physical model, which for operational NWP means the method needs an adjoint of the real model; in settings where only trajectory data exists, those labels would have to be approximated, which could dilute the benefit.
  • Because Lorenz 96 is a single chaotic testbed with $F=8$, the strength of the effect for smoother, higher-dimensional, or strongly forced systems is untested; architecture and sample-size sensitivity are left to future work.
  • If Jacobian enforcement tacitly regularizes the network toward the true linear response, the same idea could be applied to other learned surrogates where tangent-linear fidelity matters beyond data assimilation, such as uncertainty quantification with neural emulators.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a Jacobian-Enforced Neural Network (JENN) training framework for neural network emulators of dynamical systems, using the Lorenz96 model as a testbed. The method first trains a standard forecast model with an RMSE loss, then refines it with a composite loss that adds RMSE terms for tangent-linear (TL) and adjoint (AD) responses. The authors claim that this two-step procedure preserves nonlinear forecast skill while substantially improving the fidelity of the network's TL, AD, and Jacobian representations, and they argue that this makes neural network emulators more suitable for data assimilation. The results are presented exclusively through figures comparing JENN to a standard neural network.

Significance. If fully substantiated, the JENN idea would be a useful and practical contribution: adding TL/AD response penalties to the training loss is a plausible regularization strategy, and the premise of adapting pretrained ML weather models for DA is timely and of broad interest. The two-phase training design that avoids architectural changes is a clear strength. However, the paper currently provides no quantitative evidence, no data assimilation experiment, and the displayed figures are not interpretable because all figure captions are duplicated. The central claim of 'improved DA consistency' is therefore not supported by the reported measurements, which reduces the current significance to that of an untested proposal.

major comments (4)
  1. [Section 3 and Abstract] The central claim of improved applicability of neural networks in data assimilation is unsupported because the paper never runs a data assimilation experiment. The statement in Section 3 that 'accurate representation of sensitivities is crucial for methods such as 4DVar and ensemble-based DA' is an assertion, not a result. Improved TL/AD/Jacobian fidelity does not automatically translate into better analysis skill, which depends on the DA algorithm, observation network, and error covariances. The paper should include a concrete DA experiment, such as a 4DVar or EnKF cycle using the trained emulator as the forecast model, comparing JENN against the standard NN on analysis accuracy.
  2. [Figures 2-5] The empirical claims are unverifiable because Figures 2, 3, 4, and 5 all carry the same caption as Figure 1, which describes the neural network architecture rather than forecast, TLM, ADJ, or Jacobian results. As a result, the reader cannot determine what is plotted, what the axes are, or how the JENN and standard NN outputs compare. The paper's conclusions rest entirely on these figures, so this placeholder/caption duplication is a load-bearing problem, not a cosmetic one. At minimum, the correct captions must be restored and each figure must be self-contained with axis labels and legends.
  3. [Section 2.1] The dataset size description is internally inconsistent. The abstract states that 80,000 data points are collected over 1,000 model times after spin-up, and Section 2.1 says the model is integrated for 80,000 time steps after spin-up, which yields 80,000 samples. However, Section 2.1 then states: 'This process yields a dataset of two million samples.' The factor-of-25 discrepancy makes the experimental setup unreproducible and undermines any quantitative interpretation of training behavior. The authors should correct the number and clarify how many samples were actually used for training and validation.
  4. [Equation (4) and Section 3] The reported improvements in TL and AD components are largely a direct consequence of optimizing L_TLM and L_ADJ in Equation (4). Because the standard NN baseline is not trained with these terms, seeing lower TL/AD RMSE for JENN is expected and does not by itself demonstrate that the Jacobian structure is accurately captured in a generalizable way. The only partially independent result is forecast preservation, yet even that is only shown qualitatively. The paper should report numerical values of the final forecast, TL, and AD RMSE, with error bars or statistics over multiple random perturbation sets and initial conditions, and should test generalization to perturbation magnitudes and locations not seen in training.
minor comments (4)
  1. [Section 2.2] The definitions of L_TLM and L_ADJ in Equation (4) are vague: the text says the network's 'predicted perturbations δx(t+Δt)' and 'adjoint x̂(t)' are compared to those of the physical model, but it does not specify how these quantities are computed for the neural network. In particular, the adjoint of a neural network is not uniquely defined without specifying whether it is the transposed Jacobian of the forward pass or the backpropagation through the entire training loss. A precise definition is needed for reproducibility.
  2. [Section 2.2] The hyperparameters α, β, γ, the perturbation scale (stated as 'approximately 1%' of state values), the number of random perturbations, and the convergence criteria for the L-BFGS phases are not reported. These parameters directly control the tradeoff between forecast accuracy and Jacobian fidelity, so the paper should list them or state that a sensitivity study was performed.
  3. [Section 3] The text uses phrases like 'significantly reduces noise,' 'closely match,' and 'substantial improvements' without any numerical support. Since no RMSE values, confidence intervals, or statistical tests are given, the qualitative claims cannot be evaluated by the reader.
  4. [General] There is a minor spelling inconsistency: the paper uses both 'Lorenz 96' and 'Lorenz96.' The reference list also does not include a distinct entry for the Lorenz 1990 paper beyond a general citation; please verify the bibliography entries.

Circularity Check

2 steps flagged · score 6.0 of 10

Reported TL/AD/Jacobian gains are the trained loss terms; DA benefit is asserted, not measured.

  1. fitted input called prediction [Section 2.2, Eq. (4), and Section 3, 'Tangent linear responses' paragraph]
    "A custom loss function combines the forecast loss and a Jacobian enforcement term: Ltotal = αLforecast + βLTLM + γLADJ, (4) where LTLM is the RMSE between the neural network’s predicted perturbations δx(t + ∆t) and the true perturbations δxtrue(t + ∆t) and LADJ RMSE between the neural network’s adjoint xˆ(t) and the true adjoint xˆtrue(t). ... The tangent linear responses exhibit notable improvements in accuracy following the JENN training."

    The results section reports that JENN reduces TL/AD errors relative to a standard NN, but those errors are precisely the quantities minimized in phase two. The standard NN was optimized only on Lforecast, so any improvement on LTLM/LADJ is the direct effect of adding those terms to the training objective; it is the fit, not an independent prediction. No train/test split, no held-out perturbation set, and no DA analysis experiment is described, so the TL/AD numbers do not support the abstract’s 'improved applicability of NNs in DA' beyond the training criterion itself.

  2. self definitional [Section 3, 'heat map of Jacobian matrix comparisons' paragraph, and Section 4]
    "The JENN framework significantly reduces the deviations between the learned Jacobian (JJEN N) and the true Jacobian (Jtrue), as evident in the bottom right panel in Fig. 5. ... The improvements in TLM and ADJ consistency, as well as the reduction in Jacobian deviations, make the JENN framework a promising candidate for integration into data assimilation systems."

    The Jacobian is the operator whose action on perturbations is exactly the TL response and whose transpose is the AD response. Since LTLM and LADJ directly penalize mismatches in those responses, the reported 'reduction in Jacobian deviations' is not an independent outcome; it is a restatement of the training loss aggregated over directions. The paper even states it relies 'instead on the tangent linear and adjoint information' rather than the full Jacobian, so the Jacobian result is a consequence of the fitted TL/AD objective, not a separate discovery.

full rationale

JENN’s headline TL/AD/Jacobian results are the direct minimization targets of Eq. (4); the paper optimizes βLTLM + γLADJ and then reports that these terms are smaller than in a baseline NN that was never trained on them. That is a fitted outcome, not a DA prediction. The only independent quantitative outcome is forecast preservation, which is shown to be roughly unchanged; this is a useful sanity check but not a demonstration of DA skill. The paper runs no 4DVar or ensemble-DA experiment, so the bridge from sensitivity accuracy to analysis accuracy is asserted in Section 3 rather than established. Self-citations (Tian et al. 2023, 2024) are used only to motivate the problem and are not load-bearing uniqueness arguments. Manuscript inconsistencies prevent independent verification of the plotted claims: Figures 2–5 repeat Figure 1’s caption verbatim, and the abstract says 80,000 samples while Section 2.1 says two million samples. Net: partial circularity at the core demonstration, with the DA benefit claim untested, so the score is 6.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

All TL/AD targets are generated from the same Lorenz 96 model used to create training data, and the evaluation metrics coincide with the training objectives. No external benchmark or DA experiment is provided, so the reported improvements are partly by construction.

free parameters (5)
  • Loss weights alpha, beta, gamma = not stated
    Chosen by hand in Eq. (4) to balance forecast, TL, and AD terms; no values or sensitivity analysis are given.
  • Perturbation scale = ~1% of state values
    Used to generate TL/AD input perturbations; assumes the tangent-linear approximation is valid at this finite amplitude, which is not tested.
  • Number and locations of random perturbations = not stated
    The secondary training phase applies random perturbations at random locations; sampling density is unspecified and affects Jacobian coverage.
  • Network architecture size = two hidden layers of 256 tanh units
    Chosen without justification or architecture search; results may depend on capacity.
  • Integration time step dt = 0.0125
    Determines the discrete map whose Jacobian is emulated; no sensitivity test is provided.
assumptions (5)
  • domain assumption Lorenz 96 numerically integrated with a fourth-order Runge-Kutta scheme is the ground-truth physical model.
    All training labels and TL/AD targets come from this discretization; no validation against a higher-accuracy reference is given.
  • standard math The neural network is differentiable and its Jacobian is computable by backpropagation.
    The tanh hidden layers and linear output layer guarantee differentiability, so this is standard.
  • domain assumption A 1% perturbation lies in the tangent-linear regime of the Lorenz 96 model for dt = 0.0125.
    TL/AD labels are computed from linearized dynamics but compared with the NN at finite perturbation amplitude; linearity is not verified.
  • ad hoc to paper Minimizing TL/AD RMSE on random perturbations improves data assimilation analysis skill.
    The paper asserts this in Section 3 but never runs a DA experiment; it is the key bridge from Jacobian metrics to DA applicability.
  • standard math L-BFGS finds a good minimum in both training phases.
    The paper relies on optimizer convergence without detailing stopping criteria, restarts, or robustness checks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Jacobian-Enforced Neural Networks (JENN) for Improved Data Assimilation Consistency in Dynamical Models." pith.science (2026). https://pith.science/paper/4GYBTSD3

@misc{pith2026241201013,
  author       = {Pith},
  title        = {Pith review of: Jacobian-Enforced Neural Networks (JENN) for Improved Data Assimilation Consistency in Dynamical Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4GYBTSD3}},
  note         = {Machine review of arXiv:2412.01013}
}
read the original abstract

Machine learning-based weather models have shown great promise in producing accurate forecasts but have struggled when applied to data assimilation tasks, unlike traditional numerical weather prediction (NWP) models. This study introduces the Jacobian-Enforced Neural Network (JENN) framework, designed to enhance DA consistency in neural network (NN)-emulated dynamical systems. Using the Lorenz 96 model as an example, the approach demonstrates improved applicability of NNs in DA through explicit enforcement of Jacobian relationships. The NN architecture includes an input layer of 40 neurons, two hidden layers with 256 units each employing hyperbolic tangent activation functions, and an output layer of 40 neurons without activation. The JENN framework employs a two-step training process: an initial phase using standard prediction-label pairs to establish baseline forecast capability, followed by a secondary phase incorporating a customized loss function to enforce accurate Jacobian relationships. This loss function combines root mean square error (RMSE) between predicted and true state values with additional RMSE terms for tangent linear (TL) and adjoint (AD) emulation results, weighted to balance forecast accuracy and Jacobian sensitivity. To ensure consistency, the secondary training phase uses additional pairs of TL/AD inputs and labels calculated from the physical models. Notably, this approach does not require starting from scratch or structural modifications to the NN, making it readily applicable to pretrained models such as GraphCast, NeuralGCM, Pangu, or FuXi, facilitating their adaptation for DA tasks with minimal reconfiguration. Experimental results demonstrate that the JENN framework preserves nonlinear forecast performance while significantly reducing noise in the TL and AD components, as well as in the overall Jacobian matrix.

Figures

Figures reproduced from arXiv: 2412.01013 by the authors.

Figure 1
Figure 1. An illustration of neural network structure to emulate the Lorenz 96 model, with an input layer of 40 nodes, [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. An illustration of neural network structure to emulate the Lorenz 96 model, with an input layer of 40 nodes, [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. An illustration of neural network structure to emulate the Lorenz 96 model, with an input layer of 40 nodes, [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: An illustration of neural network structure to emulate the Lorenz 96 model, with an input layer of 40 nodes, [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: An illustration of neural network structure to emulate the Lorenz 96 model, with an input layer of 40 nodes, [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

10 extracted references · 8 canonical work pages

  1. [1]

    E. Kalnay. Atmospheric Modeling, Data Assimilation and Predictability, volume 341. Cambridge University Press, 2003

    work page 2003

  2. [2]

    Dueben and Peter Bauer

    Peter D. Dueben and Peter Bauer. Challenges and design choices for global weather and climate models based on machine learning. Geoscientific Model Development, 11 0 (10): 0 3999--4009, 2018. ISSN 1991-959X

    work page 2018

  3. [3]

    Toward data‐driven weather and climate forecasting: Approximating a simple general circulation model with deep learning

    Sebastian Scher. Toward data‐driven weather and climate forecasting: Approximating a simple general circulation model with deep learning. Geophysical Research Letters, 45 0 (22): 0 12--616, 2018. ISSN 0094-8276

    work page 2018

  4. [4]

    Learning skillful medium-range global weather forecasting

    Remi Lam, Alvaro Sanchez-Gonzalez, Matthew Willson, Peter Wirnsberger, Meire Fortunato, Ferran Alet, Suman Ravuri, Timo Ewalds, Zach Eaton-Rosen, Weihua Hu, Alexander Merose, Stephan Hoyer, George Holland, Oriol Vinyals, Jacklynn Stott, Alexander Pritzel, Shakir Mohamed, and Peter Battaglia. Learning skillful medium-range global weather forecasting. Scien...

    doi:10.1126/science.adi2336 2023

  5. [5]

    Neural general circulation models for weather and climate

    Dmitrii Kochkov, Janni Yuval, Ian Langmore, Peter Norgaard, Jamie Smith, Griffin Mooers, Milan Klöwer, James Lottes, Stephan Rasp, and Peter Düben. Neural general circulation models for weather and climate. Nature, pages 1--7, 2024. ISSN 0028-0836

    work page 2024

  6. [6]

    A neural-network based mpas-shallow water model and its 4d-var data assimilation system

    Xiaoxu Tian, Luke Conibear, and Jeffrey Steward. A neural-network based mpas-shallow water model and its 4d-var data assimilation system. Atmosphere, 14 0 (1): 0 157, 2023. ISSN 2073-4433. URL https://www.mdpi.com/2073-4433/14/1/157

    work page 2023

  7. [7]

    Derber and F

    J. Derber and F. Bouttier. A reformulation of the background error covariance in the ecmwf global data assimilation system. Tellus A, 51 0 (2): 0 195--221, 1999. ISSN 1600-0870

    work page 1999

  8. [8]

    Variational assimilation of meteorological observations with the direct and adjoint shallow-water equations

    Philippe Courtier and Olivier Talagrand. Variational assimilation of meteorological observations with the direct and adjoint shallow-water equations. Tellus A, 42 0 (5): 0 531--549, 1990. ISSN 1600-0870

    work page 1990

Show all 10 references
  1. [9]

    Exploring the use of machine learning weather models in data assimilation

    Xiaoxu Tian, Daniel Holdaway, and Daryl Kleist. Exploring the use of machine learning weather models in data assimilation. arXiv preprint arXiv:2411.14677, 2024

    2024 arXiv

  2. [10]

    Edward N. Lorenz. Predictability: A problem partly solved. volume 1. Reading, 1990

    1990

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.