{"id":"54aca380-9a96-4433-8ae7-7ff6b7b91136","arxiv_id":"2412.01013","paper_version":1,"verdict":"REJECT","confidence":"LOW","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A two-phase training scheme that adds tangent-linear and adjoint loss terms to a neural network emulator of Lorenz 96 improves Jacobian consistency while preserving forecast accuracy.","lead":"This paper trains a neural network to mimic the Lorenz 96 weather toy model, then fine-tunes it with extra loss terms that force its tangent-linear and adjoint sensitivities to match the true model. The goal is to make machine-learned weather models usable in data assimilation, where sensitivity information matters as much as forecast skill.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Concern: the bridge from improved Jacobian/TL/AD fidelity to actual data assimilation performance is asserted, not demonstrated, and the paper reports no DA experiment, so the central DA-consistency claim is unsupported.","rationale":"I agree with the reader's weakest assumption: the paper tests a proxy (TL/AD/Jacobian fidelity) and never tests the actual DA outcome, so the central claim of improved DA applicability is unsupported. The additional red flags—placeholder captions, dataset count inconsistency, no code/data, no quantitative metrics or error bars—lower confidence but are secondary to the missing DA experiment. I do not see an internal contradiction in the training loss itself; the issue is external validity. The reader's REJECT verdict is therefore the right one, and my stress-test does not change it. A conditional path to acceptance would be adding a properly designed DA experiment with quantitative comparison against the standard NN and the true model, plus releasing code and data.","tokens_in":5920,"tokens_out":8662,"duration_ms":85674,"concrete_test":"Run a Lorenz 96 4DVar cycling experiment with identical observation network and error statistics for all forecast models: observations every 4 steps on 20 of 40 variables with Gaussian error sigma=0.1, background covariance from a long-run climatology, assimilation window of 5-10 steps at dt=0.0125, and a 2000-step cycling period. Compare three forecast models: true Lorenz 96 with its exact TL/AD, a standard NN, and the JENN-trained NN. Report mean and standard deviation of analysis RMSE and one-window forecast RMSE over at least 50 independent runs with bootstrap confidence intervals. If JENN does not improve analysis or forecast scores over the standard NN, or does not approach the true-model DA skill, then the improved TL/AD fidelity does not translate into DA benefit and the central claim should be revised from 'DA consistency' to 'Jacobian fidelity.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing bridge in the argument is the assumption, stated in Section 3, that 'accurate representation of sensitivities is crucial for methods such as 4DVar and ensemble-based DA, where the quality of the tangent linear and adjoint models directly impacts the analysis accuracy.' This is an assertion, not a result. The experiments measure exactly the quantities being optimized: L_TLM and L_ADJ in Eq. (4) are direct RMSE penalties on TL/AD responses, so reporting reduced TL/AD noise largely reports fit of the training objective. The claimed DA benefit would require a separate experiment: an actual 4DVar or ensemble-DA analysis cycle using the trained NN as the forecast model, with JENN and standard NN compared on analysis quality. Absent that, the paper's phrase 'improved DA consistency' and the abstract's 'improved applicability of NNs in DA' are unsupported even if the Jacobian plots are cleaner. The evidence base is also unverifiable: Figures 2-5 all carry the same placeholder caption as Figure 1 (the NN architecture illustration), and the dataset description is internally inconsistent (abstract: 80,000 samples; Section 2.1: two million samples), so the visual claims cannot be checked independently. The concern is not that the method is wrong, but that the central claim overreaches the measurements.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Jacobian-Enforced Neural Network (JENN) training framework for neural network emulators of dynamical systems, using the Lorenz96 model as a testbed. The method first trains a standard forecast model with an RMSE loss, then refines it with a composite loss that adds RMSE terms for tangent-linear (TL) and adjoint (AD) responses. The authors claim that this two-step procedure preserves nonlinear forecast skill while substantially improving the fidelity of the network's TL, AD, and Jacobian representations, and they argue that this makes neural network emulators more suitable for data assimilation. The results are presented exclusively through figures comparing JENN to a standard neural network.","tokens_in":6214,"tokens_out":3567,"duration_ms":33011,"significance":"If fully substantiated, the JENN idea would be a useful and practical contribution: adding TL/AD response penalties to the training loss is a plausible regularization strategy, and the premise of adapting pretrained ML weather models for DA is timely and of broad interest. The two-phase training design that avoids architectural changes is a clear strength. However, the paper currently provides no quantitative evidence, no data assimilation experiment, and the displayed figures are not interpretable because all figure captions are duplicated. The central claim of 'improved DA consistency' is therefore not supported by the reported measurements, which reduces the current significance to that of an untested proposal.","major_comments":[{"comment":"The central claim of improved applicability of neural networks in data assimilation is unsupported because the paper never runs a data assimilation experiment. The statement in Section 3 that 'accurate representation of sensitivities is crucial for methods such as 4DVar and ensemble-based DA' is an assertion, not a result. Improved TL/AD/Jacobian fidelity does not automatically translate into better analysis skill, which depends on the DA algorithm, observation network, and error covariances. The paper should include a concrete DA experiment, such as a 4DVar or EnKF cycle using the trained emulator as the forecast model, comparing JENN against the standard NN on analysis accuracy.","section":"Section 3 and Abstract"},{"comment":"The empirical claims are unverifiable because Figures 2, 3, 4, and 5 all carry the same caption as Figure 1, which describes the neural network architecture rather than forecast, TLM, ADJ, or Jacobian results. As a result, the reader cannot determine what is plotted, what the axes are, or how the JENN and standard NN outputs compare. The paper's conclusions rest entirely on these figures, so this placeholder/caption duplication is a load-bearing problem, not a cosmetic one. At minimum, the correct captions must be restored and each figure must be self-contained with axis labels and legends.","section":"Figures 2-5"},{"comment":"The dataset size description is internally inconsistent. The abstract states that 80,000 data points are collected over 1,000 model times after spin-up, and Section 2.1 says the model is integrated for 80,000 time steps after spin-up, which yields 80,000 samples. However, Section 2.1 then states: 'This process yields a dataset of two million samples.' The factor-of-25 discrepancy makes the experimental setup unreproducible and undermines any quantitative interpretation of training behavior. The authors should correct the number and clarify how many samples were actually used for training and validation.","section":"Section 2.1"},{"comment":"The reported improvements in TL and AD components are largely a direct consequence of optimizing L_TLM and L_ADJ in Equation (4). Because the standard NN baseline is not trained with these terms, seeing lower TL/AD RMSE for JENN is expected and does not by itself demonstrate that the Jacobian structure is accurately captured in a generalizable way. The only partially independent result is forecast preservation, yet even that is only shown qualitatively. The paper should report numerical values of the final forecast, TL, and AD RMSE, with error bars or statistics over multiple random perturbation sets and initial conditions, and should test generalization to perturbation magnitudes and locations not seen in training.","section":"Equation (4) and Section 3"}],"minor_comments":[{"comment":"The definitions of L_TLM and L_ADJ in Equation (4) are vague: the text says the network's 'predicted perturbations δx(t+Δt)' and 'adjoint x̂(t)' are compared to those of the physical model, but it does not specify how these quantities are computed for the neural network. In particular, the adjoint of a neural network is not uniquely defined without specifying whether it is the transposed Jacobian of the forward pass or the backpropagation through the entire training loss. A precise definition is needed for reproducibility.","section":"Section 2.2"},{"comment":"The hyperparameters α, β, γ, the perturbation scale (stated as 'approximately 1%' of state values), the number of random perturbations, and the convergence criteria for the L-BFGS phases are not reported. These parameters directly control the tradeoff between forecast accuracy and Jacobian fidelity, so the paper should list them or state that a sensitivity study was performed.","section":"Section 2.2"},{"comment":"The text uses phrases like 'significantly reduces noise,' 'closely match,' and 'substantial improvements' without any numerical support. Since no RMSE values, confidence intervals, or statistical tests are given, the qualitative claims cannot be evaluated by the reader.","section":"Section 3"},{"comment":"There is a minor spelling inconsistency: the paper uses both 'Lorenz 96' and 'Lorenz96.' The reference list also does not include a distinct entry for the Lorenz 1990 paper beyond a general citation; please verify the bibliography entries.","section":"General"}],"recommendation":"reject","confidential_remarks":"The manuscript appears to be an early-stage preprint: the duplicated figure captions suggest that the actual result figures were inadvertently omitted, and the dataset-size inconsistency (80,000 vs. 2,000,000 samples) indicates the text has not been carefully checked. Even with these presentation issues corrected, the paper would still need a substantive DA experiment and quantitative evaluation to support its central claim of improved DA consistency. The gap between the measured quantities (TL/AD/Jacobian RMSE) and the claimed outcome (better data assimilation) is not bridgeable by argument alone in this community. I recommend rejection, with an invitation to resubmit a revised version that includes a genuine DA comparison and complete, interpretable figures."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: the JENN recipe is sensible and the results are plausible, but the evidence is all eyeballed—no numbers, no DA experiment, placeholder figure captions. The phrase \"improved DA consistency\" is asserted, not demonstrated.\n\nWhat's actually new: the two-phase training scheme—forecast-only first, then a second phase with TL and AD RMSE terms in the loss—is a concrete recipe that I don't see in the cited papers. That is a legitimate extension, and the pitch that it could be applied to pretrained models like GraphCast or Pangu without architectural changes is reasonable. The forecast-preservation check is the one partly independent result, and it's reassuring that the second phase doesn't wreck the nonlinear forecast.\n\nThe soft spots are substantial. The TL and AD losses are direct minimization targets in Eq. (4), so reporting reduced TL/AD noise is partly reporting the fit. The load-bearing bridge—that better Jacobian/TL/AD agreement translates into better DA performance—is never tested. There is no 4DVar or ensemble-DA cycling experiment, no RMSE tables, no error bars, and no statistical tests. The dataset description is internally inconsistent: the abstract says 80,000 data points, Section 2.1 says two million, and the actual integration time yields 80,000 samples. Figures 2 through 5 all carry the same placeholder caption as the architecture diagram, so the visual claims cannot be independently checked. The paper also misses the derivative-matching and Jacobian-regularization literature, which is directly relevant and should be cited and compared against.\n\nNone of these are fatal to the underlying idea. The method could work, and for a Lorenz 96 testbed a proper DA experiment is straightforward. But as submitted, the paper is a sketch: the central claim overreaches the measurements, and the presentation is too careless for the evidence to be trusted.\n\nWho this is for: someone working on making ML weather models usable in DA. They would find the two-phase loss idea worth trying, but they would need to run the DA experiment themselves. I'd bring it to a reading group as a discussion piece, but I wouldn't cite it as a result.\n\nRecommendation: send to peer review with major revision. The idea is relevant and the failure mode is under-evidence, not wrongness. A serious referee could push for quantitative results and a real DA experiment, which would either make or break the paper.","headline":"A sensible Jacobian-enforcement recipe for NN emulators, but the central DA claim is asserted rather than tested, and the manuscript is too unfinished to accept.","tokens_in":6704,"tokens_out":1779,"would_cite":false,"duration_ms":17629,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that adding tangent-linear and adjoint loss terms to neural-network training makes learned emulators keep their forecast skill while reproducing the true model's sensitivities, which is the structure variational data…","keywords":["Jacobian enforcement","data assimilation","neural network emulator","tangent linear model","adjoint model","Lorenz 96","variational data assimilation","two-step training"],"falsifier":"A concrete test: integrate the JENN-trained and the standard network into the same 4DVar or ensemble Kalman filter on Lorenz 96, cycle assimilation with identical observations, and compare analysis RMSE; if JENN's analysis errors are not smaller, the bridge from Jacobian consistency to DA skill fails.","tokens_in":5719,"feed_emoji":"🌀","tokens_out":7328,"duration_ms":57221,"temperature":0.7,"pith_summary":"This paper proposes Jacobian-Enforced Neural Networks (JENN), a two-phase training procedure that teaches a neural network emulator not just to predict the next state of a dynamical system but to reproduce the system's tangent-linear and adjoint sensitivities. The claim is that adding RMSE losses on perturbation propagation to the forecast loss enforces Jacobian consistency without harming nonlinear forecast skill. Using Lorenz 96 with 40 variables as the testbed, the paper reports that the trained network's tangent-linear and adjoint responses align much more closely with the true model's, with noise reduced in sharp-gradient regions. The motivation is that ML weather models forecast well but lack the sensitivity structures variational data assimilation methods need, so JENN is presented as a route to make them DA-ready with minimal reconfiguration.","feed_headline":"Jacobian loss steers neural nets toward true model sensitivities","feed_subtitle":"Fixes tangent-linear and adjoint responses for 4DVar without hurting forecast skill","key_machinery":"The machinery is the composite loss function that couples the nonlinear forward map to its linear sensitivities. During the second training phase, random perturbations are fed through the network to produce a tangent-linear response $\\delta y$ and an adjoint response $\\hat{y}$; these are compared by RMSE against the true Lorenz 96 tangent-linear and adjoint outputs. The gradients of these extra loss terms propagate back through the same weights, so the network's Jacobian is shaped without ever forming the full 40 by 40 Jacobian explicitly during training. The two-step schedule (forecast-only first, Jacobian-enforced second) is what protects the already-learned forecast skill.","core_discovery":"The central discovery is that enforcing Jacobian relationships during a second, fine-tuning training phase yields a neural network whose Jacobian matrix, tangent-linear propagator, and adjoint operator approximate those of the true Lorenz 96 dynamics far better than a standard forecast-trained network. The two-step scheme first trains on state-to-state predictions until the forecast loss converges, then continues with the composite loss $L_{\\text{total}} = \\alpha L_{\\text{forecast}} + \\beta L_{\\text{TLM}} + \\gamma L_{\\text{ADJ}}$, where $L_{\\text{TLM}}$ and $L_{\\text{ADJ}}$ compare the network's responses to random perturbations against labels produced by the physical model's tangent-linear and adjoint code. The reported results show the JENN forecast tracks the true trajectory as closely as the standard network, while absolute deviations in tangent-linear and adjoint responses shrink markedly, especially where gradients are sharp. The paper interprets this as evidence that neural emulators can carry the sensitivity infrastructure required by 4DVar-style assimilation.","pith_inferences":["The paper stops at Jacobian agreement; it does not run a data assimilation experiment. A natural next step would be to insert the JENN and standard NN emulators into the same 4DVar or ensemble filter and compare analysis RMSE, which would directly test whether cleaner Jacobians translate into better analyses.","The TL and AD labels come from the physical model, which for operational NWP means the method needs an adjoint of the real model; in settings where only trajectory data exists, those labels would have to be approximated, which could dilute the benefit.","Because Lorenz 96 is a single chaotic testbed with $F=8$, the strength of the effect for smoother, higher-dimensional, or strongly forced systems is untested; architecture and sample-size sensitivity are left to future work.","If Jacobian enforcement tacitly regularizes the network toward the true linear response, the same idea could be applied to other learned surrogates where tangent-linear fidelity matters beyond data assimilation, such as uncertainty quantification with neural emulators."],"forward_implications":["A JENN-trained emulator retains the nonlinear forecast accuracy of the standard network while its tangent-linear and adjoint responses move closer to the physical model's, so the sensitivity information DA systems rely on is no longer buried in NN noise.","Because the method adds only a training-phase loss term and does not alter the network architecture, it can be applied on top of pretrained forecast models such as GraphCast, Pangu, or FuXi with minimal reconfiguration.","With accurate TL and AD representations, ML-based weather models become more plausible candidates for integration into 4DVar and ensemble-based assimilation systems rather than remaining forecast-only tools.","The reduction in Jacobian noise, particularly in sharp-gradient regions, suggests the emulator's local linearizations are more trustworthy for computing analysis increments."],"supporting_citations":[{"why":"Supplies the Lorenz 96 model used as the testbed and the source of training and label data.","marker":"[Lorenz, 1990]"},{"why":"Establishes that tangent-linear and adjoint models are the structures variational data assimilation needs, motivating the whole approach.","marker":"[Courtier and Talagrand, 1990]"},{"why":"Provides the prior neural-network-based model and its 4DVar assimilation system, the work this method builds on and improves.","marker":"[Tian et al., 2023]"},{"why":"Explores the use of ML weather models in data assimilation, framing the difficulty JENN addresses.","marker":"[Tian et al., 2024]"},{"why":"Documents design choices and challenges for machine-learning weather and climate models.","marker":"[Dueben and Bauer, 2018]"},{"why":"An example of a pretrained ML weather model that JENN claims to be readily applicable to.","marker":"[Lam et al., 2023]"},{"why":"Another pretrained ML weather model mentioned as a candidate for JENN-style DA adaptation.","marker":"[Kochkov et al., 2024]"}],"fun_headline_variants":["Jacobian-enforced training makes neural nets DA-consistent","JENN: two-step training yields true tangent-linear and adjoint responses","Neural emulators learn correct Jacobians for 4DVar via fine-tuning","Jacobian-enforced neural nets improve data assimilation consistency","Fine-tuning neural nets with Jacobian loss fixes adjoint for DA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that making a neural network's tangent-linear and adjoint responses look like the physical model's actually improves data assimilation performance, since the paper demonstrates the sensitivity improvements but never runs a data assimilation experiment.","fun_headline_variants_meta":{"raw":{"variants":["Jacobian-enforced training makes neural nets DA-consistent","JENN: two-step training yields true tangent-linear and adjoint responses","Neural emulators learn correct Jacobians for 4DVar via fine-tuning","Jacobian-enforced neural nets improve data assimilation consistency","Fine-tuning neural nets with Jacobian loss fixes adjoint for DA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000847,"raw_usage":{"total_tokens":3747,"prompt_tokens":1069,"completion_tokens":2678,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":685,"completion_tokens_details":{"reasoning_tokens":2587}},"tokens_in":685,"tokens_out":2678,"duration_ms":17868,"temperature":1.0,"reasoning_tokens":2587,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:45:46.407408+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: integrate the JENN-trained and the standard network into the same 4DVar or ensemble Kalman filter on Lorenz 96, cycle assimilation with identical observations, and compare analysis RMSE; if JENN's analysis errors are not smaller, the bridge from Jacobian consistency to DA skill fails.","supporting_citations":[{"cited_title":"Variational assimilation of meteorological observations with the direct and adjoint shallow-water equations","cited_arxiv_id":null,"evidence_quote":"Establishes that tangent-linear and adjoint models are the structures variational data assimilation needs, motivating the whole approach."},{"cited_title":"A neural-network based mpas-shallow water model and its 4d-var data assimilation system","cited_arxiv_id":null,"evidence_quote":"Provides the prior neural-network-based model and its 4DVar assimilation system, the work this method builds on and improves."},{"cited_title":"Neural general circulation models for weather and climate","cited_arxiv_id":null,"evidence_quote":"Another pretrained ML weather model mentioned as a candidate for JENN-style DA adaptation."}],"review_version":1}