{"id":"16acd285-3d8f-407f-97e2-b83566bd3645","arxiv_id":"2412.11301","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SINODE uses IMEX-RK time integration with a linear-nonlinear partition and a discrete adjoint to train stiff neural ODEs stably and quickly, including cases where explicit and fully implicit methods fail.","lead":"This paper introduces SINODE, a neural ODE training method that splits the dynamics into a nonlinear part and a linear part and integrates them with an implicit-explicit Runge-Kutta scheme. This allows stiff problems, such as chaotic PDEs and graph diffusion, to be trained with far fewer steps and much less time than explicit or fully implicit approaches.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim is contingent on a suitable constant linear-nonlinear split; every experiment supplies J externally, so the general claim about challenging neural ODEs is not yet demonstrated.","rationale":"The reader's weakest assumption correctly identifies the linear-nonlinear split as the load-bearing condition. The paper's efficiency and stability arguments all route through Eq. (12), which is only linear because H is assumed to be a constant linear map J u. The demonstrated applications are precisely those where physics or architecture supplies J: the identity in GRAND and fixed finite-difference/convolution operators in KS and Burgers. The paper gives no general procedure for constructing a useful split, and the Limitation section does not mention this requirement. This does not invalidate the reported experiments, which are concrete and reproducible (code is released, NFEs and wall-clock times are reported), but it does mean the abstract's strongest phrasing overstates the demonstrated scope. A learned-J experiment would settle whether the method's advantages survive outside the known-linear-operator setting. I also note the discrete adjoint proof in Appendix A derives only the ERK special case and asserts the IMEX extension; a finite-difference gradient check would be a useful secondary check, but the split assumption is the more central risk to the headline claim. The reader's CONDITIONAL verdict remains appropriate: the paper is strong for the problem class it actually tests, but the general claim needs either a demonstration with learned/non-physics J or a clear scope restriction.","tokens_in":18069,"tokens_out":17397,"duration_ms":163819,"concrete_test":"Train SINODE on a stiff neural ODE benchmark where J is a learned, trainable linear layer rather than a fixed physics-derived operator, using the same architecture and step sizes as the KS experiment. Compare wall-clock time per epoch, loss curves, and final test error against explicit RK4/Dopri5 and a fully implicit Crank-Nicolson baseline. If SINODE retains its large speedup and stable training, the split concern is resolved; if it degrades toward explicit-method performance or diverges, the abstract's general 'challenging neural ODEs' claim must be restricted to problems with a known, constant linear part.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract promises that SINODE 'can train challenging neural ODEs where both explicit methods and fully implicit methods are intractable.' The mechanism that makes this possible is the constant linear part J in Eq. (10): it converts each IMEX stage (11) into the linear system (12), enables LU reuse across stages and time steps (Sec. 4.1), and keeps the backward linear solves cheap. Yet every experiment supplies J externally: J = -I in GRAND (Eq. 19), a fixed finite-difference convolution for KS (Sec. 5.2), and a fixed finite-difference layer for Burgers (Sec. 5.3). No experiment learns J or shows how to discover a useful split when the stiff dynamics is not already separated into known linear and nonlinear parts. If J must be learned, it changes every optimizer step and the reusable-factorization argument in Sec. 4.1 weakens to at most once per training iteration; if no good linear split exists, the explicit part G may still be stiff and the IMEX stability advantage can disappear. The Limitations section (Sec. 7) does not list the need for a known, constant linear part as a limitation, which makes the scope of the headline claim easy to overread. This is a scope limitation rather than an internal inconsistency, but it is load-bearing for the strongest version of the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SINODE, a semi-implicit approach for training neural ODEs whose right-hand side can be additively split into a nonlinear part G(u) and a linear part J u. The forward pass uses IMEX-RK methods, treating G explicitly and the linear part implicitly, which reduces each stage to a linear system when J is constant. The backward pass uses a discrete adjoint (Theorem 4.1) to obtain reverse-accurate gradients. Experiments on graph classification with GRAND, and on learning the Kuramoto–Sivashinsky and viscous Burgers equations, show that SINODE can use much larger time steps than explicit methods and remains stable where fully implicit methods fail, yielding substantial speedups in training time.","tokens_in":18321,"tokens_out":5758,"duration_ms":51743,"significance":"If the central claim holds, SINODE provides a practical way to train stiff neural ODEs by leveraging well-understood IMEX time integrators and off-the-shelf linear solvers, with the advantage of reverse-accurate discrete adjoints. The paper includes a concrete algorithm, a stated adjoint formula, and a reproducible code repository, and the experiments demonstrate clear stability and efficiency gains on problems with a known linear part. The main caveat is that the method's applicability currently hinges on the availability of a constant linear-nonlinear split, which is supplied externally in every experiment; the paper's headline claim about general challenging neural ODEs is therefore broader than what is demonstrated.","major_comments":[{"comment":"The abstract claims SINODE 'can train challenging neural ODEs where both explicit methods and fully implicit methods are intractable.' This claim is not qualified by the requirement, evident from Secs. 5.1–5.3, that a constant linear part J must be known a priori or supplied externally (e.g., Eq. (19) for GRAND, the fixed finite-difference convolution for KS, and the fixed finite-difference layer for Burgers). The paper does not propose a procedure for discovering a suitable split when the dynamics are not already separated, and if J were learned, the reusable-factorization argument in Sec. 4.1 would weaken to at most once per training iteration. The Limitations section does not list this as a limitation, which makes the scope of the headline claim easy to overread. The authors should either provide a method for constructing the split or explicitly restrict the central claims to problems with a known constant linear part.","section":"Abstract and Sec. 7 (Limitations)"},{"comment":"The GRAND experiments report NFEs per epoch (Table 1) and show testing accuracy versus training time (Fig. 1), but no final test accuracy table is provided, and no standard deviations across multiple seeds are reported for any dataset or method. Given that graph classification results are typically reported as mean±std over seeds, the absence of such numbers makes it difficult to assess whether the observed accuracy differences are significant. The authors should add a table of final test accuracies with error bars for all methods on Cora, CoauthorCS, and Photo.","section":"Sec. 5.1 (GRAND experiments)"},{"comment":"The NFE comparisons exclude the cost of linear solves, which are central to IMEX and fully implicit methods. While the paper also shows wall-clock training-time curves (Figs. 1, 2, 4, 6), these are not tabulated, and the reader cannot easily compare the efficiency of the methods in a quantitative way. The claim that NFE is 'usually a good metric' (Sec. 5.1) needs support: for implicit methods, the per-step linear solve cost can dominate the NN evaluation cost, especially at higher grid resolutions. The authors should either justify NFE as a fair metric in the presence of linear solves or report per-epoch wall-clock times in a table for all datasets and step sizes.","section":"Tables 1 and 2; Sec. 5.1-5.3"},{"comment":"Theorem 4.1 is the backbone of the training algorithm, but the proof in Appendix A is only carried out for explicit RK (Eqs. (22)–(28)); the extension to IMEX-RK is asserted in a single sentence without a derivation. Since the IMEX adjoint involves two coefficient tableaux and mixed explicit/implicit stage treatments, a complete derivation (or a precise reference to one) is needed to make the reverse-accuracy claim fully verifiable. Additionally, the appendix is titled 'Proof of Theorem 5.1' while the theorem in the main text is numbered 4.1.","section":"Appendix A (Proof of Theorem 4.1)"}],"minor_comments":[{"comment":"The text says the same strategy as Sec. 4.1 can be used to solve the transposed linear system, but for a non-symmetric J the transpose is generally not the same matrix as the forward system, so the LU factorization from the forward pass cannot be reused directly; this should be clarified.","section":"Sec. 4.2, after Eq. (15)"},{"comment":"The Kronecker-product expression in Eq. (13) is not fully clear: the identity matrix I_m acts on the batch dimension, but the product with (I - Δt a_ij J) seems to mix state and batch indices. The matrix-free reshape interpretation in Eq. (14) is correct, but the notation in (13) should be cleaned up to avoid confusion.","section":"Sec. 4.1, Eq. (13)"},{"comment":"The adjoint equations in the ERK proof appear to have a minor indexing issue in the last line of Eq. (26) (the factor b_i is omitted on the right-hand side of the second equation). While the final implementation in Eq. (30) looks correct, the proof's intermediate formulas should be corrected for consistency.","section":"Appendix A, Eq. (26)"},{"comment":"The exact time-step sizes used for each method and dataset are mentioned in the text but are not collected in one place. A table summarizing the step sizes for explicit, implicit, and IMEX methods across GRAND, KS, and Burgers would improve reproducibility.","section":"Sec. 5 and Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is sound and the experimental results are encouraging for the demonstrated cases. However, the generality of the method is overstated in the abstract, and the experimental validation lacks statistical rigor (no error bars, no final test accuracy table for GRAND). The adjoint proof is only sketched. These issues are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a solid engineering paper. The genuinely new piece is the combination of IMEX-Runge-Kutta time integration with a reverse-accurate discrete adjoint and a Kronecker-product mini-batching trick that reduces the implicit solve to a single small linear system across the batch, solved with off-the-shelf PETSc linear solvers. That combination works: on Kuramoto-Sivashinsky and Burgers, they report wall-clock speedups of 47x over Dopri5 and 33x over Crank-Nicolson, and the convergence plots look consistent with the claimed stability gains. The code is available and built on a real solver stack, so the results are reproducible in principle.\n\nWhere I would push back is the scope of the headline claim. 'Can train challenging neural ODEs where both explicit and fully implicit methods are intractable' is conditional on having a known, constant, linear part J in the right-hand side, and every experiment supplies J externally: -I for GRAND, finite-difference stencils for the PDEs. Nothing in the paper shows how to get a good split when one isn't handed to you by the physics or the architecture. If J must be learned, the LU-reuse that makes the linear solves cheap degrades to once per optimizer iteration; if no useful linear part exists, the explicit part can still be stiff and the stability story weakens. The Limitations section does not list this condition, which makes it easy to overread the contribution.\n\nSmaller soft spots: the appendix proof of the discrete adjoint is carried out for ERK and the extension to IMEX is asserted rather than shown. The GRAND experiments would be stronger with a final accuracy table and error bars across restarts. The NFE numbers in Tables 1-2 exclude linear-solver costs, which are central to implicit methods, though the wall-clock times do include them. None of these are fatal. The core method is sound and the efficiency claims for partitioned stiff systems appear to hold.\n\nI would send this to review. The referee should ask the authors to state the constant-linear-part assumption as a limitation, add a learned-J or sensitivity experiment if they can, complete the adjoint proof, and report final test accuracies. For someone working on neural ODEs for scientific ML, this is a practical, useful contribution.","headline":"A practical semi-implicit neural ODE trainer with real speedups on stiff partitioned systems, but the headline claim is narrower than it looks: every experiment hard-codes the linear part.","tokens_in":18867,"tokens_out":3209,"would_cite":true,"duration_ms":26963,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["65L06","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that splitting a neural ODE into a nonlinear part handled explicitly and a fixed linear part handled implicitly, with a discrete-adjoint backward pass, allows stable training of stiff systems at time steps two orders of…","keywords":["neural ordinary differential equations","semi-implicit integration","implicit-explicit Runge-Kutta","stiff ODEs","discrete adjoint","graph neural networks","scientific machine learning","chaotic dynamics"],"falsifier":"Run the GRAND Cora benchmark with the linear operator learned instead of fixed to $-I$ in Eq. (19), updating it every optimizer step so factorizations must be recomputed each epoch; if per-epoch time and loss no longer beat explicit Adams and RK4 by a large margin, then the reusable constant linear system, not IMEX stability alone, is the load-bearing mechanism and the method's generality is limited to problems with a known fixed linear split.","tokens_in":17856,"feed_emoji":"⚡","tokens_out":14275,"duration_ms":113001,"temperature":0.7,"pith_summary":"Neural ODEs trained with explicit time integrators become unstable on stiff problems, forcing tiny time steps that make training impractical. This paper claims that the instability can be removed by splitting the right-hand side into a nonlinear part $G(u)$, handled explicitly, and a fixed linear part $J u$, handled implicitly by an implicit-explicit Runge--Kutta (IMEX-RK) integrator. Because the implicit part is linear, each stage solve is a linear system with a fixed matrix whose factorization can be reused, and gradients are computed by a discrete adjoint that matches the forward discretization. The result, called SINODE, is claimed to train stiff graph and PDE models with time steps two orders of magnitude larger than explicit methods, and to succeed where both explicit and fully implicit training fail.","feed_headline":"Semi-implicit split trains stiff neural ODEs at 100x larger time steps","feed_subtitle":"Treating the linear part implicitly and nonlinear part explicitly keeps training stable where full solvers fail.","key_machinery":"The load-bearing object is the additive split of the vector field into a nonlinear $G(u)$ and a linear $J u$, plus an IMEX-RK time integrator that advances $G$ explicitly and $Ju$ implicitly. Each implicit stage reduces to the constant-coefficient linear system in Eq. (12), $(I-\\Delta t\\,\\tilde a_{ii}J)U^{(i)}=u_n+\\Delta t\\sum_{j<i}a_{ij}G^{(j)}+\\Delta t\\sum_{j<i}\\tilde a_{ij}JU^{(j)}$, which avoids nonlinear Newton iteration altogether; with $J$ constant, the factorization of $I-\\Delta t\\,\\tilde a_{ii}J$ is reused across stages, steps, and mini-batches. The backward pass is the discrete adjoint of the same IMEX-RK map (Theorem 4.1), giving a reverse-accurate gradient whose memory cost depends on the network, not on the number of solver iterations or time steps.","core_discovery":"The paper's central claim is that a learnable ODE $\\frac{du}{dt}=G(u)+Ju$ can be trained stably and efficiently when $G$ is integrated explicitly and $Ju$ implicitly, using IMEX-RK time stepping and a reverse-accurate discrete adjoint. The linearity of the implicit term turns every implicit stage into the linear system $(I-\\Delta t\\,\\tilde a_{ii}J)U^{(i)}=\\text{known}$, whose coefficient matrix does not change across stages, time steps, or (when $J$ is fixed) mini-batches, so off-the-shelf linear solvers and reused factorizations become available during training. On the GRAND graph benchmark and on Kuramoto--Sivashinsky and viscous Burgers learning problems, the paper reports stable training at step sizes roughly two orders of magnitude larger than explicit Runge--Kutta methods permit, with far fewer network evaluations per epoch and successful training in cases where fully implicit methods blow up or fail to converge.","pith_inferences":["Beyond the tested benchmarks, any stiff learning problem with a known linear operator, such as a diffusion, Laplacian, or spectral term, could use the same split, but the paper does not demonstrate a general recipe for finding such splits in arbitrary neural networks.","Allowing $J$ to be learned while held fixed for several epochs at a time would likely preserve most of the factorization-reuse benefit and extend SINODE to problems without a prescribed linear part; this is an extension, not a paper claim.","Because each fixed factorization is reused more often as the grid resolution or the number of time steps grows, the speed advantage over explicit training should widen on larger problem sizes than those tested.","The failures reported for fully implicit methods suggest that the practical bottleneck in stiff neural ODE training is the cost and fragility of Newton iterations on ill-conditioned systems, and that replacing only the stiff part with a linear implicit solve removes that bottleneck; this diagnosis is inferred from the reported blow-ups rather than proved."],"forward_implications":["Stiff neural ODEs that previously needed step sizes near $10^{-3}$ can use step sizes near $1$ in the tested problems, cutting per-epoch network evaluations from tens of thousands to tens or hundreds.","On the GRAND graph benchmark, SINODE trains stably where explicit RK4 and explicit Adams diverge and where fully implicit Crank--Nicolson and implicit Adams blow up on Cora.","For the Kuramoto--Sivashinsky and viscous Burgers learning tasks, SINODE remains trainable at fine grid resolutions where explicit methods require step sizes around $10^{-7}$ and fully implicit methods suffer nonlinear-solver divergence.","The discrete adjoint computes gradients consistent with the discrete forward pass, so the large-step training is not biased by the discretization mismatch of continuous adjoint methods.","When $J$ is fixed, one LU factorization is shared across every stage, time step, and mini-batch within an epoch, so training cost scales with network evaluations rather than with the cost of solving the linear systems."],"supporting_citations":[{"why":"introduces neural ODEs and the continuous adjoint training paradigm that SINODE's discrete adjoint backward pass replaces.","marker":"Chen et al. 2018"},{"why":"supplies the IMEX-RK framework and stability properties that the SINODE forward pass exploits.","marker":"Ascher, Ruuth, and Spiteri 1997"},{"why":"provides the additive Runge--Kutta coefficients used for the high-order IMEX-RK3/4/5 integrators in the experiments.","marker":"Kennedy and Carpenter 2003a"},{"why":"gives the IMEX-RK2 tableau used as the second-order semi-implicit integrator in comparisons.","marker":"Pareschi and Russo 2005"},{"why":"defines the GRAND graph diffusion model whose stiff ODE is split into a nonlinear attention term and a linear identity term.","marker":"Chamberlain et al. 2021"},{"why":"provides the discrete-adjoint and checkpointing neural ODE framework that SINODE builds on for reverse-accurate gradients.","marker":"Zhang and Zhao 2022"},{"why":"supplies the discrete adjoint derivation methodology used to prove the gradient formula in Theorem 4.1.","marker":"Zhang and Sandu 2014"}],"fun_headline_variants":["Semi-implicit neural ODEs handle stiffness at 100x steps","Stable and efficient neural ODE training via IMEX splitting","IMEX-RK time stepping stabilizes stiff neural ODE training","Train stiff neural ODEs with 100x larger steps via splitting","Reverse-accurate adjoint enables stable IMEX neural ODEs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole speed and stability advantage rests on the model's dynamics being expressible as a nonlinear part plus a fixed linear part, because without a fixed linear part there is no cheap reusable linear equation to solve.","fun_headline_variants_meta":{"raw":{"variants":["Semi-implicit neural ODEs handle stiffness at 100x steps","Stable and efficient neural ODE training via IMEX splitting","IMEX-RK time stepping stabilizes stiff neural ODE training","Train stiff neural ODEs with 100x larger steps via splitting","Reverse-accurate adjoint enables stable IMEX neural ODEs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000397,"raw_usage":{"total_tokens":2034,"prompt_tokens":853,"completion_tokens":1181,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":469,"completion_tokens_details":{"reasoning_tokens":1087}},"tokens_in":469,"tokens_out":1181,"duration_ms":10534,"temperature":1.0,"reasoning_tokens":1087,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:04:39.409923+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the GRAND Cora benchmark with the linear operator learned instead of fixed to $-I$ in Eq. (19), updating it every optimizer step so factorizations must be recomputed each epoch; if per-epoch time and loss no longer beat explicit Adams and RK4 by a large margin, then the reusable constant linear system, not IMEX stability alone, is the load-bearing mechanism and the method's generality is limited to problems with a known fixed linear split.","supporting_citations":[{"cited_title":"Q.; Rubanova, Y.; Bettencourt, J.; Duvenaud, D.; Chen, R","cited_arxiv_id":null,"evidence_quote":"introduces neural ODEs and the continuous adjoint training paradigm that SINODE's discrete adjoint backward pass replaces."},{"cited_title":"M.; Ruuth, S","cited_arxiv_id":null,"evidence_quote":"supplies the IMEX-RK framework and stability properties that the SINODE forward pass exploits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"gives the IMEX-RK2 tableau used as the second-order semi-implicit integrator in comparisons."},{"cited_title":"I.; Bronstein, M.; Webb, S.; and Rossi, E","cited_arxiv_id":null,"evidence_quote":"defines the GRAND graph diffusion model whose stiff ODE is split into a nonlinear attention term and a linear identity term."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the discrete-adjoint and checkpointing neural ODE framework that SINODE builds on for reverse-accurate gradients."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the discrete adjoint derivation methodology used to prove the gradient formula in Theorem 4.1."}],"review_version":1}