REVIEW 3 major objections 5 minor 55 references
Separable nonlinear least-squares parameter estimation for complex dynamic systems
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read For ODE models with partly linear parameters, separable nonlinear least squares matches or beats traditional nonlinear least squares in accuracy while running substantially faster.
desk verdict A systematic simulation study of separable integral matching undermined by an overbroad claim: the NLS baseline is not traditional NLS. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the factorization $F(x;\theta)=g(x;\theta_{\mathrm{NL}})\theta_L$, which separates the parameter vector into a nonlinear block $\theta_{\mathrm{NL}}$ and a linear block $\theta_L$. The machinery is variable projection: define $\hat G(t)=\int_0^t g(\hat x(s);\theta_{\mathrm{NL}})ds$, $\hat B=\int_0^T \hat G^\top(t)\hat G(t)dt$, so that for fixed $\theta_{\mathrm{NL}}$ the optimal linear parameters are $\hat\theta_L(\theta_{\mathrm{NL}})=\hat B^{-1}\int_0^T \hat G^\top(t)(\hat x(t)-\hat\xi)dt$, with $\hat\xi$ given by a similar closed form. Substituting these expressions back into the integral criterion yields the reduced function $M(\theta_{\mathrm{NL}})$, whose dimension equals the number of genuinely nonlinear parameters only. This reduction supplies the three advertised benefits: no initial guesses for linear parameters, a better-conditioned problem, and faster convergence.
What would settle it
Run the same Monte-Carlo benchmarks with an independent, well-established NLS solver—for example a Levenberg-Marquardt or trust-region implementation unrelated to the package used in the paper—and compare MSE and runtime; if the SLS advantage shrinks or disappears, the gain is attributable to the package's NLS implementation rather than to separability.
Extended reading notes
Core claim
The paper studies noisy observations $Y_j(t_i)=x_j(t_i;\theta,\xi)+\epsilon_{ij}$ of a system of ordinary differential equations whose vector field is partly linear in the parameters, $F(x;\theta)=g(x;\theta_{\mathrm{NL}})\theta_L$. After replacing the state $x$ by a smoothed estimate $\hat x$, it minimizes the integral criterion $\int_0^T\|\hat x(t)-\xi-\int_0^t g(\hat x(s);\theta_{\mathrm{NL}})\theta_L\,ds\|^2\,dt$. For fixed nonlinear parameters $\theta_{\mathrm{NL}}$, the initial value $\xi$ and the linear parameters $\theta_L$ are recovered by closed-form linear least squares, leaving only a low-dimensional nonlinear optimization over $\theta_{\mathrm{NL}}$. Across four benchmark systems—an age-structured SIR epidemic model, a seasonally forced Lotka-Volterra model, a GMA metabolic system, and the FitzHugh-Nagumo neuron model—the resulting SLS estimator matches NLS when initial guesses are excellent and beats it, often by large MSE margins, when prior information is medium or low, with shorter computation times.
Load-bearing premise
The paper's numerical evidence assumes that the nonlinear least-squares routine inside the software it used is a fair representative of traditional NLS; if that routine is weaker than other available implementations, the observed advantage of the separable method could be an artifact of the software rather than a general property.
Editorial extensions
If this is right
- For any ODE model of the form $x' = g(x;\theta_{\mathrm{NL}})\theta_L$, SLS eliminates the need for initial guesses on the linear parameters $\theta_L$.
- The MSE advantage of SLS over NLS grows as the quality of prior information on linear parameters degrades from high to medium to low.
- Larger sample sizes widen SLS's advantage, while higher noise reduces it but does not reverse it.
- Errors in linear-parameter estimates propagate into NLS's nonlinear-parameter estimates, so SLS also improves accuracy of nonlinear parameters when linear prior information is poor.
Reading between the lines
- We infer that the closed-form elimination of linear parameters could be paired with sparsity-inducing penalties to extend SLS to high-dimensional ODE systems where the number of rate constants is large, though the paper only discusses regularization as future work.
- Because the reduction exploits only algebraic structure, it should transfer to other inverse problems with linear-in-parameter forward maps (e.g., delay or PDE models), but the paper's evidence is limited to four ODE benchmarks.
- A hybrid strategy that starts with SLS and, when high-quality priors are available, switches to NLS might capture the best of both; the paper notes the crossover point but does not propose such a procedure.
- An independent implementation of NLS (separate from the package used in the paper) would isolate whether the benefit comes from separability or from the package's particular optimizer.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents a simulation study comparing separable nonlinear least squares (SLS) with nonlinear least squares (NLS) for parameter estimation in ODE models that are linear in a subset of parameters. The authors propose profiling out the linear parameters, yielding a reduced integral criterion (Eq. (7)), and compare this with an NLS estimator that minimizes the full integral criterion (5) via the same smoothing approach. The comparison is carried out with the R package simode on four benchmark models (SIR, Lotka-Volterra, GMA, FitzHugh-Nagumo), varying sample size, noise level, and quality of prior information for the linear parameters. The primary evidence is the MSE ratios for linear and nonlinear parameters in Tables 1-4, supported by loss scatterplots and computation-time comparisons. The paper concludes that SLS is at least as accurate as NLS and usually superior, with substantial computational savings, and recommends SLS for dynamic systems with significant linear features.
Significance. If the comparison were against a representative implementation of traditional nonlinear least squares, the results would provide practically useful evidence for the value of separability in ODE parameter estimation. The paper has clear strengths: it provides reproducible code on GitHub, uses a publicly available package (simode), and the experimental design holds nonlinear-parameter priors fixed between methods, so the comparison is not obviously biased in that respect. The MSE tables show a consistent and often large advantage for SLS when prior information on linear parameters is medium or low. However, the central claim as stated in the abstract and conclusions overreaches: the NLS baseline used throughout is not the standard data-fitting NLS used in the ODE literature, and several high-prior cells in the paper's own tables contradict the 'at least as accurate' phrasing. Thus the significance of the empirical finding depends on a revision that either benchmarks against a true traditional NLS implementation or carefully reframes the claims to refer to integral-matching NLS within the simode framework.
major comments (3)
- [Section 2.2, Eq. (6); Section 3] The estimator called NLS in this paper is defined by Eq. (6) as the minimizer of the smoothed integral criterion (5), implemented in simode with default smoothing and optimization settings. This is not the traditional nonlinear least-squares estimator for dynamic systems, which minimizes the sum of squared deviations between the observed data and the ODE solution obtained by numerical integration. The abstract and Section 4 generalize to 'traditional nonlinear least-squares,' but the paper never benchmarks against such an independent implementation. The headline claim therefore rests on a comparison between two integral-matching estimators inside one R package, and the stated generality to 'traditional NLS' is not supported by the presented evidence.
- [Tables 1 and 2, high-prior rows] The abstract claims SLS is 'at least as accurate' as NLS, but the paper's own MSE tables contain high-prior cells where NLS is clearly more accurate. For example, Table 1 low noise: SIR high 0.9 and GMA high 0.9; Table 2 low noise: GMA high 0.7, and high noise: SIR high 0.7 and GMA high 0.5. These ratios mean NLS MSE is roughly half to two-thirds of SLS MSE in some cases. The paper's inline conclusion (i) appropriately says accuracy is comparable under high-quality priors, but the abstract's unqualified 'at least as accurate' is contradicted by these cells. This needs either a qualification in the abstract or a demonstration that these differences are within Monte Carlo error.
- [Figure 2; Section 3.2] Figure 2 plots NLS losses evaluated on criterion (5) against SLS losses evaluated on criterion (7). These are different objective functions, so the scatterplot does not provide a valid apples-to-apples visual comparison of fit quality. The authors state in Section 3.1 that the two criteria are different, but the figure is still used in Section 3.2 as evidence of SLS superiority. The MSE tables are the primary evidence and are not subject to this problem, but the figure as presented could mislead readers; either compare both methods on a common criterion (e.g., prediction error or MSE) or remove the figure's comparative interpretation.
minor comments (5)
- [Section 2.2] To prevent confusion with the standard NLS estimator in the ODE literature, consider renaming the estimator in Eq. (6) as 'integral-based NLS' or 'smoothed NLS,' and use that terminology consistently in the abstract and conclusions.
- [Section 3.2, Tables 1-4] The MSE ratios are reported without Monte Carlo standard errors or confidence intervals. Ratios close to 1 (e.g., Table 3 ltk medium 0.9) may be indistinguishable from noise; adding uncertainty measures would strengthen the quantitative claims.
- [Section 3.1.4] The footnote that initial guesses for parameters were 'assured to be positive' is not defined in the main text; for reproducibility, specify how this assurance was implemented, especially since the FitzHugh-Nagumo parameters in the chosen setup include positive values only.
- [Figure 3, panel B] The observation that NLS computation time increases with the quality of prior information is counterintuitive and deserves a brief explanation, as one might expect better starting values to speed convergence.
- [Abstract] The phrase 'novel method of separable nonlinear least-squares optimization' overstates novelty; variable projection and separable NLS date back to Golub and Pereyra (1973). The contribution here is the application to ODE parameter estimation via integral matching, which should be stated more precisely.
Circularity Check
No circularity: the SLS estimator is derived in-paper from the integral criterion, and the simulation comparison is benchmarked against known parameters rather than fitted to the conclusion.
full rationale
The paper's derivation chain is self-contained with respect to its central claim. Section 2.2 introduces the integral criterion (5), defines NLS as its minimizer in (6), and then algebraically profiles the linear parameters to obtain the SLS estimator in (7) and (8). This is a mathematical derivation, not a circular definition. The simulation study in Section 3 generates synthetic data from known ODE parameters and compares MSEs and losses, so the reported accuracy is assessed against independently specified truth, not against any quantity fitted from the conclusion. The paper cites prior work by the authors, including Dattner & Klaassen (2015) and the simode package (Yaari & Dattner 2019), but these citations provide context, statistical background, and software implementation rather than a load-bearing uniqueness or existence theorem; the SLS formulas themselves are exhibited in the paper. One could question whether the NLS baseline in (6), based on integral matching, represents 'traditional' data-fitting NLS as practitioners use it, but that is a benchmarking-fairness concern, not a circularity of derivation. Therefore no circular step is identified.
Assumptions & free parameters
assumptions (4)
- domain assumption The integral-matching estimator based on a smoother is statistically valid, as established in Dattner & Klaassen (2015) and Gugushvili & Klaassen (2012).
- domain assumption The simulation study assumes the ODE models are correctly specified and the measurement errors are independent with zero mean and finite variance, as stated in equation (4).
- domain assumption The smoother x_hat(t) chosen by cross-validation is an accurate approximation to the true trajectory, and numerical integration of the ODE is accurate.
- domain assumption The benchmark models are statistically identifiable under the simulated observation schemes.
Cite this review
Pith. "Pith review of Separable nonlinear least-squares parameter estimation for complex dynamic systems." pith.science (2026). https://pith.science/paper/I73X4BWT
@misc{pith2026190803717,
author = {Pith},
title = {Pith review of: Separable nonlinear least-squares parameter estimation for complex dynamic systems},
year = {2026},
howpublished = {\url{https://pith.science/paper/I73X4BWT}},
note = {Machine review of arXiv:1908.03717}
}
read the original abstract
Nonlinear dynamic models are widely used for characterizing functional forms of processes that govern complex biological pathway systems. Over the past decade, validation and further development of these models became possible due to data collected via high-throughput experiments using methods from molecular biology. While these data are very beneficial, they are typically incomplete and noisy, so that inferring parameter values for complex dynamic models is associated with serious computational challenges. Fortunately, many biological systems have embedded linear mathematical features, which may be exploited, thereby improving fits and leading to better convergence of optimization algorithms. In this paper, we explore options of inference for dynamic models using a novel method of {\it separable nonlinear least-squares optimization}, and compare its performance to the traditional nonlinear least-squares method. The numerical results from extensive simulations suggest that the proposed approach is at least as accurate as the traditional nonlinear least-squares, but usually superior, while also enjoying a substantial reduction in computational time.
Figures
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sentence := ...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Basu, S., & Bresler, Y. (2000). The stability of nonlinear least squares problems and the Cram \'e r-Rao bound . IEEE Transactions on Signal Processing\/ , 48\/ (12), 3426--3436
work page 2000
-
[4]
Brunel, N. J. B. (2008). Parameter estimation of ODE's via nonparametric estimators . Electron. J. Stat.\/ , 2\/ , 1242--1267
work page 2008
-
[5]
Campbell, D., & Steele, R. J. (2012). Smooth functional tempering for nonlinear differential equation models. Stat. Comput.\/ , 22\/ (2), 429--443
work page 2012
-
[6]
Chen, S., Shojaie, A., & Witten, D. M. (2017). Network reconstruction from high-dimensional ordinary differential equations. Journal of the American Statistical Association\/ , 112\/ (520), 1697--1707
work page 2017
-
[7]
Chou, I.-C., & Voit, E. O. (2009). Recent developments in parameter estimation and structure identification of biochemical and genomic systems. Math. Biosci.\/ , 219\/ (2), 57--83
work page 2009
-
[8]
Chung, J., & Nagy, J. G. (2010). An efficient iterative approach for large-scale separable nonlinear inverse problems. SIAM J. Sci. Comput.\/ , 31\/ (6), 4654--4674
work page 2010
Show all 55 references
-
[9]
Dattner, I. (2015). A model-based initial guess for estimating parameters in systems of ordinary differential equations. Biometrics\/ , 71\/ (4), 1176--1184
2015
-
[10]
Dattner, I., & Gugushvili, S. (2018). Application of one-step method to parameter estimation in ODE models. Stat. Neerl.\/ , 72\/ (2), 126--156
2018
-
[11]
Dattner, I., & Huppert, A. (2018). Modern statistical tools for inference and prediction of infectious diseases using mathematical models. Stat. Methods Med. Res.\/ , 27\/ (7), 1927--1929
2018
-
[12]
Dattner, I., & Klaassen, C. A. J. (2015). Optimal rate of direct estimators in systems of ordinary differential equations linear in functions of the parameters. Electron. J. Statist.\/ , 9\/ (2), 1939--1973
2015
-
[13]
E., Jurkevitch, E., & Huppert, A
Dattner, I., Miller, E., Petrenko, M., Kadouri, D. E., Jurkevitch, E., & Huppert, A. (2017). Modelling and parameter inference of predator--prey dynamics in heterogeneous environments using the direct integral approach. Journal of The Royal Society Interface\/ , 14\/ (126), 20160525
2017
-
[14]
B., Zheng , P., Manohar , K., Brunton , S
Erichson , N. B., Zheng , P., Manohar , K., Brunton , S. L., Kutz , J. N., & Aravkin , A. Y. (2018). Sparse principal component analysis via variable projection . arXiv e-prints\/ . ://arxiv.org/abs/1804.00341
2018 arXiv
-
[15]
Fan , J., & Gijbels , I. (1996). Local polynomial modelling and its applications \/ , vol. 66 of Monographs on Statistics and Applied Probability \/ . London: Chapman & Hall
1996
-
[16]
FitzHugh, R. (1961). Impulses and physiological states in theoretical models of nerve membrane. Biophysical Journal\/ , 1\/ (6), 445--466
1961
-
[17]
FitzHugh, R. (1969). Mathematical models of excitation and propagation in nerve. In H. P. Schwan (Ed.) Biological engineering\/ , vol. 9 of Inter-university electronics series\/ , chap. 1, (pp. 1--85). New York, NY: McGraw-Hill
1969
-
[18]
Gan , M., Chen , C. L. P., Chen , G., & Chen , L. (2018). On some separated algorithms for separable nonlinear least squares problems. IEEE Transactions on Cybernetics\/ , 48\/ (10), 2866--2874
2018
-
[19]
Gennemark , P., & Wedelin , D. (2007). Efficient algorithms for ordinary differential equation model identification of biological systems. IET Systems Biology\/ , 1\/ (2), 120--129
2007
-
[20]
Girolami , M., & Calderhead , B. (2011). Riemann manifold Langevin and Hamiltonian Monte Carlo methods. With discussion and authors' reply . J. R. Stat. Soc., Ser. B, Stat. Methodol. \/ , 73\/ (2), 123--214
2011
-
[21]
Golub, G., & Pereyra, V. (2003). Separable nonlinear least squares: The variable projection method and its applications . Inverse Probl. \/ , 19\/ (2), 1--26
2003
-
[22]
H., & Pereyra, V
Golub, G. H., & Pereyra, V. (1973). The differentiation of pseudo-inverses and nonlinear least squares problems whose variables separate. SIAM J. Numer. Anal.\/ , 10\/ (2), 413--432
1973
-
[23]
J., & Silverman , B
Green , P. J., & Silverman , B. W. (1994). Nonparametric regression and generalized linear models: a roughness penalty approach \/ , vol. 58 of Monographs on Statistics and Applied Probability \/ . London: Chapman & Hall
1994
-
[24]
Gugushvili, S., & Klaassen, C. A. J. (2012). \( n \)-consistent parameter estimation for systems of ordinary differential equations: bypassing numerical integration via smoothing . Bernoulli \/ , 18\/ (3), 1061--1098
2012
-
[25]
Himmelblau, D., Jones, C., & Bischoff, K. (1967). Determination of rate constants for complex kinetics models. Industrial & Engineering Chemistry Fundamentals\/ , 6\/ (4), 539--543
1967
-
[26]
L., & Huxley, A
Hodgkin, A. L., & Huxley, A. F. (1952). A quantitative description of membrane current and its application to conduction and excitation in nerve. The Journal of Physiology\/ , 117\/ (4), 500--544
1952
-
[27]
M., & Silverman , B
Johnstone , I. M., & Silverman , B. W. (2005). Empirical Bayes selection of wavelet thresholds . Ann. Stat. \/ , 33\/ (4), 1700--1752
2005
-
[28]
H., & Sylvestre, E
Lawton, W. H., & Sylvestre, E. A. (1971). Elimination of linear parameters in nonlinear regression. Technometrics\/ , 13\/ (3), 461--467
1971
-
[29]
Lotka , A. J. (1956). Elements of mathematical biology \/ . New York : Dover Publications, Inc. Unabridged republication of the first edition published under the title: Elements of physical biology
1956
-
[30]
May , R. M. (2001). Stability and complexity in model ecosystems. With a new introduction by the author \/ . Princeton, NJ: Princeton University Press, 2nd ed
2001
-
[31]
McGoff, K., Mukherjee, S., & Pillai, N. (2015). Statistical inference for dynamical systems: A review . Statist. Surv.\/ , 9\/ , 209--252
2015
-
[32]
Mullen, K., & van Stokkum, I. (2007). TIMP: An R package for modeling multi-way spectroscopic measurements . Journal of Statistical Software\/ , 18\/ (3), 1--46
2007
-
[33]
Mullen, K. M. (2008). Separable nonlinear models: theory, implementation and applications in physics and chemistry\/ . Phd thesis, Vrije Universiteit
2008
-
[34]
Nagumo, J., Arimoto, S., & Yoshizawa, S. (1962). An active pulse transmission line simulating nerve axon. Proceedings of the IRE\/ , 50\/ (10), 2061--2070
1962
-
[35]
Peschel, M., & Mende, W. (1986). The predator-prey model: do we live in a Volterra world?\/ . Springer
1986
-
[36]
Ramsay , J., & Hooker , G. (2017). Dynamic data analysis. Modeling data with differential equations \/ . Springer Series in Statistics . New York, NY: Springer
2017
-
[37]
O., Hooker, G., Campbell, D., & Cao, J
Ramsay, J. O., Hooker, G., Campbell, D., & Cao, J. (2007). Parameter estimation for differential equations: a generalized smoothing approach. J. R. Stat. Soc., Ser. B, Stat. Methodol. \/ , 69\/ (5), 741--796
2007
-
[38]
Ruhe, A., & Wedin, P. . (1980). Algorithms for separable nonlinear least squares problems. SIAM Review\/ , 22\/ (3), 318--337
1980
-
[39]
Savageau , M. A. (1976). Biochemical systems analysis. A study of function and design in molecular biology \/ , vol. 6739 of Advanced Book Program \/ . London etc. : Addison-Wesley Publishing Company
1976
-
[40]
Schittkowski , K. (2002). Numerical data fitting in dynamical systems. A practical introduction with applications and software \/ . Dordrecht: Kluwer Academic Publishers
2002
-
[41]
Tufte, E. R. (2001). The visual display of quantitative information\/ . Cheshire, CT: Graphics Press, 2nd ed
2001
-
[42]
Varah, J. (1982). A spline least squares method for numerical parameter estimation in differential equations. SIAM Journal on Scientific and Statistical Computing\/ , 3\/ (1), 28--46
1982
-
[43]
N., & Ripley , B
Venables , W. N., & Ripley , B. D. (2002). Modern applied statistics with S \/ . Statistics and Computing . New York, NY: Springer, 4th ed
2002
-
[44]
Vilela, M., Borges, C. C. H., Vinga, S., Vasconcelos, A. T. R., Santos, H., Voit, E. O., & Almeida, J. S. (2007). Automated smoother for the numerical decoupling of dynamics models. BMC Bioinformatics\/ , 8\/ (1), 305
2007
-
[45]
Vissing Mikkelsen , F., & Hansen , N. R. (2017). Learning large scale ordinary differential equation systems . arXiv e-prints\/ . ://arxiv.org/abs/1710.09308
2017 arXiv
-
[46]
Voit, E. O. (2000). Computational analysis of biochemical systems: A practical guide for biochemists and molecular biologists\/ . Cambridge University Press
2000
-
[47]
Voit, E. O. (2013). Biochemical systems theory: a review. ISRN Biomathematics\/ , 2013\/ , Article ID 897658
2013
-
[48]
Volterra , V. (1926). Fluctuations in the abundance of a species considered mathematically . Nature \/ , 118\/ , 558--560
1926
-
[49]
Vuja c i \'c , I., Dattner, I., Gonz \'a lez, J., & Wit, E. (2015). Time-course window estimator for ordinary differential equations linear in the parameters. Stat. Comput.\/ , 25\/ (6), 1057--1070
2015
-
[50]
Wasserman , L. (2006). All of nonparametric statistics \/ . Springer Texts in Statistics . New York, NY: Springer
2006
-
[51]
Wickham , H. (2009). ggplot2. Elegant graphics for data analysis \/ . Use R! New York, NY: Springer
2009
-
[52]
Wu, L., Qiu, X., Yuan, Y.-x., & Wu, H. (2019). Parameter estimation and variable selection for big systems of linear ordinary differential equations: A matrix-based approach. J. Amer. Statist. Assoc.\/ , 114\/ (526), 657--667
2019
-
[53]
Yaari , R., & Dattner , I. (2018). simode: R Package for statistical inference of ordinary differential equations using separable integral-matching . arXiv e-prints\/ . ://arxiv.org/abs/1807.04202
2018 arXiv
-
[54]
Yaari, R., & Dattner, I. (2019). simode: Statistical inference for systems of ordinary differential equations using separable integral-matching\/ . R package version 1.1.4. ://CRAN.R-project.org/package=simode
2019
-
[55]
Yaari, R., Dattner, I., & Huppert, A. (2018). A two-stage approach for estimating the parameters of an age-group epidemic model from incidence data. Stat. Methods Med. Res.\/ , 27\/ (7), 1999--2014
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.