{"id":"99165934-d189-4349-8b50-73a6b81fb0ee","arxiv_id":"2411.16475","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"NonSysId is an open-source MATLAB package that combines iterative orthogonal forward regression with PRESS-based term selection and simulation-based model choice for NARMAX identification.","lead":"NonSysId is a new open-source MATLAB package for building NARMAX models of nonlinear systems from input-output data. It combines iterative orthogonal forward regression with PRESS-based cross-validation and simulation-based model choice, aiming to work without a separate validation dataset.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The iOFRS stability filter in Sec. II.A tests only constant inputs u=0 and u=1; for nonlinear ARX models this does not certify stability under other bounded inputs, so the 'robust generalisation without validation' claim rests on an unproven heuristic.","rationale":"The reader's weakest-assumption pick is exactly the one I would choose: the constant-input stability filter in Section II.A is the key mechanism separating NonSysId from earlier OFR packages, and it is the only guard against long-horizon divergence when no validation set exists. My stress test sharpens the concern by giving a concrete polynomial NARX form that passes the two-point test yet is unstable for u=0.5, showing that the filter is not a general stability certificate. This is not a stylistic objection: if the filter can admit unstable models, then the BIC-based selection in step 3 of iOFRS has no way to know that a model which fits the training record well will diverge on unseen data. The paper provides no theorem, no counterexample analysis, and no experimental stress test of this heuristic; the synthetic examples are noise-free and essentially recover the exact generating equation, so they do not exercise the stability filter in a meaningful way. I therefore keep the reader's CONDITIONAL verdict: the package may well work in practice and the open-source implementation is a useful contribution, but the advertised claims of robust generalisation and simulation stability are not supported by the submitted evidence. The proposed random-search test is cheap and would settle whether the heuristic is merely unproven or actually false.","tokens_in":51,"tokens_out":8820,"duration_ms":131023,"concrete_test":"Generate 10,000 random polynomial NARX candidate models (degree <= 3, lags <= 2) with coefficients normalised so that a nontrivial fraction pass the Section II.A filter (u=0, u=1, epsilon=10^-2), using both zero and small nonzero initial conditions. Simulate each surviving model free-run under u(t)=0.5, u(t)=0.5+0.5sin(2*pi*t/10), and a PRBS switching between 0 and 1. If any survivor diverges or exceeds a boundedness threshold, the u=0/u=1 test is not a stability certificate and the generalisation claim is unproven; if none diverge across a broad search, the heuristic is empirically supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the iOFRS stability filter in Section II.A (steps 1-2): candidate models are retained only if free-run output stays bounded with variance at most epsilon for the two constant inputs u(t)=0 and u(t)=1. Since the central claims are robust generalisation and long-horizon simulation stability without a validation dataset, this filter is the only mechanism that prevents iOFRS from selecting a model that fits the training record but diverges in free run under the actual excitation. But boundedness on two constant inputs is not a stability certificate for a nonlinear ARX model: stability of polynomial maps can depend on the input trajectory. For example, y(t)=0.5y(t-1)+4u(t-1)(1-u(t-1))y(t-1)+y(t-1)^2 reduces to y=0.5y+y^2 when u(t)=0 or u(t)=1, so from a small or zero initial condition it passes the Section II.A test, yet for u(t)=0.5 the linear coefficient becomes 1.5 and any nonzero initial condition diverges. Thus a model can pass the filter and still be unstable under another bounded input. The paper gives no proof, no counterexample analysis, and no empirical stress test for this heuristic; the noise-free synthetic examples in Section III do not exercise it. If the filter is unsound, the BIC/simulation-error selection in step 3 has no safety net, and the headline claim of robust generalisation without validation data is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents NonSysId, an open-source MATLAB package for identifying polynomial NARX/NARMAX models. It implements an iterative Orthogonal Forward Regression (iOFR) variant with PRESS-statistic-based term selection and a simulation-based model selection step, claiming robust generalization without a separate validation dataset. The manuscript describes the algorithm, four computational-speed-up methods, and two examples: a noise-free synthetic DC-motor NARX model and a real electromechanical system.","tokens_in":14463,"tokens_out":5023,"duration_ms":44131,"significance":"NonSysId addresses a real need: open-source NARMAX toolkits based on plain OFR/ERR can overfit and produce unstable free-run models, and validation data are often scarce. The package integrates known methods (iOFR, PRESS, linear-ARX-based candidate reduction) and provides reproducible code and examples. The claimed contribution—simulation-stable, parsimonious model selection without held-out data—is attractive for online and neurophysiological applications, but the current evidence and the stability-filter justification are insufficient to establish it. If the stability heuristic can be justified or replaced, the package would be a useful methodological contribution.","major_comments":[{"comment":"The stability filter in Section II.A (steps 1-2) tests only u(t)=0 and u(t)=1 with Var(ŷ(t)) ≤ ε, ε=10^-2. Boundedness under two constant inputs is not a stability certificate for polynomial NARX models: for example, y(t)=0.5y(t-1)+4u(t-1)(1-u(t-1))y(t-1)+y(t-1)^2 satisfies the filter for u=0 and u=1 from a zero initial condition, yet for u(t)=0.5 the linear coefficient becomes 1.5 and nonzero initial conditions diverge. Because step 3 (BIC on simulation error) is the only remaining selection mechanism, the headline claim that iOFRS can 'produce models with long-horizon prediction capabilities and simulation stability' is not supported by the stated test. The paper provides no proof, counterexample analysis, or empirical stress test of this heuristic, and the synthetic examples in Section III do not exercise it (they use zero noise and inputs unlike constant signals). Please either remove the strong claim, add a rigorous sufficient condition, or conduct experiments with a diverse set of inputs and initial conditions.","section":"II.A"},{"comment":"The experimental evidence does not substantiate 'robust model generalisation without the need for a separate validation dataset.' The synthetic case (Section III.A) is noise-free (residual variances 1.60e-25 and 8.22e-18 versus training variances 0.069 and 0.0581), so it provides no evidence about generalisation under realistic disturbances, and no comparison with the original OFR/ERR algorithm or existing packages is reported. Moreover, the paper's own validation tests fail: the text states that residuals in both cases (a) and (b) 'are not entirely white noise' and that in Fig. 5 the cross-correlation functions 'are not completely within the tolerance bounds', which is explicitly acknowledged as indicating some model bias. Since these tests are the paper's only evidence of model validity on training data, the generalization claim requires either a properly designed noise experiment, an independent held-out record (even if the method does not require one), or a benchmark against an existing validation-based approach.","section":"III.A-B"},{"comment":"The BIC-based simulation selection in Section II.A (step 3) and Algorithm 1 (line 35) is not specified precisely enough to be reproducible or to assess the parsimony claim. The manuscript states only that BIC is 'calculated based on the simulated error variance (MSSE)' but does not give the formula, the effective number of parameters (including the bias term and any initial-condition parameters), or the initialization rule for the free-run simulation used to compute MSSE. Since the free-run trajectory of a nonlinear ARX model can depend sensitively on initial conditions, and since the central claim is that iOFRS selects models that simulate stably over long horizons, the missing initialization and BIC details make the selection rule ambiguous. Please provide the exact BIC definition and the simulation protocol (initial conditions, burn-in, and the exact training/validation split within the record).","section":"II.A step 3 / Algorithm 1"}],"minor_comments":[{"comment":"The title header contains a typo: 'im proved' should be 'improved'.","section":"Title"},{"comment":"Section II.A uses both 'iFRO' and 'iOFR' for the same algorithm; please standardize the notation.","section":"II.A"},{"comment":"Algorithm 1 (line 8) composes D'' = D' ∪ f(D') and line 10 D''_R = D'_m ∪ f(D'_m), but the function f is never defined; state that f generates all monomials up to degree Np.","section":"Algorithm 1"},{"comment":"The ordering of Table I does not match the description in Section III.A that sorting by descending mean squared PRESS error reveals the term-addition sequence; for example, u(t-1) has the largest PRESS value (0.47871) but is listed second, and several entries contain malformed scientific notation such as '1.9004e × 10−5'.","section":"Table I"},{"comment":"The Figure 6 caption refers to 'the system given in Eq. (8)', but the real-data example is from [46] and Eq. (8) is the synthetic DC motor; correct the cross-reference.","section":"Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is a software-description manuscript. The package itself appears to work on the demonstrated examples, but the advertised stability guarantee is not supported by the constant-input test. The revision should either provide a rigorous stability analysis or substantially soften the claims; with that change the paper would be within scope for a software journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: NonSysId is a real, potentially useful package. It implements iterative OFR with PRESS-based term selection, adds a simulation-based selection step (iOFRS), and includes two computational shortcuts (RCT methods 3 and 4). The code is on GitHub, and the synthetic examples recover the generating model almost exactly. That is a solid engineering contribution, and the paper deserves a serious referee.\n\nWhat is genuinely new: the iOFRS variant that selects models by free-run simulation error rather than one-step-ahead error, the integration of PRESS with iOFR, and the two new RCT methods. The package fills a real gap, since no open-source implementation of iOFR existed. The synthetic DC motor example is clean, and the real electromechanical example is plausible, though not benchmarked against other tools.\n\nNow the soft spots, in order of importance. The load-bearing stability filter in Section II.A tests candidate models only on constant inputs u=0 and u=1, accepting a model if the free-run output stays bounded with small variance. For a nonlinear ARX model, boundedness on two constant inputs does not certify stability under other inputs. The stress-test counterexample—a simple polynomial map that passes the test but diverges for u=0.5—is valid. The paper gives no proof, no counterexample analysis, and no empirical stress test for this heuristic. Since the headline claim is \"robust generalisation without a separate validation dataset,\" this filter is the only safety net, and it is unproven. That is not a deal-breaker for the software, but it is a real gap between claim and evidence.\n\nRelated issues: the synthetic example is noise-free, there are no baseline comparisons against SysIdentPy or narmax, and the paper's own residual correlation tests fail in places. The authors acknowledge the residual issue and then argue the residual variance is tiny. That is fair as a description, but it does not support the stronger generalization claim.\n\nAll that said, the paper is honest about its limitations, and the code appears to work as advertised on the examples shown. The math is standard OFR/PRESS material, correctly cited. Self-citation is not a concern here; the cited prior work is the actual basis for the method.\n\nWho is this for? Practitioners who want an iOFR/PRESS NARMAX tool in MATLAB and are willing to treat the stability filter as a heuristic rather than a guarantee. A serious referee should engage with it, with the expectation of major revision: either justify or replace the stability filter, add a noisy synthetic case and at least one baseline comparison, and soften the \"robust generalisation\" claim to match the evidence.","headline":"NonSysId is a legitimate software contribution with a real gap: the open-source iOFR/PRESS implementation is useful, but the central generalization claim rests on an unproven constant-input stability heuristic that the paper neither justifies nor stress-tests.","tokens_in":14983,"tokens_out":2124,"would_cite":true,"duration_ms":19919,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces NonSysId, a MATLAB package that selects NARMAX model terms using iterative orthogonal forward regression, PRESS-statistic cross-validation, and simulation-based stability filtering, allowing parsimonious models to be…","keywords":["system identification","NARX","NARMAX","orthogonal forward regression","PRESS statistic","iterative OFR","simulation-based model selection","MATLAB package"],"falsifier":"Take a nonlinear ARX system, identify a model with NonSysId using a short non-persistently exciting input, then run the model free on a longer input record drawn from the same distribution; if the simulated output diverges while a model selected by conventional cross-validation on a separate validation set tracks the data, the constant-input stability filter has failed to certify generalization.","tokens_in":13933,"feed_emoji":"🧮","tokens_out":7511,"duration_ms":61259,"temperature":0.7,"pith_summary":"This paper presents NonSysId, an open-source MATLAB package for identifying polynomial NARX and NARMAX models. Its central claim is that combining iterative Orthogonal Forward Regression (iOFR) with PRESS-statistic term selection and simulation-based stability filtering yields parsimonious models that generalize well without a separate validation dataset. This matters for real-time applications—structural health monitoring, fault diagnosis, and biomedical signal processing—where obtaining consistent validation data is often impractical. The paper backs the claim with two demonstrations: a synthetic DC-motor NARX model excited by white noise and by a multi-tone sinusoid, and a real electro-mechanical system driven by a PRBS. A careful reader would take away that the package targets long-horizon free-run simulation accuracy, not just one-step-ahead fit.","feed_headline":"New MATLAB tool picks NARMAX terms via simulation stability","feed_subtitle":"Iterative forward regression plus leave-one-out cross-validation builds free-run models from one data record.","key_machinery":"The load-bearing machinery is the iOFRS procedure: an extension of the iterative OFR algorithm that (step 1) seeds orthogonalization paths from each pre-selected term, (step 2) admits only candidate models whose simulated output for u(t)=0 and u(t)=1 has variance at most ε=$10^{{-2}}$, and (step 3) ranks survivors by BIC computed from simulated mean-squared error. Inside the forward search, PRESS-statistic term selection estimates leave-one-out cross-validation error in closed form from the OLS orthogonalization, replacing ERR-based selection. Complementing the search are four RCT methods that shrink the candidate term set (e.g., via a linear ARX pre-model) and sharpen the initial pre-select set, reducing runtime while preserving the simulation-stability search.","core_discovery":"The central discovery is a concrete algorithmic pipeline, iOFRS, in which model structure is selected by simulation behaviour rather than by one-step-ahead explained variance alone. At each iteration, the algorithm pre-selects candidate terms, follows multiple orthogonalization paths to produce candidate models, filters out models whose free-run output under constant inputs u=0 and u=1 is not bounded with small variance, and then chooses the model with the lowest BIC computed from simulation error variance. Within each path, PRESS-statistic term selection replaces the classical error reduction ratio, giving an efficient leave-one-out cross-validation during forward selection. The paper claims this produces models that are both parsimonious and stable under free-run simulation, without needing a held-out validation set.","pith_inferences":["The constant-input filter is a heuristic, not a general stability certificate; a polynomial NARX model can be bounded under u=0 and u=1 yet diverge under other bounded inputs, so the 'no validation data needed' claim should be read as conditional on the input class.","The paper's own residual plots show residual autocorrelation not fully white in both examples; the claim of robust generalization would be strengthened by reporting out-of-sample free-run error on an unseen segment rather than training-residual variance alone.","A natural benchmark, not run in the paper, is a systematic comparison against OFR-ERR and against a standard validation-split approach on systems with weak excitation, measuring both free-run error and model parsimony.","The same iOFRS machinery could be lifted to non-polynomial basis functions and to MIMO systems, which the authors list as future work; the stability filter and BIC selection would need re-testing in those settings."],"forward_implications":["A user can identify a NARX or NARMAX model from a single input-output record, with leave-one-out cross-validation built into term selection rather than requiring a held-out split.","Selected models are chosen by free-run simulation error, so they should support long-horizon prediction and simulation-based analysis such as NOFRF frequency-domain studies.","The stability filter and PRESS criterion should reduce the redundant-term and overfitting problems reported for OFR-ERR, especially when the input is not persistently exciting.","The four RCT methods let practitioners trade a smaller search space for lower computation time in real-time monitoring and diagnosis tasks."],"supporting_citations":[{"why":"Supplies the iterative OFR algorithm that explores multiple orthogonalization paths to approximate globally optimal model structure.","marker":"[32]"},{"why":"Introduces PRESS residuals for dynamic system identification, grounding the leave-one-out cross-validation term criterion.","marker":"[21]"},{"why":"Combines forward regression with PRESS statistic for automatic nonlinear predictive model construction, integrated here with iOFR.","marker":"[22]"},{"why":"Documents limitations of OFR-ERR and motivates simulation-error-based selection for NARX models.","marker":"[14]"},{"why":"Shows that optimal terms are found only along orthogonalization paths beginning with correct terms, justifying iOFR's path exploration.","marker":"[15]"},{"why":"Provides the simulation-error perspective used to define the stability filter and BIC-on-simulation selection.","marker":"[33]"},{"why":"Defines the orthogonal least squares base algorithm on which OFR and iOFR are built.","marker":"[18]"},{"why":"Proposes candidate-term reduction from a linearised ARX model, used as RCT Method 1 to cut search space.","marker":"[39]"}],"fun_headline_variants":["NonSysId: NARMAX term selection by free-run stability","iOFRS picks NARMAX terms via simulation, not one-step fit","New MATLAB package selects NARMAX terms for stable free-run models","NonSysId: PRESS-based forward regression for parsimonious NARMAX"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole approach takes the two constant-input stability tests (u=0 and u=1) as sufficient evidence that a candidate model will simulate stably under the actual, possibly non-persistently exciting input; if that inference fails, the simulation-based BIC selection can pick a model that diverges in free run even though the package judged it stable.","fun_headline_variants_meta":{"raw":{"variants":["NonSysId: NARMAX term selection by free-run stability","iOFRS picks NARMAX terms via simulation, not one-step fit","New MATLAB package selects NARMAX terms for stable free-run models","NonSysId: PRESS-based forward regression for parsimonious NARMAX"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000559,"raw_usage":{"total_tokens":2626,"prompt_tokens":882,"completion_tokens":1744,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":1663}},"tokens_in":498,"tokens_out":1744,"duration_ms":13043,"temperature":1.0,"reasoning_tokens":1663,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:04:20.933500+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a nonlinear ARX system, identify a model with NonSysId using a short non-persistently exciting input, then run the model free on a longer input record drawn from the same distribution; if the simulated output diverges while a model selected by conventional cross-validation on a separate validation set tracks the data, the constant-input stability filter has failed to certify generalization.","supporting_citations":[{"cited_title":"Orth ogonal parameter estimation algorithm for non-linear stoc hastic systems,","cited_arxiv_id":null,"evidence_quote":"Supplies the iterative OFR algorithm that explores multiple orthogonalization paths to approximate globally optimal model structure."},{"cited_title":"Nonlinear interacti ons in the thalamocortical loop in essential tremor: A model-based frequency domain analysis,","cited_arxiv_id":null,"evidence_quote":"Introduces PRESS residuals for dynamic system identification, grounding the leave-one-out cross-validation term criterion."},{"cited_title":"Nonlinear system identification of ne ural systems from neurophysiological signals,","cited_arxiv_id":null,"evidence_quote":"Combines forward regression with PRESS statistic for automatic nonlinear predictive model construction, integrated here with iOFR."},{"cited_title":"Billings, Nonlinear System Identification: NARMAX Methods In The Time , Frequency, And Spatio-Temporal Do- mains, vol","cited_arxiv_id":null,"evidence_quote":"Documents limitations of OFR-ERR and motivates simulation-error-based selection for NARX models."},{"cited_title":"Identification of continuous-t ime systems: A tutorial,","cited_arxiv_id":null,"evidence_quote":"Shows that optimal terms are found only along orthogonalization paths beginning with correct terms, justifying iOFR's path exploration."},{"cited_title":"Use of press residuals in dynam ic system identification,","cited_arxiv_id":null,"evidence_quote":"Provides the simulation-error perspective used to define the stability filter and BIC-on-simulation selection."},{"cited_title":"Estimating the incidence of spontaneous breathing effort o f mechanically ventilated patients using a non-linear auto regressive (narx) model,","cited_arxiv_id":null,"evidence_quote":"Defines the orthogonal least squares base algorithm on which OFR and iOFR are built."},{"cited_title":"The necessity of leave one subject out (loso) cross v alidation for eeg disease diagnosis,","cited_arxiv_id":null,"evidence_quote":"Proposes candidate-term reduction from a linearised ARX model, used as RCT Method 1 to cut search space."}],"review_version":1}